The metric used here to compare different tuning of the autocomplete algorithm represents the probablistic number of characters typed by a typical user before selecting their desired result from the autocomplete drop down. This metric first looks at real user sessions to estimate how likely a user is to continue typing even when their result is presented in the autocomplete, conditioned on the position the result is displayed at. Individual user sessions, represented in the data as (prefix_typed, page_id_clicked), are then simulated with prefixes from length 1 to the full prefix typed. From this simulation we determine the expected number of characters typed for an individual search clickthrough.
The dataset used contains 7.77K clickthroughs from Mar 01 - Mar 18 in the training set, and another 7.77K clicks from Mar 18 - Mar 31 in the test set.
The graph below shows bootstrapped probability densities for each bucket. Tick marks are shown at the 95% confidence levels.
The following graphs show the before and after effects of tuning. This shows strong improvement, up to a full character, from percentiles 4-76 with minor improvements up to percentile 92. Tail queries show some decline in performance, but only slightly.
This shows on a per-session basis the change in number of characters typed between the baseline scoring and the scoring after parameter tuning. A session is defined as the user submitting a series of autocomplete queries for a single input. This suggests 33% of sessions save between 1 and 6 characters typed. Another 32% save a fractional character, and 18% have no impact. Around 18% of sessions are impacted negatively, future inspection of what makes these sessions different may be useful for investigating new scoring signals.
The orange line shows the training run with the best score. The faint blue lines show other training runs that have a mean score less than the upper 95% CI of the best score.
0.71 : query/dismax/0/constant_score/labels.it.near_match/boost:0
0.56 : query/dismax/1/constant_score/labels.it.near_match_folded/boost:0
0.54 : query/dismax/2/constant_score/labels.it.prefix/boost:0
0.87 : query/dismax/3/constant_score/labels_all.near_match_folded/boost:0
0.71 : query/dismax/4/constant_score/labels.en.near_match/boost:0
0.26 : query/dismax/5/constant_score/labels.en.near_match_folded/boost:0
0.57 : query/dismax/6/constant_score/labels.en.prefix/boost:0
0.29 : query/dismax/tie_breaker:0
0.58 : query/title.keyword/boost:0
0.07 : query/title.keyword/k1:0
0.47 : rescore/0/function_score/0/satu/incoming_links/a:0
286.12 : rescore/0/function_score/0/satu/incoming_links/k:0
0.51 : rescore/0/function_score/0/weight:0
6.12 : rescore/0/function_score/1/satu/sitelink_count/a:0
254.83 : rescore/0/function_score/1/satu/sitelink_count/k:0
0.01 : rescore/0/function_score/1/weight:0
0.04 : rescore/0/query_weight:0
0.70 : rescore/0/rescore_query_weight:0
To get an idea of how much influence individual parameters have on the final score, and to estimate how sensitive those variables are to small changes, the graphs below plot the sensitivity of individual parameters. This is performed by holding all variables except one as a constant, and sweeping a set of values around the chosen point. The graphs then show how the final score changes based on changes to that variable. Dots on the graphs are additionally colored by their score. A graph of a single color suggests the variable in question has a relatively small influence on the final output.
$wgWBCSPrefixSearchProfiles['wikibase_config_prefix_query-202204-it'] = [
"any" => 0.867,
"en-exact" => 0.708,
"en-folded" => 0.26,
"en-prefix" => 0.57,
"it-exact" => 0.706,
"it-folded" => 0.556,
"it-prefix" => 0.536,
"language-chain" => [
"it",
"en"
],
"tie-breaker" => 0.294
]
$wgWBCSRescoreProfiles['wikibase_config_entity_weight-202204-it'] = [
"i18n_msg" => "wikibase-rescore-profile-prefix",
"query_weight" => 0.0425,
"rescore" => [
"function_chain" => "wikibase_config_entity_weight",
"function_chain_overrides" => [
"functions.0.params.a" => 0.473,
"functions.0.params.k" => 286.0,
"functions.0.weight" => 0.509,
"functions.1.params.a" => 6.12,
"functions.1.params.k" => 255.0,
"functions.1.weight" => 0.0065
],
"score_mode" => "total",
"type" => "function_score",
"window" => 8192,
"window_size_override" => "EntitySearchRescoreWindowSize"
],
"rescore_query_weight" => 0.699,
"supported_namespaces" => "all"
]