As part of T306644 the weights of the wbsearchentities prefix search on www.wikidata.org were tuned using historical click logs as a guide. To determine the effectiveness of this tuning an AB test was run from May 1 at 00:00 UTC until May 15 at 00:00 UTC. The test was limited to users performing searches for wikidata items in the English language. Users were divided equally on a per-page load basis into either the control or test bucket.
The analytics for this test collect usage from the various entity selectors used throughout the interface for editing Wikidata. The data also includes usage of the autocomplete on the top-right of all Wikidata pages, but due to a bug in the data collection the usage of the top-right autocomplete was only logged from entity pages.
The graphs below are all probability densities and annotated with 95% confidence intervals. The confidence intervals are constructed by running five thousand rounds of the bootstrap method.
| bucket | event | count |
|---|---|---|
| T306644-fr | start | 7000 |
| T306644-fr | click | 3677 |
| control | start | 7023 |
| control | click | 3673 |
The number of characters typed in each session is a proxy for the amount of effort a user must exert to find the item they are looking for. The control has a small but statistically signficant lead over the test tuning, suggesting the tuned ranking is not an improvement.
The ratio of page loads with start events against the page loads with click events is interepreted loosely as the abandonment rate of search. The 95% CI completely overlaps, suggesting the test treatment had no effect on abandonment rates. While this is consistent with previous tests, we don't know why so many sessions start but don't complete the entity selection.
The position of clicked result is another proxy for the amount of effort a user must exert to find the item they are looking for. The mean position clicked increased (became worse) from 1.53 to 1.63, which is statistically significant.
Looking into this result closer, the change that occured was an decrease in Clicks@1 from 74% to 70%. These click throughs look to mostly have moved to the second and fourth position.