Reading/Web/Content Discovery Experiments/Search Suggestions - MediaWiki
Jump to content
From mediawiki.org
Reading
Web
Content Discovery Experiments
Translate this page
Languages:
italiano
Wikimedia Foundation projects
Search Suggestions
Start:
2024-11
Team members:
Web
Backlog:
#fy2024-25_kr_3.1_content_discovery/
Lead:
Olga Vasileva
(Product Owner)
Content Discovery Experiments
Search suggestions is a feature for the mobile and desktop websites of Mediawiki wikis that aims to help readers discover related content before they begin typing a search query. The feature uses the MoreLike API to provide article recommendations related to the page the reader was viewing when they click into the empty search bar.
What are search suggestions?
edit
Search suggestions show three pre-filled suggested articles in the empty state of the search bar on the desktop and mobile website. These articles serve as recommendations on what the reader could read next.
Where do search suggestions come from?
edit
The feature provides search suggestions generated by the MoreLike API in
Extension:CirrusSearch
, the extension which implements searching for MediaWiki.
This API is also currently in use in the
Related Pages
feature and
Extension:RelatedArticles
The suggestions in search suggestions will be identical to the suggestions in Related Pages, which looks at pages similar to the current page to generate the suggestions.
Design
edit
The search suggestions will appear once the user clicks into the search bar on the mobile or desktop website.
The suggestions will clear once text is entered into the search bar, and will be replaced by the typeahead search suggestions relevant to the query.
Screenshot of search Suggestions feature on Italian Wikipedia:
Search suggestions on Italian Wikipedia
Experimentation
edit
The search suggestions experiments proposed the development of a feature which provides suggestions to readers when they open the search bar. This feature was proposed as a tool that could make browsing easier for readers by providing a quicker way to explore topics similar to the ones the reader is currently on.
Browser extension experiment
edit
Goal
edit
This experiment sought to add suggestions to the empty state of our search bar, providing readers with recommendations on pages they could read next. We wanted to experiment with whether readers click on recommendations in the empty state of the search bar to see if this type of feature is helpful to readers.
In order to perform this experiment, we built a browser extension which injects the proposed recommendations into the search bar. Then we tracked clicks on the provided suggestions. We also considered using the extension itself to communicate to readers when the experiment is over.
Our hypothesis was that if we display recommendations in search, people will be interested in reading the recommended pages, as shown through a clickthrough rate similar to existing recommendation entry points (Related pages).
Experimental setup
edit
A quicksurvey was displayed to a small percentage of readers on English and Spanish Wikipedias, inviting them to download a browser extension.
Interested readers were prompted to download a browser extension which added the feature into the page.
The feature provides search suggestions generated by the MoreLike API in
Extension:CirrusSearch
, the extension which provides search results.
The suggestions are also currently available in the
Related Pages
feature and
Extension:RelatedArticles
Data was collected on how often readers clicked on the suggestions generated by the extension. No user or session data was collected.
Design
edit
The search suggestions will appear once the user clicks into the search bar.
The suggestions will clear once text is entered into the search bar, and will be replaced by the typeahead search suggestions relevant to the query.
Results
edit
We found that the readers interacted with the recommendations shown in the search bar at a rate higher than recommendations shown at the bottom of the page.
Report on the overall clickthrough rate (CTR) for search recommendations coming from the browser extension:
On English wikipedia (enwiki) desktop, the overall CTR for search recommendation is 1.53%
On Spanish wikipedia (eswiki) desktop, the overall CTR for search recommendation is 0.72%
The overall CTR across both wikis is 1.44%
Comparison to clickthrough rate for related pages on desktop, mobile: we observed that the CTR for search recommendations is higher than that for related pages on desktop for both English Wikipedia and Spanish Wikipedia.
English Wikipedia (enwiki): 1.53% vs 0.41% (CTR of related pages on desktop, enwikivoyage), diff 1.12 pp
Spanish Wikipedia (eswiki): 0.72% vs 0.13% (CTR of related pages on desktop, eswikivoyage), diff 0.59 pp
Detailed results are available on
T374965
Next steps
edit
On mobile: Proceed with A/B test for the feature on one or more pilot wikis (pilot wiki list to be selected). Details on A/B test setup are available in
phab:T378094
On desktop: Wait until the results of the article-level recommendations are ready and compare and proceed with the feature which shows better results.
A/B test
edit
Goal
edit
After the successful browser extension experiment, we wanted to test the feature in a mobile environment, as well as gather data on its effects on higher order metrics such as session length. If the feature was successfully engaging people to learn more, we expected to see an increase in session length due to the feature in addition to clickthrough rates similar to the results we saw in the browser extension. To do this, we decided to test the feature on mobile in the context of an A/B test. Here, the control group was the current search bar, and the test group was the search bar with the search suggestions.
Our hypothesis was that users who interact with this feature will have sessions that are 5% longer than those who do not.
Experimental setup
edit
The A/B test was run on anonymous mobile web users with the Minerva skin. It was enabled in tiers—starting with all wikis except enwiki on March 4 (UTC), and then deployed on enwiki on March 6 (UTC). The test was disabled on March 24, 2025. For this analysis, we reviewed A/B test data recorded from March 4 through March 23, 2025.
In this test, a randomly selected half of anonymous user sessions were assigned to the treatment group, where they saw empty search recommendations. The remaining half were assigned to the control group, where they experienced the current search setup and did not see any recommendations until they began typing in the search box.
Results
edit
Click-through rate (CTR): 3.4% of searches resulted in clicks on the empty state recommendations. We estimate that 0.8% of these clicked searches would have engaged with autocompleted search recommendations even without the empty state feature. The remaining 2.6% of clicks represent boosted engagement driven by our new feature.
Session length:
For all tested user sessions, being assigned to the treatment group increased session length by 0.1%, compared to sessions in the control group that focused on the search box but were not exposed to the empty search recommendations. The odds of a session having a length shorter than 30 seconds decreased by 0.68% when assigned to the treatment group. The overall impact is small, as only around 2.6% of sessions interacted with the empty-state search recommendations.
For sessions that interacted with the empty state recommendations, engaging with the empty search recommendations increased session length by 15.4%, compared to sessions in the control group that focused on the search box but were not exposed to the empty search recommendations. Additionally, the odds of a session having a length shorter than 30 seconds decreased by 48% when users interacted with the empty search recommendations. However, the results may be influenced by selection bias, as sessions that interacted with the recommendations likely represent more active users.
Based on these results, we conclude that our hypothesis is
confirmed,
with our target of 5% being surpassed by the test results in which we saw a 15.4% increase.
See the full results
Deployment timeline
edit
Based on the results of the A/B test and browser extension, we will proceed with scaling this feature and making it available to all wikis. Our current deployment plan is as follows:
May 7 - May 21: Test feature and prepare for deployment. Discuss upcoming change with communities
May 21 - June 4: Deploy feature to
group1
wikis on desktop
June 4 - June 15: Deploy feature to all wikis on desktop, with the exception of English Wikipedia. Deploy feature to group1 wikis on mobile
June 15 - June 30: Deploy feature to all wikis on mobile, with the exception of English Wikipedia
July 1 - July 30: Deploy feature to English Wikipedia on mobile and desktop
Retrieved from "
Category
WMF Projects
Hidden categories:
WMF Projects 2024q4
WMF Projects 2025q1
WMF Projects 2025q2
WMF Projects 2025q3
WMF Projects 2025q4
WMF Projects 2026q1
WMF Projects 2026q2
Reading/Web/Content Discovery Experiments/Search Suggestions
Add topic