Task: T410178

Introduction

The Reader Experience team developed image browsing for logged-out readers on mobile web (Hypothesis FY25–26 WE3.1.3). An A/B test was conducted to evaluate its impact on reader retention, feature adoption, and usage.

Methodology

The A/B test was run on logged-out mobile web users. It was enabled in tiers—starting with 5 pilot wikis on November 3, and then deployed on English Wikipedia on November 17. For this analysis, we reviewed A/B test data recorded from November 3 through December 17, 2025.

In this test, a randomly selected half of logged-out users were assigned to the treatment group, where they saw the image carousel when the article page contained images. If they clicked on any of the images, they were directed to the image detail view. The remaining half were assigned to the control group, where they experienced the existing interface. We tracked pageviews and reviewed reader retention rates on a per-user basis for all tested users. We also tracked click events related to image browsing and feature-level retention to understand feature usage.

Summary of results

Feature usage

7.8% of 3,856,289 users exposed to this feature clicked an image in the image carousel—i.e., 301,824 unique users—resulting in 558,266 image views from the image carousel. They also viewed images 35,496 times from the visual table of contents and clicked the View in article button 8,173 times.
Among clicks from the detail view, the majority were on full image (41,877 clicks), accounting for around 61% of all clicks . Download was the second most common action (15%, 10,581 clicks). The remaining actions—share, commons, caption link, and more — together make up the remaining ~24%.
28% of image views from the image carousel came from clicks on non-first images.

Feature retention

10.2% of users who clicked images in the image carousel have reused the image browsing at least once by the end of day 21.
5.9% of users who clicked images in the image carousel have reused the image browsing at least twice by the end of day 21.
14.1% of users who clicked on image detail view reused the image browsing at least once by the end of day 21.

Reader retention (guardrail)

The A/B test showed a 0.1 percentage point increase in 21-day cumulative retention for non-English wikis. While the result was statistically significant, it did not meet our 0.2 percentage point threshold for practical significance. The A/B test retention results are available on the Test Kitchen dashboard. (Dashboard for 1st-tier test wikis; Dashboard for 2nd-tier test wikis)

Setup

Code

shhh <- function(expr) suppressPackageStartupMessages(suppressWarnings(suppressMessages(expr)))
shhh({
library(tidyverse); 
library(lubridate); 
library(scales);
library(magrittr); 
library(dplyr);
})

Code

# For summary tables   
library(gt)
library(gtsummary)
library(IRdisplay)
library(htmltools)

Code

# For visulization
options(repr.plot.width = 15, repr.plot.height = 10)
# colorblind color friendly pallette:
cbPalette <- c("#999999", "#E69F00", "#56B4E9", "#009E73", "#F0E442", "#0072B2", "#D55E00", "#CC79A7")

Analysis

The data was extracted in the data-gathering notebook. Please refer to it for details on the queries used.

Feature usage

Primary metric: Click-through Rate

What percentage of users exposed to this feature clicked the image in image carousel?
Metric definition: the ratio of the number of unique users who clicked an image in the image carousel to the number of unique users who have been exposed to this feature.

Code

df_ctr  <-
  read.csv(
    file = 'Data_out/primary_ctr.tsv',
    header = TRUE,
    sep = "\t",
    stringsAsFactors = FALSE
  )

Code

df <- df_ctr %>%
    group_by(variation) %>%
    summarize( 
             exposed_users=n(),
             clicked_users=sum(outcome),
            ctr=mean(outcome))

Code

display_html(
  as_raw_html(
    df %>%
      gt() %>%
      tab_header(
        title = md("Click-through Rate")
      ) %>% 
      cols_label(
        variation = "Experiment Group",
        exposed_users = "Exposed users",
        clicked_users = "Clicked users",
        ctr = "Click-through Rate"
      ) %>%
      fmt_percent(
        columns = c(ctr),
        decimals = 2
      ) %>%
      fmt_number(
        columns = c(exposed_users, clicked_users),
        decimals = 0,
        use_seps = TRUE
      ) %>%
      opt_stylize(6) %>%
      cols_width(everything() ~ px(150)) %>%
      tab_source_note(
        source_note = md(
          "Timeframe: November 3 - December 17, 2025 <br>
           Platform: Mobile Web <br>"
        )
      ) %>%
      tab_style(
        style = cell_text(align = "left"),
        locations = cells_source_notes()
      )
  )
)

Experiment Group	Exposed users	Clicked users	Click-through Rate
Click-through Rate
treatment	3,856,289	301,824	7.83%
Timeframe: November 3 - December 17, 2025 Platform: Mobile Web

7.8% of users exposed to this feature clicked an image in the image carousel.

Secondary Metric: Number of image views

Metric definition: Total clicks on images from the image carousel and the Visual Table of Contents (VTOC)

Code

df_feature_usage  <-
  read.csv(
    file = 'Data_out/feature_usage.tsv',
    header = TRUE,
    sep = "\t",
    stringsAsFactors = FALSE
  )

Code

df <- df_feature_usage %>%
    filter(action_subtype=='view_image')%>%
    group_by(action,action_subtype,action_source) %>%
    summarize(total_clicks=sum(events),
              .groups = "drop")

Code

display_html(
  as_raw_html(
    df %>%
      select(action_source,total_clicks) %>%
      gt() %>%
      tab_header(
        title = md("Number of Image Views")
      ) %>% 
      cols_label(
        action_source = "Action source",
        total_clicks = "Number of image views"
      ) %>%
      fmt_number(
        columns = c(total_clicks),
        decimals = 0,
        use_seps = TRUE
      ) %>%
      opt_stylize(6) %>%
     cols_width(everything() ~ px(300)) %>%
      tab_source_note(
        source_note = md(
          "Timeframe: November 3 - December 17, 2025 <br>
           Platform: Mobile Web <br>"
        )
      ) %>%
      tab_style(
        style = cell_text(align = "left"),
        locations = cells_source_notes()
      )
  )
)

Action source	Number of image views
Number of Image Views
image_carousel	558,266
visual_table_of_contents	35,496
Timeframe: November 3 - December 17, 2025 Platform: Mobile Web

Code


g <- df %>%
    ggplot(aes(x= action_source, y =total_clicks, fill = action_source)) +
    geom_col(position = 'dodge', width = 0.5, show.legend = FALSE) +
    scale_y_continuous(labels = scales::label_number(scale = 1/1000, suffix = "K")) +
    geom_text(aes(label = paste(total_clicks), fontface=2), vjust=1.2, size = 10, color = "white") +
    labs (y = "Number of image views",
           x = "Action source",
          title = "Number of Image Views")  +
    scale_fill_manual(values=c("#56B4E9","#E69F00"), name = "")  +
   theme(
        panel.grid.minor = element_blank(),
        panel.background = element_blank(),
        plot.title = element_text(hjust = 0.5),
        plot.margin = margin(t = 10, r = 10, b = 10, l = 30),
        text = element_text(size=24),
        #legend.position= "bottom",
        #legend.text = element_text(size = 18),
        #axis.text.x = element_blank(),
        axis.ticks.x = element_blank(),
        axis.title.y = element_text(vjust = 3),
        axis.line = element_line(colour = "black"))

      
g

558,266 image views are from the image carousel, and 35,496 are from the visual table of contents.

Code

ggsave("Graphs/image_views.png",
    plot = g , width = 60, height = 30, units = "cm", dpi = "screen");

Number of clicks on links in detail view

Code

df <- df_feature_usage %>%
    filter(action_source=='detail_view') %>%
    group_by(action,action_subtype,action_source) %>%
    summarize(clicks=sum(events),.groups = "drop") %>%
    mutate(
        click_percentage=round(clicks/sum(clicks),4)
        ) %>%
    arrange(desc(clicks))

Code


g <- df %>%
    ggplot(aes(x= action_subtype, y =click_percentage, fill = action_subtype)) +
    geom_col(position = 'dodge', show.legend = FALSE) +
    scale_y_continuous(labels = scales::percent) +
    geom_text(aes(label = scales::percent(click_percentage, accuracy = 0.1), fontface=2), vjust=1.2, size = 10, color = "white") +
    labs (y = "Percent of clicks",
           x = "Types of Links",
          title = "Detail View Clicks")  +
    scale_fill_manual(values=cbPalette, name = "")  +
   theme(
        panel.grid.minor = element_blank(),
        panel.background = element_blank(),
        plot.title = element_text(hjust = 0.5),
        plot.margin = margin(t = 10, r = 10, b = 10, l = 30),
        text = element_text(size=24),
        legend.position= "bottom",
     #   legend.key.width = unit(2, "lines"),  # space for each legend item
        legend.text = element_text(size = 18),
       # axis.text.x = element_blank(),
        axis.ticks.x = element_blank(),
        axis.title.y = element_text(vjust = 3),
        axis.line = element_line(colour = "black"))

      
g

Among clicks from the detail view, the majority were on full_image, accounting for around 61% of all clicks (41,877).
Download was the second most common action (15%, 10,581 clicks).
The remaining actions—share, commons, caption_link, and more—each accounted for roughly 5–9% of clicks, together making up the remaining ~24%.

Number of clicks on ‘view in article’ button

Code

 df <- df_feature_usage %>%
    filter(action_subtype=='view_in_article') %>%
    group_by(action,action_subtype,action_source) %>%
    summarize(clicks=sum(events),
              users=n(),
              .groups = "drop")

Code

display_html(
  as_raw_html(
    df %>%
      select(action_source,clicks, users) %>%
      gt() %>%
      tab_header(
        title = md("Total Clicks on the “View in Article” Button")
      ) %>% 
      cols_label(
        action_source = "Action source",
        clicks = "Number of clicks",
        users = "Number of users"
      ) %>%
      fmt_number(
        columns = c("clicks","users"),
        decimals = 0,
        use_seps = TRUE
      ) %>%
      opt_stylize(6) %>%
     cols_width(everything() ~ px(300)) %>%
      tab_source_note(
        source_note = md(
          "Timeframe: November 3 - December 17, 2025 <br>
           Platform: Mobile Web <br>"
        )
      ) %>%
      tab_style(
        style = cell_text(align = "left"),
        locations = cells_source_notes()
      )
  )
)

Action source	Number of clicks	Number of users
Total Clicks on the “View in Article” Button
visual_table_of_contents	8,173	7,315
Timeframe: November 3 - December 17, 2025 Platform: Mobile Web

Among the page loads to the detail view, 1.5% (8173/558266) clicked view in article.

How frequently did users click on non-first images in the image carousel?

Code

df <- df_feature_usage %>%
    filter(action_source=='image_carousel') %>%
    group_by(action,action_subtype,action_source,action_context) %>%
    summarize(clicks=sum(events),.groups = "drop") %>%
    mutate(
        click_percentage=round(clicks/sum(clicks),4),
        image=case_when(
            action_context=='image1' ~ 'First image in carousel',
            TRUE ~ 'Other images in carousel'
            )
        )

Code

display_html(
  as_raw_html(
    df %>%
      select(image,clicks, click_percentage) %>%
      arrange(desc(image == "First image in carousel")) %>%
      gt() %>%
      tab_header(
        title = md("Image Clicks by Image Location in the Carousel")
      ) %>% 
      cols_label(
        image = "Which image",
        clicks = "Number of clicks",
        click_percentage = "Percent of clicks"
      ) %>%
      fmt_number(
        columns = c("clicks"),
        decimals = 0,
        use_seps = TRUE
      ) %>%
      fmt_percent(
        columns = c("click_percentage"),
        decimals = 2
      ) %>%
      opt_stylize(6) %>%
     cols_width(everything() ~ px(300)) %>%
      tab_source_note(
        source_note = md(
          "Timeframe: November 3 - December 17, 2025 <br>
           Platform: Mobile Web <br>"
        )
      ) %>%
      tab_style(
        style = cell_text(align = "left"),
        locations = cells_source_notes()
      )
  )
)

Which image	Number of clicks	Percent of clicks
Image Clicks by Image Location in the Carousel
First image in carousel	400,709	71.78%
Other images in carousel	157,557	28.22%
Timeframe: November 3 - December 17, 2025 Platform: Mobile Web

28% of image views from the image carousel came from clicks on non-first images.

What percentage of users used the feature again?

Feature 21-day cumulative retention - at least one reuse

Metric definition: Percentage of users who reuse the feature by the end of day 21 at least once, one day after their initial use, within a given cohort window.

Code

df_retention_1reuse  <-
  read.csv(
    file = 'Data_out/retention_21day_cum_m.tsv',
    header = TRUE,
    sep = "\t",
    stringsAsFactors = FALSE
  )

Code

df <- df_retention_1reuse %>%
    group_by(variation) %>%
    summarize(
        cohort_size= n(),
        returned_users=sum(outcome),
        retention_rate=mean(outcome))

Code

display_html(
  as_raw_html(
    df %>%
      gt() %>%
      tab_header(
        title = md("Feature Retention Rate <br> At Least One Reuse by Day 21")
      ) %>% 
      cols_label(
        variation = "Experiment Group",
        cohort_size = "Cohort Users",
        returned_users = "Returning Users",
        retention_rate = "Retention Rate"
      ) %>%
      fmt_percent(
        columns = c(retention_rate),
        decimals = 1
      ) %>%
      fmt_number(
        columns = c(cohort_size, returned_users),
        decimals = 0,
        use_seps = TRUE
      ) %>%
      opt_stylize(6) %>%
      cols_width(everything() ~ px(150)) %>%
      tab_source_note(
        source_note = md(
          "Cohort Users: users who used the feature during the first 14 days <br>
           Returning Users: users who reused the feature at least once by the end of Day 21<br>"
        )
      ) %>%
      tab_style(
        style = cell_text(align = "left"),
        locations = cells_source_notes()
      )
  )
)

Experiment Group	Cohort Users	Returning Users	Retention Rate
Feature Retention Rate At Least One Reuse by Day 21
treatment	154,627	15,758	10.2%
Cohort Users: users who used the feature during the first 14 days Returning Users: users who reused the feature at least once by the end of Day 21

10.2% of users who clicked images in the image carousel have reused the image browsing at least once by the end of day 21.

Feature 21-day cumulative retention - at least two reuses

Metric definition: Percentage of users who reuse the feature by the end of day 21 at least twice, one day after their initial use, within a given cohort window.

Code

df_retention_2reuses  <-
  read.csv(
    file = 'Data_out/retention_21day_2reuse_cum_m.tsv',
    header = TRUE,
    sep = "\t",
    stringsAsFactors = FALSE
  )

Code

df <- df_retention_2reuses %>%
    group_by(variation) %>%
    summarize(
        cohort_size= n(),
        returned_users=sum(outcome),
        retention_rate=mean(outcome))

Code

display_html(
  as_raw_html(
    df %>%
      gt() %>%
      tab_header(
        title = md("Feature Retention Rate <br> At Least Two Reuses by Day 21")
      ) %>% 
      cols_label(
        variation = "Experiment Group",
        cohort_size = "Cohort Users",
        returned_users = "Returning Users",
        retention_rate = "Retention Rate"
      ) %>%
      fmt_percent(
        columns = c(retention_rate),
        decimals = 1
      ) %>%
      fmt_number(
        columns = c(cohort_size, returned_users),
        decimals = 0,
        use_seps = TRUE
      ) %>%
      opt_stylize(6) %>%
      cols_width(everything() ~ px(150)) %>%
      tab_source_note(
        source_note = md(
          "Cohort Users: users who used the feature during the first 14 days <br>
           Returning Users: users who reused the feature at least twice by the end of Day 21<br>"
        )
      ) %>%
      tab_style(
        style = cell_text(align = "left"),
        locations = cells_source_notes()
      )
  )
)

Experiment Group	Cohort Users	Returning Users	Retention Rate
Feature Retention Rate At Least Two Reuses by Day 21
treatment	154,627	9,181	5.9%
Cohort Users: users who used the feature during the first 14 days Returning Users: users who reused the feature at least twice by the end of Day 21

5.9% of users who clicked images in the image carousel have reused the image browsing at least twice by the end of day 21.

Conditional feature 21-day cumulative retention: at least one reuse among users who clicked the image detail view

Metric definition: Percentage of users who reuse the feature by the end of day 21 at least once, one day after their initial clicks on the image detail view , within a given cohort window.

Code

df_retention_conditional_1reuse  <-
  read.csv(
    file = 'Data_out/retention_21day_1reuse_cum_conditional_m.tsv',
    header = TRUE,
    sep = "\t",
    stringsAsFactors = FALSE
  )

Code

df <- df_retention_conditional_1reuse %>%
    group_by(variation) %>%
    summarize(
        cohort_size= n(),
        returned_users=sum(outcome),
        retention_rate=mean(outcome))

Code

display_html(
  as_raw_html(
    df %>%
      gt() %>%
      tab_header(
        title = md("Conditional Feature Retention Rate <br> At Least One Reuse by Day 21")
      ) %>% 
      cols_label(
        variation = "Experiment Group",
        cohort_size = "Cohort Users",
        returned_users = "Returning Users",
        retention_rate = "Retention Rate"
      ) %>%
      fmt_percent(
        columns = c(retention_rate),
        decimals = 1
      ) %>%
      fmt_number(
        columns = c(cohort_size, returned_users),
        decimals = 0,
        use_seps = TRUE
      ) %>%
      opt_stylize(6) %>%
      cols_width(everything() ~ px(150)) %>%
      tab_source_note(
        source_note = md(
          "Cohort Users: users who clicked on image detail view during the first 14 days <br>
           Returning Users: users who reused image browsing at least once by the end of day 21<br>"
        )
      ) %>%
      tab_style(
        style = cell_text(align = "left"),
        locations = cells_source_notes()
      )
  )
)

Experiment Group	Cohort Users	Returning Users	Retention Rate
Conditional Feature Retention Rate At Least One Reuse by Day 21
treatment	14,603	2,061	14.1%
Cohort Users: users who clicked on image detail view during the first 14 days Returning Users: users who reused image browsing at least once by the end of day 21

14.1% of users who clicked on image detail view reused the image browsing at least once by the end of day 21.