I have drafted the instrumentation spec for empty search recommendation AB test and listed the events that need to be tracked to calculate each metric. Please review and decide whether/how to instrument them. Additionally, in preparing for the legal/safety/security review, I started drafting a measurement plan. Feel free to replace the placeholders with the specifics of our final plan.

Completed Tasks
The daily trends of the following metrics for pilot wikis have been added to the deployment dashboard.

• New regular account creations
• Checkuser admins
• Revert rate of all type of users

Add documentation to the dashboard.

@Jdrewniak, thanks for the instrumentation proposal. Does the team plan to instrument the eventlogging in metric platform schema? If so, the platform has designed several fields to capture the info, like action, action_subtype, action_source, action_context.

Completed Tasks
The daily trends of the following metrics for pilot wikis have been added to the deployment dashboard.

@ovasileva, the plan looks okay to me.

HI @ovasileva , here is the summary of the analysis results of new donate link. Let me know if you have any questions.

Status update

Completed Tasks

• Enable a data pipeline on daily granularity (with 1-4 weeks delay)
• Enable a data pipeline on monthly granularity

• Enable a data pipeline on monthly granularity

Next steps

Discussed the data issue on enwiki (T375162#10208484) with @ovasileva. As the result, we decided to use eswiki and jawiki for comparison instead of enwiki in the later analysis of new donate link. Therefore, I ran data for these two wikis for the same period, from September 1 to 23, 2024, and documented the result as the baseline.

Daily trend of English Wikipedia (enwiki)

DAG files for trust_safety_block_daily and trust_safety_block_monthly have been checked-in to gitlab. Here is the merge request, which is pending review. link

DAG files for following tables have been checked-in to gitlab. Here is the merge request, which is pending review. link

• trust_safety_admin_action_monthly
• trust_safety_block_monthly
• trust_safety_admin_monthly
• trust_safety_new_admin_monthly

Hi @ovasileva, I have drafted an instrumentation spec for Summarization feature, based on our discussion. As we plan to instrument using statsd, which is quite different from EventLogging. I pass it onto the engineers to evaluate its feasibility.

Thanks for the explanation, @Jdlrobson. Agree it's not within the scope of this ticket. I am going to resolve this ticket.

If the labels small, regular, and large in the action_source can be changed to numbers that reflect the font values (1, 2, 3), that would be great. However, I understand if that’s a significant undertaking.

Hi, the font field looks good. It reflects the font value when the page was loaded.

Priority and where to track were discussed with @kostajh and @Niharika. The decision was documented in columns D&E of google sheet.

Summarize the discussion on slack regarding two potential data issues we discovered in mediawiki_web_ui_actions schema.

The code changes for the queries have been submitted for code review.
https://gitlab.wikimedia.org/repos/product-analytics/data-pipelines/-/merge_requests/25

• Enable a data pipeline on daily granularity (with 1-4 weeks delay)

The data pipeline was enabled by the two check-ins below.
https://gitlab.wikimedia.org/repos/product-analytics/data-pipelines/-/merge_requests/22
https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/843

The click-through rate is defined as the number of clicks on the old 'donate' link out of the total number of initiated pageviews. This rate is likely an underestimation, as users might not have seen the donate link. As the new donate link was deployed on September 24, our baseline uses data collected before the deployment date, between September 1 and 23, 2024.

Baseline for desktop anonymous users across wikis

• Vector’s “Small” (14px) → Log as 0
• Minerva’s “Standard” (16px) → Log as 1
• Vector’s “Standard” (16px) → Log as 1
• Minerva’s “Medium” (18px) → Log as 2
• Vector’s “Large” (18px) → Log as 2
• Minerva’s “Large” (20px) → Log as 3

Thanks, @KSarabia-WMF , The mapping looks good to me. Can we document this somewhere?

Analysis Summary

• Report on the overall clickthrough rate for search recommendations coming from the browser extension
  • On English wikipedia (enwiki) desktop, the overall CTR for search recommendation is 1.53%
  • On Spanish wikipedia (eswiki) desktop, the overall CTR for search recommendation is 0.72%
  • The overall CTR across both wikis is 1.44%.

is also present in analytics_privatedata_users , analytics-product-users, shall we remove them from those groups as well?

Hi, @Jdrewniak, I'd like to confirm the data in the dashboard to calculate CTR rate.

@Jdlrobson, can you share a screenshot of the donate link which fired the above event? We might not be referring to the same link.

@ssingh, My manager (@mpopov ) told me I should request for airflow-analytics-product-admins group, instead of deployment group. deployment granted me wider permissions than I need. Could you remove me from deployment group and add me into airflow-analytics-product-admins group?

HI @KSarabia-WMF, thanks for looking into it. The above event is from LandingPageImpression schema right?
As the article page initialization is captured in mediawiki_web_ui_actions schema, we'd better to use the click events in the same schema. Can you check how the click on donate link was captured in mediawiki_web_ui_actions schema?

Hi, @KSarabia-WMF, Can you provide how clicks on the donate link are captured in event logging? Thanks!

Based on the discussion with @Jdlrobson on slack, it would be helpful to have the daily clickthrough rate for the Related Pages feature on mobile and desktop on English Wikivoyage. Here is the data.

Daily overall percentage of internal referrals generated by the Related Pages feature on mobile

• Based on the data between 2024-08-09 and 2024-09-08: around 4.20% of internal pageviews are generated by the Related Pages feature on mobile

Daily clickthrough rate for the Related Pages feature on mobile

• Based on the data between 2024-08-09 and 2024-09-08: around 5.96% of Related Articles presentations result in a click.

• Daily trend

@Jdrewniak , what is morelike API request? what action triggers such a request? For context, I am looking for the "impression" events to estimate the click through rate of Related Articles link.

I have documented the data source and estimated measurement frequency and their efforts in the google sheet tab existing metrics.

Here is the investigation summary.

Number of rate limit trips for temporary account creations per IP (default is 6 temp account creations per IP address per day)