Katherine's manager is on leave, skip-level here approving.

Just confirmed with Google that they've fixed this problem. I checked several examples. If you find broken ones, please add them here and reopen.

Thank you.

1. What form does this model take?
2. Where is the code?
3. How can MW code use this model?
4. Where's the documentation that explains how to retrain the model / deploy it?

Skip manager approves in manager's absence.

Manager is currently on leave, skip manager approves.

Manager approves.

@Mvolz @Krinkle I know that at least one of the properties throttling us is doing so with 403s and not 429s. They said as much.

@Robertsky We heard back from the property owner who showed us graphs of traffic and traffic levels and said that this was clearly more than what they considered acceptable. Their concern was purely about volume and not about suspicious looking user agent strings.

I've reached out to Partnerships to surface this to Google.

We're learning more as we are talking to owners of the sites who've decided to block Citoid, while also trying to be sure about how much traffic we're sending to them.

Relevant: https://searchengineland.com/google-search-fixes-issues-with-site-names-not-appearing-for-internal-pages-442933

Ah you're right. It seems to have fixed it for English Wiktionary, but not for French. Try the query "wiktionary" and see what it says for english results. There's a chance the french ones have just not been scraped yet, but I can't see that on search console yet (I don't have access for wiktionary). Let me get access and see when they were last indexed.

For Wiktionary, adding og:site_name seems to have fixed the problem.
We should probably look at this with a pinch of salt because it also works correctly for commons without it.

Reasoning as requested by @Reedy

Here's some context that I should have included when I filed the ticket:

@Krinkle This could use a second pair of eyes to make sure I'm not missing something completely obvious if you could spare the time.

Granted. Please check and close the task if everything is in order.

In the Grafana dash, Saturation -> Total Network says it's about 10 MB/s. Does this count everything the job is doing, including what Zotero might be sending out?

Idea 1 has the level of information and detail that my teams would find useful, especially in conversations around browser versions, support levels, and such.

Looks good to me.

I'm comfortable closing this task as resolved given that I've been getting search console requests from people following the process as documented and have been resolving them.

Hello Taavi,

I think ThisDot should be in the table here: https://www.mediawiki.org/wiki/Gerrit/Privilege_policy#Expedited_process_for_trusted_organisations

Just escalated to the folks talking to Google one more time.

Fixed, thank you.

The search console says this page is simply unknown to Google.

I have a patch ready if someone would like to review it.

Here is a summary of our discussions with Google (they proofread this summary):

Update: First Input Delay is going to be deprecated in favour of Interaction to Next Paint.

@Nicholas_Perry is the person talking to them.

Partnerships just told me they will reach out to Google soon. We'll post updates here.

I've requested partnerships to look into this too.

That bit you're seeing is called the "Site Name", as distinct from the page title. The idea behind it seems to be to tell you what site you're looking at (without needing to look at the URL itself, or relying on the page title to include the name of the site).

The important takeaway from this (as per our discussion) was this bit:

We met with Google to discuss this further. Google will provide more details on this soon, but the crux of the matter is that not all pages are guaranteed to be crawled, indexed, and served, as is stated on Search Central documentation.

@Seddon Could you please post an update here and link to relevant tickets?

Folks, here's a plan of action.

Seddon and I are meeting on Friday. We'll have a concrete action plan (or
the beginnings of one) to share on Monday.

We're meeting with them in the next couple of weeks to troubleshoot our scraping problems. Will report back once we learn more.

Manager here: I approve.

This has been reported to Google. We're waiting for them to get back.

@Jdlrobson Did the frontend standards group meeting come to a conclusion, or did it bring up any new insights not already shared in this ticket?

Thanks for all the inputs.

@elukey Do we have another idea on the table aside from asking a team of Android devs (3) to maintain a recommendations service?

This has been granted for a period of one year.

@Soda I need a gmail account to provide access to.

OK, having talked to some folks in the Enterprise org and other teams and having eliminated a few possible problems, the one I'm investigating now is the possibility that for some reason Google's bot is ratelimiting itself. I'll continue to post any findings here.

Thank you for the supporting links.

Here are some unindexed articles (confirmed from search console and from Google). I came upon them by simply hitting "Zufällige Seite" (Random Page) on de Wikisource and checking if the resulting page is indexed at all.

@Xover That's very useful to know.

@Soda Yeah Navboxes would indeed have helped.

@Krinkle Could you please answer the question I had above when you have a chance?

As far as I can tell, a lot of the pages that aren't appearing in the index are simply not linked to from within the Wiki. There are no sitemaps any more, and Special:Lonelypages is uncrawlable because there's a robots.txt rule blocking all Special: pages from being crawled.

I think we can close this task for now.

If the volunteer is given access to Search Console data, they will be able
to examine the following information:

• which queries lead to search results on Google that have Wikis in them
• which results get clicked on by users
• the total query volume that has search results from the Wikis
• Whether a given page from a Wiki property has been indexed or has

problems preventing it from being indexed

• a breakdown of the above information by device platform, country, and

date.

Manager approval if needed: approved

I've responded on the ticket with the volunteer. I'll handle it once they get the NDA and C-level approval out of the way.

Please assign this to me once C-level approval and NDA have been taken care of.

No strong opinions either way, but it seems to me that the examples that @pmiazga provides aren't all user types. "admin", "group:something", or "en_wiki:admin" are not really user types. They're memberships or privileges, which can grow and change as needed. Isn't there another way to represent those? I would imagine that a user type is something that's largely permanent, while privileges (through memberships) are expected to change.

Silly question: @Krinkle Given that robots.txt has a Disallow: /wiki/Special: rule, how do search engines read the LonelyPages or RecentChanges pages? As far as I can tell these pages aren't being indexed. Am I missing something?

Apologies for the ridiculous delay. I have Wikisource search console access now and am looking at it.

Manager approves.

WMF Staff who will need this on an ongoing basis. No expiry.

Perhaps @Dbrant knows? I remember Jdlrobson mentioning that they did some work together on this.

Rather than speculate on this one feature in isolation, I recommend letting @ovasileva decide and prioritize based on what else needs doing. She likely to have a more holistic view and more insights from the community.

Just had a conversation with @Catrope . I'm comfortable owning the risk and understand what that entails.

That is not the case. Quoting from T326816 (please read for details):

Please note that any changes to this extension might first need to be
discussed in T326956.

I've mostly focused on search performance and stats for the Wikipedias and haven't had a chance to set up and build an understanding of where we are with Wikisource yet. That's why I don't have an immediate answer for this problem.

The vendor replied with numbers over the last two days. Here are some numbers that might help decide on a strategy. There is a new dump of IPs every day. Between the last two successive dumps (today and yesterday) there is:

• 10-12% daily change
• 1.7M New entries
• 2M Aged off entries (i.e. dropped/removed)

Thanks for replying, @Ladsgroup

Are DBA and SRE folks aware that this entire database will essentially be wiped every one or two days and recreated from a dump? Does that complicate things in any way?

Apologies, I don't have access to wikisource. @mpopov does probably.