Some ideas:

• Because this will be visible to new editors, then the regular volunteers at enwiki Teahouse and the frwiki Wikipédia:Forum des nouveaux need to know about this, so they can answer questions about it.
• We should talk to RecentChanges reviewers, as they may have advice about which problems they encounter (some of which may not be related at all). It might be possible to find these people by seeing who uses specific tools (e.g., Huggle, RedWarn, STiki) or, more generically, by seeing who hits the revert button the most often. Seeing who uses Template:Uw-unsourced1 might also be a way to find interested people.
• It could be interesting to talk to editors who are involved in writing policies/guidelines/advice. The most efficient starting point for this group might be people involved in Growth's Newcomer Homepage, as there are similar constraints in software (e.g., a five-thousand-word guideline can't be fit into a tiny box).

Meeting Notes: 19 January
Building on what @Whatamidoing-WMF shared in T327330#8542022, in the meeting we had offline, we came to think:

1. To start, we ought to prioritize talking with and learning from Senior Contributors who are already working to:
   • Proactively educate and guide newcomers to make changes they feel proud of and changes that improve Wikipedia (e.g. people who are active at Teahouse-like pages)
   • Prevent people from publishing destructive changes (e.g. people who contribute to AbuseFilters and Edit Notices, people who are creating new gadgets/scripts), and
   • React to and moderate changes to Wikipedia articles (e.g. people who are monitoring recent changes, patrolling new articles, etc.)
2. It is important that we make it clear to the Senior Contributors we'll be speaking with about Edit Check that we're intending it to be a tool they feel empowered to configure so that the prompts and actions it surfaces to people align with the workflows and culture projects have established for themselves

Per what @nayoub shared offline today, we also ought to look at the Community Wishlist for people who have submitted and voted for EditCheck-related proposals.

Some other questions we might consider asking Senior Contributors:

• What edits that involve people adding net new content require an accompanying reference?
• What edits that involve people adding net new content do NOT require an accompanying reference?

Context: T326856#8566055.

• What edits that involve people adding net new content do NOT require an accompanying reference?

Bad edits.

OK, approaching this more seriously, there are very few instances I can think of here. They are:

• When there is already a reference for the material, and the editor is just adding more content from that reference. (Note: Something to consider as you ponder future edit checks is that this is often someone violating WP:SUMMARYSTYLE, e.g. by adding a WP:UNDUE amount of detail to a controversy that was previously described concisely.)
• When someone is writing the lead section, which does not normally need to be cited per WP:LEADCITE. (In fact, if someone is trying to add references to the lead, they should probably be informed of LEADCITE and told they should probably place them in the body instead.)
• For plot sections, which don't need citations per WP:PLOTCITE.
• For images and image captions, which don't need citations per WP:ORIMAGE.

For pretty much everything else, we're going to want a citation.

What additional information/context might you find helpful in assessing the quality of an edit and the intentions of the person making it so that you can decide how best to respond to it/them?

Well, to assess the quality of the edit, all I really need is the edit itself, plus access to the source it uses. From there, I can look up the reliability of the source (via WP:RSN) or judge it for myself and check whether the information added is actually in the source (good sources with e.g. page numbers are helpful here, so it's notable that VE doesn't facilitate those yet per T216817). I can look at the nearby content of the article and the talk page to see if there's any prior consensus in the area (e.g. an RfC that decided not to include this piece of info).

For the intentions of the person making it, I certainly want to know if they have any paid or unpaid conflict of interest. I want to see their edit history and user talk page to help determine if they're a WP:SPA (single-purpose account) or a point-of-view pusher. The operating question is "What made you want to edit this page?"

• What patterns have you noticed in the newcomers/edits you feel compelled to assist with (e.g. write a personalized message on their talk page, add a reference to a content addition that lacked one)?
• What patterns have you noticed in the newcomers/edits you do not feel compelled to assist with (e.g. revert the edits they made, post a templated warning message on their talk page)

There are a bunch of factors that influence this. I'll be less friendly to COI editors, and least friendly to paid COI editors. A big positive factor is when the editor shows that they've done work to try to understand guidance themselves rather than immediately asking for help and expecting us to do the bulk of the work for them. Being articulate about what specific problem they're facing and otherwise indicating general competence both makes it easier for me to help and more likely to want to. If the editor is working in an underrepresented content realm (e.g. women, non-Western topics) I'll be more likely to help than if they're working on a well-represented topic. Countervailing that, I'll be more likely to help if they're working in an area I find interesting (which, if you multiply by all editors, means systemic bias). If the editor has a history of talk page warnings or other indications of bad behavior, that's a negative factor.

Thinking ahead, what other kinds of checks can you imagine being useful/valuable?

Oh, so many! I'd look through the editnotices to find a bunch of examples, as well as the abuse filters, maintenance tags, and guidance pages. Here are some (some for more specific situations than others):

• Adds a redlinked entry to a dynamic list
• Adds an image lacking alt text to an article related to accessibility, e.g. Blindness
• Links to a projectspace page from the mainspace body of an article
• Links to a disambiguation page from an article body (already implemented)
• Uses a word from a variety of English different from the one the article is tagged as having
• Uses a word that is likely a typo
• Adds a section titled "criticism" or "controversy"
• Adds a link to an article that is already linked in that section (very likely a WP:DUPLINK violation)
• Adds a reference with the same URL as an existing reference (likely a duplicate)
• Uses a relative time word like "recently" in an article body
• Use a word or phrase indicating a likely personal attack on a talk page
• Adds a reference containing only a bare URL
• Adds an external link in an article body
• Adds many entries to an external links section (likely a violation of WP:ELNO)
• Uses a phrase indicative of likely promotional language (e.g. "mission statement", "award-winning")
• Expands a section titled "Plot" beyond, say, 1400 words (recommend max length is 700)
• Uses a word or phrase listed at MOS:Words to Watch
• Adds a link to a word that is likely an overlink (e.g. a year, or a major country/language/ethnicity/etc.)
• Adds an image gallery to an article on an ethnicity (likely a violation of MOS:PEOPLEGALLERY)
• Adds a short description longer than 40 characters (see WP:SDSHORT)
• Adds a very long quote (likely a summary style violation, if not also a copyright one)
• Adds an article to a category when the article is already in a subcategory

You'd find many more examples talking to other editors. The potential extent of the applications is staggering, which is one reason I hope that once this feature is more fully developed it'll be possible for the community to create our own ones without developer assistance, just as we currently do for e.g. edit filters.

What rules ought to determine whether a change someone is making warrants the reference check being triggered?

I think the ideal situation here is one where there's a medium-importance issue — one not so dire that it warrants an abuse filter, but also not one so minor it can easily just be fixed up later. There are also some issues that are easier to fix when they're introduced rather than later on — citations is the big one here, since it's so much easier for someone to just add the source they used than for someone later to try to find it.

We should consider a few different factors, including:

• How easy is it to offer useful guidance to help newcomers with this issue? (e.g. "use 'colour' here, not 'color'" is easy; "use less promotional language" is not)
• How bad would it be if this issue went unaddressed? (e.g. it's a lot worse if uncited info goes in a medical article than a film article)
• How confident are we that the algorithm correctly identified an issue here?

I'd add to @Sdkb 's list:

• How universal is it?

The goal is to offer this tool to all the Wikipedias, not just the English one. Things that are specific to the English language (e.g., British vs American spelling) or that apply solely to the English Wikipedia (e.g., adds Template:Cite_web to an article that already includes Template:Bluebook_website, because mixing the two templates will never be accepted by the English Wikipedia's Featured Articles process) are not necessarily appropriate.

if someone is trying to add references to the lead, they should probably be informed of LEADCITE and told they should probably place them in the body instead.

This isn't true. That guideline says things like "Any statements about living persons that are challenged or likely to be challenged must have an inline citation every time they are mentioned, including within the lead" and "The presence of citations in the introduction is neither required in every article nor prohibited in any article."

In T327330#8586331, @Whatamidoing-WMF wrote:

if someone is trying to add references to the lead, they should probably be informed of LEADCITE and told they should probably place them in the body instead.

This isn't true. That guideline says things like "Any statements about living persons that are challenged or likely to be challenged must have an inline citation every time they are mentioned, including within the lead" and "The presence of citations in the introduction is neither required in every article nor prohibited in any article."

Fair; my "probably place them" should probably have been a "possibly place them". We're seeing here one early example of the difficulty of boiling down Wikipedia's (very complex and very caveated) guidance into something simple enough to present in an edit check. I'm sure we'll see many more.

In T327330#8581224, @Sdkb wrote:

Thinking ahead, what other kinds of checks can you imagine being useful/valuable?

Oh, so many! I'd look through the editnotices to find a bunch of examples, as well as the abuse filters, maintenance tags, and guidance pages. Here are some (some for more specific situations than others):

• Adds a redlinked entry to a dynamic list

(and many more...)

What I'd like to see is that this be developed more as a platform + api approach, so that in the end, the actual list of rules are maintainable by wikipedia editors, generally expert editors, and sometimes with review required. This could be somewhat like the way AWB rules about fixing typos and other things are maintained by editors expert in writing regular expressions, templates are written by editors with template privileges, bots must be approved at Bot requests, and so on. So I'm envisioning each rule you develop no longer living embedded in the software you are developing, nor in a library in the code base, but rather interpreted from an external source that lives at en-wiki (or any other wiki). You would provide the platform, the API, and the doc, and we'd write the rules.

As for "we'd write the rules", actually, ideally you would provide a "starter set" of rules developed by your team, which could be, for example, the list provided by Skdb above, or whatever basic set you settle upon. Doing so would have multiple advantages, not least of which is that by designing well formed rules, you'd be in effect providing "best practices" models, because many Wikipedia editors with expertise in technical areas may prefer to look at real world examples and create new content by copy & modify, than by reading a long doc page and starting with a blank sheet. It won't escape your notice, that this would offload future complaints and arguments about existing rules, as well as requests for new ones off of your stack, and onto the community of editors interested in the topic, who would deal with it by discussion and consensus, just as we do with everything else (including AWB, Templates, Modules, scripts, bots, and so on). Every once in a blue moon, perhaps some tricky situation would require your intervention, or perhaps some new edit check request might require an upgrade to the API and you'd have to tweak it to make it available, but that should be the rare exception.

If you consider this approach, which I very much hope you do, then a major decision is what kind of format would the "rules" that live at en-wiki take? Lua? Javascript? Regex? Template scripting? Or some home-brew descriptor that you come up with, which specifies "when X happens, then do Y"? Conceptually, this places the "if... then..." as your responsibility as part of the API definition, and the "X" and "Y" as our responsibility. The simpler the format, the better, but you also want it to be able to cover all the reasonable situations. If you defined it based on template language (parser functions and so on), then you have a built in community of editors at Wikipedia with an associated privilege, so that might be an easy approach.

So this might mean, for example, that each rule would be implemented as a template (or possibly, a subtemplate of "Template:EditCheck", to keep them all in one place, kind of like how AWB does it). You write the initial template, the starter set of subtemplates, the doc, and let us take care of the rest. Whatever approach is decided upon, your software then slurps the set of rules when it fires up and interprets them. Then we WP editors can argue on our own time about subtemplate CiteLead, and what we want to do about advising editors who are adding references in the lead, and under what conditions, and what wording to use, and what colors and borders and all the rest of it; all of the business end of gets shifted to the community, where it actually belongs, and not in the mw software, where I presume you care very little about such things, and which will just become a tarpit for you if you do it there. Also, community consensus on such things may shift over time, and having it modifiable by the community puts the onus on us to do it, instead of having an endless stream of requests to change your software.

I'm most familiar with templates, which is why I used that as an example. If you wanted to consider this further, I have ideas about how such an API might work with templates in order to provide the "if X happens, then do Y" functionality, but that's a more detailed (and technical) conversation that should perhaps be held elsewhere than Phabricator.

P.S., I'm not crazy about the word "rule" here by the way; maybe "plug-in", or something. Hope this gives you something to think about, and I hope the suggestion is not too late in the process to be considered.

In T327330#8566397, @Sdkb wrote:

• What edits that involve people adding net new content do NOT require an accompanying reference?

OK, approaching this more seriously, there are very few instances I can think of here. They are...

These examples are wonderful; thank you, @Sdkb. If more ideas emerge in your mind, we'd value knowing.

In the meantime, I've added the examples you shared in T327330#8566397 to T324730 in this edit: T324730#8620525.

In T327330#8566409, @Sdkb wrote:

What additional information/context might you find helpful in assessing the quality of an edit and the intentions of the person making it so that you can decide how best to respond to it/them?

Well, to assess the quality of the edit...

Note: I moved this thread over to T329593 in T329593#8620553.

In T327330#8581224, @Sdkb wrote:

Thinking ahead, what other kinds of checks can you imagine being useful/valuable?

Oh, so many! I'd look through the editnotices to find a bunch of examples, as well as the abuse filters, maintenance tags, and guidance pages. Here are some (some for more specific situations than others)...

Woah, what a great list, @Sdkb. I think it would useful to document the ideas you shared in a centralized place where volunteers and staff A) are likely to discover and B) would feel welcomed to quickly jot down ideas for potential future checks themselves.

Do you think a page like Edit check/Ideas mediawiki.org would work?

Regardless of where we put it, I'm thinking it would be useful for each check idea to include "If this, then that" style information about each one along with links to, as you said, abuse filters, maintenance tags, and guidance pages, etc.

hi @Mathglot! Thank you for sharing this wonderful feedback with us and for being patient with me in returning back to what you've shared.

I'm going to respond to the points you raised in T327330#8607428 in separate comments in an effort to make it easier for us to track and explore these different threads...

In T327330#8607428, @Mathglot wrote:

In T327330#8581224, @Sdkb wrote:

Thinking ahead, what other kinds of checks can you imagine being useful/valuable?

Oh, so many! I'd look through the editnotices to find a bunch of examples, as well as the abuse filters, maintenance tags, and guidance pages. Here are some (some for more specific situations than others):

• Adds a redlinked entry to a dynamic list

(and many more...)

@Mathglot: excellent.

I'm thinking I'll create a page on mediawiki.org so that we (e.g. you, @Sdkb, members of the Editing Team and other volunteers, etc.) can collaborate on documenting ideas we think could make for useful checks.

In T327330#8607428, @Mathglot wrote:

What I'd like to see is that this be developed more as a platform + api approach, so that in the end, the actual list of rules are maintainable by wikipedia editors, generally expert editors, and sometimes with review required. This could be somewhat like the way AWB rules about fixing typos and other things are maintained by editors expert in writing regular expressions, templates are written by editors with template privileges, bots must be approved at Bot requests, and so on. So I'm envisioning each rule you develop no longer living embedded in the software you are developing, nor in a library in the code base, but rather interpreted from an external source that lives at en-wiki (or any other wiki). You would provide the platform, the API, and the doc, and we'd write the rules.

I feel energized hearing you suggest this!

Reason being: we (read: you and the Editing Team) seem to be squarely aligned in seeing both:

1. The need for volunteers, on a per-project basis, to be able to audit and iterate upon the rules Edit Check is built upon so that they can ensure it is promoting behavior that aligns with the projects they contribute to's policies and conventions
2. The potential in volunteers being able to author new checks so that they can accelerate the rate at which people who are new to contributing to Wikipedia are able to make edits they are proud of experience volunteers deem productive

T327959 is an effort to start bringing shape to what I was describing above.

@Mathglot: would you be open to reviewing what's currently described in T327959's description and letting me know how/if what you see there matches up with what you're thinking in terms of how Edit Check could be built as a platform?

In T327330#8607428, @Mathglot wrote:
As for "we'd write the rules", actually, ideally you would provide a "starter set" of rules developed by your team, which could be, for example, the list provided by Skdb above, or whatever basic set you settle upon. Doing so would have multiple advantages, not least of which is that by designing well formed rules, you'd be in effect providing "best practices" models, because many Wikipedia editors with expertise in technical areas may prefer to look at real world examples and create new content by copy & modify, than by reading a long doc page and starting with a blank sheet.

What you're describing matches up with how the Editing Team has been thinking about Edit Check evolving: we'd author an initial set of checks so that we can iterate towards the design that is most effective in helping newcomers publish edits they are proud of and experienced volunteers deem useful.

And then with that initial set of, as you said, "best practices" defined, volunteers can use them to modify existing checks or author new ones.

Do you think a page like Edit check/Ideas mediawiki.org would work?

Yes, that would be great! Really, the potential domain for this is everything that goes against best practice but isn't so universally guaranteed bad that we block it with an edit filter or fix it with a bot. That's a huge domain that encompasses a good portion of how experienced editors spend our time, and while not everything in it is potentially machine-readable, a sizeable portion of it is.