Wikipedia:Wikipedia Signpost/2019-10-31/News from the WMF
Welcome to Wikipedia! Here's what we're doing to help you stick around
Originally published on the WMF blog on 21 October 2019
Someone signing up to edit a Wikipedia page for the first time may feel a bit like a visitor to a new planet—excited, to be sure, but wildly overwhelmed.
Many new editors must wonder: Just how many millions of articles are in this vast online resource? And which of those millions of articles need that particular person's help?
More than 10,000 users open accounts on Wikipedia each day, and the English language version alone contains nearly six million articles. Many new users give up before they start, however, intimidated by how many different ways there are to get started, and how challenging it can be to learn to edit. Wikipedia wants their help, of course; its success requires an engaged community of contributors, providing wisdom at all levels to enhance the product and enrich the greater good.
With the goal of increasing new users' engagement, the Wikimedia Foundation Research team and their collaborators in the Data Science Lab at EPFL, a technical university in Switzerland, undertook a study to see how Wikipedia might solve this so-called "cold start" problem. How, by asking a few simple questions at the outset of someone's volunteering as an editor, it could nurture and coax that person, guiding them to the places where their expertise could be most helpful—easing them into the experience, and making it satisfying enough to encourage them to stick around. They published the results in the proceedings of the Thirteenth International AAAI Conference on Web and Social Media (ICWSM 2019).
Assembling data without feedback
Wikipedia needs to start by asking questions because of an age-old problem: Because the users are only just then signing up, Wikipedia doesn't yet know anything about them, and can't immediately guide them to a soft landing in their area of interest.
Moreover, other sites, like Amazon, Facebook, YouTube or Spotify, immediately generate a wealth of data on their users, from things they buy, to things they "like", to things they watch or listen to (and how long they watch or listen), to things they recommend.
That's explicit feedback. Wikipedia, however, has an implicit feedback system. It must infer from its data how a user feels about one of its articles, which is harder to do; if someone edits an article, do they like it? Or do they contribute in order to fix the issues they see in the article, even though they are not necessarily interested in the subject? You can't tell.
To set up its recommender system, the researchers sought to establish a questionnaire for new editors. Researchers had a general architecture in mind. They didn't want to ask too many questions, fearing a burdensome process would turn people away. So they wanted to get as accurate a preference as possible, as quickly as possible.
The main concept was to have pre-loaded lists of articles that users could react to: "I want to edit articles in list A, but not list B". The lists, the researchers knew, would have to have articles that were distinctive enough for the preferences to be meaningful. Once a user identified enough lists, the engine would have the data to extract articles that need editing that—hopefully—align with the user's interests.
With that structure in mind, researchers considered populating lists based on the content of Wikipedia articles. That's an obvious choice, but it carried a drawback: some words have multiple meanings, and putting it in a list wouldn't necessarily answer the question of the user's interest.
The other option was picking topics based on how much editing they had received, which researchers called the "collaborative" method. That, too, had a drawback. Some topics get a lot of edits because they may have a lot of typos, for instance, while important topics may be non-controversial and not generate so many edits.
So the researchers decided to use a little bit of both the content method and the collaborative method, to leverage both of their strengths. That involved a complex series of calculations, ultimately optimized into a series of lists that could be presented to new editors.
One list might include the musicians Elvis Costello, Stevie Nicks, and Eddie Vedder. Another could itemize entries on the cities Sikeston, Missouri; Selma, Alabama; and Baltimore, Maryland. One list could feature scientific terms, such as Apomecynini, Desmiphorini, and Agaristinae, while another list offered other terms, like affective computing, isothermal microcalorimetry, and positive feedback.
Each term in a list would be related to another. As new users chose one list compatible with their interests and rejected others outside their expertise, the system would then learn which types of Wikipedia entries to ask those users to help edit.
Reaction from users
Getting to that point took a lot of complicated math and computer science, partly because the researchers didn't want to mess with the actual workings of Wikipedia. (The site has a policy: "Wikipedia is not a laboratory".)
Once the surveys were established and the research project fully described on Wikimedia's Meta-Wiki, the researchers launched the experiment. They took all the users who signed up for an account on English Wikipedia from September to December 2018 and divided them into two sets: those with more than three edits, and those with three or fewer edits.
Once the groups were further sliced and diced, they were sent questionnaires of 20 questions each, in which each "question" was actually a comparison of two lists of 20 articles each—an extension of the illustration above. Once the users made their selections, they received 12 lists of recommendations, each with five articles to read or edit.
After meticulously analyzing the responses, the researchers found that the users were generally satisfied with the recommendations. Naturally, they also found room for improvement. For instance, they could do a better job of distinguishing whether users are editors or readers, and tailoring recommendations that way. And they could make recommendations for articles that really need editing, such as stubs or articles of low quality.
Another fix could be a timesaver for the users. Each question took about a minute to answer; the entire process took 20 minutes. That may drive some users away, but the researchers theorized that they could simply allow a user to answer as many questions as they would like.
Long term changes
In this experiment, researchers focused only on making sure the recommendations were of high quality, and that the users were satisfied. They did not get at the bigger question the work is designed to ensure: Will the users stay on the site? In the future, the researchers will need to see if they can improve retention, but noted that "will need to be a different and longer experiment".
One alternative to the questionnaires could be establishing connections with other Wikipedia users. If the researchers pair a new user with other new users with similar interests, they might form a group that would collectively contribute to Wikipedia. And if they pair a new user with a veteran editor, that person could serve as a mentor to help them along.
Anything that encourages new users to stay and contribute should also help diversity Wikipedia. "The gender and ethnicity distribution of active Wikipedia users is currently very biased, and having a system for holding the hands of newcomers can help alleviate this problem", the researchers noted.
If you like the idea of that—either teaming with other users, or having a mentor—please let us know, and maybe we can arrange it. Perhaps you'll be part of the next experiment!
And if you're a new user, we look forward to playing 20 Questions with you!
Dan Fost is a freelance writer based in San Rafael, Calif.
Discuss this story