Bowser, A, et al. 2020.

Still in Need of Norms: The State of the

Data in Citizen Science. Citizen Science: Theory and Practice,
5(1): 18, pp. 1–16. DOI: https://doi.org/10.5334/cstp.303

RESEARCH PAPER

Still in Need of Norms: The State of the Data in Citizen

Science
Anne Bowser*, Caren Cooper†, Alex de Sherbinin‡, Andrea Wiggins§, Peter Brenton‖,
Tyng-Ruey Chuang¶, Elaine Faustman**, Mordechai (Muki) Haklay†† and Metis Meloche*

This article offers an assessment of current data practices in the citizen science, community science, and
crowdsourcing communities. We begin by reviewing current trends in scientific data relevant to citizen
science before presenting the results of our qualitative research. Following a purposive sampling scheme
designed to capture data management practices from a wide range of initiatives through a landscape
sampling methodology (Bos et al. 2007), we sampled 36 projects from English-speaking countries. The
authors used a semi-structured protocol to interview project proponents (either scientific leads or data
managers) to better understand how projects are addressing key aspects of the data lifecycle, reporting
results through descriptive statistics and other analyses. Findings suggest that citizen science projects
are doing well in terms of data quality assessment and governance, but are sometimes lacking in provid-
ing open access to data outputs, documenting data, ensuring interoperability through data standards, or
building robust and sustainable infrastructure. Based on this assessment, the paper presents a number
of recommendations for the citizen science community related to data quality, data infrastructure, data
governance, data documentation, and data access.

Keywords: citizen science; crowdsourcing; data; data management; FAIR; data quality

Introduction information and communication technologies (ICT); rec-

Citizen science refers to a spectrum of activities where sci- ognition from scientists that involving volunteers can sup-
entists and members of the public collaborate in scientific port and augment their work; funder requirements for
work. While conversations to more concretely define and public engagement or outreach; and the rapid increase in
bound “citizen science” are underway (Eitzel et al. 2017), global education (Silvertown 2009; Cooper 2016). Millions
we consider citizen science inclusive of projects across of people now contribute to citizen science each year.
domains and scales, to include both local, place-based SciStarter, a United States (US)–based directory of citizen
initiatives and broader, crowdsourcing solutions.1 Though science projects and related activities, recorded an average
the phrase citizen science entered the vernacular in the of 30 projects added per month during the course of 2018.
mid-1990s (Bonney 1996; Irwin 1995), members of the Educators in formal and informal settings introduce
lay public have been involved in science for centuries. citizen science with the goal of enhancing topical knowl-
Driving and enabling factors for the current prolifera- edge and public understanding of science (Bonney et al.
tion of activities include the rise of the Internet; increased 2016). Scientists in academic institutions incorporate
smartphone penetration along with the spread of other citizen science into their research programs, with biblio-
metric analysis demonstrating the exponential growth of
publications referencing citizen science in recent years
(Follett and Strezov 2015). Citizen science is also enjoying
* Woodrow Wilson International Center for Scholars, US
increased attention on the policy level, as seen in Europe
†
North Carolina (NC) State University, US
and the US (Nascimento et al. 2018). Members of profes-
‡
Center for International Earth Science Information Network
(CIESIN), The Earth Institute, Columbia University, US
sional and public communities engage in diverse citizen
science activities for a wide range of reasons. Some seek to
§
University of Nebraska Omaha, US
advance the research enterprise, for example, by enabling
‖
Atlas of Living Australia, CSIRO, AU
data collection on scales and resolutions not possible
¶
Institute of Information Science, Academia-Sinica, TW
through professional activities alone (Cooper et al. 2012).
**
Institute for Risk Analysis and Risk Communication (IRARC),
School of Public Health, University of Washington, Seattle,
Others seek to bridge the science-society gap by making
WA, US professional researchers and citizens more accountable to
††
Extreme Citizen Science (ExCiteS), Department of Geography, UCL, each other (Irwin 1995).
GB The growth and formalization of citizen science is sup-
Corresponding author: Anne Bowser (anne.bowser@wilsoncenter.org) ported by professional associations based in Australia,
Art. 18, page 2 of 16 Bowser et al: Still in Need of Norms

Europe, and the US, as well as emerging associations in decision-making, will be limited if the field does not fur-
Asia, South America, and Africa. These organizations pro- ther advance norms around high-quality data collection
vide convening power, and help collect and distribute and management. Several researchers have offered case
best practices on the science of citizen science, includ- studies of individual citizen science projects that excel at
ing through conferences and a peer-reviewed journal various aspects of data collection, management, and use.
(Storksdieck et al. 2016). As further evidence for global These case studies generally document effective practices
reach, the Citizen Science Global Partnership was launched within a specific project, and sometimes offer more gen-
in collaboration with United Nations Environment eralized recommendations in areas including avian pres-
Programme as a network-of-networks supporting global ence and distribution (Sullivan et al. 2017), marine debris
coordination and linking citizen science to the UN. (van der Velde et al. 2017), urban tree inventories (Roman
Sustainable Development Goals (SDGs). Beyond the estab- et al. 2016), and invasive species (Crall et al. 2011).
lishment of new organizations, existing governments and Other researchers have identified and analyzed, for
NGOs are developing resources for their employees, grant- example, data quality practices and fitness-for-use assess-
ees, and partners to conduct citizen science. For example, ments across citizen science initiatives (see for example
the US. Federal Government and partners launched the Specht and Lewandowski 2018; Kelling 2018; Aceves-
CitizenScience.gov platform in 2016, which included a Bueno et al. 2017; Kosmala et al. 2016; Lukyanenko et al.
toolkit, a catalogue of federal citizen science projects, and 2016; Sheppard, Wiggins, and Terveen 2014; Wiggins et
a community page (Nascimento et al. 2018). al. 2011). Still others delved into issues related to stand-
One common theme across these citizen science initia- ardized data collection (Higgins et al. 2018; Sturm et al.
tives is the central importance of data collected or gen- 2017), data management (Schade et al. 2017; Bastin et
erated by the efforts of volunteers who are not typically al. 2017) or concepts like fitness to purpose or fitness for
from scientific professions. As the common denominator use (Parrish et al. 2018). But with the exception of Schade
in nearly all citizen science projects, data are the founda- et al. (2017), who collected data focused on citizen sci-
tion of citizen science: Without proper handling of such ence data access, standardization, and preservation via
data, projects will have limited success. However, the an online survey, little published work in the context of
potential to generate knowledge through primary research citizen science evaluates practices related to the full data
and the reuse of data, and to inform evidence-based lifecycle as defined in Box 1.

Box 1: Data Lifecycle and Data Management.

This box provides definitions of different aspects of the data lifecycle and data management. The purpose is to pro-
vide a high-level overview for citizen science researchers who may be less familiar with terminology and approaches
taken by the research data community.
Data acquisition: Collection, processing, and curation of scientific information. Acquisition can occur through
human observation or automated sensors.
Data quality: Quality assurance/quality control (QA/QC) checks taken across the data lifecycle, from acquisition to
archiving to dissemination. These include validation, cleaning, and checks for data integrity.
Data infrastructure: Tools and technologies including hardware and software that support data collection, man-
agement, and access.
Data security: Methods of protecting data from unauthorized access, modification, or destruction through proper
system security and staff training.
Data governance: Rules for the control of data including provisions for stewardship, privacy, and ethical use,
including ensuring the protection of personally identifiable information (PII).
Data documentation: Discovery metadata (structured descriptive information about data sets used by catalog
search tools) and documents describing data inputs and methods used to develop data sets.
Data access: The conditions required for users to find and use data, including metadata and licensing. The research
community has variously adopted standards of open access or FAIR (Findable Accessible, Interoperable, and Reus-
able) data. This includes long-term preservation.
Data services: Tools and web-based applications built with data sets and computer code.
Data integration: The process of combining data from different sources, which requires interoperability enabled
through the use of data and service standards.
For additional information on any of these aspects, visit the World Data System training resources page (https://
www.icsu-wds.org/services/training-resources-guide) or the ESIP Federation data management training clearing-
house (http://dmtclearinghouse.esipfed.org/).
Bowser et al: Still in Need of Norms Art. 18, page 3 of 16

We sought to advance conversations about the state lab notes and code; open peer review; and open access dis-
of the data in citizen science through structured inter- semination of results and data. Within open science, much
views with 36 citizen science projects around the world, of the emphasis to date has been on open data sharing,
representing many scientific domains, and to provide with a strong focus on licensing. Clear data licensing helps
recommendations for improved practice. This research enable open data by clarifying to third party users the sta-
was conducted by citizen science and data experts work- tus of a data set and their ability to apply the data for dif-
ing under the auspices of the International Science ferent purposes and under different conditions. Common
Council Committee on Data (CODATA) and World Data ways to release open data include the Creative Commons
System (WDS). Together, CODATA and WDS formed a task Public Domain Dedication (CC0), the Creative Commons
group, Citizen Science and the Validation, Curation, and Attribution license (CC BY), the Creative Commons
Management of Crowdsourced Data, in 2016. The objec- Attribution-NonCommercial license (CC-BY-NC), or the
tives of the task group were to better understand the Creative Commons Attribution-ShareAlike (CC-BY-SA).
ecosystem of data-generating citizen science, scientific These latter licenses include restrictions that can be prob-
crowdsourcing, and volunteered geographic information lematic, an issue we discuss further in Section 5.
(VGI) to characterize the potential and challenges of these A second, related movement is emerging around making
developments for science as a whole, and data science in data more FAIR. Many of the ideals behind FAIR match the
particular. rhetoric around open science; guiding principles include
Following this introduction, we review current trends transparency, reproducibility, and reusability (Wiklinson
in science and scientific data relevant to citizen science, et al. 2016). Calls for open and FAIR data differ on a few
and then examine current issues around data quality and key points. First, all FAIR data do not necessarily need to
fitness for use in citizen science. The first contribution be open. FAIR is about enabling, rather than securing,
of this paper is an exploratory empirical investigation access to information. Whereas open data are necessarily
into the state of the data in citizen science. We present free of charge, FAIR data could be accessible but behind a
our methods and results of the survey of practices before paywall. Second, while open science can be described as
discussing the results. This paper also contributes practi- a paradigm, or an approach to scientific research, FAIR is
cal and research-oriented recommendations. As an initial more prescriptive, offering concrete guidelines and even
step toward offering concrete guidelines, we identify a list checklists for researchers to follow (Wilkinson et al. 2016).
of good data management practices that may be helpful Practices around cataloguing and metadata documenta-
for citizen science projects to consider, particularly if they tion help make data FAIR.
wish to elevate the value of their data for reuse. We also
suggest areas where more research is needed to under- The state of the data in scientific research
stand more about our findings, and maximize the impact Understanding the current state of data management is
of this steadily growing field. critical for understanding and charting progress moving
forward. Notably, the larger scientific community has only
Trends in Science and Scientific Data recently begun to adopt practices related to open and FAIR
Shifting norms around open and FAIR data. One benchmark study of 1,329 researchers across
Norms and practices governing data management are still scientific domains explored practices and perceptions of
emerging in conventional science, and are not yet firmly data sharing (Tenopir et al. 2011).2 At the time of publi-
established across disciplines. One important develop- cation in 2011, 29% of respondents had data manage-
ment in scientific research is the emergence of open and ment plans, while 55% did not and 16% were uncertain.
FAIR (Findable, Accessible, Interoperable, and Reusable) Regarding data access, 38.5% of respondents stored their
principles. Broadly, open science is research conducted data in an organization-specific system. A follow-up study
in a way that allows others to collaborate and contrib- conducted shortly after National Science Foundation
ute (OECD 2020). As a movement or paradigm, open (NSF) policies went into effect reported mixed progress.
science can be traced to the Scientific Revolution of the Perceptions of the value of data sharing increased, but so
late 16th and early 17th centuries when rapid dissemi- did perception of threats, and progress on self-reported
nation of knowledge became a guiding principle for sci- practices was mixed (Tenopir et al. 2015).
entific research (David 2008). Contemporary advocates A number of factors contribute to suboptimal data
argue that open science strengthens research by facilitat- management in scientific research. While researchers are
ing reproducibility through transparency (Munafò et al. generally satisfied with tools for short-term storage and
2017), and makes science more accessible to stakeholders documentation of their data, access to longer-term reposi-
including the general public, though important power dif- tories may be lacking (Tenopir et al. 2011), and citizen
ferences often remain (Levin and Leonelli 2017). Recently, science practitioners may not be familiar with the many
open science has been accelerated by policy initiatives in domain-specific repositories—though in recent years
Australia, the European Union, the United Kingdom, and open repositories such as Dryad and FigShare have grown
the US (Tenopir et al. 2015). in popularity. Beyond the provision of technical tools,
As an umbrella term, open science encompasses a range “Barriers to effective data sharing and preservation are
of components, including participatory research; open deeply rooted in the practices and culture of the research
access to research publications and pre-prints; open access process as well as the researchers themselves” (Tenopir et
to data and methodologies, including processes such as al. 2011, p.1). Incentives are often missing for researchers
Art. 18, page 4 of 16 Bowser et al: Still in Need of Norms

to invest the time and effort required to make their data quality is a key concern in the scientific enterprise because
open or FAIR, since data cleaning and documentation perceptions of poor data quality can influence the willing-
are time-consuming activities that lack the same incen- ness of scientists or policy makers to trust the results of
tives as, for example, publication. Further, an academic citizen science. In the context of a research project, the
culture that tethers scholarly publication to professional construct of data quality means that data are high enough
milestones like the tenure process may actively disin- quality to serve a project’s goals: there are no universal
centive openness and sharing if researchers fear getting criteria to establish quality in scientific data because it is
scooped. And volunteer citizen scientists are not necessar- inherently contextual. In acknowledgement of this real-
ily motivated by the same incentives as researchers, but ity, the concept of fitness for use is frequently applied in
rather factors such as personal interest, learning, creativ- citizen science (Kosmala et al. 2016), with the focus on
ity, socialization, and the desire to contribute to scientific designing project processes with the end in mind (Parrish
research (Jennett et al. 2016; Rotman et al. 2012). et al. 2018). For example, in air-quality monitoring, low-
Researchers have also started to study data reuse, cost sensors cannot currently compete with professional
defined as the use of data by the original data collector or instruments for achieving precision and accuracy at
third-party users, sometimes by combining the data with the levels necessary for regulation (Castell et al. 2017).
other data, for the same or different purposes for which Therefore, one goal of citizen science air-quality projects
they were originally collected. One study found that per- may be to get regulators to take notice when systemati-
ceived utility of a data set was the single strongest factor cally collected data indicates a potential problem meriting
leading to reuse, and concluded that the value of reuse further investigation. Low-cost (including commercial or
should be more widely demonstrated to the academic open-source/do-it-yourself) sensors are of suitable quality
community (Curty et al. 2017). Efforts to make data discov- to be fit for this, and often other, purposes.
erable, promote the use of strong metadata, and improve When used to describe an individual data record, data
norms and practices around data attribution and citation quality typically refers to the accuracy and precision with
could all lead to more data reuse. Regarding citation, the which a data value represents a measurable parameter of
use of persistent identifiers (e.g., Digital Object Identifiers an entity or phenomenon. At a whole dataset level, data
[DOIs]) can ensure that researchers are able to refer to quality refers to all attributes being accurately measured
a unique data set produced at a given point in time by using a standard/common protocol and accurate instru-
providing persistent URLs to data that are retained even mentation.3 Higher-quality data accurately and precisely
if, for example, data moves from a project website to a represent reality, whereas low-quality data are a poor or
longer-term repository. This is important for traceability inconsistent representation. Errors in measurement can
in scientific findings as well as for appropriate attribution. be random (scattered) or systematic (always wrong or
biased in the same direction), and they can arise owing
The state of the data in citizen science to poor instrumentation (imprecise, poorly calibrated,
The White House memorandum Addressing Society and or old) and operator errors, which usually introduce sys-
Scientific Challenges through Citizen Science and Crowd- tematic biases in data. Therefore, measurement accuracy
sourcing (Holdren 2015) offers three core principles for may be affected by several factors, including the training
citizen science: Contributions of volunteers should be 1) and competence of volunteers; sensitivity, calibration and
fully voluntary, 2) meaningful, and 3) acknowledged. Simi- construction quality of measuring instruments; establish-
larly, the European Citizen Science Association (ECSA)’s 10 ment of a consistent sampling frame; the methods used in
Principles of Citizen Science (ECSA 2016) include “citizen taking/determining measurements and their consistency
science project data and metadata are made publicly avail- over time and space and across volunteers; and delays
able and where possible, results are published in an open between sample collection and measurement (in lab
access format.” These codes suggest that data sharing, settings). With respect to field-based observational facts
including through publication, may be necessary to fulfill such as species occurrence recording, competency and
a core best practice of citizen science. Some researchers attention to detail by citizen scientists can affect factors
document the importance of report-backs, or the process such as correct identification, spatial accuracy, precision
of sharing individual and collective results with volunteers and uncertainty, and date/time precision. In addition,
in ways that are meaningful and useful to them (Bonney third-party perceptions of data quality can be affected by
et al. 2009; Morello-Frosch et al. 2009; Gallo and Waitt whether records have been verified or validated by experts
2011). Related, there is often a noticeable commitment or if there are methods or additional data sets available to
within citizen science projects to publish academic pub- cross-validate or even triangulate results.
lications in open access journals (although fees can be a However, the actual quality of data has significance
barrier to follow-through). However, the realities of data only in the context of usage. This is a relative concept that
sharing may suggest differently: One study of open bio- relates to fitness for use (Chapman 2005), i.e., for some
diversity data available through the Global Biodiversity applications, low-quality data may be acceptable. One of
Information Facility (GBIF) found that citizen science the underlying premises of citizen science in the field of
datasets were among the least open (Groom et al. 2016). biology, for example, is that scores of amateur scientists
Beyond open data, a significant portion of research on can collect data over much larger areas and longer periods
data practices addresses data quality. The topic of data than would ever be possible by highly trained biologists
Bowser et al: Still in Need of Norms Art. 18, page 5 of 16

alone. Thus, in some studies, the lower quality is balanced We reviewed citizen science typologies and other classifi-
by a far wider scope, demonstrating that almost all data cation schemes to create the sampling framework. Typolo-
has value depending on the purpose for which it is to be gies were largely drawn from academic research, and cov-
used. In addition, citizen science data may be analyzed ered aspects of citizen science including governance model
along with other scientific or instrumental observations (Haklay 2013; Shirk et al. 2012) and scientific research
as a method of either validating or cross-validating the discipline (Kullenberg and Kasperowski 2016; Follett and
data, or complementing data of known quality with a Strezov 2015). Other classification schemes included UN
larger sample size. regions for capturing geographic distribution, and con-
Researchers typically consider data quality and fitness trolled vocabularies used to document variables including
for use in individual project design, explaining the factors type of hosting organization (e.g., university, community-
affecting data quality within the text of research papers. based group, etc.).
However, such explanations are not always documented The sampling framework allowed us to search for pro-
in metadata accompanying the primary raw and processed jects representing different types of diversity (e.g., in
datasets used in the research, and if they are documented, governance, in scientific research discipline, and in geo-
it is rarely in structured, standardized formats. These cases graphic distribution). Using this framework, we recruited
both create a number of significant problems and con- participants through a three-step process. Our partici-
straints for secondary users of the primary data. pants were recruited following a purposive sampling
For a variety of reasons, researchers are increasingly scheme designed to capture data management practices
turning to individual and aggregated datasets collected from a wide range of initiatives through a landscape sam-
by other projects as primary or secondary data (e.g., to pling methodology (Bos et al. 2007).4 First, we pulled
augment their own original datasets) for their research. a random sample of citizen science projects from the
These secondary applications of data are highly depend- SciStarter database, requesting the listed contact for each
ent on researchers having a clear understanding of the project to participate in our study. In this initial sample,
provenance, methods, data-quality constraints, and prior we found that projects in environmental citizen science,
treatments of datasets in order to support decisions about particularly biodiversity, and projects based in the US were
fitness for use in their particular application of the data. over-represented.
For secondary users of data to be able to assess fitness for We then used our sampling framework to identify gaps
use, they must be able to efficiently filter, sort, and select in the sample and sought out projects not necessarily
particular datasets that satisfy the quality criteria for their listed on SciStarter to fill the gaps. As gaps were filled, the
purpose. To accomplish this, it is critical for dataset meta- research team met numerous times to discuss our evolv-
data to describe the quality aspects of the data as compre- ing sample and early findings. The research team then
hensively as possible, including its provenance, treatment, conducted additional purposive sampling until theoretical
constraints, and biases, in a structured, standardized way. saturation was reached (Weed 2006) at 36 interviews with
Our results indicate that currently, well-documented data citizen science projects and platforms. Note that while
are not always the norm in citizen science. this sampling strategy appears successful in covering a
In summary, while the citizen science community may wide range of citizen science projects, it is not intended to
lag slightly behind ideals, this is probably in part because be statistically representative of the field as a whole, and
of the rapid evolution of scientific norms of open data, only English-speaking projects were represented.
data publication, metadata, data documentation, and data
reuse over the past decade, which in fact means that many Data collection
corners of the global scientific enterprise are rushing to We began our structured interview protocol with ques-
catch up. We turn now to our methods and results, before tions from our sampling framework. Interview questions
turning to a discussion of what the evolution of norms addressed various practices related to data quality and
means for the citizen science community and how the data management (see Appendix B for the interview pro-
community can improve its data practices. tocol). In addition to supporting our sampling methodol-
ogy, these questions enabled us to collect valuable infor-
Methods mation to help characterize our sample. The second part
The level of detail we sought about data management of our interview protocol addressed practices related to
practices was rarely conveyed on project websites. There- data quality and data management. We focused on these
fore, to better understand the state of the data in citizen practices because our review of the literature suggested
science, we conducted structured interviews with project that practices related to data quality and data manage-
managers or key personnel working on the data manage- ment (as opposed to, for example, data security) may be
ment aspects of 36 citizen science projects (see Appendix unique to citizen science compared with other forms of
A for a full list). scientific research. Grounding our protocol in the existing
literature allowed us to create a structured protocol with
Sampling framework multiple choice rather than open-ended questions. For
Members of the Task Group began by reviewing a range example, rather than asking participants “Where can your
of literature on citizen science data practices across the data be accessed?” we asked, “Can your data be accessed
data lifecycle to inform development of study methods. from: a) Project website; b) Institutional repository; c) Top-
Art. 18, page 6 of 16 Bowser et al: Still in Need of Norms

ical or field-based repository; and/or, d) Public sector data archiving. But in others, an interviewee offered informa-
repository?” tion that was factually incorrect, for example, suggesting
Regarding data acquisition, we asked our participants that a project launched with support from iNaturalist did
to describe the full range of data collection or process- not have the option to apply standardized data licenses
ing tasks that were used in their citizen science research. when, in actuality, iNaturalist does offer this functional-
For data management (including data quality), we asked ity. Because many of the details we asked about were not
about quality assurance/quality control (QA/QC) pro- directly observable in projects’ online presence, we were
cesses, including those related to data collection but also not able to systematically verify all of the data collected.
human aspects such as targeted recruitment or training; This finding informed our analysis and presentation
instrument control such as the use of a standardized of our results. For example, while our sample was large
instrument; and, data verification or validation strategies, enough to support descriptive statistics such as tabula-
such as voucher collection (e.g., through a photo or speci- tions, we believe that the format of statistical analysis
men) or expert review. We asked questions on data access, implies a certainty and confidence in the findings that
including whether access to analyzed, aggregated, and/ is not fully appropriate. A narrative reporting structure
or raw data were provided, and how data discovery and more closely aligns with the relatively exploratory nature
dissemination retrieval were supported (if at all). Because of this study, and emphasizes the reliance of our method-
they relied on known practices identified through existing ology on self-report.
literature, the vast majority of our questions were multi-
ple choice, though participants were encouraged to elabo- Characteristics of sample
rate on their answers or provide additional information. The average start year of the projects in our sample was
Members of the research team conducted interviews or 2011, with the earliest year being 1992 and the most
surveys, either in person, by phone, by Skype, or by email.5 recent being 2017. Our sample was heavily weighted
Each team member followed the same structured protocol toward the environmental and biological sciences (n = 29,
during the interview process, although open-ended ques- 81%), reflecting the early genesis of citizen science in
tions allowed for the collection of richer detail on selected these communities (Schade et al. 2017), but also included
cases. several health-related projects (n = 7, 19%), two VGI ini-
tiatives, a general-purpose crowdsourcing initiative, and
Data analysis a technology development project. Most of the projects
Analysis was conducted through tallying responses, com- were hosted in North America (n = 19, 53%). The remain-
paring responses with previous research, and augmenting ing sample was from Europe (n = 7, 19%), Oceania (n = 7,
structured responses with unstructured comments. We 19%), Asia (n = 6, 17%), South America (n = 2, 6%), and
also compared results with prior quantitative assessments Africa (n = 1, 3%). Host organizations included nonprofit
of citizen science data practices, including Schade et al. organizations (n = 14, 39%); academic institutions (n =
(2017) and Wiggins et al. (2011). 12, 33%); government agencies, including federal, state,
and tribal (n = 7, 19%); and for-profit companies (n = 3,
Results 8%). Partnerships were plentiful, with ten projects (28%)
Although we did not structure interviews directly fol- designating one or more type of organization as host. Our
lowing the data life cycle (Box 1), we solicited responses sample included all participation models according to
relevant to each step in the data lifecycle, except for data the Haklay (2013) typology, though not evenly. The sam-
integration. Note that counts often exceed the total sam- ple included a majority of participatory science projects
ple size because response categories are not mutually (n = 21, 58%), followed by crowdsourcing (n = 13, 36%),
exclusive and many citizen science projects selected mul- distributed intelligence (n=2, 6%), extreme citizen sci-
tiple response options for each item. ence (n = 2, 6%), and volunteered computing (n = 1, 3%).
Early in our analysis, we found a number of discrepan- Several projects reported multiple participation models,
cies between self-reported information and actual prac- for example offering options that included participatory
tices. For example, our protocol asked project personnel science contributions as well as crowdsourcing tasks. In
to tell us, “Does the data set or access point include the terms of geographic scope, 11 projects (31%) were global
name of a person to contact with questions?” A number of in reach, 11 (31%) were national, six (17%) were tied to
people we interviewed responded in the affirmative, and a locality such as a city or specific site, five (14%) were
even suggested a specific name of their designated data regional, and three (8%) involved online-only participa-
point of contact, but a quick review of that project’s digital tion with no geographic component. Most of the projects
presence in data catalogues, websites, and/or data reposi- involved data collection at sites chosen by the contribu-
tories suggested that either no contact was given or the tors, but several involved assignments to work in specific
email listed was a generic one (e.g., info@projectname. locations.
org). In addition, the participants we interviewed, typically
the scientific research leads, were not always familiar with Data life cycle
the details of how their research was being supported Data acquisition
by technological platforms or how their data were being Observational or raw data collection and/or interpreta-
managed. In some cases, an interviewee reached out to tion tasks (e.g., bird watching or monitoring poaching pat-
a colleague to provide follow-up information on data terns) were by far the most prevalent form of research (n =
Bowser et al: Still in Need of Norms Art. 18, page 7 of 16

27, 75%). Specimen or sample collection (e.g., water sam- Third, many projects approached data quality through
ples or animal scat) was also common (n = 13, 36%). Other standardizing data collection or analysis processes.
projects engaged volunteers in cognitive work (e.g., self- Twenty-two (61%) used a standardized protocol. In addi-
reporting of dreams; n = 7, 19%); categorization or clas- tion, five (14%) used disciplinary data standards (e.g.,
sification tasks (e.g., classifying images or labeling points Darwin Core for biodiversity data), and five (14%) used
of interest on a map; n = 4, 11%); digitization/transcrip- cross-domain standards (e.g., of the Open Geospatial
tion (n = 3, 8%); annotation (n = 2, 6%); and specimen Consortium [OGC]).
analysis (including lab or chemical analysis; n = 2, 6%). Fourth, many projects enabled data quality through
Thirteen projects (36%) were classified as having only one instrument control. Fourteen (39%) used a standard-
general task type, typical of many crowdsourcing, distrib- ized instrument for data collection or measurement. Five
uted intelligence (Haklay 2013), and contributory-style (14%) reported processes for instrument calibration.
citizen science projects (Shirk et al. 2012). Twenty-four Finally, a handful of projects documented their data
projects (67%) involved volunteers in multiple research quality practices. Seven projects (19%) shared what was
tasks, suggesting participatory science, extreme citizen classified as other documentation on a project website,
science (Haklay 2013), collaborative, or co-created (Shirk while one project in our sample (3%) offered a formal QA/
et al. 2012) models. QC plan.
In addition to reporting on practices, many participants
Data quality spoke at length about their data quality practices. Some
Interview participants reported a high number of QA/QC indicated that data quality was secured through “very
mechanisms (Figure 1). All projects used at least one simple protocols and instructions.” Upon reflection, one
QA/QC method, while 34 (94%) used more than one noted that the use of simple protocols led to data collec-
method, and 22 (61%) utilized five methods or more. tion practices that were “standardized, but not deliberately
First, twenty projects (56%) conducted expert review, standardized.” Participants also taught us that data qual-
and six (17%) leveraged human expertise through crowd- ity practices are often rich and contextual. One explained
sourced review. Additional data validation strategies how data were vetted according to a six-pronged approach,
included voucher collection (n = 9, 25%), algorithmic fil- where all published observations must be specific, com-
tering or review (n = 5, 14%), and replication or calibration plete, and appropriate (“the content is professional and
across volunteers (n = 4, 11%). Fourteen projects (39%) for the purpose of education, not political or to further
removed data considered suspect or unreliable, while nine personal agendas”). A second described how traditional
(25%) contacted volunteers to get additional information data quality metrics, such as temporal accuracy, were less
on questionable data. relevant to their work than the ability to offer a detailed
Second, projects focused on the human aspects of data reporting of a phenomenon of interest.
quality through training before data collection (n = 25,
69%) and/or on an ongoing basis (n = 11, 31%). Seven Data infrastructure
projects (19%) used targeted recruiting to find highly Our survey did not delve heavily into infrastructure for
qualified volunteers. Four (11%) conducted volunteer two primary reasons. First, the topic of infrastructure did
testing or skill assessment. not emerge as significantly as other topics in our initial

Number of QA/QC methods

8

6
Number of cases

0
1 2 3 4 5 6 7 8 9 10
QA/QC methods Quality Assurance /Quality Control Methods

Figure 1: Number of quality assurance/quality control (QA/QC) methods per project.

Art. 18, page 8 of 16 Bowser et al: Still in Need of Norms

literature review and scoping process. Second, during the Data documentation
interview process, many of the project principals we inter- Regarding accompanying documentation about data col-
viewed were not very familiar with the back end infra- lection activities, 13 (36%) included information on envi-
structure supporting their projects. With this in mind, we ronmental conditions (e.g., weather, location details), 11
noted that many projects adopted existing data collection (31%) identified the methodology or protocol for data
applications and online communities, such as iNatural- collection, three (8%) provided information about vol-
ist, BioCollect, CitSci.org, and Spotteron; leveraged exist- unteers including characteristics or training levels, and,
ing crowdsourcing platforms, such as Zooniverse or Open two (6%) included equipment details or device settings.
Street Map; or developed their own fit-for-purpose plat- In addition, eight projects (22%) included multiple pieces
form with robust infrastructure, often including backups of information from the foregoing categories, most com-
and redundancies. Smaller projects may rely on a volun- monly environmental conditions and protocol details.
teer technician to manage the data infrastructure, and Only twelve (33%) provided no additional information
here the lack of familiarity with the details of the IT back- whatsoever. Participants were also asked a series of ques-
end of projects on the part of project managers may sug- tions about documentation of the research study. Thir-
gest some underlying fragilities. teen (36%) mentioned publishing information about the
methodology or protocol, while eight (22%) documented
Data security limitations. Five projects (14%) offered fitness-for-use
For reasons similar to those offered above, the team did statements or use cases. Sometimes these were simply dis-
not pose specific questions related to the security of the claimers, such as “data is provided as is.” Participants also
systems used to store data (e.g., passwords, encryption, or identified information on different types of documenta-
two factor authentication), nor did we examine provisions tion that might be helpful in fitness-for-use assessments,
for long-term data stewardship (e.g., archiving in trusted including whether a designated contact was available to
digital repositories). As with issues around data infrastruc- answer additional questions.
ture, it is likely that citizen science projects vary in matu-
rity levels with regards to adherence to standard security Data access
protocols, from relatively weak to very robust. And as with Questions on data discovery were designed to probe
larger findings on data infrastructure, we noted that many whether potential users could find information on the
citizen science project leaders struggled to articulate spe- project or data. Questions on access covered raw data, ana-
cifics around data security approaches when the topic lyzed or aggregated data, and digital data services.
arose organically through the interview process. Finally, Eighteen projects made their data discoverable through
while no projects reported data losses or breaches, it is the project website. Ten projects (28%) made data avail-
conceivable that these may have occurred as a result of able through a topical or field-based repository (such as
piecemeal approaches to infrastructure, which itself may GBIF). Further, eight projects (22%) shared their data
reveal the often limited funding available for more robust through an institutional repository, four (11%) through
approaches. a public sector data repository, and two (6%) through a
publication-based repository. Only nine projects (25%)
Data governance did not easily enable secondary users to find their data.
In citizen science and other scientific research, sensitive Notably, some projects’ data were known to be redistrib-
data are often obscured. For the purpose of this study, uted by third parties, but interviewees were unable to
data sensitivity addressed both data or information about specify the full range of discovery and access points (at
citizen scientists or crowdsourcing volunteers who are least three projects).
contributing to research, and sensitive data that is col- Access to cleaned, aggregated data was mixed. Fourteen
lected by a citizen science community (Bowser et al. 2014; projects (39%) published open data, defined as “available
Bowser and Wiggins, 2015.) Twelve projects (33%) speci- for human or machine download without restriction.”
fied that they removed or anonymized personally iden- Thirteen (36%) offered data upon request, including by
tifiable information (PII), five projects (14%) obscured emailing the principal investigator (PI). Interestingly, one
location information (typically for sensitive species), four of the projects that made data available on request had
projects (11%) reported obscuring other confidential actually developed a sophisticated data dashboard and
information, and one specifically did not record individ- gave permissions to 15 local government agencies, not
ual-level information in the first place. One project had advertising this because they lacked the capacity to handle
a social networking model, whereby only members could more subscribers. Six projects (17%) published open data,
view identifying information about other members and but required processes like creating user accounts that
the observations they had made: in essence members effectively prohibited automated access. Seven projects
opted in and volunteered information about themselves. (19%) stated that their data were never available, though
Six projects (17%) that made their data openly available one respondent commented that access “varies,” and
deliberately avoided any obscuring, with one noting that another indicated that data were available “only to project
an informed consent process was used to make sure par- partners.” An additional interviewee noted that “my prior-
ticipants understood and were comfortable with what ity is to publish first the results, and then I want to look
was shared. for the ways that are in place to open those data as well.”
Bowser et al: Still in Need of Norms Art. 18, page 9 of 16

Participants were asked about their use of a persistent willing to provide data “on request.” However, 14 pro-
and unique identifier, such as a GUID (globally unique jects (39%) provided no specific tools for accessing data
identifier) or DOI (digital object identifier), and their resources.
use of a standardized data license. Eleven projects (31%)
offered a persistent and unique identifier to support reuse Discussion
and citation; the other 26 (72%) either did not offer one, Adoption of Best Practices
or participants did not know. Only 16 projects (44%) had a We found projects were generally implementing best
standardized license to support data reuse. For those pro- practices with regard to data quality (as described by Wig-
jects licensing their data, Creative Commons licenses were gins et al. 2011), but were not implementing, and gener-
the most common. CC-BY and CC-BY-SA licenses, which ally not aware of, best practices with regard to aspects of
require attribution, were most frequently adopted (n = data management such as data documentation, discovery,
8, 22%), with five projects embracing CC0 public domain and access.
dedication (14%) and three projects (8%) using another In regard to data quality, we were encouraged to see
license, such as CC BY-NC or CC BY-NC-SA, that prohibited the wide range of practices that projects employed. The
commercial use. Beyond CC licenses, three projects (8%) majority of our sample (34 projects, 94%) used more than
reported holding or co-owning copyright, one project one method to ensure data quality, and 20 projects (56%)
(3%) reported using an Open Database License (ODbL), used five methods or more. That said, many could only
one project (3%) reported another unnamed license. articulate data quality methods when prompted, and only
However, 18 participants (50%) did not identify any one had a systematic documentation of QA/QC through a
standardized license for their data, and two participants formal plan. This suggests that, contrary to some external
(6%) didn’t know whether their project had a license or skepticism (e.g., Nature 2015), the issue with citizen sci-
not. Numerous participants provided commentary. Some ence and data quality is not in actual practices, but with
suggested that licensing was the responsibility of another the documentation—or lack thereof—to describe the care
team member. Others indicated a general desire to “keep it and consideration taken with QA/QC.
open access” or believed that even if a standardized license Many projects demonstrated willingness to make their
was not used, “the site has a FAQ that somehow addresses data available, for example by suggesting that data would
these questions.” Notably, data provided without a license be shared upon request. But we found that such de facto
or explicit terms of use cannot really be considered open attitude to open access was not always backed by the
data, an important detail discussed in greater depth later appropriate licensing required to establish the legal (and
on. ethical) conditions required for reuse, nor was provision of
Projects were typically open to inquiries about their access in formats accessible to human and machine users
data: Twenty-six projects (72%) provided some form of alike a dominant practice. This finding supports prior
contact information for data inquiries, although seven research conducted within the field of biodiversity, which
(19%) had a general project contact but no data-specific found that out of different types of data hosted in GBIF,
contact person, and eight (22%) provided no contact citizen science data were among the worst documented
details at all. and most restrictive (e.g., especially by prohibiting com-
mercial reuse; Groom, Weatherdon, and Geijzendorffer
Data services 2016). While seemingly egalitarian, progressive, and in
Access to analyzed (cleaned, aggregated, summarized, keeping with the community ethos of some citizen science
visualized) data were provided in a variety of forms. Nine- initiatives, the restriction on commercial uses or the inap-
teen projects (53%) shared findings through project pub- propriate application of share-alike licenses6 can prevent
lications or whitepapers, while 16 (44%) shared findings third parties from providing value-added data and services
through peer-reviewed publications. Many projects noted based on raw data, and may stymie private sector research
that scholarly publication was “a longer-term goal.” Only and innovation that could be in keeping with project and
six projects (17%) provided no access to analyzed data. participant values. It may also hinder a project’s goals; for
Many projects used other mechanisms for sharing, some example, a primary customer of citizen science data for
of which were specific to the audiences they served. For mosquito-vector monitoring could be commercial mos-
example, one project offered a dashboard for State gov- quito control groups. In addition, if citizen science data
ernment agencies with explicit partnership agreements are enhanced owing to significant investments by com-
to access data, but did not make this service available to panies, they may represent a real value proposition for all
others. data consumers, including citizen scientists themselves. In
Twenty-three projects (64%) offered digital data ser- such cases, CC-BY-NC and CC-BY-SA licenses can be viewed
vices. Of these, 16 (44%) provided tools for user-speci- as regressive and not in keeping with open science princi-
fied queries or downloads (with several also providing ples, though the debate is nuanced and open. For exam-
application programming interfaces [APIs] for machine ple, biodiversity observations shared under an NC license
queries), 14 (39%) made data available through web ser- cannot be used on Wikipedia (which supports a broader
vices or data visualizations, including maps, 10 (28%) open data policy) to illustrate articles about species for
offered bulk download options, and 5 (14%) provided which citizen science data may be the primary or best
custom analyses or services. In addition, 1 project was available records.
Art. 18, page 10 of 16 Bowser et al: Still in Need of Norms

Some citizen science projects implemented best prac- This was particularly notable for projects whose data
tices in regard to data governance, including access and access, services, and persistent identifiers were provided
control. Many solutions, such as location obscuration by a platform that offered data hosting. While this may
or masking PII, were designed to protect the privacy of be a reasonable option, particularly for smaller or start-
humans and/or sensitive species. Further, at least a hand- up citizen science projects, and whereas taking advan-
ful of the projects that did not leverage these solutions tage of the expertise of an interdisciplinary team is often
had thought about implications like privacy and made a advocated, it can clearly lead to a lack of awareness about
deliberate decision to prioritize, for example, principles data practices, with potential consequences for data
like notice and informed consent (see also Bowser et al. strategies. Outsourcing may lead to, or be a sign of, inat-
2017). tention to the importance of decisions made by the data
In respect to data provenance and traceability, the use host. This inattentiveness could lead to issues down the
of DOIs and appropriately explicit licensing statements is road if infrastructure should fail or security be lax. As one
an issue for establishing scientific merits. One respond- respondent explained, each project that was affiliated
ent indicated that “The data could have been referenced with the larger program made their own data sharing
in publications, but we don’t know about it,” a situation decisions, but deciding to make data openly available did
that could be remedied by the use of DOIs. The global not mean that that the lead researcher assumed respon-
biodiversity informatics community has long recognized sibility for depositing the data into an open access data-
this issue too, and has made some progress on data archiv- base with a persistent identifier. Project managers were
ing (Higgins et al. 2014). As one example, GBIF, together not always sure who was responsible to carry out policy-
with its partners and members, implemented DOI mint- oriented dictates for data management and preservation.
ing and tracking mechanisms to link publications citing While not all data need to be archived, at present prob-
data sources with the original source data, which include ably too little are being proactively preserved for the long
datasets sourced from citizen science projects. While users term.
are not required to use DOIs or even to attribute to the In some cases, the adoption of best practices in citizen
referenced data (CC-BY is a “requirement” that may not be science data management may be similar to or lagging
enforced), it is becoming an increasingly prevalent prac- only slightly behind those of conventional science. For
tice in the science community. This is a persistent issue example, we found that in regard to data discovery and
related to data citation practices: It’s harder to establish access, ten projects (28%) made data available through
impacts for fully open access data. And some practices, a topical or field-based repository (such as GBIF), eight
such as requiring registration for access to data can help (22%) through an institutional repository, four (11%)
to track usage, but may serve as an impediment for some through a public sector data repository, and two (6%)
users (Wiggins et al. 2018). through a publication-based repository. In comparison,
Finally, when datasets are not adequately described Tenopir et al. (2015) found that 27.5% of the research-
with relevant metadata, their potential for secondary ers in their survey made their data available through a
uses is significantly compromised, frequently resulting discipline-based repository, 32.8% through an institu-
in whole datasets being discounted as untrustworthy and tional repository, and 18.4% through a publication-based
reinforcing the perceptions of poor rigor in citizen sci- repository. Comparing these studies suggests that both
ence. Addressing this perception is critically important for citizen and conventional science lag far behind the ideal.
citizen science–generated data to gain more trust within But the consequences are more significant for citizen sci-
the research sector. ence. Widespread adoption of best practices in data man-
Across all aspects of data management, we found a few agement in citizen science would provide much needed
projects following best practices in every category, but transparency about data collection and cleaning practices
most projects had a mishmash of practices and a clear and could go a long way in advancing the reputation of
work-in-progress narrative with respect to evolving prac- the field. It could also help satisfy citizen science’s com-
tices as project activities progressed. As one respondent mitment to ethical principles, as outlined in the Holdren
commented, “We really want scientists to use the data Memorandum and ECSA’s 10 principles of citizen science
but we’re not at a point where we would recommend (Holdren 2015).
that they use the data,” and multiple projects reported While the questions on data management and discovery
plans to achieve higher levels of data management for practices often focused on a scientific user audience, it is
several items we asked about. Further, many respond- important to recall that the scientific research community
ents, including project managers who had dedicated IT isn’t always the primary audience for a citizen science pro-
support or leveraged an external platform, often did not ject: Local communities, students, or other parties may be
know details of their data management practices, as a target audience, for whom access through a project web-
these duties were delegated to others (consistent with site is preferable and analyzed products may be preferred
Wiggins et al. 2011). In a similar vein, several respond- over raw data access. However, data access is also reflective
ents noted that they had not written their project’s of current archival practices and long-term stewardship
data management plans nor designed the technological choices. From this perspective, most of the projects in
workflows themselves; these tasks had been outsourced, this study were not positioned to ensure long-term access
leaving our respondents unable to fully answer the to data, and in the majority of cases, data sustainability
questions asked. appears tenuous at best.
Bowser et al: Still in Need of Norms Art. 18, page 11 of 16

Infrastructure and technology impacts interviewee noted that adopting a third-party platform to
Databases, software applications, mobile apps, and other manage their data did not allow them to direct data man-
e-infrastructures supporting citizen science have a sig- agement practices because they didn’t have control of the
nificant role to play in facilitating improvements in data technical infrastructure to impose their own field-specific
quality. Such infrastructures can, if they conform to appro- or project-specific preferences. This presents a significant
priate standards and use good design principles, make the challenge for infrastructure providers, as it suggests that
data more discoverable, more accessible, more reusable, software is expected to be both highly configurable around
more trusted, more interoperable with other systems, individual user needs while applying standards, rules,
more accurate, and less prone to human-induced errors and workflows that assist users to apply best practices in
(Brenton et. al. 2018). Good design and open infrastruc- data collection and management. At the extremes, these
tures enable efficient and simple data recording and are diametrically opposed concepts, but it is possible to
management by using workflows, processes, and user- provide flexible solutions within a standards-constrained
centered design to minimize the risk of user errors and environment. Achieving the right balance between flexi-
ensure that consistent data formats and mandatory attrib- bility and appropriately structured constraints will require
utes are recorded correctly, along with consistent use of both project owners and infrastructure providers to be
vocabularies, spatial referencing, and dates. At the same aware of standards and best practices, as well as for pro-
time, providing project managers with adequate and eas- viders to be transparent as to if or how they are applied in
ily understood reference information about the default their platforms.
policies that apply to hosted data seemed to be a clear gap
for our respondents. The human dimension
At the global scale, and indeed in many countries, it A fundamental rationale for improving data management
would be fair to say that the e-infrastructures currently practices in citizen science is to ensure the ability of citi-
supporting the majority of citizen science projects are zens, scientists, and policy makers to reuse the data for
largely functioning independently of each other and are scientific research or policy purposes. Mayernik (2017)
not often adequately ascribing metadata to describe the explores how hard and soft incentives can help support
datasets and methods. In addition, very few e-infrastruc- open data initiatives. Hard incentives include require-
tures are currently implementing any commonly used data ments by funders like the National Science Foundation
standards. This effectively isolates these systems from each (NSF) in the USA for researchers to supply data manage-
other and from being able to share data in ways that can ment plans or requirements from publishers that man-
open doors to important new scientific insights through, date publishing data in conjunction with a research arti-
for example, larger aggregated views and analyses based cle. Mayernik also uses the concepts of accountability and
on spatially and temporally dense datasets. transparency to explore additional factors that may limit
However, there are examples in some countries where reuse. Transparency includes requirements for making
efforts are being made to bridge the e-infrastructure data discoverable and can be charted on a spectrum. For
divide. Firstly, the Public Participation in Scientific example, providing a link to data online with brief tex-
Research-Core (PPSR-Core) project is an initiative of the tual descriptions is less transparent than registering data
citizen science associations (US, European, and Australian in a catalogue (metadata repository) with standardized
Citizen Science Associations) in partnership with the descriptions and/or tags.
OGC and World Wide Web Consortium (W3C) to develop Culture also has a significant role to play. In line with
a set of standards for citizen science data, metadata, and broader discussions of open science (David 2008; Levin
data exchange protocols. Within each of the association and Leonelli 2017; Munafò et al. 2017), traditional aca-
regions there are separate third-party platform-based demic cultures often fail to incentivize researchers for
initiatives to support individual citizen science projects good data management to enable reuse. Here, the use of
(e.g., CitSci.org, Zooniverse, iNaturalist and SciStarter [US]; DOIs can be a technical solution that also enables cultural
BioCollect [Australia]; and Spotteron [Europe]). Some of change if researchers can get credit when other researchers
these multi-project platforms are already implementing are able to find, use, and ultimately cite their data. There is
the PPSR-Core standards as they evolve and are already also an opportunity for cultural change specifically within
sharing project-level metadata amongst each other to the citizen science community. By evoking aspirational
improve the discoverability of citizen science projects. guidelines such as those outlined in the Holdren Memo
As a next step, researchers working with Earth Challenge and ECSA’s 10 principles (Hodren 2015), linking good data
2020 and the Frontiers open access publication series are management practices to already-articulated community
creating a metadata repository to facilitate the discovery values like transparency can create pressure for research-
and access of citizen science data. ers to make their data more discoverable and accessible as
Assuming that standards and best practices already an ethical imperative.
exist in an accessible and usable form (which was not
universally the case at the time of writing) to apply them Conclusions and Recommendations
in e-infrastructure and data management solutions, pro- While citizen science has emerged as a promising means
viders should codify them into their software to ensure to collect data on a massive scale and is maturing in regard
consistency and offer guidance for users, particularly to data practices, there is still much progress to be made in
those inexperienced with such matters. However, one approaches to the data lifecycle from acquisition to man-
Art. 18, page 12 of 16 Bowser et al: Still in Need of Norms

agement to dissemination. This reflects the speed of devel- self-reported information by respondents. Reliance on
opment of scientific data management norms and the fact self-reported information is particularly challenging given
that the scientific community as a whole has difficulty the discrepancy between self-reported information and
keeping up. However, it may also reflect lack of resources, actual practices, as described above.
particularly for smaller or startup citizen science efforts These discrepancies offer significant opportunities for
that struggle to maintain staff and funding and perhaps research and practical work. While the finding that pro-
find that data management falls to the bottom of the to-do ject leaders do not necessarily understand their data man-
list. Finally, the fact that many of those who start citizen agement practices offers an important insight, there is a
science projects are motivated primarily by intellectual need for clarity regarding what actual practices are most
curiosity, educational goals, environmental justice, or the and least common. A follow-up study could compare self-
desire to inform society about significant challenges, may reported with actual practices by, for example, comple-
be reflected in project founders who may lack the back- menting self-report methodologies with desk research,
ground in data practices that could carry their work to the perhaps developing profiles of projects with certain data
next level. The characterization of data practices in this management practices, or even quantifying the strength
paper is not intended as a criticism of the field, but rather of data management approaches. There is a related oppor-
an effort to identify areas where improvements are needed tunity to conduct studies of research role differentiation
and to provide a call to action and greater maturation. We within citizen science projects, and map the different
will have succeeded to the degree that we have educated types of expertise, such as scientific, technological, or
the citizen science community about emerging practices educational knowledge, represented on a project sup-
that can help to improve the usability of their data for not port team, which may be distributed across a number of
only scientific research but also to solve important societal departments or institutions.
and environmental problems. Our landscape sampling framework sought to identify
For projects that seek to elevate the value of their data and characterize a wide range of practices across differ-
for reuse, we propose a number of steps that could help ent types of citizen science projects. Others, including
to increase conformity to data management best practices Schade and colleagues, have leveraged different meth-
(Box 2). odologies, such as large-scale surveys, that attempt to
There are a number of limitations to this research, gain a more representiative view (2017). Future research
including the small sample size and the reliance on could leverage random or purposive sampling to build

Box 2: Recommendations.
This box provides key recommendations for improving data management practices that can be applied across a wide
range of citizen science initiatives. Recommendations are offered for individual researchers, and for the field writ
large. Additional helpful information may be found in a primer published by DataONE (Wiggins et al. 2013), though
more work may be needed to identify an updated set of best practices for broad citizen science communities to use.
Data quality: While significant quality assurance/quality control (QA/QC) checks are taken across the data lifecy-
cle, these are not always documented in a standardized way. Citizen science practitioners should document their
QA/QC practices on project websites and/or through formal QA/QC plans. Researchers seeking to advance the field
could help develop controlled vocabularies for articulating common data-quality practices that can be included in
metadata for data sets and/or observations.
Data infrastructure: Citizen science practitioners should consider leveraging existing infrastructures across the data
lifecycle, such as for data collection and data archiving, e.g., in large and stable data aggregation repositories. Research-
ers seeking to advance the field should fully document supporting infrastructures to make their strengths and limita-
tions transparent and increase their utility, as well as develop additional supporting infrastructures as needed.
Data governance: Relevant considerations include privacy and ethical data use, such as ensuring the protection
of sensitive location-based information, personally identifiable information (PII), and proper use of licensing. Citi-
zen science practitioners should carefully consider tradeoffs between openness and privacy. Researchers seeking
to advance the field could develop standard data policies, including privacy policies and terms of use, that clearly
outline data governance practices.
Data documentation: Citizen science practitioners should make discovery metadata (structured descriptive infor-
mation about data sets) available through data catalogues, and should share information on methods used to
develop data sets on project websites. Researchers seeking to advance the field could develop controlled vocabular-
ies for metadata documentation, particularly to enable fitness for purpose assessments.
Data access: In addition to discovery metadata, citizen science practitioners should select and use one or more open,
machine-readable licenses like the Creative Commons licenses. Researchers seeking to advance the field should iden-
tify, share information about, and if necessary develop long-term infrastructures for data discovery and preservation.
Bowser et al: Still in Need of Norms Art. 18, page 13 of 16

on these studies and potentially investigate the role of a Ethics and Consent
single variable, such as project governance model, in data NC State University’s Institutional Review Board (IRB) clas-
management. sified this research as not involving human subjects and
Finally, future work could expand across the data life- thus not requiring IRB review.
cycle to focus on such aspects as data infrastructure and
data security, or seek to do a direct comparative study Acknowledgements
between citizen science and research conducted through Rorie Edwards at WDS was a critical contributor who
other means. To the final point, we believe that given facilitated Task Group online meetings as well as prepar-
the ethical imperatives around good data practices that ing the initial Task Group proposal that was submitted to
enable open and FAIR data, citizen science could play a CODATA. In addition, Carolynne Hultquist of the Earth
strong leadership role in the broader community of sci- Institute at Columbia University provided feedback that
entific research. helped improve our final manuscript.

Data Accessibility Statement Funding Information

Because of the potentially sensitive nature of participant The authors would like to acknowledge financial support
responses, qualitative data are not available for reuse. from CODATA for a consultancy that greatly facilitated this
research. Participation from AB and MM was supported
Notes by the Alfred P. Sloan Foundation. CC recognizes support
1
Although citizen science and crowdsourcing dif- from NSF #1835352, Establishing Norms of Data Ethics in
fer in some respects, here the authors collectively Citizen Science. ADS recognizes support from NASA con-
refer to projects gathering data principally through tract NNG13HQ04C for the continued operation of the
the engagement of volunteers as citizen science Socioeconomic Data and Applications Center (SEDAC) and
projects. PB from the National Collaborative Research Infrastruc-
2
The date of this study is notable, as 2011 marked the ture Strategy (NCRIS). TRC was funded in part by the Min-
year that the US National Science Foundation (NSF) istry of Science Technology, Taiwan (grant no. 108-2621-
began mandating that principal investigators (PIs) M-001-006 and 109-2621-M-001-001) and the Research
must include a Data Management Plan as a core com- Center for Information Technology Innovation, Academia
ponent of their proposal. The publication authored by Sinica. EF recognizes NIH, EPA, and Nippon Foundation.
Tenopir and colleagues in 2011, reporting on research MH is funded by European Research Council (ERC) under
activities conducted in 2010, can therefore be helpful the European Union’s Horizon 2020 research and innova-
as a benchmark for understanding norms before NSF tion programme (Grant agreement No. 694767, ERC-2015-
policies took effect. AdG.
3
Accuracy is the degree to which a measurement meas-
ures the actual or real value (proximity to reality), and Competing Interests
precision is the degree to which measurements of the The authors have no competing interests to declare.
same parameter real value are close to each other and/
or are consistent over time. Author Contributions
4
Landscape sampling is not a methodology that seeks ADS, CC, and TRC co-chaired the CODATA Task Group and
to produce a sample that fully and comprehensively obtained funding. AB led development of the sampling
reflects trends within a population—rather, the goal of frame and MH interviewed protocol. CC hosted AB on a
landscape sampling is to uncover a wide diversity of research sabbatical. AB, CC, ADS, AW, PB, TRC, EMF, and
practices within a population. MH interviewed project managers. AB, ADS, AW, CC, and
5
NC State University’s Institutional Review Board (IRB) PB wrote and edited the initial draft of the manuscript; CC,
classified this research as not involving human sub- ADS, AW, PB, TRC, EMF, and MH advised on the research
jects and thus not requiring IRB review. question and study design, and provided edits to the
6
Share-alike licenses require users of data to contribute to manuscript. MM contributed figures and copious edits.
the community any newly developed data or value-added
services that build upon the original raw data, with the References
same license as initially assigned. Depending on the initial Aceves-Bueno, E, Adeleye, AS, Feraud, M, Huang, Y,
license, this may or may not result in derivative products Tao, M, Yang, Y and Anderson, SE. 2017. The accuracy
being made available as open and free of charge. of citizen science data: a quantitative review. The Bulle-
tin of the Ecological Society of America, 98(4): 278–290.
Supplementary Files DOI: https://doi.org/10.1002/bes2.1336
The supplementary files for this article can be found as Bastin, L, Schade, S and Schill, C. 2017. Data and meta-
follows: data management for better VGI reusability. Citizen
Sensor, 249. DOI: https://doi.org/10.5334/bbf.k
• Appendix A. Citizen Science Projects Reviewed. DOI: Bonney, R. 1996. Citizen science: a lab tradition. Living
https://doi.org/10.5334/cstp.303.s1 Bird, 15(4): 7–15.
• Appendix B. Interview Protocol. DOI: https://doi. Bonney, R, Cooper, C, Dickinson, J, Kelling, S, Phillips,
org/10.5334/cstp.303.s2 T, Rosenberg, KV and Shirk, J. 2009. Citizen science:
Art. 18, page 14 of 16 Bowser et al: Still in Need of Norms

a developing tool for expanding science knowledge entists’ data reuse. PloS one, 12(12): e0189288. DOI:
and scientific literacy. BioScience, 59: 977–984. DOI: https://doi.org/10.1371/journal.pone.0189288
https://doi.org/10.1525/bio.2009.59.11.9 David, PA. 2008. The Historical Origins of ‘Open Science’:
Bonney, R, Phillips, T, Ballard, H and Enck, J. 2016. An essay on patronage, reputation and common agency
Can citizen science enhance public understanding contracting in the scientific revolution. Capitalism and
of science? Public Understanding of Science, 25(1): Society, 3(2): 1. DOI: https://doi.org/10.2202/1932-
2–16. DOI: https://doi.org/10.1177/096366251560 0213.1040
7406 ECSA (European Citizen Science Association). 2016.
Bos, N, Zimmerman, A, Olson, J, Yew, J, Yerkie, J, 10 Principles of Citizen Science. 2016. Available at
Dahl, E and Olson, G. 2007. From shared databases https://ecsa.citizen-science.net/engage-us/10-princi-
to communities of practice: A taxonomy of collabora- ples-citizen-science.
tories. Journal of Computer-Mediated Communication, Eitzel, MV, Cappadonna, JL, Santos-Lang, C, Duerr,
12(2): 652-672. DOI: https://doi.org/10.1111/j.1083- RE, Virapongse, A, West, SE, Kyba, CCM, Bowser,
6101.2007.00343.x A, Cooper, CB, Sforzi, A, Metcalfe, AN, Harris,
Bowser, A, Shilton, K, Preece, J and Warrick, E. 2017. ES, Thiel, M, Haklay, M, Ponciano, L, Roche, J,
Accounting for privacy in citizen science: Ethical Ceccaroni, L, Shilling, FM, Dörler, D, Heigl, F,
research in a context of openness. In: Proceedings Kiessling, T, Davis, BY and Jiang, Q. 2017. Citizen sci-
of the 17th ACM conference on Computer supported ence terminology matters: exploring key terms. Citizen
cooperative work & social computing, 2124–2136. Science: Theory and Practice, 2(1): 1. DOI: https://doi.
Seattle WA, February 2017. DOI: https://doi. org/10.5334/cstp.96
org/10.1145/2998181.2998305 Follett, R and Strezov, V. 2015. An analysis of citizen
Bowser, A, Wiggins, A, Shanley, L, Preece, J and Hen- science based research: usage and publication pat-
derson, S. 2014. Sharing data while protecting pri- terns. PloS one, 10(11): e0143687. DOI: https://doi.
vacy in citizen science. Interactions, 21(1): 70–73. DOI: org/10.1371/journal.pone.0143687
https://doi.org/10.1145/2540032 Gallo, T and Waitt, D. 2011. Creating a successful citi-
Bowser, A and Wiggins, A. 2015. Privacy in participatory zen science model to detect and report invasive spe-
research: Advancing policy to support human compu- cies. BioScience, 61(6): 459–465. DOI: https://doi.
tation. Human Computation, 2(1): 19–44. DOI: https:// org/10.1525/bio.2011.61.6.8
doi.org/10.15346/hc.v2i1.3 Groom, Q, Weatherdon, L and Geijzendorffer, IR.
Brenton, P, von Gavel, S, Vogel, E and Lecoq, ME. 2016. Is citizen science an open science in the case of
2018. Technology infrastructure for citizen science. biodiversity observations? Journal of Applied Ecology,
In: Hecker, S, Haklay, M, Bowser, A, Makuch, Z, Vogel, 54(2): 612–617. DOI: https://doi.org/10.1111/1365-
J and Bonn, A (eds). Citizen Science: Innovation in Open 2664.12767
Science, Society and Policy. London: UCL Press. DOI: Haklay, M. 2013. Citizen science and volunteered
https://doi.org/10.14324/111.9781787352339 geographic information: Overview and typology
Castell, N, Dauge, FR, Schneider, P, Vogt, M, Lerner, U, of participation. In: Sui, D, Elwood, S, and Good-
Fishbain, B, Broday, D and Bartonova, A. 2017. Can child, L (eds). Crowdsourcing Geographic Knowledge,
commercial low-cost sensor platforms contribute to 105–122. Dordrecht: Springer. DOI: https://doi.
air quality monitoring and exposure estimates? Envi- org/10.1007/978-94-007-4587-2_7
ronment International, 99: 293–302. DOI: https://doi. Higgins, JW, Baillie, SR, Boughey, K, Bourn, NA,
org/10.1016/j.envint.2016.12.007 Foppen, RP, Gillings, S, Gregory, RD, Hunt, T,
Chapman, AD. 2005. Principles of Data Quality, version Jiguet, F, Lehikoinen, A and Musgrove, AJ. 2018.
1.0. Copenhagen: Global Biodiversity Information Overcoming the challenges of public data archiving
Facility ISBN 87-92020-03-8. for citizen science biodiversity recording and monitor-
Cooper, C. 2016. Citizen science: How ordinary people are ing schemes. Journal of Applied Ecology, 55(6): 2544–
changing the face of discovery. New York: The Overlook 2551. DOI: https://doi.org/10.1111/1365-2664.13180
Press, Peter Mayer Publishers, Inc. Holdren, J. 2015. White House Memorandum Addressing
Cooper, CB, Hochachka, WM and Dhondt, AA. 2012. Society and Scientific Challenges through Citizen Science
The opportunities and challenges of Citizen Science and Crowdsourcing. Washington, DC: White House
as a tool for ecological research. In: Dickinson, JL and Office of Science and Technology Policy.
Bonney, R (eds). Citizen Science: Public Collaboration in Irwin, A. 1995. Citizen Science: A Study of People, Expertise,
Environmental Research. Ithaca, NY: Cornell University and Sustainable Development. New York: Routledge.
Press. Jennett, C, Kloetzer, L, Schneider, D, Iacovides, I,
Crall, A, Newman, G, Stohlgren, T, Holfelder, K, Graham, Cox, A, Gold, M, Fuchs, B, Eveleigh, A, Mathieu,
J and Waller, D. 2011. Assessing citizen science data K, Ajani, Z and Talsi, Y. 2016. Motivations, learning
quality: An invasive species case study. Conservation and creativity in online citizen science. Journal of Sci-
Letters, 4(6): 1–10. DOI: https://doi.org/10.1111/ ence Communication, 15(3): A05. DOI: https://doi.
j.1755-263X.2011.00196.x org/10.22323/2.15030205
Curty, RG, Crowston, K, Specht, A, Grant, BW and Kullenberg, C and Kasperowski, D. 2016. What is citi-
Dalton, ED. 2017. Attitudes and norms affecting sci- zen science?–A scientometric meta-analysis. PloS one,
Bowser et al: Still in Need of Norms Art. 18, page 15 of 16

11(1): e0147152. DOI: https://doi.org/10.1371/jour- ACM conference on Computer supported coopera-

nal.pone.0147152 tive work & social computing, 217–226. Vancouver,
Kelling, S. 2018. Improving Data Quality in eBird-the Canada, on March 15–18, 2012. DOI: https://doi.
Expert Reviewer Network. Biodiversity Information org/10.1145/2145204.2145238
Science and Standards, 2: e25394. DOI: https://doi. Schade, S, Tsinaraki, C and Roglia, E. 2017. Scientific
org/10.3897/biss.2.25394 data from and for the citizen. First Monday, 22(8). DOI:
Kosmala, M, Wiggins, A, Swanson, A and Simmons, B. https://doi.org/10.5210/fm.v22i8.7842
2016. Assessing data quality in citizen science. Fron- Shirk, J, Ballard, H, Wilderman, CC, Phillips, T,
tiers in Ecology and the Environment, 14(10): 551–560. Wiggins, A, Jordan, R, McCallie, E, Minarchek, M,
DOI: https://doi.org/10.1002/fee.1436 Lewenstein, BV, Krasny, ME and Bonney, R. 2012.
Levin, N and Leonelli, S. 2017. How does one “open” sci- Public participation in scientific research: A frame-
ence? Questions of value in biological research. Sci- work for deliberate design. Ecology and society, 17(2):
ence, Technology, & Human Values, 42(2): 280–305. 29. DOI: https://doi.org/10.5751/ES-04705-170229
DOI: https://doi.org/10.1177/0162243916672071 Sheppard, SA, Wiggins, A and Terveen, L. 2014. Cap-
Lukyanenko, R, Parsons, J and Wiersma, YF. 2016. turing quality: Retaining provenance for curated vol-
Emerging problems of data quality in citizen science. unteer monitoring data. In Proceedings of the 17th
Conservation Biology, 30(3): 447–449. DOI: https:// ACM conference on Computer supported cooperative
doi.org/10.1111/cobi.12706 work & social computing, 1234–1245. Baltimore,
Mayernik, MS. 2017. Open data: Accountability and trans- MD, on 15–18 February 2014. DOI: https://doi.
parency. Big Data & Society, 4(2): 2053951717718853. org/10.1145/2531602.2531689
DOI: https://doi.org/10.1177/2053951717718853 Silvertown, J. 2009. A new dawn for citizen science.
Morello-Frosch, R, Brody, JG, Brown, P, Altman, RG, Trends in Ecology & Evolution, 24(9): 467–471. DOI:
Rudel, RA and Pérez, C. 2009. Toxic ignorance and https://doi.org/10.1016/j.tree.2009.03.017
right-to-know in biomonitoring results communica- Specht, H and Lewandowski, E. 2018. Biased Assump-
tion: A survey of scientists and study participants. tions and Oversimplifications in Evaluations of Citi-
Environmental Health, 8(1): 6. DOI: https://doi. zen Science Data Quality. The Bulletin of the Ecological
org/10.1186/1476-069X-8-6 Society of America, 99(2): 251–256. DOI: https://doi.
Munafò, MR, Nosek, BA, Bishop, DV, Button, KS, org/10.1002/bes2.1388
Chambers, CD, Du Sert, NP, Simonsohn, U, Wagen- Storksdieck, M, Shirk, JL, Cappadonna, JL, Domroese,
makers, EJ, Ware, JJ and Ioannidis, JP. 2017. A M, Göbel, C, Haklay, M, Miller-Rushing, AJ,
manifesto for reproducible science. Nature Human Roetman, P, Sbrocchi, C and Vohland, K. 2016.
Behaviour, 1(1): 0021. DOI: https://doi.org/10.1038/ Associations for Citizen Science: Regional Knowledge,
s41562-016-0021 Global Collaboration. Citizen Science: Theory and Prac-
Nascimento, S, Rubio Iglesias, JM, Owen, R, Schade, tice, 1(2). DOI: https://doi.org/10.5334/cstp.55
S and Shanley, L. 2018. Citizen science for policy for- Sturm, U, Gold, M, Luna, S, Schade, S, Ceccaroni, L,
mulation and implementation. In: Hecker, S, Haklay, Kyba, CCM, Claramunt, B, Haklay, M, Kasperowski,
M, Bowser, A, Makuch, Z, Vogel, J and Bonn, A (eds). D, Albert, A and Piera, J. 2017. Defining principles for
Citizen Science: Innovation in Open Science, Soci- mobile apps and platforms development in citizen sci-
ety and Policy. London: UCL Press. DOI: https://doi. ence. Research Ideas and Outcomes, 3: e21283. DOI:
org/10.14324/111.9781787352339 https://doi.org/10.3897/rio.3.e21283
Nature. 2015. Editorial: Rise of the citizen scien- Sullivan, B., Phillips, T, Dayer, AA, Wood, CL, Farns-
tist. Nature, 524(7565): 265. DOI: https://doi. worth, A, Iliff, MJ, Davies, IJ, Wiggins, A, Fink, D,
org/10.1038/524265a Hochachka, WM and Rodewald, AD. 2017. Using
OECD. 2020. Open Science. OECD, 2020. Available at: open access observational data for conservation
https://www.oecd.org/science/inno/open-science. action: A case study for birds. Biological Conserva-
htm. tion, 208: 5–14. DOI: https://doi.org/10.1016/j.bio-
Parrish, JK, Burgess, H, Weltzin, JF, Fortson, L, Wig- con.2016.04.031
gins, A and Simmons, B. 2018. Exposing the science Tenopir, C. Allard, S, Douglass, K, Aydinoglu, A, Wu, L,
in citizen science: Fitness to purpose and intentional Read, E, Manoff, M and Frame, M. 2011. Data Shar-
design. Integrative and comparative biology, 58(1): ing by scientists: Practices and perceptions. PLoS One,
150–160. DOI: https://doi.org/10.1093/icb/icy032 6(6): e21101. DOI: https://doi.org/10.1371/journal.
Roman, L, Scharenbroch, B, Ostberg, J, Mueller, L, Hen- pone.0021101
ning, J, Koeser, A, Sanders, J, Betz, D and Jordan, R. Tenopir, C, Dalton, ED, Allard, S, Frame, M, Pjesivac,
2016. Data quality in citizen science urban tree inven- I, Birch, B, Pollock, D and Dorsett, K. 2015. Changes
tories. Urban Forestry & Urban Greening, 22: 124–135. in data sharing and data reuse practices and percep-
DOI: https://doi.org/10.1016/j.ufug.2017.02.001 tions among scientists worldwide. PloS One, 10(8):
Rotman, D, Preece, J, Hammock, J, Procita, K, e0134826. DOI: https://doi.org/10.1371/journal.
Hansen, D, Parr, C, Lewis, D and Jacobs, D. 2012. pone.0134826
Dynamic changes in motivation in collaborative van der Velde, T, Milton, DA, Lawson, TJ, Wilcox, C,
citizen-science projects. In Proceedings of the 15th Lansdell, M, Davis, G and Hardesty, BD. 2017. Com-
Art. 18, page 16 of 16 Bowser et al: Still in Need of Norms

parison of marine debris data collected by researchers December, 2011. DOI: https://doi.org/10.1109/
and citizen scientists: Is citizen science data worth the eScienceW.2011.27
effort? Biological conservation, 208: 127–138. DOI: Wiggins, A, Bonney, R, Graham, E, Henderson, S,
https://doi.org/10.1016/j.biocon.2016.05.025 Kelling, S, LeBuhn, G, Litauer, R, Lots, K, Michener,
Wilkinson, M, et al. 2016. The FAIR guiding principles W and Newman, G. 2013. Data management
for scientific data management and stewardship. Sci. guide for public participation in scientific research.
Data, 3: 160018. Albuquerque, NM: DataOne Public Participation in Sci-
Wiggins, A, Newman, G, Stevenson, RD and Crowston, entific Research (PPSR).
K. 2011 December. Mechanisms for data quality Wiggins, A, Bonney, R, LeBuhn, G, Parrish, JK and
and validation in citizen science. In e-Science Work- Weltzin, JF. 2018. A Science Products Inventory for Cit-
shops (eScienceW), 2011 IEEE Seventh International izen-Science Planning and Evaluation. BioScience, 68(6):
Conference, 14–19. Stockholm, Sweden, on 5–8 436–444. DOI: https://doi.org/10.1093/biosci/biy028

How to cite this article: Bowser, A, Cooper, C, de Sherbinin, A, Wiggins, A, Brenton, P, Chuang, T-R, Faustman, E, Haklay, MM and
Meloche, M. 2020. Still in Need of Norms: The State of the Data in Citizen Science. Citizen Science: Theory and Practice, 5(1): 18,
pp. 1–16. DOI: https://doi.org/10.5334/cstp.303

Submitted: 12 December 2019 Accepted: 02 July 2020 Published: 04 September 2020

Copyright: © 2020 The Author(s). This is an open-access article distributed under the terms of the Creative Commons
Attribution 4.0 International License (CC-BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium,
provided the original author and source are credited. See https://creativecommons.org/licenses/by/4.0/.

Citizen Science: Theory and Practice is a peer-reviewed open access journal published by
Ubiquity Press. OPEN ACCESS