Augmentative and Alternative Communication

ISSN: 0743-4618 (Print) 1477-3848 (Online) Journal homepage: http://www.tandfonline.com/loi/iaac20

A systematic quality review of high-tech AAC

interventions as an evidence-based practice

Kristi L. Morin, Jennifer B. Ganz, Emily V. Gregori, Margaret J. Foster,

Stephanie L. Gerow, Derya Genç-Tosun & Ee Rea Hong

To cite this article: Kristi L. Morin, Jennifer B. Ganz, Emily V. Gregori, Margaret J. Foster,
Stephanie L. Gerow, Derya Genç-Tosun & Ee Rea Hong (2018): A systematic quality review
of high-tech AAC interventions as an evidence-based practice, Augmentative and Alternative
Communication, DOI: 10.1080/07434618.2018.1458900

To link to this article: https://doi.org/10.1080/07434618.2018.1458900

View supplementary material

Published online: 26 Apr 2018.

Submit your article to this journal

Article views: 7

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

http://www.tandfonline.com/action/journalInformation?journalCode=iaac20
AUGMENTATIVE AND ALTERNATIVE COMMUNICATION, 2018
https://doi.org/10.1080/07434618.2018.1458900

RESEARCH ARTICLE

A systematic quality review of high-tech AAC interventions as an

evidence-based practice
Kristi L. Morina, Jennifer B. Ganzb, Emily V. Gregoric, Margaret J. Fosterb, Stephanie L. Gerowd,
Derya Genç-Tosune and Ee Rea Hongf
a
Frank Porter Graham Child Development Institute, University of North Carolina, Chapel Hill, NC, USA; bEducational Psychology, Texas A&M
University, College Station, TX, USA; cEducational Studies, Purdue University, West Lafayette, IN, USA; dEducational Psychology, Baylor
University, Waco, TX, USA; eDepartment of Special Education, Faculty of Education, Eskisehir Osmangazi University, Eskisehir, Turkey; fFaculty
of Human Sciences, University of Tsukuba, Tsukuba, Japan

ABSTRACT ARTICLE HISTORY

Although high-tech augmentative and alternative communication (AAC) is commonly used to teach Received 21 June 2017
social-communication skills to people with autism spectrum disorder or intellectual disabilities who Revised 15 March 2018
have complex communication needs, there is a critical need to evaluate the efficacy of this Accepted 21 March 2018
approach. The aim of this systematic review was to evaluate the quality of single-case experimental
KEYWORDS
design research on the use of high-tech AAC to teach social-communication skills to individuals with Evidence-based practice;
autism spectrum disorder or intellectual disabilities who have complex communication needs, to autism spectrum disorder;
determine if this intervention approach meets the criteria for evidence-based practices as outlined intellectual disability;
by the What Works Clearinghouse. Additionally, information on the following extended methodo- augmentative and
logical standards is reported on all included studies: participant description, description of setting alternative communication;
and materials, interventionist description, baseline and intervention description, maintenance, general- high-tech AAC; quality of
ization, procedural integrity, and social validity. The results from 18 multiple-baseline or multiple- design and evidence;
probe experiments across 17 studies indicate that using high-tech AAC to teach social-communica- quality of methodology;
tion skills to individuals with autism spectrum disorder or intellectual disabilities and complex com- systematic review
munication needs can be considered an evidence-based practice, although the review of comparison
(i.e., alternating treatment) design studies did not indicate that high-tech AAC is significantly better
than low-tech AAC.

A systematic quality review of high-tech AAC are often considered a major factor that negatively influences
interventions as an evidence-based practice short- and long-term outcomes (American Psychiatric
Association, 2000). Deficits in social and communication skills,
It is estimated that between 0.8–1.2% of the total US popula-
such as those associated with ASD or intellectual disability,
tion experience communication difficulties due to language
put individuals at heightened risk of experiencing academic
and communication impairments, and, therefore, may require failure, negative social interactions, and employment chal-
additional supports to supplement their spoken communica- lenges (Estes, Rivera, Bryan, Cali, & Dawson, 2011;
tion (American Speech-Language-Hearing Association [ASHA], McNaughton, Bryen, Blackstone, Williams, & Kennedy, 2012).
2005). Increasing incidence rates of autism spectrum disorder Augmentative and alternative communication (AAC) can
(ASD) seem to have partially contributed to the number of remediate some of the communication difficulties experi-
people who present language and communication deficits. enced by individuals with complex communication needs
The global reported prevalence rates of ASD and intellectual (i.e., individuals who have incomprehensible speech, have dif-
disability are 6.2 and 10.37 per 1,000 individuals, respectively ficulty comprehending speech, and/or have limited semantic
(see Elsabbagh et al., 2012; Maulik, Mascarenhas, Mathers, or syntactic skills; Iacono & Johnson, 2004). Furthermore, use
Dua, & Saxena, 2011). ASD and intellectual disability are life- of AAC can contribute to more positive outcomes for this
long conditions that cause a significant impact on the indi- population (Ganz, 2015). The use of AAC supports in general,
viduals and their families. Social and behavioural and high-tech AAC in particular, has risen over the last three
presentations of individuals with ASD, intellectual disability, decades (Light & McNaughton, 2012), possibly due to the
or both may vary depending on their developmental and portability, affordability, and social acceptability of these
chronological ages. However, communication difficulties are devices (Still, Rehfeldt, Whelan, May, & Dymond, 2014).
a common hallmark of ASD and intellectual disability, and Although individuals with ASD or intellectual disabilities

CONTACT Kristi Morin klmorin@unc.edu Frank Porter Graham Child Development Institute, University of North Carolina at Chapel Hill, Chapel Hill, NC
27516, USA
Supplemental data for this article can be accessed here.
ß 2018 International Society for Augmentative and Alternative Communication
2 K. L. MORIN ET AL.

appear to prefer high-tech AAC (Couper et al., 2014; Ganz, 2001), and to inform practitioners regarding which interven-
Hong, & Goodwyn, 2013; Lorah et al., 2013), and a number of tions have the most evidence to support their use with chil-
recent systematic reviews and meta-analyses provide evi- dren with ASD and intellectual disabilities (Camargo
dence of positive outcomes associated with the use of high- et al., 2014).
tech AAC by this population, questions still remain. For Making evidence-based determinations using the guide-
example, only one systematic review has evaluated the meth- lines set forth by the WWC involves several steps (Kratochwill
odological quality of studies involving AAC for participants et al., 2013; Maggin, Briesch, & Chafouleas, 2013; U.S.
with ASD (Logan, Iacano, & Trembath, 2017). Another focused Department of Education [USDE], 2016). First, basic design
solely on communication outcomes related to expressing standards are applied to all included experiments to deter-
wants and needs (Still et al., 2014). In addition, only two mine whether the experiment (a) meets design standards, (b)
studies specifically focused on high-tech AAC: a companion meets standards with reservations, or (c) does not meet
article to the current study (Ganz et al., 2017), and Still et al. standards. These standards evaluate single-case studies
Of these, only the companion article included participants according to the systematic manipulation of the independent
with intellectual disabilities. Drawing on the same data set as variable, the quality and frequency of inter-observer agree-
the current study, the companion article focused on provid- ment data, the ability to demonstrate experimental control,
ing estimates of effects, rather than determining whether or and the number of data points per phase. Second, multiple-
not the use of high-tech AAC met the standard for evidence- probe designs are evaluated according to three additional
based practice (EBP). Furthermore, although three systematic standards that focus on the number of data points within
reviews reviewed the methodological quality of the included the first three sessions, just prior to intervention, and when
studies (Ganz et al., 2017; Hong et al., 2017; Logan et al., the independent variable is introduced at each level. An
2017), only one (Logan et al., 2017) evaluated the mainten- overall evaluation is then applied, based on the experiment’s
ance, generalization, and social validity of the included stud- scores on the standards just described. Once all experiments
ies, and it was restricted to participants with ASD. Finally, have been reviewed, the ones that met the design standards
none of these reviews or meta-analyses provided an over- or met them with reservations are further evaluated with an
arching determination regarding the status of high-tech AAC evidence standard protocol that uses visual analysis (e.g.,
as an EBP for individuals with ASD or intellectual disabilities. consistency of level, trend, and variability within each phase
Evaluating the quality of research on a given topic is crit- and the immediacy of effect, proportion of overlap, and con-
ical for several reasons, including the need to inform the field sistency of data across phases; Kazdin, 2011) to determine
about methodological shortcomings that should be addressed whether the experiment exhibits strong evidence, moderate
by future studies, and to emphasize the ongoing need to con- evidence, or no evidence of a causal relationship. The third
duct high-quality research (Camargo et al., 2014; Jitendra, step in the process is to determine whether (a) a minimum
Burgess, & Gajria, 2011). In addition, identifying methodo- of five studies met the evidence standards with strong or
logically sound studies through the application of quality indi- moderate evidence, (b) included studies were conducted by
cators heightens the credibility of recommendations of at least three different research teams with no overlapping
effective interventions for use in applied settings (Camargo authorship at three different institutions, and (c) the com-
et al., 2014; Kratochwill et al., 2010; Reichow, Volkmar, & bined number of cases (e.g., participants, classrooms, etc.)
Cicchetti, 2008). Finally, using a set of quality indicators to totals at least 20 (USDE, 2016). If these criteria are met, then
objectively evaluate the methodological quality of research is the intervention can be considered an EBP according to the
necessary for determining whether or not an intervention is guidelines set forth by the WWC.
an EBP (Horner et al., 2005; Kratochwill et al., 2010). Although the WWC basic design standards are useful for
Although there is no single set of criteria for identifying determining evidence-based practices, they do not incorpor-
EBPs that is universally accepted in the field of special educa- ate indicators of methodological quality identified by other
tion, several researchers have developed and proposed meth- research teams and professional organizations (e.g., Cook
odologically rigorous guidelines and a set of research quality et al., 2014; Council for Exceptional Children, 2014; Horner
standards to help conduct and evaluate studies that specific- et al., 2005), such as indicators related to (a) the description
ally include single-case experimental designs (e.g., Cook of participants, settings, materials, interventionists, and base-
et al., 2014; Horner et al., 2005; Kratochwill et al., 2010, 2013). line and intervention phases; (b) whether or not the depend-
Among these, guidelines set forth by the What Works ent variable is operationally defined; and (c) the collection of
Clearinghouse (WWC; Kratochwill et al., 2010) have been maintenance, generalization, procedural integrity, and social
used to review research studies using single-case experimen- validity data. The collection and reporting of data on these
tal designs in special education (Kratochwill et al., 2013) and indicators are important for the field for a number of reasons,
to make EBP determinations (e.g., Boles et al., 2016; Camargo including to determine (a) if the intervention is internally
et al., 2014; Hong et al., 2015). EBPs exist when there is valid; (b) whether or not the intervention will generalize to
methodologically sound empirical research to inform the field other people, settings, or behaviours; (c) whether the behav-
and provide guidance on the best interventions available iour or skill will maintain in the absence of intervention; and
(Camargo et al., 2014; Reichow et al., 2008). It is important to (d) whether key stakeholders find the intervention accept-
determine which practices are evidence-based for several rea- able. Adequate descriptions of participants, settings, materi-
sons, including the need to comply with federal mandates als, interventionists, and phases are also important for
regarding the implementation of EBPs (IDEA, 2004; NCLB, replicability and external validity purposes. Without adequate
 AUGMENTATIVE AND ALTERNATIVE COMMUNICATION 3

descriptions of these variables, researchers cannot replicate database are detailed in the companion article (Ganz et al.,
the experiment to determine if the results can be repro- 2017; Appendix A). Supplementary Appendix A provides a
duced, and practitioners cannot determine for whom and graphic depiction of the number of documents screened and
under what circumstances the intervention is effective. excluded at this and remaining steps, and a list of the pri-
The purpose of this review is to expand on the results in mary reasons why articles were excluded during the title/
the companion article (Ganz et al., 2017) by evaluating the abstract and full-text review.
quality of research on high-tech AAC for individuals who
have complex communication needs co-morbid with ASD or
intellectual disabilities. This study extends prior research on Exclusion criteria for title/abstract review
the topic by including participants with intellectual disabil- The following exclusion criteria were used to evaluate the
ities, evaluating the included studies on both basic design titles and abstracts of identified documents: the article did
and extended methodological standards, and making an EBP not include (a) any participants with complex communication
determination. The research questions were: (a) Does the needs co-morbid with ASD, pervasive developmental dis-
body of literature on using high-tech AAC to teach social- order, developmental delay, global developmental disability,
communication skills to people with ASD or intellectual dis- Down syndrome, Rett’s syndrome, intellectual disability, or
ability who have complex communication needs meet the cerebral palsy; (b) a dependent variable focused on receptive
criteria for EBP as outlined by Kratochwill et al. (2010); and or expressive social-communication skills, AAC use, joint
(b) Does the same body of literature meet extended meth- attention, social play with others, or imitation; (c) an inde-
odological standards (i.e., participant description, setting and pendent variable focused on a technology-based interven-
materials description, interventionist description, baseline and tion, intervention for participants with disabilities, or high-
intervention description, maintenance, generalization, proced- tech AAC; or (d) a single-case design (e.g., alternating treat-
ural integrity, and social validity)? ment, multiple-baseline, multiple-probe, multi-element,
changing criterion, reversal/withdrawal). Although alternating
Method treatment designs were not used in the EBP determination,
they were evaluated using both the design quality standards
Article identification and the extended methodological standards to answer the
The search and coding procedures for the title/abstract, full- second research question. The decision to include only sin-
text, and What Works Clearinghouse (WWC) Pilot Single- gle-case research was made when designing the meta-ana-
Case Design Standards (USDE, 2016) review were identical lysis that uses this data set (see Ganz et al., 2017). Because
to the procedures described in the companion paper (Ganz there is not consensus in the field on an effect size measure
et al., 2017). Although the prior work used meta-analytic that can aggregate effects from both group and single-case
techniques to calculate an overall effect size and moderator design research, all group design studies were excluded.
analyses on the data set, the current study applied the Furthermore, single-case designs are commonly used to
WWC Design Standards, extended-methodological standards, evaluate interventions for low-incidence populations (Horner
and evidence-quality standards (USDE, 2016) to the data set et al., 2005) and are the predominant methodology used
to evaluate whether it meets the criteria for an EBP, as out- with individuals with autism spectrum disorder (Wong et al.,
lined by Kratochwill et al. (2010). Readers who are inter- 2013). In addition to these exclusion criteria, articles were
ested in learning about the magnitude of intervention also excluded if they were not peer-reviewed (e.g., editorials,
effect for individuals who have complex communication book reviews, conference papers, etc.) or if they were not in
needs co-morbid with ASD or intellectual disability, as well English. In instances when the information presented in the title
as moderator analyses related to implementer, setting, and abstract was not sufficient to make a determination regard-
implementation strategies (i.e., errorless learning, time delay, ing whether or not the inclusion/exclusion criteria were met,
and verbal prompting), participants’ age, and communica- the document was further evaluated in the full-text review.
tion function are referred to the companion article (Ganz
et al., 2017).
Inclusion and exclusion criteria for full-text review
Documents not excluded during the title/abstract review
Search procedures
were further evaluated using the following full-text inclusion
A research librarian, the fourth author, with expertise in con- criteria: the article included (a) a line graph or other repre-
ducting systematic reviews, searched the following databases sentation of time-series data with at least three individual
through EBSCO on December 10, 2015: ERIC, Academic data points per baseline and intervention phase; (b) a single-
Search Complete, and PsycINFO. The search combined the case experimental design with at least three attempts to
following concepts using keywords and thesaurus terms demonstrate an effect (i.e., multiple-baseline or multiple-
within the database: ‘autism or developmental disability/delay’; probe with at least three levels, reversal/withdrawal with at
‘communication’; and ‘technology’. In addition to the original least four phases, multi-element or alternating treatment
search, a first author and ancestral search were also con- design with at least four data points per intervention
ducted from the included studies using the same search pro- and baseline phases, or changing criterion design with
cedures as the original search. The search strategies for each at least four phases); (c) a receptive or expressive
4 K. L. MORIN ET AL.

social-communication dependent variable (i.e., speaking, Application of design quality standards

speech, aided and unaided AAC use, joint attention, initiating
WWC pilot single-case design standards (USDE, 2016)
and responding, social play with others, imitation, non-verbal
A total of 59 articles met the inclusion criteria for both title/
social interaction, gestures, or paralinguistic communication);
abstract and full-text review and were further evaluated
(d) at least one participant who had complex communication
using the basic design standards for single-case research pro-
needs (i.e., used AAC, solely communicated via echolalia,
posed by What Works Clearinghouse (WWC; Kratochwill
used imitated or prompted communication only, did not use
et al., 2010) and adapted by Maggin et al. (2013). Although
conventional speech, or used 10 or fewer words spontan-
prior versions of the WWC Design Standards (USDE, 2011)
eously and at a level below age expectations); and (e) a tech-
consisted of four standards, two sub-standards, and an over-
nology-based intervention for the participant with disabilities,
all evaluation, the current version (Version 3.0; USDE, 2016)
including high-tech interventions (e.g., speech-generating
includes five standards, four sub-standards, and an overall
device), interventions that require electricity (i.e., requires evaluation. The standards and sub-standards that were added
plugging in or the use of batteries), video, Bluetooth or bug- pertain to evaluating multiple-probe design experiments only
in-ear, or a mobile device used to implement the interven- (i.e., Design Standards 5A, 5B, and 5C). A description of each
tion. Documents were also excluded if they included the fol- standard and the codes used to evaluate each experiment
lowing: (a) AB designs, multiple-baseline/multiple-probe can be found in Supplementary Appendix B.
designs with only two levels, ABA designs, ABAC designs, Documents were evaluated at the experiment level for
ABC designs, or variations of these designs; (b) dependent this stage of the review, meaning that, if an article included
variables that involved expressive communication but only to multiple single-case design experiments (i.e., multiple-base-
respond to academic lessons, such as phonics instruction, line, multiple-probe, reversal, or alternating treatment
spelling, responding to questions about a science lesson, etc.; designs), each experiment was coded individually. As per the
or (c) a technology-based intervention used solely for the recommendation in the WWC Procedures and Standards
teacher or instructor. Handbook (USDE, 2016) any experiment that scored a ‘0’ on
Because the data set reported here is the same as that the overall evaluation, meaning that it did not meet stand-
used in the meta-analysis in the companion article (Ganz ards, was excluded from further evaluation. After application
et al., 2017), articles from the initial search were coded for of the WWC Design Standards (USDE, 2016), eight articles
the following variables after the full-text review: imple- were excluded because they did not include any experiments
menter, setting, implementation strategies (i.e., errorless that met the basic standards or met them with reservations.
learning, time delay, and verbal prompting), participant age, An additional 27 articles were excluded either because they
and communication function. After reviewing the data set, were duplicates (n ¼ 2), the independent variable was not
it was noted that the majority of articles used AAC as the AAC (n ¼ 20), or the participants’ primary disability was not
intervention and there were too few articles in the other ASD or intellectual disability (n ¼ 5). Another article (Sigafoos,
categories (e.g., modelling, social stories, visual strategies, Didden, & O’Reilly, 2003) was removed at this stage because
etc.) to analyse. Therefore, any articles that used other tech- the experiments compared high-tech to no-tech, whereas all
nology-based interventions (i.e., modelling, prompting, social other alternating treatment designs compared high-tech to
stories, other visual strategies) were excluded, and the full- low-tech; therefore, the experiment was not comparable to
text criteria were revised to include only technology-based the other studies in this review. This resulted in a total of 28
AAC interventions. Although the magnitude of effect using additional articles that were excluded prior to being coded
statistical analyses was not calculated in this article, the for the evidence evaluation (see Supplementary Appendix A),
revised full-text criteria were used to review all documents leaving 23 articles that were coded for the additional meth-
identified from the ancestral and first author searches. As odological standards and the quality of evidence standards.
such, studies that used interventions other than high-tech Please see the inclusion/exclusion criteria described earlier in
AAC were not included in this review because they were the text for a full review of why these articles were excluded
not identified in later stages of the original study. at a later stage of the review.
Additionally, whereas the original inclusion criteria included
participants with a variety of disabilities, after conducting Extended methodological standards
moderator coding for the meta-analysis (Ganz et al., 2017), Experiments that met the WWC Design Standards (USDE,
any articles that did not include at least one participant 2016) or met them with reservations were further evaluated
with ASD or intellectual disability were excluded to remove for the following additional methodological standards: par-
individuals with disabilities who were less likely to have dif- ticipant description, setting and materials descriptions, inter-
ficulty with the social aspect of language (e.g., a person ventionist description, baseline and intervention descriptions,
with a physical disability who could not speak due to oral dependent variable, maintenance data, generalization data,
motor difficulties). The full-text criteria were revised as procedural integrity, and social validity. These additional
described previously, and subsequent searches (e.g., ances- standards were adapted from prior work by the Council for
tral, first author, etc.) used the revised full-text criteria. A list Exceptional Children (2014), Horner et al. (2005), and
of articles excluded at this and subsequent stages are avail- Reichow et al. (2008), and were not used to exclude articles
able by request from the first author. from further evaluation. Operational definitions and codes for
 AUGMENTATIVE AND ALTERNATIVE COMMUNICATION 5

each of these standards can be found in Supplementary Instead, these initial analyses prime the rater to provide rat-
Appendix C. ings that integrate these factors via the overall effectiveness
ratings, which are used in a more formulaic manner to allow
a determination of strong, moderate, or weak/no effects.
Application of evidence standards Ratings for each experiment resulted in a categorization of
After experiments were evaluated for basic design and the evidence from the study as weak/no evidence, moderate,
extended methodological standards, they were further or strong evidence (USDE, 2016). A description of the rating
evaluated for quality of evidence, as recommended in the system, including the codes and operational definitions that
WWC Procedures and Standards Handbook (USDE, 2016). were used when evaluating experiments, can be found in
The evidence standards relied on visual analysis and used Supplementary Appendix D. The list of the articles that were
the following features to evaluate each experiment: (a) level, evaluated using the evidence standards can be found in
(b) trend, (c) variability, (d) immediacy of effect, (e) propor- Supplementary Appendix E.
tion of overlap, and (f) consistency of data across phases. In
single-case research, level, trend, and variability are evaluated
Rater descriptions
within each phase, whereas immediacy of effect, proportion
of overlap, and consistency of data are evaluated by visually A total of five raters coded articles throughout each stage of
analysing adjacent phases (e.g., baseline and intervention). the review. One rater was a full professor in the department
When evaluating data within a phase, “level” refers to the of special education at a university in the southern United
mean score, “trend” refers to the slope of the data, and States (second author), with extensive experience conducting
“variability” refers to the range of the data. For between- and mentoring doctoral students in single-case research and
phase evaluations, “immediacy of effect” refers to the change meta-analyses in the area of augmentative and alternative
in level between the last three data points in one phase to communication. The other four raters were doctoral students
the first three data points in the next phase; “overlap” refers in special education with experience conducting systematic
to the proportion of data in one phase that overlaps with literature reviews and single-case experimental design
data in the adjacent phase; and “consistency of data” in simi- research. Because every article was coded by a minimum of
lar phases refers to the extent to which there is consistency two raters at every stage of the review, any articles authored
in data patterns across all phases within the same condition by one of the raters was independently reviewed by at least
(e.g., all baseline phases in a multiple-baseline design). one other rater that was not part of the author team for
For multiple-baseline and multiple-probe experiments, that article.
these six features were used to evaluate the baseline phases,
the intervention phases, between-phase effects, and an over-
Inter-rater reliability
all evaluation of the effectiveness of the intervention.
Because alternating treatment designs are different from Two or more raters evaluated 100% of the included articles
other single-case designs in that effectiveness is evaluated in for each stage of the review. When a disagreement occurred,
terms of whether one treatment or condition is more effect- either the original reviewers came to a consensus after dis-
ive than one or more other treatments or conditions, the cussing the agreement, or a third reviewer was employed. All
standards were modified for experiments with these designs. evaluators were trained to 80% accuracy or above for each
For alternating treatment designs, the six features were used stage of the review, using a sub-set of the included articles.
to evaluate the target independent variable (i.e., high-tech Training occurred in two stages. First, one or more articles
AAC) and non-target independent variable (i.e., low-tech were reviewed by the evaluators as a group to clarify the cri-
AAC) separately for the intervention phase. Evaluation of teria used to rate the articles. Second, once all evaluators
between-phase basic effects for alternating treatment designs were clear on the criteria, they evaluated a small sub-set of
involved comparing two treatments or conditions, rather articles independently and met as a group to discuss their
than adjacent phases, to determine effectiveness. Finally, the results. Any disagreements were discussed and the criteria
quality of the baseline data when evaluating the overall were further clarified. Articles used for training purposes
effectiveness was evaluated. Although this rating is not were not included in the percentage agreement calculations
included in the WWC document (USDE, 2016), it was added for inter-rater reliability (IRR).
because baseline data should be considered when determin- Both percentage agreement and Cohen’s kappa (K; Cohen,
ing the overall effectiveness of an intervention. That is, the 1960) were used to calculate IRR. Kappa is sometimes a pre-
WWC Design Standards (USDE, 2016) do not consider, in the ferred method of calculating IRR, due to its ability to account
overall evaluation, whether there is a significant increasing for chance agreement; however, it has several limitations
trend in baseline, in which case it would be more difficult to that necessitate the inclusion of a second method for calcu-
determine the validity of the intervention data with regard to lating IRR in many cases (McHugh, 2012). For example, kappa
effects of the treatment. The within-phase and initial is an estimate of IRR and, therefore, can lead to instances
between-phase analyses are designed to focus the visual ana- when percentage agreement is high but kappa is quite low
lysis on the individual elements of demonstrations of evi- (Feinstein & Cicchetti, 1990; McHugh, 2012). This issue with
dence within single-case experimental deigns (SCED) and, as kappa occurs frequently enough that it is commonly referred
such, are not intended to be used as summative evaluations. to in the field as the paradox of kappa and occurs when a
6 K. L. MORIN ET AL.

particular rating is prevalent in the data set (i.e., the preva- primarily due to the phase change lines on graphs not lining
lence paradox) or when observers differ in how frequently up with the tick marks. As a result, the raters disagreed as to
they code an occurrence of a particular rating (i.e., the bias which sessions occurred just prior to intervention.
paradox; Byrt, Bishop, and Carlin, 1993). Due to these limita- Additionally, because there were fewer ratings for these
tions, it is generally advisable to include a second measure of design standards due to the low number of multiple-probe
IRR when reporting kappa (McHugh, 2012). Consequently, in designs that were evaluated, when a disagreement did occur
this study percentage agreement is also reported. Although it had a more substantial impact on the overall score. For the
percentage agreement is not without its own limitations, (i.e., extended standards, the lower range of scores were from dis-
it can inflate IRR due to the inclusion of chance agreement), agreements on the setting and materials description, the par-
it is a direct measure, and can provide a clearer picture of ticipant description, and the interventionist description and
IRR scores when presented alongside kappa (McHugh, 2012). were primarily due to articles not describing these variables
Kappa was calculated according to the steps outlined in thoroughly within the original studies. This is a common limi-
Cohen (1960), whereas percentage agreement was calculated tation of intervention research, and has led to the develop-
by dividing agreements by disagreements plus agreements ment of reporting guidelines (see Hoffman et al., 2014). For
and multiplying the resulting number by 100 to obtain a per- the quality of evidence standards, the lower range of scores
centage. To interpret Kappa, the guidelines outlined by was primarily from disagreements on the baseline phase ana-
Landis and Kock (1977) were used: < 0.00 ¼ poor agreement, lysis and within phase analysis. These disagreements were
0–0.20 ¼ slight agreement, 0.21–0.40 ¼ fair agreement, mostly due to the subjective nature of visual analysis, which
0.41–0.60 ¼ moderate agreement, 0.61–0.80 ¼ substantial has historically resulted in poor agreement among raters,
agreement, 0.81–1.00 ¼ almost perfect agreement. According particularly among more experienced raters (Ninci, Vannest,
to the guidelines set by the WWC (USDE, 2016), levels of Willson, & Zhang, 2015). Due to the highly variable IRR
agreement at or above 0.60 for Kappa and 0.80 for percent- scores, two independent raters reviewed every article at
age agreement are considered acceptable. IRR calculations each stage.
for title/abstract and full-text review resulted in 93% and
89% agreement using percentage agreement and 0.79 and
0.67 using kappa, respectively. IRR for the WWC Design Results
Standards (USDE, 2016), extended standards, and quality of
Three primary types of designs were included in this review,
evidence standards resulted in a mean agreement of 89%
multiple probe (MPD), multiple baseline (MBD), and alternat-
(range ¼ 67–100%), 78% (range ¼ 59–98%), and 83%
ing treatment designs (ATD). To aid in the discussion of the
(range ¼ 73–100%) using percentage agreement and 0.63
results, they are divided by the purpose of the design. While
(range ¼ 0.00–1.00), 0.66 (range ¼ 0.00–0.76), and 0.69
the purpose of multiple probe and multiple baseline designs
(range ¼ 0.04–0.80) using kappa, respectively. A more
detailed report of IRR results for the WWC Design Standards, is to determine if an intervention is efficacious by comparing
extended standards, and quality of evidence standards can data in baseline and intervention phases (Kazdin, 2011), the
be found in Supplementary Appendix F. purpose of alternating treatment designs is to determine
Some of the limitations of kappa are illustrated in the whether one treatment or condition is more effective than
scores reported in Supplementary Appendix F. For example, one or more others. As such, only multiple probe and mul-
kappa could not be calculated for Design Standard 1, tiple baseline designs can be used to determine whether or
because the denominator equaled to zero (i.e., 1 – the pro- not a treatment may be deemed an EBP. Although alternat-
portion of units for which agreement is expected by chance) ing treatment designs could not be used in making the EBP
due to both raters coding every article as ‘yes’ for the pres- determination, they were reviewed to provide information on
ence of this variable (i.e., the prevalence paradox). In Design how they fared against the extended methodological stand-
Standards 3 and 5C, kappa was zero because both the pro- ards. The quality of design, methodology, and evidence
portion of units on which the judges agreed and the propor- standards are divided into groups by the types of design,
tion of units for which agreement would be expected by and are summarized in Tables 1–3 and Supplementary
chance were equal. Analysing the 2  2 table that was used Appendices G and H.
to calculate kappa, it is apparent that the prevalence paradox In addition to information on the quality of design, meth-
was responsible for this kappa score, due to an imbalance in odology, and evidence standards, information on the charac-
the scores arising from the fact that both raters used a singu- teristics of the included studies is also presented (see
lar code (i.e., ‘yes’ for DS3 and ‘no’ for DS5C) to rate all Supplementary Appendix I). Of the 23 studies reviewed, most
articles for that variable. Although it is likely that the paradox participants were elementary aged, with only one adult par-
of kappa had an effect on other IRR scores in this data set ticipant. Many different behavioural procedures were utilized
given the extremely low kappa scores for some variables, to teach participants to use high-tech AAC devices, with time
these are some of the more obvious examples. delay and least-to-most prompting being most common.
For the WWC Design Standards (USDE, 2016), the lower Additionally, the most common outcome variable across all
ranges of scores were from disagreements on the additional studies was requesting, supporting the finding from a previ-
design standards for multiple-probe designs (i.e., Design ous review that requesting is the dominant dependent vari-
Standards 5A and 5B). The disagreements for Design able in AAC studies (Wendt, 2009). Although this study
Standard 5B (i.e., probe points prior to intervention) were provides summary information on the included studies,
 AUGMENTATIVE AND ALTERNATIVE COMMUNICATION 7

Table 1. What works clearinghouse pilot single-case design standards (US Department of Education (WWC), 2016) ratings for experiments that met standards,
met standards with reservations, or did not meet standards.
Design standards (DS)
Participant/
Study Design Figure Condition 1 2A 2Ba 2Ca 3 4 5Ab 5Bb 5Cb Overall
Met overall standards
Ganz et al. (2015) MBD 1 N/A 1 1 2 2 1 2 N/A N/A N/A 2
McLay et al. (2015) ATD 1 Mika 1 1 2 2 1 2 N/A N/A N/A 2
ATD 2 Hemi 1 1 2 2 1 2 N/A N/A N/A 2
Met overall standards with reservations
Boesch et al. (2013a) MBD 1 N/A 1 1 2 1 1 1 N/A N/A N/A 1
Boesch et al. (2013b) MBD 1 N/A 1 1 2 1 1 1 N/A N/A N/A 1
Couper et al. (2014) ATD 1 Cameron 1 1 1 1 1 2 N/A N/A N/A 1
ATD 1 Henry 1 1 1 2 1 2 N/A N/A N/A 1
ATD 2 Andrew 1 1 1 1 1 2 N/A N/A N/A 1
ATD 2 Simon 1 1 1 1 1 2 N/A N/A N/A 1
ATD 3 Edward 1 1 1 1 1 2 N/A N/A N/A 1
ATD 3 Jimmy 1 1 1 1 1 2 N/A N/A N/A 1
ATD 3 Shane 1 1 1 1 1 2 N/A N/A N/A 1
Durand (1993) MBD 2 N/A 1 1 2 1 1 1 N/A N/A N/A 1
Ganz et al. (2013) MPD 1 N/A 1 1 2 2 1 1 0 0 0 1
Johnson, McDonnell, Holzwarth, and MBD 3 N/A 1 1 1 1 1 1 N/A N/A N/A 1
Hunter (2004)
King et al. (2014) MPD 1–2 N/A 1 1 2 2 1 1 1 2 0 1
Lorah, Karnes, and Speight (2015) MBD 2 N/A 1 1 2 2 1 1 N/A N/A N/A 1
Lorah, Parnell, and Speight (2014) MBD 1 N/A 1 1 1 1 1 1 N/A N/A N/A 1
Lorah et al. (2013) ATD 1 Joel 1 1 1 1 1 2 N/A N/A N/A 1
ATD 4 Peter 1 1 2 1 1 2 N/A N/A N/A 1
ATD 5 Rick 1 1 2 1 1 2 N/A N/A N/A 1
McLay et al. (2015) ATD 1 Pene 1 1 2 2 1 1 N/A N/A N/A 1
ATD 2 Afasa 1 1 2 2 1 1 N/A N/A N/A 1
Olive et al. (2007) MPD 1 N/A 1 1 2 2 1 1 0 1 0 1
Olive, Lang, and Davis (2008) MPD 3 N/A 1 1 2 2 1 1 1 0 0 1
Sigafoos et al. (2003) ATD 1 Jason 1 1 2 1 1 2 N/A N/A N/A 1
ATD 1 Michael 1 1 2 1 1 2 N/A N/A N/A 1
ATD 2 Jason 1 1 2 1 1 2 N/A N/A N/A 1
ATD 2 Michael 1 1 2 1 1 2 N/A N/A N/A 1
Son, Sigafoos, O’Reilly, and Lancioni (2006) ATD 1 Bruce 1 1 1 1 1 2 N/A N/A N/A 1
ATD 1 Kim 1 1 1 1 1 2 N/A N/A N/A 1
ATD 1 Lucy 1 1 1 1 1 2 N/A N/A N/A 1
Soto, Belfiore, Schlosser, and Haynes (1993) ATD 1 Group home 1 1 2 1 1 1 N/A N/A N/A 1
ATD 1 Work 1 1 2 1 1 2 N/A N/A N/A 1
Steege et al. (1990) MBD 5 N/A 1 1 2 2 1 1 N/A N/A N/A 1
Strasberger and Ferreri (2014) MBD 1 N/A 1 1 2 2 1 1 N/A N/A N/A 1
Trembath, Balandin, Togher, and Stancliffe, (2009) MPD 1 N/A 1 1 1 1 1 1 0 0 0 1
van der Meer, Didden et al. (2012a) MPD 1 N/A 1 1 1 1 1 1 0 1 0 1
van der Meer, Kagohara et al. (2012b) MPD 1 N/A 1 1 1 1 1 1 0 1 0 1
van der Meer, Sutherland, O’Reilly, Lancioni, and ATD 1 Hannah 1 1 1 1 1 2 N/A N/A N/A 1
Sigafoos (2012c)
ATD 1 Jack 1 1 1 1 1 2 N/A N/A N/A 1
ATD 1 Jason 1 1 1 1 1 2 N/A N/A N/A 1
Waddington et al. (2014) MPD 2 N/A 1 1 2 1 1 1 0 0 0 1
Did not meet overall standards
Beck, Stoner, and Dennis (2009) Reversal 2–3 N/A 1 1 2 1 1 0 N/A N/A N/A 0
Buzolich, King, and Baroody (1991) MBD 1 N/A 1 1 2 2 1 0 N/A N/A N/A 0
Cannella-Malone, DeBar, and Sigafoos (2009) MBD 1 N/A 1 1 2 1 1 0 N/A N/A N/A 0
Chung and Douglas (2015) MBD 1 N/A 1 1 1 0 1 1 N/A N/A N/A 0
Couper et al. (2014) ATD 1–2 Andy, Nico 1 1 1 1 1 0 N/A N/A N/A 0
Hanser and Erickson (2007) MBD 2–3 N/A 1 0 0 0 1 2 N/A N/A N/A 0
Lorah, Crouser, Gilroy, Tincani, and Hantula (2014) MBD 1 N/A 1 1 1 1 1 0 N/A N/A N/A 0
Lorah et al. (2015) MBD 1 N/A 1 1 2 2 1 0 N/A N/A N/A 0
Lorah et al. (2013) ATD 2–3 Axel, Aaron 1 1 2 1 1 0 N/A N/A N/A 0
McMillan (2008) MBD 1 N/A 1 0 0 0 1 2 N/A N/A N/A 0
Schepis, Reid, Behrmann, and Sutton (1998) MBD 1 N/A 1 1 0 0 1 2 N/A N/A N/A 0
Schlosser et al. (2007) ATD 1–11 All 1 1 0 1 1 2 N/A N/A N/A 0
van der Meer et al. (2012c) ATD 1 Ian 1 1 1 1 1 0 N/A N/A N/A 0
MBD: multiple baseline design; MPD: multiple probe design; ATD: alternating treatment design; N/A: not applicable; DS1: Systematic manipulation of the inde-
pendent variable; DS2A: inter-observer agreement reported; DS2B: inter-observer agreement frequency; DS2C: inter-observer agreement quality; DS3: attempt to
demonstrate experimental control; DS4: number of data point per phase; DS5A (MPD only): initial baseline sessions; DS5B (MPD only): probe points prior to inter-
vention; DS5C (MPD only): additional probe point considerations.
For Design Standards 2B, 2C, 4, 5A, 5B, and the Overall Evaluation, 2 ¼ meets standard, 1 ¼ meets standard with reservations, and 0 ¼ did not meet standards;
For Standards 1, 2A, 3, and 5C, 1 ¼ meets standards and 0 ¼ did not meet standards.
a
Ratings for Design Standards 2B–2C were modified from the original Design Standards (2016) to add an intermediary score of meets with reservations.
b
Ratings for Design Standards 5A–5C were not included in the overall evaluation.
Definitions and coding procedures for each of the design standards are available in Supplementary Appendix B.
8 K. L. MORIN ET AL.

Table 2. Extended methodological quality standard ratings for experiments that met WWC pilot single-case design standards (US Department of Education
(WWC), 2016) or met with reservations.
Study Figure Participant/Condition PD SMD ID BID DV MD GD PI SV
Alternating treatment designs
Couper et al. (2014) 1 Cameron 1 2 2 2 1 2 0 1 0
1 Henry 1 2 1 2 1 1 0 1 0
2 Andrew 1 2 2 2 1 1 0 1 0
2 Simon 1 2 2 2 1 0 0 1 0
3 Edward 1 2 2 2 1 2 0 1 0
3 Jimmy 1 2 2 2 1 1 0 1 0
3 Shane 1 2 2 2 1 1 0 1 0
Lorah et al. (2013) 1 Joel 2 2 1 2 1 1 0 2 0
4 Peter 2 2 1 2 1 1 0 2 0
5 Rick 2 2 1 2 1 1 0 2 0
McLay et al. (2015) 1 Pene 1 2 1 2 1 2 1 1 1
1 Mika 1 2 1 2 1 2 1 1 1
2 Hemi 1 2 1 2 1 2 1 1 1
2 Afasa 1 2 1 2 1 2 1 1 1
Sigafoos et al. (2003) 1 Jason 1 2 0 2 1 0 0 0 1
1 Michael 1 2 0 2 1 0 0 0 1
2 Jason 1 2 0 2 1 0 0 0 1
2 Michael 1 2 0 2 1 0 0 0 1
Son et al. (2006) 1 Bruce 1 2 1 2 1 0 0 0 1
1 Kim 1 2 1 2 1 0 0 0 1
1 Lucy 1 2 1 2 1 0 0 0 1
1 Group home 1 1 0 2 1 1 0 0 1
Soto et al. (1993) 1 Work 1 1 0 2 1 1 1 0 1
van der Meer et al. (2012c) 1 Hannah 1 2 1 2 1 2 0 1 1
1 Jack 1 2 1 2 1 0 0 1 1
1 Jason 1 2 1 2 1 1 0 1 1
Multiple baseline designs
Boesch et al. (2013a) 1 N/A 1 2 1 2 1 2 0 1 1
Boesch et al. (2013b) 1 N/A 1 1 0 2 1 2 0 2 0
Durand (1993) 2 N/A 1 2 1 2 1 0 0 0 1
Ganz et al. (2015) 1 N/A 2 2 1 2 1 0 0 2 0
Johnson et al. (2004) 3 N/A 1 1 1 2 1 1 0 1 1
Lorah et al. (2015) 2 N/A 1 1 1 2 1 1 0 2 0
Lorah, Parnell, and Speight et al. (2014) 1 N/A 1 2 0 2 1 0 0 2 0
Steege et al. (1990) 5 N/A 1 2 0 1 1 1 0 0 1
Strasberger & Ferreri (2014) 1 N/A 1 1 1 2 1 1 1 1 1
Multiple probe designs
Ganz et al. (2013) 1 N/A 2 2 0 2 1 0 0 2 0
King et al. (2014) 1 N/A 1 2 1 2 1 0 0 2 0
2 N/A 1 2 1 2 1 0 0 2 0
Olive et al. (2007) 1 N/A 2 1 2 2 1 0 0 1 1
Olive et al. (2008) 3 N/A 2 1 1 2 1 0 0 1 1
Trembath et al. (2009) 1 N/A 1 2 1 2 1 0 1 0 1
van der Meer et al. (2012a) 1 N/A 1 2 1 2 1 1 0 1 1
van der Meer et al. (2012b) 1 N/A 1 2 1 2 1 2 0 2 0
Waddington et al. (2014) 2 N/A 1 2 1 2 1 2 1 2 0
PD: participant description; SMD: setting and materials description; ID: interventionist description; BID: baseline and intervention descriptions; DV: dependent vari-
able; MD: maintenance data; GD: generalization data; PI: procedural integrity; SV: social validity; N/A: not applicable; MPD: multiple probe design; MBD: multiple
baseline design.
For the dependent variable standard, 1 ¼ meets standard and 0 ¼ does not meet standard; For all other standards, 2 ¼ meets standard, 1 ¼ meets standard with
reservations, and 0 ¼ does not meet standard.
Definitions and coding procedures for each of the extended design standards are available in Supplementary Appendix C.

further information regarding the magnitude of intervention reservations, and 27 experiments (nine MBD, 16 ATD, two
effect and moderator analyses related to implementer, set- reversal designs) did not meet the standards and were
ting, implementation strategies, participants’ age, and com- excluded from further review. Because the two reversal
munication function can be found in the companion article designs were the only experiments with this design, they are
(Ganz et al., 2017). not discussed in further detail. Regarding those that did not
meet standards, 16 failed to meet standards related to collec-
tion of inter-observer agreement (IOA) data. That is, they
The WWC basic design standard review (USDE, 2016)
either failed to report IOA, collected too little IOA, or failed
A total of 27 multiple baseline and multiple probe experi- to meet the minimal expectations for reliable observations.
ments and 42 alternating treatment design experiments from The other nine experiments failed to meet standards, due to
34 articles were analysed using the basic design standards too few data points in one or more phases. Of all the experi-
from the WWC (see Table 1). Overall three experiments (one ments that met standards with reservations, the primary rea-
MBD, two ATD) met all basic design standards, 41 experi- son they did not fully meet standards was failure to
ments (eight MBD, nine MPD, 24 ATD) met them with disaggregate IOA results by phase or condition. For multiple
 AUGMENTATIVE AND ALTERNATIVE COMMUNICATION 9

Table 3. Overall effectiveness quality of evidence ratings.

Participant/ BL Data Treatment Effect Evidence
Study Design Figure Condition quality points effect ratio of effect
Strong evidence
Boesch et al. (2013a) MBD 1 N/A 2 2 2 2 2
Moderate evidence
Couper et al. (2014) ATD 3 Edward 1 2 1 1 1
ATD 3 Shane 0 2 1 2 1
Durand (1993) MBD 2 N/A 1 1 1 2 1
Johnson et al. (2004) MBD 3 N/A 1 1 1 2 1
King et al. (2014) MPD 1 N/A 2 1 1 2 1
Lorah et al. (2015) MBD 2 N/A 2 1 1 2 1
Lorah, Parnell, and Speight et al. (2014) MBD 1 N/A 2 1 1 2 1
Lorah et al. (2013) ATD 1 Joel 1 2 1 1 1
ATD 4 Peter 2 2 1 1 1
McLay et al. (2015) ATD 1 Mika 1 1 1 1 1
Olive et al. (2007) MPD 1 N/A 2 1 1 2 1
Steege et al. (1990) MBD 5 N/A 2 1 1 2 1
Strasberger and Ferreri (2014) MBD 1 N/A 1 1 1 2 1
van der Meer et al. (2012a) MPD 1 N/A 2 1 1 2 1
van der Meer et al. (2012b) MPD 1 N/A 1 1 1 2 1
van der Meer et al. (2012c) ATD 1 Jason 2 2 1 2 1
Waddington et al. (2014) MPD 2 N/A 2 1 1 2 1
Weak or no evidence
Boesch et al. (2013b) MBD 1 N/A 1 2 0 0 0
Couper et al. (2014) ATD 1 Cameron 2 0 0 0 0
ATD 1 Henry 2 2 0 0 0
ATD 2 Andrew 1 2 0 0 0
ATD 2 Simon 2 2 0 0 0
ATD 3 Jimmy 0 1 0 0 0
Ganz et al. (2013) MPD 1 N/A 2 1 0 0 0
Ganz et al. (2015) MBD 1 N/A 1 2 0 0 0
King et al. (2014) MPD 2 N/A 2 1 0 0 0
Lorah et al. (2013) ATD 5 Rick 2 2 0 0 0
McLay et al. (2015) ATD 1 Pene 2 1 0 0 0
ATD 2 Hemi 2 1 0 0 0
ATD 2 Afasa 0 1 0 0 0
Olive et al. (2008) MPD 3 N/A 0 1 0 0 0
Son et al. (2006) ATD 1 Bruce 2 2 0 0 0
ATD 1 Kim 2 2 0 0 0
ATD 1 Lucy 2 2 0 0 0
Soto et al. (1993) ATD 1 Group home 1 1 0 0 0
ATD 1 Work 2 2 0 0 0
Trembath et al. (2009) MPD 1 N/A 2 1 0 0 0
van der Meer et al. (2012c) ATD 1 Hannah 2 2 0 0 0
ATD 1 Jack 2 2 0 0 0
BL: baseline; N/A: not applicable; ATD: alternating treatment design; MPD: multiple probe design; MBD: multiple baseline design; 2 ¼ strong evidence,
1 ¼ moderate evidence, 0 ¼ no evidence; Definitions and coding procedures for each of the quality of evidence standards are available in Supplementary
Appendix D.

baseline and multiple probe designs, the most common res- methodological standard review, given disagreement in the
ervation was inclusion of too few data points to meet the field regarding minimum standards (Moeller, Dattilo, & Rusch,
ideal standard. The MPD experiments were subjected to add- 2015). All but one experiment met the highest standard
itional review, based on new basic design standards by the related to providing thorough descriptions of baseline and
WWC (USDE, 2016), regarding the amount of data that over- intervention procedures, and all met the highest standards
lap across the levels/cases in implementation while baseline with regard to describing the dependent variables. Regarding
and intervention phases are introduced in other levels/cases. thorough descriptions of the participants, settings, and mate-
These standards were not used to exclude experiments from rials, all articles met the standards or met them with reserva-
further review because they are new and there does not tions, with five fully meeting standards for participant
appear to be consensus in the field regarding these stand- descriptions and 17 fully meeting them for setting and
ards; however, this additional review indicated that none of material descriptions. Ratings were mixed for descriptions of
the MPD experiments met all of the added standards.
interventionists, maintenance and generalization data collec-
tion, procedural integrity, and social validity.
Extended methodological standard review
A total of 44 experiments (18 MBD/MPD, 26 ATD) from 24 The WWC evidence standard review (USDE, 2016)
articles were evaluated using the extended methodological
standards to allow a more fine-grained analysis of the litera- Multiple baseline and multiple probe designs
ture (see Table 2 and Supplementary Appendix C). No experi- A total of 18 multiple baseline or multiple probe design
ments were excluded as a result of the extended experiments were evaluated using the WWC evidence
10 K. L. MORIN ET AL.

standards (USDE, 2016; see Table 3 and Supplementary interventions for the target client group, therefore exceeding
Appendices G and H). Only one experiment (Boesch, Wendt, the first criterion for determination of EBPs. These articles
Subramanian, & Hsu, 2013a,b) demonstrated strong evidence were published by nine teams of researchers from varied
of effect, 11 demonstrated moderate evidence, and six dem- institutions, exceeding the second criterion for EBPs. Across
onstrated weak or no evidence (see Table 3). All of the these experiments, 33 participants were included, exceeding
experiments that had overall moderate or strong evidence the third criterion. On the basis of this analysis, implementa-
had acceptable baseline data patterns. When considering all tion of high-tech AAC can be considered an EBP for support-
experiments, 11 demonstrated clear baseline patterns with ing social-communication skills in individuals with ASD or
flat or decreasing trends, six had slight increasing trends or intellectual disabilities and complex communication needs.
were so variable that trend was difficult to determine, and
one did not meet minimum standards. Regarding data points
per phase, three of the experiments included at least five Discussion
data points per phase and all others included at least three This study expands upon previous systematic reviews and
data points per phase. No studies included less than three meta-analyses of high-tech AAC interventions for individuals
data points per phase because those that did were screened with ASD and complex communication needs by (a) includ-
out at an earlier stage. Regarding the number of demonstra- ing participants with intellectual disabilities, (b) evaluating
tions of treatment effect, there were at least three presenta- the included studies on both basic design and extended
tions of effect for all of the studies with moderate-to-strong methodological standards, and (c) making evidence-based
evidence (n ¼ 12); the other studies did not meet this stand- practice determinations. One finding of this quality review is
ard. This pattern was also true with regard to the ratio of that implementation of high-tech AAC may be considered an
effects to non-effects. EBP for use with this population. This is encouraging, particu-
larly considering the rapid increase in the availability of AAC
Alternating treatment designs apps for mobile devices. Additionally, this study investigated
A total of 22 ATD experiments were evaluated using the evi- the effectiveness of high-tech AAC interventions via visual
dence standards (see Table 3 and Supplementary Appendices analysis for the purpose of the evidence-based determin-
G and H). None of the experiments were determined to have ation; however, it does not report results from statistical anal-
strong evidence of superior performance of high-tech AAC yses. Information regarding the magnitude of intervention
compared to low-tech, six were categorized as having mod- effect across studies, as well as moderator analyses related to
erate evidence, and the remaining experiments were found implementer, setting, implementation strategies, participants’
to have weak or no evidence of superior outcomes (see age, and communication function can be found in the com-
Table 3). These superior and moderate experiments spanned panion article to this study (Ganz et al., 2017).
six articles, four of which included experiments rated as As a result of the design quality review, only three experi-
superior evidence, and all of which included experiments ments across two articles met all basic design standards. Of
rated as moderate evidence. Those with weak or no evidence the 40 experiments that met the design standards with reser-
primarily failed to meet standards due to an insufficient num- vations, the major reasons they received a reduced score
ber of effects and ratio of effects to non-effects, when data were because they failed to collect five data points per phase
points for each condition were compared side-by-side. All or because they collected or reported a mean IOA score
but one of the six experiments that met criteria for moderate rather than disaggregating the results by phase. Although
evidence had acceptable baseline data patterns, and all prior guidance was released in 2005 regarding design quality
included a sufficient number of treatment effects. for single case research (see Horner et al., 2005), these guide-
Considering the number of data points per condition and lines did not require a minimum of five data points or the
the ratio of treatment effects to non-effects, the six moder- disaggregation of IOA scores, by phase. This guidance was
ately effective-rated experiments were rated as strong or not released until September 2011 when the WWC published
moderate evidence for both these features. Supplementary the Single-Case Pilot Design Standards (USDE, 2011). Given
Appendices G and H provide more detailed information the delay to publication, one would expect studies published
regarding the scores for baseline, intervention, and between in 2013 or later to incorporate these standards; however, of
phases. Although no standards currently exist to determine the 22 articles that met the design standards with reserva-
whether treatments qualify as superior to other treatments, tions, 10 were published in or after 2013. This is concerning,
given these ATD data, it appears clear that high-tech AAC and indicates a need for increased rigor when designing and
may not be considered superior to low-tech AAC for people implementing single-case research.
with ASD or intellectual disability and complex communica- When evaluating studies using extended methodological
tion needs. standards, mixed results were found. Studies scored the low-
est on standards relating to generalization, maintenance, pro-
cedural integrity, and social validity. Regarding
Evidence-based practice evaluation
generalization, none of the studies met the highest standard,
A total of 12 multiple baseline or multiple probe design which requires collection of generalization data during both
experiments across 12 articles demonstrated strong or mod- baseline and intervention phases with at least three data
erate evidence of the effectiveness of high-tech AAC points per phase. When generalization data were collected in
 AUGMENTATIVE AND ALTERNATIVE COMMUNICATION 11

the studies included here, they were often only collected efficacious for more complex skills such as exchanging infor-
during the intervention phase with a limited number of data mation and social closeness (Light, 1989). Practitioners may
points. This is problematic because, without generalization confidently use AAC to teach expression of wants and needs,
data in baseline, there is nothing against which to compare but should also implement AAC for other communicative func-
the intervention data. In addition, it is difficult to determine tions, while collecting data to determine whether or not it is as
trend, variability, and change in level with the collection of effective for people with more abstract and complex lan-
only one or two data points. Although several studies did guage use.
meet the highest standard for maintenance data, most did Furthermore, much of the research cited in this review
not. Often, only a few maintenance data points were col- was implemented by researchers, who are often highly
lected, and very soon after the conclusion of intervention, trained in AAC implementation, calling into question whether
raising questions about whether those data truly constitute or not high-tech AAC would be as successful if implemented
maintenance. Finally, failure to meet standards related to in real world settings. Because the findings indicate that
procedural integrity and social validity brings into question highly trained individuals are likely to implement AAC with
the internal validity of the study and the importance of the positive outcomes, practitioners are encouraged to receive
research to key stakeholders. If the individuals who are specialized training in AAC implementation when working
expected to implement the findings of research do not deem with individuals who rely on AAC. Finally, it is important to
the research itself to be socially valid, then sustainability will note that techniques used to implement high-tech AAC were
likely not be achieved.
not evaluated, only the use of the modes themselves.
Although it was not possible to include alternating treat-
Intervention strategies likely have a large impact on the suc-
ment designs in the evidence-based determinations, the data
cess of AAC implementation. That said, it is clear that high-
from these designs were visually analysed using the WWC
tech AAC has the potential to be effective with people with
standards in order to determine whether high-tech or low-
ASD or intellectual disability when implemented in natural
tech AAC interventions were more effective. Many alternating
contexts by highly trained professionals.
treatment designs that compared high- to low-tech AAC
were identified; however, few of these demonstrated superior
performance of high-tech AAC when the studies were visu-
ally analysed using the WWC standards. Most demonstrated Limitations and future research
approximately equal responsiveness to the two types of
This review has several limitations that should be considered
treatments. This is a promising result, as it indicates that low-
when interpreting the results. Although every attempt was
tech, less-expensive AAC (McNaughton & Light, 2013) may be
made to ensure the data were accurate (i.e., 100% reliability
just as effective as high-tech AAC implementation. The cur-
at all stages of the review process), some standards had low
rent review contradicts prior work that indicated high-tech
reliability (e.g., Design Standard 5C: additional probe point
AAC was significantly more effective with individuals with
considerations; descriptions of participants, settings and
ASD and intellectual disabilities than low-tech AAC (Ganz
materials, and interventionists; quality of evidence standards
et al., 2014); however, this contrast may be due to the previ-
related to predictability and variability) primarily due to inad-
ous meta-analysis evaluating for statistical differences
between data gleaned primarily from multiple-baseline equate descriptions of the variables coded or the subjective
designs rather than direct comparison designs (e.g., alternat- nature of visual analysis. As such, the results associated with
ing treatment design). these standards should be viewed with caution. Another limi-
tation is that two of the included articles were evaluated by
a reviewer who was also an author of the articles (see Ganz
Implications et al., 2013; Ganz, Hong, Goodwyn, Kite, & Gilliland, 2015);
The finding that high-tech AAC can be considered an evi- however, given that the final score for both of these articles
dence-based practice for individuals with ASD or intellectual was ‘weak-to-no evidence’, the likelihood of reviewer bias
disabilities and complex communication needs is reassuring, is low.
particularly given the now ubiquitous nature of mobile tech- Reviewing the literature base on high-tech AAC revealed
nology (McNaughton & Light, 2013). However, some caution several areas that would benefit from additional research.
is warranted. The purpose of this review was to make an First, future research could differentiate between high-tech
over-arching determination about whether or not high tech- apps and older speech-generating devices. Given the speed
AAC is an EBP for the stated population; therefore, it did at which technology is expanding, it would be beneficial to
not include a breakdown of studies by specific communica- determine whether there are differential effects between the
tive functions, in order to determine under what conditions different types of AAC technologies. Second, little research
high-tech AAC is most effective. These comparisons were has evaluated how to teach navigation of dynamic AAC dis-
made in the companion article, which found that the major- plays and the impact of varied arrangement of symbols, and
ity of studies used high-tech AAC to teach requesting (Ganz this work is sorely needed (Reichle & Drager, 2010). Finally,
et al., 2017). Thus, while high-tech AAC is useful for teach- given the broader focus on multimodal communication inter-
ing a limited number of communicative functions, more ventions, investigations of communication across many
research is needed to determine if this intervention is modes is warranted.
12 K. L. MORIN ET AL.

Conclusions Beck, A.R., Stoner, J.B., & Dennis, M.L. (2009). An investigation of aided
language simulation: Does it increase AAC use with adults with devel-
Given the increase in the use of handheld mobile devices, it opmental disabilities and complex communication needs?
is critical that new technology is investigated with respect to Augmentative and Alternative Communication, 25, 42–54. doi:10.1080/
07434610802131059
its benefits for enhancing the communication of individuals
Boesch, M.C., Wendt, O., Subramanian, A., & Hsu, N. (2013a). Comparative
with disabilities, including those with ASD or intellectual dis- efficacy of the Picture Exchange Communication System (PECS) versus
abilities. The use of AAC apps has increased among commu- a speech-generating device: Effects on requesting skills. Research in
nicators with ASD or intellectual disabilities who also have Autism Spectrum Disorders, 7, 480–493. doi:10.1016/j.rasd.2012.12.002
complex communication needs. Although the current study Boesch, M.C., Wendt, O., Subramanian, A., & Hsu, N. (2013b). Comparative
found evidence supporting the efficacy of high-tech AAC efficacy of the Picture Exchange Communication System (PECS) versus
a speech-generating device: Effects on social-communicative skills and
interventions, because the literature on mobile apps is still speech development. Augmentative and Alternative Communication,
emerging, the same cannot be unequivocally stated specific- 29, 197–209. doi:10.3109/07434618.2013.818059
ally for mobile apps, only generally for electronic AAC sys- Boles, M., Ganz, J.B., Hagan-Burke, S., Gregori, E.V., Neely, L.C., Mason,
tems. Furthermore, because the literature base has primarily R.A., … , Wilson, V.L. (2016). Quality review of single-case studies con-
evaluated use of high-tech AAC with younger individuals, cerning employment skill interventions for individuals with develop-
mental disabilities. Cadernos De Educacao (UFPel), 53, 15–51.
with more introductory communicative functions, and in Buzolich, M.J., King, J., & Baroody, S. (1991). Acquisition of the comment-
more highly controlled research contexts, it cannot be stated ing function among system users. Augmentative and Alternative
with confidence that these results can generalize beyond Communication, 7, 88–99. doi:10.1080/07434619112331275753
these contexts. Finally, although most of the papers in this Byrt, T., Bishop, J., & Carlin, J.B. (1993). Bias, prevalence, and kappa.
review were published within the last 10 years, a large num- Journal of Clinical Epidemiology, 46, 423–429. doi:10.1016/0895-
4356(93)90018-V
ber were excluded because they did not meet basic meth- Camargo, S., Rispoli, M., Ganz, J., Hong, E.R., Davis, H., & Mason, R. (2014).
odological quality standards. Several key areas were A review of the quality of behaviorally-based intervention research to
identified where SCED researchers should include or expand improve social interaction skills of children with ASD in inclusive set-
their descriptions of methodological components, including tings. Journal of Autism and Developmental Disorders, 44, 2096–2116.
descriptions of participants (e.g., standardized assessment doi:10.1007/s10803-014-2060-7
Cannella-Malone, H.I., DeBar, R.M., & Sigafoos, J. (2009). An examination
results) and interventionists (e.g., education, training, and of the preference for augmentative and alternative communication
relationship to the participant); collection and reporting of devices with two boys with significant intellectual disabilities.
maintenance and generalization data; and collection and Augmentative and Alternative Communication, 25, 262–273.
reporting of procedural integrity and social validity data. doi:10.3109/07434610903384511
Therefore, SCED researchers are encouraged to expand the Chung, Y-C., & Douglas, K.H. (2015). A peer interaction package for stu-
dents with autism spectrum disorders who use speech-generating
quality of their future research designs and their reporting of
devices. Journal of Developmental and Physical Disabilities, 27,
said research. That said, this review provides justification for 831–849. doi:10.1007/s10882-015-9461-1
using high-tech AAC apps with individuals with ASD or intel- Cohen, J. (1960). A coefficient of agreement for nominal scales.
lectual disability who have complex communication needs, Educational and Psychological Measurement, 20, 37–46. doi:10.1177/
particularly given the ubiquity of AAC apps, their ease of use, 001316446002000104
Cook, B., Buysse, V., Klingner, J., Landrum, T., McWilliam, R., Tankersley,
and their acceptability to most stakeholders.
M., & Test, D. (2014). Council for Exceptional Children: Standards for
evidence-based practices in special education. Teaching Exceptional
Children, 46, 206–212. doi:10.1177/0040059914531389
Disclosure statement Council for Exceptional Children. (2014). Council for Exceptional Children
No potential conflict of interest was reported by the authors. standards for evidence-based practices in special education. Retrieved from
http://www.cec.sped.org/Standards/Evidence-Based-Practice-Resources-
Original
Funding Couper, L., van der Meer, L., Sch€afer, M.C., McKenzie, E., McLay, L., O’reilly,
M.F., … Sutherland, D. (2014). Comparing acquisition of and preference
The research reported here was supported by the Institute of Education for manual signs, picture exchange, and speech-generating devices in
Sciences, US Department of Education through Grant R324B160038 nine children with autism spectrum disorder. Developmental
awarded to UNC-Chapel Hill and by the Office of Special Education Neurorehabilitation, 17, 99–109. doi:10.3109/17518423.2013.870244
Programs, U.S. Department of Education through Grant H325D110046 Durand, V. M. (1993). Functional communication training using assistive
awarded to Texas A&M University. The opinions expressed represent devices: Effects on challenging behavior and affect. Augmentative and
those of the authors and do not represent the Institute of the US Alternative Communication, 9, 168–176. doi:10.1080/
Department of Education. 07434619312331276571
Elsabbagh, M., Divan, G., Koh, Y.J., Kim, Y.S., Kauchali, S., Marcın, C., …
Yasamy, M.T. (2012). Global prevalence of autism and other pervasive
developmental disorders. Autism Research, 5, 160–179. doi:10.1002/
References aur.239
Estes, A., Rivera, V., Bryan, M., Cali, P., & Dawson, G. (2011). Discrepancies
American Psychiatric Association. (2000). Diagnostic and Statistical between academic achievement and intellectual ability in higher-func-
Manual of Mental Disorders (4th ed.). Washington, DC: Author. text tioning school-aged children with autism spectrum disorder. Journal
revision. doi:10.1007/springerreference_179660 of Autism and Developmental Disorders, 41, 1044–1052. doi:10.1007/
American Speech Language Hearing Association. (2005). Roles and s10803-010-1127-3
responsibilities of speech-language pathologists with respect to aug- Feinstein, A.R., & Cicchetti, D.V. (1990). High agreement but low kappa: I.
mentative and alternative communication: Position statement. The problems of two paradoxes. Journal of Clinical Epidemiology, 43,
doi:10.1044/policy.PS2005-00113 543–549. doi:10.1016/0895-4356(90)90158-L
 AUGMENTATIVE AND ALTERNATIVE COMMUNICATION 13

Ganz, J.B., Hong, E.R., Goodwyn, F., Kite, E., & Gilliland, W. (2015). Impact Light, J.C. (1989). Toward a definition of communicative competence for
of PECS tablet computer app on receptive identification of pictures individuals using augmentative and alternative communication sys-
given a verbal stimulus. Developmental Neurorehabilitation, 18, 82–87. tems. Augmentative and Alternative Communication, 5, 137–144.
doi:10.3109/17518423.2013.821539 doi:10.1080/07434618912331275126
Ganz, J.B., Hong, E.R., & Goodwyn, F.D. (2013). Effectiveness of the PECS Light, J., & McNaughton, D. (2012). Supporting the communication, lan-
Phase III app and choice between the app and traditional PECS guage, and literacy development of children with complex communi-
among preschoolers with ASD. Research in Autism Spectrum Disorders, cation needs: State of the science and future research priorities.
7, 973–983. doi:10.1016/j.rasd.2013.04.003 Assistive Technology, 24, 34–44. doi:10.1080/10400435.2011.648717
Ganz, J.B., Mason, R.A., Goodwyn, F.D., Boles, M.B., Heath, A.K., & Davis, Logan, K., Iacano, T., & Trembath, D. (2017). A systematic review of
J.L. (2014). Interaction of participant characteristics and type of AAC research into aided AAC to increase social-communication functions
with individuals with ASD: A meta-analysis. American Journal on in children with autism spectrum disorder. Augmentative and
Intellectual and Developmental Disabilities, 119, 516–535. doi:10.1352/ Alternative Communication, 33, 51–64. doi:10.1080/
1944-7558-119.6.516 07434618.2016.1267795
Ganz, J.B., Morin, K.L., Foster, M.J., Vannest, K.J., Tosun, D.G., Gregori, E.V., Lorah, E.R., Crouser, J., Gilroy, S.P., Tincani, M., & Hantula, D. (2014).
& Gerow, S.L. (2017). High-technology augmentative and alternative Within stimulus prompting to teach symbol discrimination using an
communication for individuals with intellectual and developmental iPad® Speech Generating Device. Journal of Developmental and
disabilities and complex communication needs: A meta-analysis. Physical Disabilities, 26, 335–346. doi:10.1007/s10882-014-9369-1
Augmentative and Alternative Communication, 33, 224–238. Lorah, E.R., Karnes, A., & Speight, D.R. (2015). The acquisition of intraver-
doi:10.1080/07434618.2017.1373855 bal responding using a speech generating device in school aged chil-
Hanser, G.A., & Erickson, K.A. (2007). Integrated word identification and dren with autism. Journal of Developmental and Physical Disabilities,
communication instruction for students with complex communication 27, 557–568. doi:10.1007/s10882-015-9436-2
needs: Preliminary Results. Focus on Autism and Other Developmental Lorah, E.R., Parnell, A., & Speight, D.R. (2014). Acquisition of sentence
Disabilities, 22, 268–278. doi:10.1177/10883576070220040901 frame discrimination using the iPad as a speech generating device in
Hoffmann, T.C., Glasziou, P.P., Boutron, I., Milne, R., Perera, R., Moher, D., young children with developmental disabilities. Research in Autism
… Michie, S. (2014). Brief reporting of interventions: Template for Spectrum Disorders, 8, 1734–1740. doi:10.1016/j.rasd.2014.09.004
Lorah, E.R., Tincani, M., Dodge, J., Gilroy, S., Hickey, A., & Hantula, D.
intervention description and replication (TIDieR) checklist and guide.
(2013). Evaluating picture exchange and the iPadTM as a speech gen-
BMJ, 7, 348. doi:10.1136/bmj.g1687
erating device to teach communication to young children with aut-
Hong, E.R., Ganz, J.B., Ninci, J., Neely, L., Gilliland, W., & Boles, M. (2015).
ism. Journal of Developmental and Physical Disabilities, 25, 637–649.
An evaluation of the quality of research on evidence-based practices
doi:10.1007/s10882-013-9337-1
for daily living skills for individuals with autism spectrum disorder.
Maggin, D.M., Briesch, A.M., & Chafouleas, S.M. (2013). An application of
Journal of Autism and Developmental Disorders, 45, 2792–2815.
the What Works Clearinghouse standards for evaluating single-subject
doi:10.1007/s10803-015-2444-3
research synthesis of the self-management literature base. Remedial
Hong, E.R., Gong, L., Ninci, J., Morin, K., Davis, J., Kawaminami, S., …
and Special Education, 34, 44–58. doi:10.1177/0741932511435176
Noro, F. (2017). A meta-analysis of single-case research on the use of
Maulik, P.K., Mascarenhas, M.N., Mathers, C.D., Dua, T., & Saxena, S.
tablet-mediated interventions for persons with ASD. Research in
(2011). Prevalence of intellectual disability: A meta-analysis of popula-
Developmental Disabilities, 70, 198–214. doi:10.1016/j.ridd.2017.09.013
tion-based studies. Research in Developmental Disabilities, 32, 419–436.
Horner, R.H., Carr, E.G., Halle, J., McGee, G., Odom, S., & Wolery, M.
doi:10.1016/j.ridd.2010.12.018
(2005). The use of single-subject research to identify evidence-based
McHugh, M.L. (2012). Interrater reliability: The kappa statistic. Biochemia
practice in special education. Exceptional Children, 71, 165–179.
Medica, 22, 276–282. doi:10.11613/BM.2012.031
doi:10.1177/001440290507100203 McLay, L., van der Meer, L., Schafer, M.C.M., Couper, L., McKenzie, E.,
Iacono, T., & Johnson, H. (2004). Patients with disabilities and complex O’Reilly, M.F., …Sutherland, D. (2015). Comparing acquisition, general-
communication needs: The GP consultation. Australian Family ization, maintenance, and preference across three AAC options in four
Physician, 33, 585–589. children with autism spectrum disorder. Journal of Developmental and
Individuals with Disabilities Education Improvement Act. (IDEA; 2004). Physical Disabilities, 27, 323–339. doi:10.1007/s10882-014-9417-x
PLU.S.C. (pp. 108–406). McMillan, J.M. (2008). Teachers make it happen: From professional devel-
Jitendra, A., Burgess, C., & Gajria, M. (2011). Cognitive strategy instruction opment to integration of augmentative and alternative communica-
for improving expository text comprehension of students with learn- tion technologies in the classroom. Australasian Journal of Special
ing disabilities: The quality of evidence. Exceptional Children, 77, Education, 32, 199–211. doi:10.1017/S1030011200025859
135–159. doi:10.1177/001440291107700201 McNaughton, D., Bryen, D., Blackstone, S., Williams, M., & Kennedy, P.
Johnson, J.W., McDonnell, J., Holzwarth, V.N., & Hunter, K. (2004). The effi- (2012). Young adults with complex communication needs: Research
cacy of embedded instruction for students with developmental disabil- and development in AAC for a “diverse” population. Assistive
ities enrolled in general education classes. Journal of Positive Behavior Technology, 24, 45–53. doi:10.1080/10400435.2011.648715
Interventions, 6, 214–227. doi:10.1177/10983007040060040301 McNaughton, D., & Light, J. (2013). The iPad# and mobile technology
Kazdin, A.E. (2011). Single-case research designs: Methods for clinical and revolution: Benefits and challenges for individuals who require aug-
applied settings (2nd ed.). New York, NY: Oxford University Press. mentative and alternative communication. Augmentative and
King, M.L., Takeguchi, K., Barry, S.E., Rehfeldt, R.A., Boyer, V.E., & Mathews, Alternative Communication, 29, 107–116. doi:10.3109/
T.L. (2014). Evaluation of the iPad# in the acquisition of requesting 07434618.2013.784930
skills for children with autism spectrum disorder. Research in Autism Moeller, J.D., Dattilo, J., & Rusch, F. (2015). Applying quality indicators to
Spectrum Disorders, 8, 1107–1120. doi:10.1016/j.rasd.2014.05.011 single-case research designs used in special education: A systematic
Kratochwill, T.R., Hitchcock, J., Horner, R.H., Levin, J.R., Odom, S.L., review. Psychology in the Schools, 52, 139–153. doi:10.1002/pits.21801
Rindskopf, D.M., & Shadish, W.R. (2010). Single-case designs technical Ninci, J., Vannest, K.J., Willson, V., & Zhang, N. (2015). Interrater agree-
documentation. Retrieved from http://ies.ed.gov/ncee/wwc/pdf/wwc_ ment between visual analysts of single-case data: A meta-analysis.
scd.pdf Behavior Modification, 39, 510–541. doi:10.1177/0145445515581327
Kratochwill, T.R., Hitchcock, J.H., Horner, R.H., Levin, J.R., Odom, S.L., No child left behind act. (NCLB; 2001). 20 U.S.C. $6319.
Rindskopf, D.M., & Shadish, W.R. (2013). Single-case intervention Olive, M.L., de la Cruz, B., Davis, T.N., Chan, J.M., Lang, R.B., O’Reilly, M.F.,
research design standards. Remedial and Special Education, 34, 26–38. & Dickson, S.M. (2007). The effects of enhanced milieu teaching and a
doi:10.1177/0741932512452794 voice output communication aid on the requesting of three children
Landis, J.R., & Kock, G.G. (1977). The measurement of observer agreement with autism. Journal of Autism and Developmental Disorders, 37,
for categorical data. Biometrics, 33, 159–174. doi:10.2307/2529310 1505–1513. doi:10.1007/s10803-006-0243-6
14 K. L. MORIN ET AL.

Olive, M.L., Lang, R.B., & Davis, T.N. (2008). An analysis of the effects of behaviors of children with autism. Journal of Developmental and
functional communication and a voice output communication aid for Physical Disabilities, 26, 513–526. doi:10.1007/s10882-013-9358-9
a child with autism spectrum disorder. Research in Autism Spectrum Trembath, D., Balandin, S., Togher, L., & Stancliffe, R.J. (2009). Peer-
Disorders, 2, 223–236. doi:10.1016/j.rasd.2007.06.002 mediated teaching and augmentative and alternative communication
Reichle, J., & Drager, K.D.R. (2010). Examining issues of aided communica- for preschool-aged children with autism. Journal of Intellectual &
tion display and navigational strategies for young children with devel- Developmental Disability, 34, 173–186. doi:10.1080/
opmental disabilities. Journal of Developmental and Physical 13668250902845210
Disabilities, 22, 289–311. doi:10.1007/s10882-010-9191-3 U.S. Department of Education, Institute of Education Sciences, What
Reichow, B., Volkmar, F.R., & Cicchetti, D.V. (2008). Development of the Works Clearinghouse. (2016, September). What Works Clearinghouse:
evaluative method for evaluating evidence-based practices in autism. Procedures and Standards Handbook (Version 3.0). Retrieved from
Journal of Autism and Developmental Disorders, 38, 1311–1319. http://whatworks.ed.gov
doi:10.1007/s10803-007-0517-7 U.S. Department of Education, Institute of Education Sciences, What
Schepis, M.M., Reid, D.H., Behrmann, M.M., & Sutton, K.A. (1998). Works Clearinghouse (USDE). (2011, September). What Works
Increasing communicative interactions of young children with autism Clearinghouse: Procedures and Standards Handbook (Version 2.1).
using a voice output communication aid and naturalistic teaching. Retrieved from http://whatworks.ed.gov
Journal of Applied Behavior Analysis, 31, 561–578. doi:10.1901/ van der Meer, L., Didden, R., Sutherland, D., O’Reilly, M.F., Lancioni, G.E.,
jaba.1998.31-561 & Sigafoos, J. (2012a). Comparing three augmentative and alternative
Schlosser, R.W., Sigafoos, J., Luiselli, J.K., Angermeier, K., Harasymowyz, U., communication modes for children with developmental disabilities.
Journal of Developmental and Physical Disabilities, 24, 451–468.
Schooley, K., & Belfiore, P.J. (2007). Effect of synthetic speech output
doi:10.1007/s10882-012-9283-3
on requesting and natural speech production in children with autism:
van der Meer, L., Kagohara, D., Achmadi, D., O’Reilly, M.F., Lancioni, G.E.,
A preliminary study. Research in Autism Spectrum Disorders, 1,
Sutherland, D., Sigafoos, J. (2012b). Speech-generating devices versus
139–163. doi:10.1016/j.rasd.2006.10.001
manual signing for children with developmental disabilities. Research
Sigafoos, J., Didden, R., & O’Reilly, M. (2003). Effects of speech output on
in Developmental Disabilities, 33, 1658–1669. doi:10.1016/
maintenance of requesting and frequency of vocalizations in three
j.ridd.2012.04.004
children with developmental disabilities. Augmentative and Alternative
van der Meer, L., Sutherland, D., O’Reilly, M.F., Lancioni, G.E., & Sigafoos, J.
Communication, 19, 37–47. doi:10.1080/0743461032000056487
(2012c). A further comparison of manual signing, picture exchange, and
Son, S.H., Sigafoos, J., O’Reilly, M., & Lancioni, G.E. (2006). Comparing two
speech-generating devices as communication modes for children with
types of augmentative and alternative communication systems for autism spectrum disorders. Research in Autism Spectrum Disorders, 6,
children with autism. Pediatric Rehabilitation, 9, 389–395. doi:10.1080/ 1247–1257. doi:10.1016/j.rasd.2012.04.005
13638490500519984 Waddington, H., Sigafoos, J., Lancioni, G.E., O’Reilly, M.F., van der Meer,
Soto, G., Belfiore, P.J., Schlosser, R.W., & Haynes, C. (1993). Training spe- L., Carnett, A., … Marschik, P.B. (2014). Three children with autism
cific requests: A comparative analysis on skill acquisition and prefer- spectrum disorder learn to perform a three-step communication
ence using two augmentative and alternative communication aids. sequence using an iPad-based speech-generating device. International
Education and Training in Mental Retardation, 28, 169–178. Journal of Developmental Neuroscience, 39, 59–67. doi:10.1016/
Steege, M.W., Wacker, D.P., Cigrand, K.C., Berg, W.K., Novak, C.G., Reimers, j.ijdevneu.2014.05.001
T.M., … DeRaad, A. (1990). Use of negative reinforcement in the treat- Wendt, O. (2009). Research on the use of manual signs and graphic sym-
ment of self-injurious behavior. Journal of Applied Behavior Analysis, bols in autism spectrum disorders: A systematic review. In P. Mirenda
23, 459–467. doi:10.1901/jaba.1990.23-459 & Iacono (Eds.), Autism spectrum disorders and AAC (pp. 83–140).
Still, K., Rehfeldt, R.A., Whelan, R., May, R., & Dymond, S. (2014). Baltimore, MA: Paul H. Brookes.
Facilitating requesting skills using high-tech augmentative and alter- Wong, C., Odom, S.L., Hume, K., Cox, A.W., Fettig, A., Kucharczyk, S., …
native communication devices with individuals with autism spectrum Schultz, T.R. (2013). Evidence-based practices for children, youth, and
disorders: A systematic review. Research in Autism Spectrum Disorders, young adults with Autism Spectrum Disorder. Chapel Hill: The
8, 1184–1199. doi:10.1016/j.rasd.2014.06.003 University of North Carolina, Frank Porter Graham Child Development
Strasberger, S.K., & Ferreri, S.J. (2014). The effects of peer assisted Institute, Autism Evidence-Based Practice Review Group. Retrieved
communication application training on the communicative and social from http://cidd.unc.edu/Registry/Research/Docs/31.pdf