0% found this document useful (0 votes)
96 views16 pages

Bullying and Suicide in High School Students: Findings From The 2015 California Youth Risk Behavior Survey

penelitian
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
96 views16 pages

Bullying and Suicide in High School Students: Findings From The 2015 California Youth Risk Behavior Survey

penelitian
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 16

JOURNAL OF HUMAN BEHAVIOR IN THE SOCIAL ENVIRONMENT

2018, VOL. 28, NO. 6, 695–709


https://doi.org/10.1080/10911359.2018.1456389

Bullying and suicide in high school students: Findings from the


2015 California youth risk behavior survey
Yong Lia and Junrong Shib
a
Department of Social Work, California State University Bakersfield, Bakersfield, California, USA; bSchool of
Social Welfare, University at Albany, Albany, New York, USA

ABSTRACT KEYWORDS
School bullying and cyberbullying have been linked to suicidal beha- Suicide; bullying; substance
viors through depression and alcohol, tobacco, marijuana, and other abuse; race/ethnicity; youth
drug use. However, how these associations may differ across racial/
ethnic groups remains relatively unknown. Using data from the 2015
California Youth Risk Behavior Survey, this study aims to examine two
questions in different racial/ethnic subgroups: (1) Does bullying affect
suicide? and (2) Does bullying have an indirect effect on suicide
through depression and use of alcohol, tobacco, marijuana and
other drugs? The sample consisted of 1,765 Californian youth attend-
ing grades 9–12. Logistic regression analyses indicate that being
bullied is associated with increased odds of suicide across all racial/
ethnic groups; depression mediates the effect of bullying on suicide
for all racial/ethnic groups; alcohol use mediates the effect only for
Hispanic youth; other drug use mediates the effect only for White
youth; marijuana and tobacco use have no mediating role. These
findings suggest that bullying may lead to suicide through different
risk behaviors for youths of different racial/ethnic groups.
Professionals who work with bullied youths need to treat depression
more effectively to prevent suicide in service planning and provision.
They also need to be aware of the racial/ethnic differences in the risk
behaviors intercorrelated with bullying and suicide and provide
appropriate treatment to the youth of specific race/ethnicity.

Introduction
Youth suicide is a serious public health problem in the United States. In 2015, nearly 6,000
youths ages 10 to 24 died by suicide (CDC, 2017). The prevalence of suicide among youth
ages 15–19 has been on a rise in the last decade. It was eight deaths per 100,000 in 2005
but increased to 10 (a total of 2,061 deaths) in 2015 (CDC, 2017). In addition to completed
suicide, the term suicide is also used to denote behaviors such as suicidal ideation (i.e.,
serious thoughts about killing oneself), suicidal planning (i.e., making plans for killing
oneself), and attempted suicide (i.e., taking actions to kill oneself). The rate of youth who
considered suicide or attempted suicide is much higher. About one in six youths reported
having seriously considered suicide in 2015 (Kann et al., 2016). Meanwhile, about 8.6%
youth in grades 9 to 12 attempted suicide in the past 12 months (Kann et al., 2016).

CONTACT Yong Li yli12@csub.edu Department of Social Work, California State University Bakersfield, Dorothy
Donahoe Hall, A117, 9001 Stockdale Highway, Bakersfield, CA 93311-1022, USA
© 2018 Taylor & Francis
696 Y. LI AND J. SHI

Some health risk behaviors related to youth suicide include bullying (both school bullying
and cyberbullying), depression, and use of alcohol, tobacco, and other drugs (ATOD). School
bullying includes frequent teasing, threatening, spreading rumors about, and/or physically
harming a student. About 20.8% students ages 12–18 reported being bullied in the school
year 2014–2015 (Lessne & Yanez, 2016). Conversely, cyberbullying can be done via emails,
chat rooms, instant messaging, and/or texting (Menesini & Salmivalli, 2017). In 2012–2013,
6.9% students reported being cyberbullied (Lessne & Cidade, 2015).
In addition, depression is more prevalent among adolescents than adults: in 2015, 6.7%
of all U.S. adults aged 18 or older had at least one major depressive episode; among
adolescents aged 12 to 17, the percentage was 12.5% (Center for Behavioral Health
Statistics and Quality, 2016). Finally, the most recent national survey of high school
youth (2015) reported that 10.8% of all students currently smoke cigarettes, 32.8%
currently drink alcohol, and 21.7% currently use marijuana (Kann et al., 2016). About
6% had used hallucinogenic drugs, which was the most prevalent illicit drug ever used.
Existing research suggests racial/ethnic disparities in youth suicide and health risk
behaviors. White youth are more likely to engage in tobacco-use related risk behaviors
than Black, Hispanic, and Asian/Pacific Islander (API) youths; they are more likely to be
victims of bullying (Kann et al., 2016). In another study, White youth were more likely to
feel depressed than Hispanic youth (Hallfors et al., 2004). Conversely, racial and ethnic
minority youths, such as Hispanic youth, may be at a higher risk for suicidal behaviors
comparing to White youth (Canino & Roberts, 2001; Miranda, Ortin, Polanco-Roman, &
Valderrama, 2017). In addition, Hispanic youth are more likely to drink alcohol before age
13 and use illicit drugs such as synthetic marijuana, cocaine, and ecstasy than White and
Black, non-Hispanic youths (Kann et al., 2016). The following review delineates the
interrelationships between bullying, depression and ATOD use, and suicide.

Bullying and suicide


Mounting evidence is available on the detrimental effects of bullying on the victim’s physical,
psychological, social, and economic well-being. These include sleep problems, abdominal
pain, headaches, bad appetite, depression, anxiety, involvement in interpersonal violence or
sexual violence, poor social functioning, and poor school performance (CDC, 2014; Fekkes,
Pijpers, Fredriks, Vogels, & Verloove-Vanhorick, 2006). The most devastating consequence
for those involved in bullying is suicide. In a meta-analysis, Holt et al. (2015) have linked
different types of involvement in bullying (as a victim, perpetrator, or bully-victim) to
suicidal behaviors. The risk for suicide is even higher among victims who are frequently
bullied (Srabstein, 2013; Takizawa, Maughan, & Arseneault, 2014). Using a longitudinal
design, for example, Takizawa and colleagues (2014) reported that the harmful effects of
being bullied are still manifested even four decades after exposure. Their participants were
diagnosed with depression and anxiety disorders and experienced suicidality at age 45,
although the victimization occurred at ages 7 and 11 years.

The effects of bullying on suicide through depression and ATOD use


Bullying may have an indirect effect on suicide through depression. For example, depres-
sion was found to mediate the relationship between bullying and suicidal ideation among
JOURNAL OF HUMAN BEHAVIOR IN THE SOCIAL ENVIRONMENT 697

New Jersey adolescents ages 10–18 years (Lardier, Barrios, Garcia-Reid, & Reid, 2016).
One national study reported that the effect of bullying was mediated by depression (Reed,
Nugent, & Cooper, 2015). Outside of the United States, results from the 2011 Eastern
Ontario Youth Risk Behavior Survey suggested that depression had a mediating effect on
the relationship between bullying and suicidal behaviors (Sampasa-Kanyinga, Roumeliotis,
& Xu, 2014).
In addition, bullying and depression have been found to be correlated. Victimization
from bullying is related to increased depressive symptoms (Juvonen & Graham, 2014;
Turner, Finkelhor, & Ormrod, 2010). This might be true for youths in both clinical and
population samples (Kaltiala-Heino & Fröjd, 2011). The link between childhood victimi-
zation from bullying and adulthood depression is evident in longitudinal studies, even
after controlling for many other childhood adversities (Klomek, Sourander, & Elonheimo,
2015; Ttofi, Farrington, Lösel, Crago, & Theodorakis, 2016).
As for the association of depression with suicide, recent research has often studied how
youth risk behaviors such as substance abuse or bullying were related to both depression
and suicide (Hallfors et al., 2004; Klomek et al., 2013, 2015; Pranjić & Bajraktarević, 2010).
However, social-psychological stressors (such as depression) implicated in youth suicidal
behaviors have captured the attention of researchers for a long time and their findings
have consistently indicated a positive relationship between depression and suicidality (e.g.,
Ayyash-Abdo, 2002; Bridge, Goldstein, & Brent, 2006; De Man & Leduc, 1995; Harter &
Marold, 1994; Runeson, 1989). Taken together, these findings suggest a potential media-
tional effect of depression on the relationship between bullying and suicide.
Bullying may also have an indirect effect on suicide through ATOD use. Two pre-
viously reviewed studies supported the mediational role of substance use (Lardier et al.,
2016; Reed et al., 2015). In addition, Litwiller and Brausch (2013) found that substance use
mediated the relationship between bullying (including cyberbullying) and suicidal beha-
viors among high school students in a Midwestern state in the United States.
Evidence is also available on the bivariate relationships between bullying, ATOD use,
and suicidal behaviors. Involvement in bullying was associated with increased odds of
alcohol use, tobacco and marijuana use, and illicit drug use among middle and high school
students (Hong et al., 2014; Peleg-Oren, Cardenas, Comerford, & Galea, 2012; Radliff,
Wheaton, Robinson, & Morris, 2012; Ttofi et al., 2016). In addition, different types of
ATOD use were associated with increased odds of suicidal behavior in a nationally
representative sample of adolescents in grades 7 through 12 in the United States
(Hallfors et al., 2004); illicit drug use was associated with higher odds of suicidal ideation
and suicide attempts in Canadian high school students (Rasic, Weerasinghe, Asbridge, &
Langille, 2013). Finally, in a comparative study (Swahn et al., 2012), cigarette smoking was
related to suicidal attempts in both American and French youth while alcohol and
marijuana use was related to suicidal attempts only in French youth. These findings
point to a potential mediational effect of ATOD use on the relationship between bullying
and suicide.
Although ample research has focused on the associations between bullying, depression
and ATOD use, and suicidal behaviors, the literature falls short in the comparison of these
associations between different racial/ethnic groups. Mediation analyses on the effects of
bullying on suicide through depression and ATOD use are also lacking. Finally, the
measurement for ATOD use often does not capture the full spectrum of substances
698 Y. LI AND J. SHI

used by youth. Our study aimed to bridge these gaps by (1) conducting comparative
analyses on different racial/ethnic groups, (2) examining the mediation effects of depres-
sion and ATOD use, and (3) incorporating separate measures on alcohol use, tobacco use,
marijuana use and illicit drug use.

This study
This study aims to answer the following questions: (a) Does bullying (including cyber-
bullying) affect suicide? And (b) Does bullying have an indirect effect on suicide through
depression and ATOD use? Using a sample of high school students, we tested the
following hypotheses: (1) those who are bullied will be more likely to engage in suicidal
behaviors than those who are not; (2) the effect of bullying on suicide will be mediated by
depression; (3) the effect of bullying on suicide will be mediated by alcohol use; (4) the
effect of bullying on suicide will be mediated by tobacco use; (5) the effect of bullying on
suicide will be mediated by marijuana use; (6) the effect of bullying on suicide will be
mediated by other drug use. Because how these relationships may differ across racial and
ethnic groups remains relatively unknown, we tested all hypotheses in four racial/ethnic
groups (White, Hispanic, API, and mixed) in order to examine the intergroup differences.
Age, grade level, and sex have been found to be associated with ATOD use, depression,
and suicidal behaviors (Grunbaum et al., 2004; Hallfors et al., 2004; Kann et al., 2016; Luk,
Wang, & Simons-Morton, 2012; McMahon & Luthar, 2006; Reed et al., 2015). For
example, Hallfors et al. (2004) reported that girls who engaged in drug use were more
likely to experience suicidality. In addition, it has been reported that youth of higher grade
levels were more likely to engage in ATOD use (Kann et al., 2016). Therefore, we included
age, grade level, and sex in our analyses as covariates.

Method
Data and sample
Data were from the 2015 California Youth Risk Behavior Survey (YRBS). We selected
California as the target state because it had the largest number of high school student
enrollment and the largest Hispanic student population. In 2015, a stratified two-stage
cluster sampling method was used to produce a representative sample of public school
students in grades 9–12 in California. In the first stage, a group of high schools was
randomly sampled with probability proportional to school enrollment; in the second stage,
multiple classes from the schools selected in the first stage were sampled randomly, and all
students in the sampled classes were asked to participate (Kann et al., 2016). The response
rate for schools and students were 75% and 89%, respectively. Parental permission was
acquired before youth who were interested in participating provided their responses to the
self-administered YRBS questionnaire in a class period. In addition to basic demographics,
the survey collected data on a wide range of health risk behaviors of respondents,
including suicide, alcohol and other drug use, tobacco use, unhealthy dietary behaviors,
sexual behaviors, etc.
A total of 1,943 youths participated in the survey. African American and American
Indian youths were removed from the analytic sample due to a small number of
JOURNAL OF HUMAN BEHAVIOR IN THE SOCIAL ENVIRONMENT 699

participants (N = 65, N = 8, respectively); two cases who reported to be in other grades


were removed; cases with missing values were also eliminated (N = 282). These procedures
resulted in an analytic sample of 1,586 youths. The sample size for the subgroups were 286
for Whites, 508 for Hispanics, 317 for APIs, and 475 for youth with mixed race/ethnicity.
Most respondents in the sample were aged 15–17 years (74.41%) and a slight majority
were male (50.47%). The proportion for each grade level was roughly equal (ranging from
22.70% to 27.48%). As for the racial and ethnic distribution, the largest group was
Hispanic (29.26%), followed by White (28.65%), mixed (26.31%), and Asian and Pacific
Islander (15.78%).

Measures
Suicide
Suicide was assessed by three questions about youths’ suicidal thoughts, planning, and
attempts in the past 12 months (i.e., “During the past 12 months, did you ever seriously
consider attempting suicide?”; “During the past 12 months, did you make a plan about
how you would attempt suicide?”; “During the past 12 months, how many times did you
actually attempt suicide?”). Suicide was coded as 1 when youth reported “yes” on the first
two questions and one or more times on the third question.

Bullying victimization
Bullying victimization was measured by both school bullying and cyberbullying. School
bullying was measured by a yes/no question: “During the past 12 months, have you ever
been bullied on school property?” Cyberbullying was measured by a yes/no question:
“During the past 12 months, have you ever been electronically bullied?” Answering “yes”
to either of the two questions was coded as 1 for the variable.

Depression
Depression was gauged by a yes/no question on youth’s feelings about sadness and
hopelessness (“During the past 12 months, did you ever feel so sad or hopeless almost
every day for two weeks or more in a row that you stopped doing some usual activities?”).
Depression was coded as 1 when respondents answered “yes.”

ATOD use
Youth also answered a set of questions on the current use of alcohol, tobacco, and
marijuana (i.e., “During the past 30 days, on how many days did you have at least one
drink of alcohol?”; “During the past 30 days, on how many days did you smoke cigar-
ettes?”; “During the past 30 days, how many times did you use marijuana?”). Other drug
use was measured by a set of questions on use of different drugs, including cocaine,
inhalants, heroin, methamphetamines, ecstasy, synthetic marijuana, and steroid (e.g.,
“During your life, how many times have you used any form of cocaine, including powder,
crack, or freebase?”).
Originally, questions on alcohol use and cigarette smoking were rated on a Likert-
type scale ranging from 0 to all 30 days; questions on the use of marijuana and each
type of other drugs were all rated on a Likert-type scale ranging from 0 to 40 or more
times. For the purpose of data analysis, they were all dummy coded in this study: 0
700 Y. LI AND J. SHI

day/time was coded as 0, indicating no use; all other categories were coded as 1,
indicating at least one-time use. Furthermore, the variable on other drug use was
coded as 1 when youth reported at least one time use of any of the drugs mentioned
previously.

Covariates
Sex (0 = male; 1 = female), age (seven categories ranging from 12 years old or younger to
18 years old or older), and grade level (four categories ranging from 1 to 4) were included
as covariates.

Analysis strategy
According to Kann et al. (2016), a weight was applied to adjust for school and student
nonresponse. Auxiliary variables including student grade, sex, and race/ethnicity were
used to calculate the weight. As a result, weighted estimates are representative of all
students in grades 9–12 attending public schools in California. Weighted logistic regres-
sion was used to test the hypotheses. Data were analyzed in Stata version 13.0.
In order to test the significance of the indirect effects, we performed Sobel tests and
computed the z score, the standard error, and the p value using the online calculator
developed by Preacher and Leonardelli (2001). If the effect is significant, we calculated the
proportion of the total effect that is mediated using a Microsoft Excel spreadsheet created
by Herr (2006).

Results
Descriptive statistics
Table 1 shows the descriptive statistics on major study variables by race/ethnicity. In the
total sample, 21.36% experienced bullying. The three most common risk behaviors were
alcohol use (28.54%), depression (28.23%), and marijuana use (21.26%). As for intergroup
differences, White youth were more likely to use alcohol, tobacco, and marijuana than
other groups (F = 5.61, p = .01; F = 3.09, p = .05; F = 4.63, p = .01, respectively). They were
also more likely to be bullied (F = 5.05, p = .007).

Table 1. Descriptive statistics of major study variables by race/ethnicity.


Total sample (%) White (%) Hispanic (%) APIs (%) Mixed (%) Weighted F
Suicide 21.21 22.5 20.69 18.91 21.76 .37
Alcohol use 28.54 35.69 25.87 16.30 31.07 5.61**
Tobacco use 7.69 10.98 5.15 2.77 9.90 3.09*
Marijuana use 21.26 27.71 17.83 9.99 24.79 4.63**
Other drug use 15.24 13.88 16.04 9.22 19.45 1.93
Depression 28.23 28.54 26.65 22.91 32.84 2.15
Bullying 21.36 30.02 16.62 21.23 17.28 5.05**
Note: *p ≤ .05, **p ≤ .01; proportions are based on youth who were coded yes on each variable.
JOURNAL OF HUMAN BEHAVIOR IN THE SOCIAL ENVIRONMENT 701

Total effects
Path c (without the mediator in regression) represents the total effect of bullying on
suicide. As shown in Table 2, being bullied was associated with increased odds of
suicide across all racial/ethnic groups (OR = 6.79, p < .001 for White youth; OR =
4.13, p = .001 for Hispanic youth; OR = 4.66, p = .002 for API youth; and OR = 4.70, p
< .001 for youth with mixed race/ethnicity). Female youth were more likely to display
suicidal behaviors than their male counterparts in two groups: OR = 1.89, p = .008 for
Hispanic youth; OR = 2.22, p = .002 for those with mixed race/ethnicity. However, sex
was not associated with suicidal behaviors among White or API youth. Age and grade
level were not significantly associated with suicide for White, Hispanic, API, or mixed
race/ethnicity youth.

Table 2. Path coefficients, odds ratio, Sobel z, and proportion mediated for the indirect effects of
bullying on suicide through ATOD by the racial/ethnic group.
Variable White Hispanic API Mixed
Coef. (OR) Coef. (OR) Coef. (OR) Coef. (OR)
Alcohol use
Path a 0.13 0.62 (1.86)* 1.39 (4.02)* 0.73 (2.08)*
Path b 0.52 0.75 (2.11)* 0.06 0.13
Path c 1.92 (6.79)*** 1.42 (4.13)*** 1.54 (4.66)** 1.55 (4.70)***
Path c′ 1.92 (6.84)*** 1.35 (3.87)*** 1.53 (4.60)** 1.53 (4.60)***
Sobel z 0.45 1.97* 0.11 0.55
Proportion mediated 7.80%
Tobacco use
Path a 0.51 0.48 2.95 (19.19)** −0.14
Path b 0.84 0.92 (2.51)* 0.23 0.69
Path c 1.92 (6.79)*** 1.42 (4.13)*** 1.54 (4.66)** 1.55 (4.70)***
Path c′ 1.90 (6.71)*** 1.41 (4.09)*** 1.52 (4.56)** 1.57 (4.80)***
Sobel z 0.79 0.54 0.40 −0.27
Marijuana use
Path a 0.94 (2.57)** 0.49 2.08 (8.01)* 0.38
Path b 0.64 1.35 (3.84)** −0.16 −0.30
Path c 1.92 (6.79)*** 1.42 (4.13)*** 1.54 (4.66)** 1.55 (4.70)***
Path c′ 1.83 (6.22)*** 1.41 (4.08)*** 1.57 (4.82)** 1.58 (4.84)***
Sobel z 1.38 1.20 −0.32 −0.84
Other drug use
Path a 0.98 (2.65)* 0.86 (2.37)* 1.95 (7.02)* 0.95 (2.58)**
Path b 1.15 (3.15)** 0.80 (2.24)* −0.76 0.20
Path c 1.92 (6.79)*** 1.42 (4.13)*** 1.54 (4.66)** 1.55 (4.70)***
Path c′ 1.83 (6.22)*** 1.34 (3.82)*** 1.67 (5.31)** 1.52 (4.56)***
Sobel z 2.02* 1.83 −0.82 0.53
Proportion mediated 10.77%
Depression
Path a 1.58 (4.84)** 1.49 (4.43)*** 2.00 (7.40)** 1.13 (3.11)***
Path b 2.15 (8.59)*** 2.46 (11.74)*** 2.22 (9.16)*** 1.97 (7.18)***
Path c 1.92 (6.79)*** 1.42 (4.13)*** 1.54 (4.66)** 1.55 (4.70)***
Path c′ 1.48 (4.40)*** 0.84 0.75 1.27 (3.58)**
Sobel z 3.03** 3.90*** 3.55*** 4.90***
Proportion mediated 36.30% 51.90% 55.26% 32.30%
Note: *p ≤ .05, **p ≤ .01; path a= the effect of the independent variable on the mediating variable, path b = the effect of
the mediating variable on the dependent variable while controlling for the independent variable, path c = the effect of
the independent variable on the dependent variable, path c′ = the effect of the independent variable on the dependent
variable while controlling for the mediating variable; for simplicity, odds ratio omitted for non-significant path
coefficients, proportion mediated omitted for non-significant Sobel tests, results of the covariates in the path models
omitted (all omitted results can be made available upon request).
702 Y. LI AND J. SHI

Direct effects
Path c′ (with the mediator in regression) represents the direct effect of bullying on suicide.
As shown in Table 2, when depression was included in the analyses, the direct effect of
bullying on suicide became non-significant among Hispanic and API youths. However, all
other direct effects remained statistically significant. These include the direct effects of
bullying on suicide among all four racial/ethnic subgroups when alcohol use, tobacco use,
marijuana use, and other drug use were added to the regression model. Also, among
White and mixed race/ethnicity youths, the direct effect was significant when depression
was included in the analysis.

Indirect effects
Table 2 includes the coefficients for the path a (the effects of bullying on depression and
ATOD use) and path b (the effect of depression and ATOD use on suicide while
controlling for bullying), which can be used to calculate the indirect effects of bullying
on suicide. As shown in Table 2, the indirect effects of bullying on suicide through
depression were significant across all four subgroups (Sobel z ranging from 3.03 to
4.90). The proportion of the total effect that was mediated was 36.30% for White youth,
51.90% for Hispanic youth, 55.26% for API youth, and 32.30% for youth with mixed race/
ethnicity. Two other significant indirect paths included the effect of bullying on suicide
through other drug use among White youth (Sobel z = 2.02) and the same effect through
alcohol use among Hispanic youth (Sobel z = 1.97), although the proportion of the effects
mediated was smaller (10.77% and 7.80%, respectively). Both path a and path b were
statistically significant in these indirect relationships (see Table 2).
As shown in Table 2, bully victimization was significantly associated with alcohol
use among API (OR = 4.02, p = .02) and mixed race/ethnicity youths (OR = 2.08, p =
.02). It was also significantly associated with tobacco use among API youth (OR =
19.19, p = .002), marijuana use among White (OR = 2.57, p = .007) and API youths
(OR = 8.01, p = .02), and other drug use among Hispanic (OR = 2.37, p = .02), API
(OR = 7.02, p = .03), and mixed race/ethnicity youths (OR = 2.58, p = .007).
Additionally, suicidal behaviors of Hispanic youth were significantly correlated with
tobacco use (OR = 2.51, p = .02), marijuana use (OR = 3.84, p = .005), and other drug
use (OR = 2.24, p = .02).

Discussion
In our analysis, bullying victimization is associated with increased suicidal behaviors
across all racial/ethnic groups, thus supporting Hypothesis 1. Previous research on bully-
ing and suicide has suggested the same relationship (e.g., Holt et al., 2015; Takizawa et al.,
2014). As for intergroup differences, White youth victimized from bullying are nearly six
times more likely to engage in suicidal behaviors than non-victims; while for the other
three groups, the odds are about three to four times. While existing literature suggests that
Hispanic youth in the general population may be at a higher risk for suicide (Grunbaum
et al., 2004; Kann et al., 2016; Miranda et al., 2017), our study underlines the detrimental
effect of bullying, especially for White youth.
JOURNAL OF HUMAN BEHAVIOR IN THE SOCIAL ENVIRONMENT 703

Previous research has suggested the mediating role of depression in suicidal


behaviors (Lardier et al., 2016; Reed et al., 2015). Our findings are consistent with
this line of research, thus supporting Hypothesis 2. In addition, our study provides
evidence on the positive interrelationships between bullying, depression, and suicide
in all four racial/ethnic groups. Existing evidence supports these relationships (Bridge
et al., 2006; Juvonen & Graham, 2014; Kaltiala-Heino & Fröjd, 2011), although few
studies have conducted comparative analysis using different racial/ethnic groups. In
an effort to explain why bullying is associated with depression, researchers have
suggested that bullying victims may blame themselves for what the bullies did
(Perren, Ettekal, & Ladd, 2013), have negative self-images (Turner et al., 2010), and
perceive neutral or positive social interactions as hostile (Prinstein, Cheah, & Guyer,
2005), all of which could potentially cause depressive symptoms among bullying
victims. In addition, our findings were consistent with previous research on depres-
sion and suicidal behaviors. For example, high school youth who had attempted
suicide were found to have more depressive symptoms than those who had no history
of self-harm or only had non-suicidal self-injury behaviors (Brausch & Gutierrez,
2010).
In addition, the results indicate that alcohol use mediates the effect of bullying on
suicide in Hispanic youth but not in other three subgroups, providing some evidence
for Hypothesis 3. Tobacco use or marijuana use does not act as a mediator in any of
the four groups, thus Hypothesis 4 and Hypothesis 5 are not supported. Use of other
drugs has a mediating role between bullying and suicide in White youth but not in
other three subgroups, providing some evidence for Hypothesis 6. Comparing to youth
from other racial/ethnic groups, White and Hispanic youth are both at higher risks for
alcohol use and illicit drug use (Patrick & Schulenberg, 2014; Swendsen et al., 2012);
when bullied, however, they may choose to use different substances, which in turn, put
them at a higher risk for suicidal behaviors. Therefore, the mediating role of ATOD
does vary by race/ethnicity: Among Hispanic youth victimized by bullying, alcohol
drinking is more likely to be associated with suicide; among White youth who are
bullying victims, illicit drug use is more likely to be associated with suicide; among API
and mixed race/ethnicity youth who are bullying victims, however, ATOD use is not
associated with heightened risks for suicide. To determine the underlying reasons for
these differences, future researchers may need to conduct qualitative studies with the
youth of different racial/ethnic groups.
Finally, results on the bivariate relationships between bullying, ATOD use, and suicide
also show intergroup similarities and differences. All bullied youths are at a higher risk for
other drug use. As for the differences, all but White youth who are bullied have a higher
risk for alcohol use. API youth victimized by bullying are at a higher risk for tobacco use.
White and API youths who are bullied have a higher risk for marijuana use. Again, these
findings suggest that bullied youth of different racial/ethnic groups may have their own
preference for substance, although illicit drugs seem to be a universal choice for youth
victimized by bullying. Additionally, Hispanic youth who use tobacco, marijuana, and
other drugs are at a higher risk for suicide; White youth who use other drugs are more
likely to engage in suicidal behaviors. These findings are interesting because ATOD use
seems to increase the odds of suicidality among White and Hispanic youth, but not among
API and mixed race/ethnicity youth.
704 Y. LI AND J. SHI

Implications
Given our findings on the destructive effect of bullying on youth behaviors, bullying
prevention and response programs are warranted among bullied youths from all four
racial/ethnical groups. Suicide prevention should be part of the conversation in pre-
venting and responding to bullying. Given the challenges in bullying prevention
(Juvonen & Graham, 2014), comprehensive and systematic approaches are to be
enforced simultaneously at school and at home (Hinduja & Patchin, 2010).
According to Hinduja and Patchin (2010), age-appropriate messages about all forms
of peer aggression (including school bullying and cyberbullying) must be conveyed to
adolescents. Parents and school personnel need to consistently regulate misbehaviors
toward peers and reinforce good behaviors. Additionally, it has been suggested that
using peer educators to raise awareness is an effective way to reduce bullying
(Palladino, Nocentini, & Menesini, 2016).
Evidence-based programs on the prevention and screening of depression and sui-
cide among adolescents are still lacking (Hallfors et al., 2004; Terzian, Moore, &
Constance, 2014). Based on our findings, these programs are necessary for all bullied
youths because they are at elevated risk for depression and suicide. Once diagnosed,
bullied youths should be referred to mental health services for the treatment of
depression. One treatment modality that might be useful in this regard is cognitive
behavioral therapy. It has proven effective for the treatment of depression among
adolescents (Klein, Jacobs, & Reinecke, 2007). Its techniques (such as cognitive
restructuring) can help bully victims correct cognitive distortions including self-
blame, negative self-images, and misperceptions about social interactions (Ayyash-
Abdo, 2002). In addition, mental health professionals need to integrate suicide pre-
vention when using these techniques to treat depression among bullied youth. This is
of vital importance because depression is correlated with increased probability of
suicide in this population.
Our analysis also reveals that Hispanic victims of bullying who use alcohol, tobacco, or
other drugs are more likely to engage in suicidal behaviors; therefore, ATOD use in this
population should be treated as red flags for suicide. This reiterates the need to create a
meaningful, holistic intervention approach to the prevention of substance abuse and
mental health problems including suicidality among Hispanic youth (Ttofi et al., 2016).
Similarly, programs addressing the use of other drugs and suicide among bullied White
youth are necessary given that use of other drugs mediates the effect of bullying on suicide
in White youth.
In addition, the prevention and treatment of alcohol and drug abuse seems to be a
viable option for all bullied youths. It is worthwhile to direct resources to help bullied
youth learn alternative coping methods, rather than turning to alcohol or other drugs.
Fortunately, these programs are more readily available than those for depression and
suicide prevention. For example, using the screening, brief intervention, and referral
to treatment (SBIRT) model, the Teen Intervene program is a brief, early intervention
program for youth who are involved in using alcohol and other drugs. Existing
research supports its effectiveness in reducing drug use behaviors among adolescents
aged 12–18 years in a school setting (Winters, Fahnhorst, Botzet, Lee, & Lalone,
2012).
JOURNAL OF HUMAN BEHAVIOR IN THE SOCIAL ENVIRONMENT 705

Limitations and directions for future research


First, our study did not establish any causal relationships among bullying, depression
and ATOD use, and suicidal behaviors due to the cross-sectional design. The causal
relationship between depression and being bullied, for example, could be true for either
way, although some studies supported that bullying led to depression rather than the
reverse (Ttofi et al., 2016). However, researchers have purported that depression may
cause bully victimization because youth with depression often face difficulties in peer
relationships and establishing friendships (Kochel, Ladd, & Rudolph, 2012; Menesini &
Salmivalli, 2017). Future research should consider using longitudinal designs to better
establish the causal relationships between bullying, depression and ATOD use, and
suicidal behaviors.
Second, we were not able to include any protective factors in our study given the
nature of the study. The YRBS questionnaire had no questions on protective factors,
such as social support and good parent-youth relationships, which are often believed to
buffer the detrimental influences of risk behaviors on youth well-being. These factors
include academic achievement, peer friendship and support, parent connectedness, and
neighborhood safety (Juvonen & Graham, 2014; Taliaferro & Muehlenkamp, 2014;
Yeung Thompson & Leadbeater, 2013). In fact, the differential effects of ATOD use
on suicide across racial/ethnic groups may be explained by the presence of these
protective factors. For example, a qualitative study among Asian American college
students revealed several protective factors including a desire not to hurt or burden
others, social support, self-reliance, etc. (Tran et al., 2015). In this direction, future
studies should investigate the protective roles of family dynamics, including parent-
child connection, parental supervision, and parental expectations, to name a few, in
preventing or abating risk behaviors in bullied youths of different racial/ethnic groups
(Holland, Vivolo-Kantor, Logan, & Leemis, 2017). The role of other protective factors
including religious involvement and teachers’ support is also worth investigating
(Terzian et al., 2014).
Third, we did not include African American and Native American youth in our analytic
sample. We acknowledge that this may tamper the representativeness of our sample.
However, the data set did not contain a sufficient number of observations of these racial
groups; therefore, we eliminated them from our analysis. If possible, future researchers
need to include these two groups in comparative studies. Studies on Native American
youth are in a desperate need because the prevalence rate of suicide among Native
Americans ages 18–24 nearly doubled the national average (Santhanam & Crigger,
2015). Another recommendation in this direction is to find out why certain groups are
more at risk for certain risk behaviors by doing qualitative studies. Moreover, our
comparative analysis revealed some between-group differences in the mediation effects;
however, we did not test how much difference there was. Future studies need to conduct
moderated mediation analyses which include the interaction term between race/ethnicity
and other study variables.
Finally, our study included only yes/no questions or dummy-coded variables to mea-
sure suicide, bullying, depression, and ATOD use, which might have introduced biases to
our findings. When possible, future research should consider incorporating validated
scales to measure these constructs.
706 Y. LI AND J. SHI

Conclusion
One of the strengths of the present study is the findings of the racial/ethnic differences in
the mechanism through which bullying is linked to youth suicide. Although depression is
a universal mediator for all youths, the role of ATOD use varies across different racial/
ethnic groups. More comparative studies on racial/ethnic differences are needed, particu-
larly on the different protective factors in buffering the negative consequence of bullying.
It is our hope that professionals working with bullied youth can use this type of research
to inform their endeavor to better prevent youth suicide by incorporating these risk and
protective factors in program development and implementation.

References
Ayyash-Abdo, H. (2002). Adolescent suicide: An ecological approach. Psychology in the Schools, 39
(4), 459–475. doi:10.1002/pits.10042
Brausch, A. M., & Gutierrez, P. M. (2010). Differences in non-suicidal self-injury and suicide
attempts in adolescents. Journal of Youth and Adolescence, 39(3), 233–242. doi:10.1007/s10964-
009-9482-0
Bridge, J. A., Goldstein, T. R., & Brent, D. A. (2006). Adolescent suicide and suicidal behavior.
Journal of Child Psychology and Psychiatry, 47(3–4), 372–394. doi:10.1111/j.1469-
7610.2006.01615.x
Canino, G., & Roberts, R. E. (2001). Suicidal behavior among Latino youth. Suicide and Life-
Threatening Behavior, 31, 122–131. doi:10.1521/suli.31.1.5.122.24218
CDC. (2014). The relationship between bullying and suicide: What we know and what it means for
schools. Atlanta, GA: National Center for Injury Prevention and Control/Centers for Disease
Control and Prevention.
CDC. (2017). Web-based Injury Statistics Query and Reporting System (WISQARS). Retrieved from
www.cdc.gov/injury/wisqars/index.html
Center for Behavioral Health Statistics and Quality. (2016). Key substance use and mental health
indicators in the United States: Results from the 2015 National Survey on Drug Use and Health
(HHS Publication No. SMA 16-4984, NSDUH Series H-51). Retrieved from http://www.samhsa.
gov/data/
De Man, A. F., & Leduc, C. P. (1995). Suicidal ideation in high school students: Depression and
other correlates. Journal of Clinical Psychology, 51(2), 173–181. doi:10.1002/1097-4679(199503)
51:2<173::AID-JCLP2270510205>3.0.CO;2-R
Fekkes, M., Pijpers, F. I. M., Fredriks, A. M., Vogels, T., & Verloove-Vanhorick, S. P. (2006). Do
bullied children get ill, or do ill children get bullied? A prospective cohort study on the relation-
ship between bullying and health-related symptoms. Pediatrics, 117(5), 1568–1574. doi:10.1542/
peds.2005-0187
Grunbaum, J. A., Kann, L., Kinchen, S., Ross, J., Hawkins, J., Lowry, R., . . . Collins, J. (2004).
Youth risk behavior surveillance–United States, 2003. MMWR Surveillance Summaries, 53(2),
1–96.
Hallfors, D. D., Waller, M. W., Ford, C. A., Halpern, C. T., Brodish, P. H., & Iritani, B. (2004).
Adolescent depression and suicide risk: Association with sex and drug behavior. American
Journal of Preventive Medicine, 27(3), 224–231. doi:10.1016/j.amepre.2004.06.001
Harter, S., & Marold, D. B. (1994). Psychosocial risk factors contributing to adolescent suicidal
ideation. New Directions for Child and Adolescent Development, 1994(64), 71–91. doi:10.1002/
cd.23219946407
Herr, N. R. (2006). Tools for automatically calculating logistic regression mediation. Retrieved from
http://www.nrhpsych.com/mediation/logmed.html
Hinduja, S., & Patchin, J. W. (2010). Bullying, cyberbullying, and suicide. Archives of Suicide
Research, 14(3), 206–221. doi:10.1080/13811118.2010.494133
JOURNAL OF HUMAN BEHAVIOR IN THE SOCIAL ENVIRONMENT 707

Holland, K. M., Vivolo-Kantor, A. M., Logan, J. E., & Leemis, R. W. (2017). Antecedents of suicide
among youth aged 11–15: A multistate mixed methods analysis. Journal of Youth and
Adolescence, 46(7), 1598–1610. doi:10.1007/s10964-016-0610-3
Holt, M. K., Vivolo-Kantor, A. M., Polanin, J. R., Holland, K. M., DeGue, S., Matjasko, J. L., . . .
Reid, G. (2015). Bullying and suicidal ideation and behaviors: A meta-analysis. Pediatrics, 135(2),
496–509. doi:10.1542/peds.2014-1864
Hong, J. S., Davis, J. P., Sterzing, P. R., Yoon, J., Choi, S., & Smith, D. C. (2014). A conceptual
framework for understanding the association between school bullying victimization and
substance misuse. The American Journal of Orthopsychiatry, 84(6), 696–710. doi:10.1037/
ort0000036
Juvonen, J., & Graham, S. (2014). Bullying in schools: The power of bullies and the plight of victims.
Annual Review of Psychology, 65, 159–185. doi:10.1146/annurev-psych-010213-115030
Kaltiala-Heino, R., & Fröjd, S. (2011). Correlation between bullying and clinical depression in
adolescent patients. Adolescent Health, Medicine and Therapeutics, 2, 37–44. doi:10.2147/AHMT.
S11554
Kann, L., McManus, T., Harris, W. A., Shanklin, S. L., Flint, K. H., Hawkins, J., . . . Zaza, S. (2016).
Youth Risk Behavior Surveillance — United States, 2015. MMWR Surveillance Summaries, 65(6),
1–174. doi:10.15585/mmwr.ss6506a1
Kann, L., Olsen, E. O., McManus, T., Harris, W. A., Shanklin, S. L., Flint, K. H., . . . Zaza, S. (2016).
Sexual identity, sex of sexual contacts, and health-related behaviors among students in grades
9–12 — United States and selected sites, 2015. MMWR Surveillance Summaries, 65(No. SS–9), 1–
202. doi:10.15585/mmwr.ss6509a1
Klein, J. B., Jacobs, R. H., & Reinecke, M. A. (2007). Cognitive-behavioral therapy for adolescent
depression: A meta-analytic investigation of changes in effect-size estimates. Journal of the
American Academy of Child and Adolescent Psychiatry, 46(11), 1403–1413. doi:10.1097/
chi.0b013e3180592aaa
Klomek, A. B., Kleinman, M., Altschuler, E., Marrocco, F., Amakawa, L., & Gould, M. S. (2013).
Suicidal adolescents’ experiences with bullying perpetration and victimization during high school
as risk factors for later depression and suicidality. Journal of Adolescent Health, 53(1), S37–S42.
doi:10.1016/j.jadohealth.2012.12.008
Klomek, A. B., Sourander, A., & Elonheimo, H. (2015). Bullying by peers in childhood and effects
on psychopathology, suicidality, and criminality in adulthood. The Lancet Psychiatry, 2(10), 930–
941. doi:10.1016/S2215-0366(15)00223-0
Kochel, K. P., Ladd, G. W., & Rudolph, K. D. (2012). Longitudinal associations among youth
depressive symptoms, peer victimization, and low peer acceptance: An interpersonal process
perspective. Child Development, 83(2), 637–650. doi:10.1111/j.1467-8624.2011.01722.x
Lardier, D. T., Jr., Barrios, V. R., Garcia-Reid, P., & Reid, R. J. (2016). Suicidal ideation among
suburban adolescents: The influence of school bullying and other mediating risk factors. Journal
of Child & Adolescent Mental Health, 28(3), 213–231. doi:10.2989/17280583.2016.1262381
Lessne, D., & Cidade, D. (2015). National center for education statistics. Retrieved from https://nces.
ed.gov/pubs2015/2015056.pdf
Lessne, D., & Yanez, C. (2016). Student reports of bullying: Results from the 2015 school crime
supplement to the national crime victimization survey. Retrieved from https://nces.ed.gov/
pubs2017/2017015.pdf
Litwiller, B. J., & Brausch, A. M. (2013). Cyber bullying and physical bullying in adolescent suicide:
The role of violent behavior and substance use. Journal of Youth and Adolescence, 42(5), 675–684.
doi:10.1007/s10964-013-9925-5
Luk, J. W., Wang, J., & Simons-Morton, B. G. (2012). The co-occurrence of substance use and
bullying behaviors among U.S. adolescents: Understanding demographic characteristics and
social influences. Journal of Adolescence, 35(5), 1351–1360. doi:10.1016/j.
adolescence.2012.05.003
McMahon, T. J., & Luthar, S. S. (2006). Patterns and correlates of substance use among affluent,
suburban high school students. Journal of Clinical Child and Adolescent Psychology, 35(1), 72–89.
doi:10.1207/s15374424jccp3501_7
708 Y. LI AND J. SHI

Menesini, E., & Salmivalli, C. (2017). Bullying in schools: The state of knowledge and effective
interventions. Psychology, Health & Medicine, 22(sup1), 240–253. doi:10.1080/
13548506.2017.1279740
Miranda, R., Ortin, A., Polanco-Roman, L., & Valderrama, J. (2017). Understanding adolescent
suicide. In N. L. Cohen, R. Miranda, A. Ortin, L. Polanco-Roman, & J. Valderrama (Eds.), Public
health perspectives on depressive disorders (pp. 211–238). Baltimore, MA: Johns Hopkins
University Press.
Palladino, B. E., Nocentini, A., & Menesini, E. (2016). Evidence-based intervention against bullying
and cyberbullying: Evaluation of the NoTrap! program in two independent trials. Aggressive
Behavior, 42(2), 194–206. doi:10.1002/ab.21636
Patrick, M. E., & Schulenberg, J. E. (2014). Prevalence and predictors of adolescent alcohol use and
binge drinking in the United States. Alcohol Research : Current Reviews, 35(2), 193–200.
Peleg-Oren, N., Cardenas, G. A., Comerford, M., & Galea, S. (2012). An association between
bullying behaviors and alcohol use among middle school students. The Journal of Early
Adolescence, 32(6), 761–775. doi:10.1177/0272431610387144
Perren, S., Ettekal, I., & Ladd, G. (2013). The impact of peer victimization on later maladjustment:
Mediating and moderating effects of hostile and self-blaming attributions. Journal of Child
Psychology and Psychiatry, 54(1), 46–55. doi:10.1111/j.1469-7610.2012.02618.x
Pranjić, N., & Bajraktarević, A. (2010). Depression and suicide ideation among secondary school
adolescents involved in school bullying. Primary Health Care Research & Development, 11(4),
349–362. doi:10.1017/S1463423610000307
Preacher, K. J., & Leonardelli, G. J. (2001). Calculation for the sobel test: An interactive calculation
tool for mediation tests. Retrieved from http://quantpsy.org/sobel/sobel.htm
Prinstein, M. J., Cheah, C. S. L., & Guyer, A. E. (2005). Peer victimization, cue interpretation, and
internalizing symptoms: Preliminary concurrent and longitudinal findings for children and
adolescents. Journal of Clinical Child and Adolescent Psychology, 34(1), 11–24. doi:10.1207/
s15374424jccp3401_2
Radliff, K. M., Wheaton, J. E., Robinson, K., & Morris, J. (2012). Illuminating the relationship
between bullying and substance use among middle and high school youth. Addictive Behaviors,
37(4), 569–572. doi:10.1016/j.addbeh.2012.01.001
Rasic, D., Weerasinghe, S., Asbridge, M., & Langille, D. B. (2013). Longitudinal associations of
cannabis and illicit drug use with depression, suicidal ideation and suicidal attempts among Nova
Scotia high school students. Drug and Alcohol Dependence, 129(1), 49–53. doi:10.1016/j.
drugalcdep.2012.09.009
Reed, K. P., Nugent, W., & Cooper, R. L. (2015). Testing a path model of relationships between
gender, age, and bullying victimization and violent behavior, substance abuse, depression,
suicidal ideation, and suicide attempts in adolescents. Children and Youth Services Review, 55,
128–137. doi:10.1016/j.childyouth.2015.05.016
Runeson, B. (1989). Mental disorder in youth suicide. Acta Psychiatrica Scandinavica, 79(5), 490–
497. doi:10.1111/j.1600-0447.1989.tb10292.x
Sampasa-Kanyinga, H., Roumeliotis, P., & Xu, H. (2014). Associations between cyberbullying and
school bullying victimization and suicidal ideation, plans and attempts among Canadian school
children. PLoS ONE, 9(7), e102145. doi:10.1371/journal.pone.0102145
Santhanam, L., & Crigger, M. (2015). Suicide among young American Indians nearly double national
rate. Retrieved from http://www.pbs.org/newshour/rundown/suicide-rate-among-young-ameri
can-indians-nearly-double-national-average/
Srabstein, J. C. (2013). News reports of bullying-related fatal and nonfatal injuries in the Americas.
Pan American Journal of Public Health, 33(5), 378–382. doi:10.1590/S1020-49892013000500010
Swahn, M. H., Bossarte, R. M., Choquet, M., Hassler, C., Falissard, B., & Chau, N. (2012). Early
substance use initiation and suicide ideation and attempts among students in France and the
United States. International Journal of Public Health, 57(1), 95–105. doi:10.1007/s00038-011-
0255-7
Swendsen, J., Burstein, M., Case, B., Conway, K. P., Dierker, L., He, J., & Merikangas, K. R. (2012).
Use and abuse of alcohol and illicit drugs in US adolescents: Results of the national comorbidity
JOURNAL OF HUMAN BEHAVIOR IN THE SOCIAL ENVIRONMENT 709

survey–adolescent supplement. Archives of General Psychiatry, 69(4), 390–398. doi:10.1001/


archgenpsychiatry.2011.1503
Takizawa, R., Maughan, B., & Arseneault, L. (2014). Adult health outcomes of childhood bullying
victimization: Evidence from a five-decade longitudinal British birth cohort. The American
Journal of Psychiatry, 171(7), 777–784. doi:10.1176/appi.ajp.2014.13101401
Taliaferro, L. A., & Muehlenkamp, J. J. (2014). Risk and protective factors that distinguish
adolescents who attempt suicide from those who only consider suicide in the past year. Suicide
and Life-Threatening Behavior, 44(1), 6–22. doi:10.1111/sltb.12046
Terzian, M. A., Moore, K. A., & Constance, N. (2014). Transitioning to adulthood: The role of
adolescent depression and suicidal ideation. Bethesda, MD: Child Trends.
Tran, K. K., Wong, Y. J., Cokley, K. O., Brownson, C., Drum, D., Awad, G., & Wang, M.-C. (2015).
Suicidal Asian American college students’ perceptions of protective factors: A qualitative study.
Death Studies, 39(8), 500–507. doi:10.1080/07481187.2014.970299
Ttofi, M. M., Farrington, D. P., Lösel, F., Crago, R. V., & Theodorakis, N. (2016). School bullying
and drug use later in life: A meta-analytic investigation. School Psychology Quarterly, 31(1), 8–27.
doi:10.1037/spq0000120
Turner, H. A., Finkelhor, D., & Ormrod, R. (2010). The effects of adolescent victimization on self-
concept and depressive symptoms. Child Maltreatment, 15(1), 76–90. doi:10.1177/
1077559509349444
Winters, K. C., Fahnhorst, T., Botzet, A., Lee, S., & Lalone, B. (2012). Brief intervention for drug-
abusing adolescents in a school setting: Outcomes and mediating factors. Journal of Substance
Abuse Treatment, 42(3), 279–288. doi:10.1016/j.jsat.2011.08.005
Yeung Thompson, R. S., & Leadbeater, B. J. (2013). Peer victimization and internalizing symptoms
from adolescence into young adulthood: Building strength through emotional support. Journal of
Research on Adolescence, 23(2), 290–303. doi:10.1111/j.1532-7795.2012.00827.x
Copyright of Journal of Human Behavior in the Social Environment is the property of Taylor
& Francis Ltd and its content may not be copied or emailed to multiple sites or posted to a
listserv without the copyright holder's express written permission. However, users may print,
download, or email articles for individual use.

You might also like