JOURNAL OF APPLIED DEVELOPMENTAL PSYCHOLOGY 16,95-106 (1995)

Classification of Problem Behaviors

in Young Children: A Comparison
of Four Systems

BEVERLY I. FAGOT
Oregon Social Learning Center and University of Oregon

Both 156 toddler-aged children (16 to 18 months old at first testing) and their parents
provided data on four measures that have been used for selecting children who
might have problems: (1) attachment classification (Ainsworth Stmnge Situation), (2)
temperament, (Toddler Temperament Scale), (3) parent mtings of child problem be-
haviors (Child Behavior Checklist for Ages 2-3), and (4) playgroup observation of
child problem behaviors, (Pagot Interactive Code). The attachment and temperament
data were collected at 18 months, the peer playgroup data from 18 to 24 months,
and the parent ratings of child problem behaviors at 24 months. The best agreement
was between the same raters on different systems (i.e., mother temperament ratings
with mother problem behavior ratings). When systems were compared across raters
(mothers versus fathers) or methods (ratings, attachment classification, and observa-
tions), most predictions were very close to or below chance. The implications for
treatment and prevention programs are discussed.

The recent rise of interest in infant psychopathology has been at least partially
fueled by evidence that behavior problems are present in very young children and
that they are stable over a number of years (Richman, Stevenson, & Graham,
1982; Rose, Rose, & Feldman, 1989). The evidence is particularly strong for the
stability of externalizing problems (Campbell & Ewing, 1990). In addition to
examining clinical problems, there is a new emphasis on prevention with young
children which, if implemented, would require identifying children at risk (Zig-
ler, Taussig, & Black, 1992). There is new urgency, therefore, in examining
existing classification systems for children below the age of 3 years. Not enough
work has been done on the development of assessment procedures for children
under 3 years of age, although this is rapidly changing.
Four different ways of classifying children were examined to determine
whether there is agreement among the systems in selecting children at risk. Each
of these measures has been used to predict or screen for risk of future prob-
lems, including both externalizing and internalizing behaviors. Two of the sys-
tems (attachment and temperament) were developed as research tools to study

This research was supported in part by NIMH grant MH 37911 and NICHD grant HD 17571.
Correspondence and requests for reprints should be sent to Beverly Fagot, Oregon Social Leam-
ing Center, 207 East 5th Avenue, Suite 202, Eugene, OR 97401.

95
96 FAGOT

individual differences within developmental psychology, but in the past few

years have come to be used as predictors of future problems. The attachment
system, as measured by the Strange Situation (Ainsworth, Blehar, Waters, &
Wall, 1978) was initially developed as an experimental procedure to study differ-
ent styles of dyadic interaction between mothers and children; however, the
technique has gained popularity as a method for predicting future behavior prob-
lems (Belsky & Nezworski, 1988), and there has been increasing reference to the
use of attachment classification as a risk factor. Temperament characteristics,
another method of classification originally used in developmental research, are
now also being used in intervention programs (Cameron & Rice, 1986). The
New York Longitudinal Study examined long-term temperament characteristics
in a small group of children (Thomas & Chess, 1977; Thomas, Chess, Birch,
Hartzig, & Kern, 1963). This study was expanded to examine the relationship
between temperament characteristics and behavioral disorders (Thomas, Chess,
& Birch, 1968). Chess and Thomas (1986) have published a book examining the
role of temperament in clinical practice.
Two more traditional assessment techniques are available for measuring prob-
lem behaviors in young children. The Child Behavior Checklist for Ages 2-3
(CBCL/2-3; Achenbach, Edelbrock, & Howell, 1987) was developed to assess
problem behaviors in young children, and it is the most clinical of the assessment
techniques. Observation procedures have been developed during the past 25
years in both research and clinical traditions, and they have been used to diag-
nose and assess change in treatment as well as to study normative interaction in
the family (Reid, Baldwin, Patterson, & Dishion, 1988). Each of these measures
has established ways to classify risk versus nonrisk scores; these will be dis-
cussed.
Two broad patterns of behavioral disorders found in adults (internalizing and
externalizing) have been identified in young children (Cicchetti & Toth, 1991).
Each of the assessment procedures discussed in this article has been considered a
predictor of both internalizing and externalizing behaviors. Each of these proce-
dures, therefore, has been used to predict the same general sets of behavioral
problems. The question addressed in this article is the extent to which the
systems overlap in their classification of children at risk.

METHOD

Subjects
The total sample consisted of 156 toddler-aged children, half boys and half girls,
who were participants in longitudinal studies, all of whom were 18 months old at
the beginning of the studies. Although all the children were observed in the
Strange Situation, not all of them had usable measures on each classification
system, with 143 children having complete data with their mother and 125 with
their fathers.
 PROBLEM CLASSIFICATION COMPARISON 97

This toddler sample represented a good cross-section from the Eugene-

Springfield, Oregon, population. Nineteen of the children were from single-
mother families, and 137 were from two-parent families. The income of the
two-parent families ranged from below $3,000 to $100,000 per year, with a
median income of $15,500. The two-parent families had been married or living
together for 1 to 19 years, with a median of 4.2 years together. Approximately
15% of the two-parent families were unmarried. The fathers’ education level
ranged from high school or equivalent to graduate school, with a median of some
college. The mothers’ education ranged from some high school to graduate
school, again with a median of some college. The socioeconomic status level of
the fathers ranged from 0 to 9 on the Hollingshead 7-factor schedule, with a
median of 4. Of the fathers, 70% worked full time, 20% part time, and 10% were
unemployed. The mothers’ occupational level ranged from 0 to 9 with a median
of 2. Of the mothers 35% worked full-time, 35% were working part-time, and
30% said they were full-time housewives. The single mothers had a mean in-
come of $9,000, ranging from $2,500 per year to $35,000. Approximately half
of the single mothers were divorced, and the other half had never been married.
Their education level did not differ from the married mothers. These figures are
very similar to overall Lane County, Oregon, statistics.
The ethnic background of the children and their parents was also representa-
tive of Eugene-Springfield. The sample was 95% Caucasian, 2% Black, 1%
Hispanic, 1% Asian, and 1% South Pacific Island.
It should be noted that although this was not a risk sample, this does not mean
that it was a sample without risk. For instance, by the time the oldest of these
children was 5 years old, eight of the families had had contact with the Oregon
Children’s Service Division for suspicion of abuse, four families had been home-
less, 3 1% had one parent who had lost a job or had had employment reduced, and
after 4 years, almost 30% had changed family status (e.g., the married couple had
divorced; the single mother had remarried so the children had a stepfather). This
was a sample of families not selected for risk who were proceeding with the risky
business of parenting children through the 1980s and 1990s in the United States. If
data from previous longitudinal samples are predictive, we would expect approx-
imately 10% of these children to encounter adjustment problems that require some
intervention and another 10% to encounter some difftculties at some point in their
childhood that bring them to the attention of the school or the law.

Procedure
The children and their families were participants in a longitudinal study examin-
ing the relationship between parenting style and child problem behaviors. The
children were assessed in the Strange Situation at 18 months of age. At the same
time, parents were asked to fill out the Toddler Temperament Scale (Fullard,
McDevitt, & Carey, 1984). The children then attended toddler playgroups in
which they were observed using the Fagot Interactive Behavior Code (Fagot,
98 FAGOT

1983) over a period of at least 6 months. Finally, at approximately 2 years of age,

the parents rated the children on the Child Behavior Checklist for Ages 2-3
(Achenbach et al., 1987).

Measures

Toddler Temperament Scale. Fullard et al. (1984) developed a toddler tem-

perament scale based on the conceptualization of Thomas and Chess ( 1977). The
scale contains the nine temperament categories-activity level, rhythmicity, ap-
proach and withdrawal, adaptability, intensity of reactions, mood, persistence,
distractibility, and frustration threshold-developed in the New York Longitudi-
nal Project (Thomas et al., 1963). Fullard et al. (1984) reported that the scales
had median alphas of .70 and median test-retest reliabilities of .81. Fullard et al.
(1984) followed the lead of Thomas et al. (1963) and created a measure of
difficulty using five of the categories-rhythmicity, approach and withdrawal,
adaptability, intensity, and mood. The scoring system of Fullard et al. can be
used to place all children into one of four ranked categories-easy, intermediate
low, intermediate high, and difficult. They reported that 12.3% of their sample
were classified as difficult.

Atkzchment Measure. The Strange Situation (Ainsworth & Wittig, 1969) is a

laboratory procedure developed to assess the theoretical area of attachment.
Specifically, it is intended to look at individual differences in the patterning of
infants’ attachment behaviors judged by their reactions to repeated separations
from and reunions with the parent. The task includes the infant, mother, and
stranger, and spans approximately 20 min, divided into eight increasingly stress-
ful episodes. The task took place in a room approximately 3.05 m X 4.27 m,
furnished with chairs and appropriate toys. The security of the child’s attachment
is assessed based on the child’s scores with the mother in the two reunion
episodes following separation. Infants are classified into one of four secure
categories (B babies) or one of two insecure classifications: insecure avoidant (A
babies) or insecure resistant (C babies). A senior member of the Oregon Social
Learning Center (OSLC) staff completed a 2-week training session with Michael
Lamb at the University of Utah along with individuals from several other labora-
tories. A criteria tape was developed in this situation with raters from five
different laboratories coming to consensus on 10 different sessions. The staff
member then trained a group of coders using a set of tapes developed at OSLC.
Kappas were computed on agreement between the newly trained observers and
the consensus classifications from the Utah training group; for each observer, the
kappa was above .75.
In middle-class, home-reared samples, approximately 70% of babies are clas-
sified as securely attached, 20% as avoidant, and 10% as resistant (Ainsworth et
al., 1978). van Ijzendoom and Kroonenberg (1988) found remarkable consisten-
cy to this norm across several cultures; they reported as much within-culture
 PROBLEM CLASSIFICATION COMPARISON 99

variation as between-culture variation. We compared the proportions of children

in each classification in the current sample to the van Ijzendoom and Kroonen-
berg norms and found no difference. In the present sample, 67% of the children
were classified as secure, 23% as avoidant, and 10% as resistant. There were no
differences between boys and girls in the pattern of classifications.

ChiM Behavior Checklist for Ages 2-3. The Child Behavior Checklist for
Ages 2-3 (CBCL/2-3; Achenbach et al., 1987) consists of 99 items chosen as
problems in children aged 2 to 3, with space for parents to write in additional
problems. Of these items, 57 overlap with items from the Child Behavior Check-
list for Ages 4-16 (CBCL/4-16; Achenbach & Edelbrock, 1981). The scoring
profile consists of six scales and two broad factors derived from factor analyses
of checklists filled out by parents of four hundred 2- to 3-year-old children; half
boys and half girls. There were no sex differences at this age; therefore, scores
were computed for boys and girls together. We have used the broad Externalizing
and Internalizing factors for this study. The Externalizing factor includes items
from the Destructive and Aggression scales and the Internalizing factor includes
items from the Depressed and Social Withdrawal scales. The l-week stability for
the two broad band factors was .88, and the l-year stability was .76 for Inter-
nalizing and .70 for Externalizing.

Observational Coding System. The code used was the Interactive Behavior
Code developed by Fagot (1983), currently in use as part of a longitudinal study
being conducted at the University of Oregon Child Research Laboratory. The
code is designed to capture as broad a sample of behavior as possible. This code
consists of five subcodes: (1) Context, (2) Interaction, (3) Recipient, (4) Reactor,
and (5) Reaction. With this code, it is possible to record all phases of what is
going on at any particular moment: Context (what the child is doing or paying
attention to, or what is happening in the very near vicinity), Interaction (the
child’s interactional behavior; e.g., talking), Recipient (to whom the interaction
is directed), Reactor (who reacts to the child’s interactive behavior), and Reac-
tion (how the reactor responds). A more thorough explanation of this coding
system and of observer agreement appears in Fagor and Hagan (1986, 1991). We
clustered two sets of behaviors for data analysis: Negative Behaviors (demand
attention, take toys, physical aggression, hit, verbal aversive, yell at, whine,
criticize, nag) with an alpha of .79 and a kappa of .85, and Passive Behaviors
(look at, passively sitting, no activity, no interaction) with an alpha of .81 and a
kappa of .82.

Statistical Analyses
In this article we examine risk classification and agreement using both continu-
ous and categorical comparisons. Each measure includes the capacity to be used
in either way, although in general both temperament ratings and attachment
100 FAGOT

classification are most often used as categorical variables, while observations

and the CBCL are most often used as continuous variables. Boys and girls were
examined separately in the first analyses; because there were no differences in the
results, analyses were then done for both sexes together. This lack of mean
differences of sex of child in problem behaviors of l- and 2-year-old children is
very consistent with the findings of others (Achenbach et al., 1987).
At this point, there is very little information on how well the different types of
assessment relate to one another in either clinical or normal populations. Even
more important is how well they do or do not identify the same children as at
risk. What method does one use to compare the different modes of assessment?
We used a measure of predictive efficiency used by Loeber, Dishion, and Patter-
son (1984) to determine the efficacy of different assessments for the prediction of
delinquency. Loeber et al. extended the work of Cronbach and Gleser (1965) and
Wiggins (1973) in determining valid positives and valid negatives to develop a
predictive measure that showed the relative improvement over chance (RIOC).
RIOC allows one to compare the predictive agreement between various nominal
classification systems despite varying marginal distributions of the categories.
RIOC is closely related to kappa, and like kappa contains a correction for
chance. In addition, RIOC has the unique advantage of correcting for the maxi-
mum value resulting from any discrepancy between base rate and selection ratio.
In addition, confidence intervals can be calculated to test significance of differ-
ence from zero (no improvement over chance) or between two or more RIOCs
(Copas & Loeber, 1990). Furthermore, RIOC is expressed in percentages over
chance (0.00) so that it can be easily understood by individuals with widely
varying backgrounds.

RESULTS

Classification of Children into Risk Groups

Children were classified into at-risk versus not-at-risk groups on each of the four
measures. For the attachment measure, the Ainsworth scoring system was used,
and children were classified into seven subgroups (Al, A2, B 1, B2, B3, B4, and
C). In one analysis, we combined the A and C groups as insecurely attached and
the B groups as securely attached. We then compared securely attached against
each of the insecure classifications separately.
All parents participating in the study filled out the temperament questionnaire
(156 mothers and 137 fathers). Approximately 11% of the mothers and 7% of the
fathers classified their children as difficult. All other categories were considered
not at risk.
For the CBCL/2-3, we used the Achenbach scales for internalizing and
externalizing behaviors. Five of the mothers and 11 of the fathers did not fill out
the CBCL/2-3 or filled it out incorrectly, so the scores were calculated on ratings
of 15 1 mothers and 126 fathers. The initial norms for the toddler age groups were
based upon a relatively small sample of 273 children (Achenbach et al., 1987),
 PROBLEM CLASSIFICATION COMPARISON 101

and there were no norms for Internalizing and Externalizing scores presented in
the initial articles. We used the 90th percentile as the cut-off for our sample
(separately for mothers and fathers). Fifteen children were classified at risk on
the Externalizing scale and 15 on the Internalizing scale when mothers were
rating, with 14 on the Externalizing scale and 12 on the Internalizing scale when
fathers were rating. We followed the same procedures Achenbach used to select a
risk group within his normative sample.
For the playgroup observation data, we used 1.3 standard deviations above the
mean for passive and negative behaviors to classify 15 children as at risk (again
putting the top 10% of children in the risk sample). Children were observed in
playgroups from the time they were 17 months old until they were 24 months
old. Each child’s score is an aggregate across at least two different terms of
playgroup observations, so that the composition of the class of peers and teachers
changed once. Of the children, 148 attended the playgroups often enough for us
to calculate scores. Two scores are examined in this article: (1) a withdrawn score
from when the child was not engaged in a task and was passively watching others
or playing alone, and (2) a set of negative behaviors, including both physically
aggressive behaviors and verbal negatives. The scores were standardized by
playgroup and aggregated across the terms. The analyses were computed using
the standardized score as the dependent variable. The numbers of children in
each category under each system are presented in Table 1. In addition, means and
standard deviations are presented for the two systems in which they are appropri-
ate (CBCL and observations).

TABLE 1
Number of Children in Each Category in the Four Classification Systems

Classification System N n M SD

Attachment classification 156

Resistant 15
Avoidant 36
Secure 105

Difficult temperament
Mother rating 156 16
Father rating 137 IO

CBCLI 2-3
Mother rating 151
Externalizing 15 46.6 5.8
Internalizing 15 45.9 5.3
Father rating 126
Externalizing 14 46.4 6.2
Internalizing 12 44.6 5.4

Observed behaviors 148

Negative 15 .04 .08
Withdrawn 14 .06 .05
102 FAGOT

TABLE 2
Overlap and Breakdown of At Risk Classification By System

Classified as at Risk With Mothers With Fathers

On 4 systems 1 0
On 3 systems 2 1
On 2 systems 15 12
On 1 system 60 51
Total classified at risk 78 (55%) 64 (52%)

Note. The maximum assessed on all four systems was 143 with
mothers and 125 with fathers.

Overlap of Classification Systems

The first question was a very simple one: Do the same children get classified as at
risk under each system? Only one child was classified at risk on all four systems
with mother ratings, and no child was classified as at risk on all four sys-
tems with father ratings. Only 15 children were classified as at risk on two
systems with mother ratings, 12 with father ratings. The results are presented in
Table 2.

Individual Comparisons of the Systems

Systems were next compared by pairs. If the percent improvement over chance is
negative, then one would be better off simply choosing risk samples at random,
whereas positive numbers indicate an improvement over chance: the larger the
number, the greater the improvement. Copas and Loeber (1990) show that the
significance of RIOC compared with chance can be tested by using the chi-
square statistic.
When chi squares were calculated, there were only two RIOCs that differed
significantly from chance (within this sample the RIOC had to be above 38% to
be significantly different than chance). Using mothers’ temperament ratings, we
found a 41.5% improvement over chance in predicting mothers’ ratings of inter-
nalizing behavior in their children, and we found that mothers’ rating of tempera-
ment difficulty produced a RIOC of 47.1% in predicting withdrawn behaviors in
the playgroups. Examining results carefully, it became clear that the rater was a
better predictor of scores than the content of the behavior scale. When systems
were compared across raters (mothers versus fathers) or methods (ratings, attach-
ment classification, and observations), the prediction was very close to or below
chance. These results are summarized in Table 3.

DISCUSSION

In this comparison of four classification systems used to predict which 2-year-old

children might be at risk for problem behaviors, there was very little overlap
 TABLE 3
Percent Relative Improvement Over Chance (RIOC) Using Four Classification Systems

System 1 2 3 4 5 6 7 6 9 10 11

Attachment
1. Insecure (all) 29.90 -8.80 -3.50 -2.00 -33.70 -4.50 -7.30 -8.30
2. Insecure-avoidant 23.50 -12.20 -7.20 -1.10 -33.20 5.80 -4.40 -2.70
3. Insecure-resistant -4.60 7.80 -3.30 -3.20 -4.50 4.50 -1.10 4.30

Difficult temperament
4. Mother ratings 2.94 28.70 47.10’ -4.50 15.40 -1.06 41.50*
5. Father ratings -6.20 -1.20 -8.3 26.28 -.38 6.99

CBCLl213
Mother ratings
6. Aggressive 3.60 -9.50 1.10 1.60
7. Withdrawn -12.30 2.30 -4.30 2.40
Father ratings
8. Aggressive 9.50 -9.50
9. Withdrawn 6.50

ObServed behaviors
10. Aggressive
11. Withdrawn

Note. All scores are percentages. Numerals in top row designate the same measures as numerals in lefthand column.
104 FAGOT

among the four systems. If one were to do a targeted prevention study using any
one of these measures, different children would be classified as at risk depending
on the measure used. While this is not surprising, considering the different
purposes behind each of the systems, at this point each system is being used to
predict similar problem behaviors (i.e., the broad categories of internalizing and
externalizing). This presents a very real quandary for researchers or clinicians
attempting to work with very young children. It may be that each system is
tapping a slightly different domain of child risk and that each will measure a
different path to the development of problems. Perhaps some combination score
will provide reasonable predictions of problem behaviors; however, so few chil-
dren showed overlapping classifications when using different methods and differ-
ent agents that the development of meaningful combination scores that would
include a reasonable number of children is very difficult. Choosing children who
are classified at risk on any one system would mean that over 50% of all children
would be considered at risk, which is quite at odds with the number of children
who actually encounter difficulty within a sample such as this one.
Even when there is agreement across measures, we are concerned that it may
be the result of agent variance rather than intrinsic child behavior. When there is
overlap among the systems, it occurs most often when using the same raters (the
CBCL/2-3 and the Toddler Temperament Scale are both parent report measures).
Is the improvement over chance in predicting problem behaviors from tempera-
ment ratings because of an overlap between temperament and problem behaviors
or because the same agent uses different scales in similar ways? We know that
there is stability in parent ratings for children from ages 2 to 5 years, but, again,
is this stability in the child or in the parent?
It is not that children are totally unstable in their behaviors. Fagot (1984)
found a good deal of stability in aggressive negative behaviors using observation
data from playgroups, but again this was using one method. Fagot and O’Brien
(1994) found that activity level was stable in toddler-aged children over a period
of 1 year, but that there was little relation between the child’s activity level in
peer groups and parents’ reports of activity level on a temperament question-
naire. Parent report measures seem to be tapping variables other than the child’s
behavior.
We are in the process of retesting this group of children as they reach the age
of 7 and will be able to examine how well each system predicts problem behav-
iors in school-aged children. One caution concerning the lack of sex differences
in any of these classification systems at ages 1 and 2 is our knowledge that the
developmental path to problems appears different for boys and girls (Zahn-
Waxler, 1993), so that the same level of behavior or the same classification may
have very different predictive meanings for boys and girls. Ultimately, it will be
the long-term predictive value of the classification scheme that will prove or
disprove its usefulness for the development of intervention programs. Although
ours was not a risk sample, it was a very good representative sample of children
 PROBLEM CLASSIFICATION COMPARISON 105

from a broad variety of backgrounds. Approximately 20% of these children will

encounter some difficulty either in behavioral adjustment or with academic
achievement during their school years. If early identification would help prevent
many of these problems, then it is worth the effort to develop measures for young
children very carefully. It does no service to the field to use existing measures as
if they were interchangeable in selecting children for either prevention or inter-
vention studies.

REFERENCES

Achenbach, T.M., & Edelbrock, C.S. (1981). Behavioral problems and compctencies reported by
parents of normal and disturbed children aged four through sixteen. Monographs of the
Society for Research in Child Development, 46( 1, Serial No. 188).
Achenbach, T.M., Edelbrock, C.S., & Howell, C.T. (1987). Empirically based assessment of the
behavioral/emotional problems of 2 and 3 year old children. Journal of Abnormal Child
Psychology, 15, 629-650.
Ainsworth, M.D.S., Blehar, M.E., Waters, E., & Wall, S. (1978). Patterns of attachment: A
psychological study of the strange situation. Hillsdale, NJ: Erlbaum.
Ainsworth, M.D.S., & Wittig, B.A. (1969). Attachment to exploratory behavior of one-year-olds in
a strange situation. In B.H. Foss (Ed.), Determinants of infant behavior (Vol. 4, pp. 113-
136). London: Mother.
Belsky, J., & Nezworski, T. (1988). Clinical implications of attachment. Hillsdale, NJ: Erlbaum.
Cameron, J.R., & Rice, D.C. (1986). Developing anticipatory guidance programs based on early
assessment of infant temperament: Two tests of a prevention model. Journal of Pediatric
Psychology, Il. 221-234.
Campbell, S.B., & Ewing, L.J. (1990). Follow-up of hard-to-manage preschoolers: Adjustment at
age 9 and predictors of continuing symptoms. Journal of Child Psychology and Psychiatry, 6,
871-889.
Chess, S., & Thomas, A. (1986). Temperament in clinical practice. New York: Guilford.
Cicchetti, D., & Toth, S .L. (Eds.). (199 1). Internalizing and externalizing expressions of dysfunction:
Rochester symposium on developmental psychopathology (Vol. 2). Hillsdale, NJ: Erlbaum.
Copas, J.B., & Loeber, R. (1990). Relative improvement over chance (RIOC) for 2 X 2 tables.
British Journal of Mathematical and Statistical Psychology, 43. 293-307.
Cronbach, L.S., & Gleser, G.C. (1965). Psychological tests and personal decisions. Urbana, IL:
University of Illinois Press.
Fagot, B.I. (1983). Training manual for the interactive code. Unpublished instrument. (Available
from B.I. Fagot, Oregon Social Learning Center, 207 East 5th Avenue, Suite 202, Eugene,
OR 97401,).
Fagot, B.I. (1984). The consequents of problem behavior in toddler children. Journal of Abnormal
Child Psychology, 12, 385-395.
Fagot, B.I., & Hagan, R. (1986). Coding of interactions: Is reliability really a problem? Canadian
Journal of Research in Early Childhood Education, I, 1 1 1 - 167
Fagot, B.I., & Hagan, R. (1991). Observation of parent reactions to sex-stereotyped behaviors: Age
and sex effects. Child Development, 62, 617-628.
Fagot, B.I., & O’Brien, M. (1994). Activity level in young children: Cross-age stability, situational
influences, correlates with temperament, and perception of problem behaviors. Merrill-
Palmer Quarterly, 40, 378-398.
Fullard, W., McDevitt, S.C., & Carey, W.B. (1984). Assessing temperament in one to three year old
children. Journal of Pediatric Psychology, 9, 205-217.
106 FAGOT

Loeber, R., Dishion, T.J., & Patterson, G.R. (1984). Multiple gating: A multistage assessment
procedure for identifying youths at risk for delinquency. Journal of Research in Crime and
Delinquency, 21, l-32.
Reid, J.B., Baldwin, D.V., Patterson, G.R., & Dishion, T.J. (1988). Some problems relating to the
assessment of childhood disorders: A role for observational data. In M. Rutter, A.H. Tuma, &
I. Lann (Eds.), Assessment and diagnosis in child and adolescent psychopathology (pp. 156-
195). New York: Guilford.
Richman, N., Stevenson, J., & Graham, P.J. (1982). Preschool co school: A behavioral study. New
York: Academic.
Rose, S.L., Rose, S.A., & Feldman, J.F. (1989). Stability of behavior problems in very young
children. Development and Psychopathology, I, S-20.
Thomas, A., & Chess, S. (1977). Temperament and development. New York: Brunner/Mazel.
Thomas, A., Chess, S., & Birch, H. (1968). Temperament and behavior disorders in children. New
York: New York University Press.
Thomas, A., Chess, S., Birch, H., Hartzig, H., & Kom, S. (1963). Behaviors and individuality in
early childhood. New York: New York University Press.
van Ijzendoom, M.H., & Kroonenberg, P.M. (1988). Cross-cultural patterns of attachment: A meta-
analysis of the Strange Situation. Child Development, 59, 147-156.
Wiggins, J.S. (1973). Personality and prediction: Principles of personality assessment. Reading,
MA: Addison-Wesley.
Zahn-Waxler, C. (1993). Warriors and worriers: Gender and psychopathology. Developmental Psy-
chopathology, 5, 79-90.
Zigler, E., Taussig, C., & Black, K. (1992). Early childhood intervention: A promising preventative
for juvenile delinquency. American Psychologist, 47, 997-1006.