ARTICLE IN PRESS

Evaluation and Program Planning 31 (2008) 299–310

www.elsevier.com/locate/evalprogplan

Systems thinking
Derek Cabreraa,b,d,, Laura Colosic,a, Claire Lobdellc
a
Cornell University, Ithaca, NY USA
b
Santa Fe Institute, Santa Fe, NM USA
c
Department of Policy Analysis and Management, Cornell University, USA
d
ThinkWorks, Ithaca, NY USA
Received 6 July 2007; received in revised form 26 November 2007; accepted 28 December 2007

Abstract

Evaluation is one of many fields where ‘‘systems thinking’’ is popular and is said to hold great promise. However, there is disagreement
about what constitutes systems thinking. Its meaning is ambiguous, and systems scholars have made diverse and divergent attempts to
describe it. Alternative origins include: von Bertalanffy, Aristotle, Lao Tsu or multiple aperiodic ‘‘waves.’’ Some scholars describe it as
synonymous with systems sciences (i.e., nonlinear dynamics, complexity, chaos). Others view it as taxonomy—a laundry list of systems
approaches. Within so much noise, it is often difficult for evaluators to find the systems thinking signal. Recent work in systems thinking
describes it as an emergent property of four simple conceptual patterns (rules). For an evaluator to become a ‘‘systems thinker’’, he or she
need not spend years learning many methods or nonlinear sciences. Instead, with some practice, one can learn to apply these four simple
rules to existing evaluation knowledge with transformative results.
r 2008 Elsevier Ltd. All rights reserved.

Keywords: Systems thinking; Systems evaluation; Concept ecology; Patterns of thinking; DSRP

1. Introduction Systems thinking as an idea permeates both popular

culture and numerous scientific fields including: planning
This paper offers insight into why people in many fields, and evaluation, education, business and management,
including evaluation, are drawn to and motivated to public health, sociology and psychology, cognitive science,
implement systems thinking. The reasons for its growing human development, agriculture, sustainability, environ-
popularity are likely as diverse as those who believe it holds mental sciences, ecology and biology, earth sciences, and
great promise. Yet beneath these reasons may lay a more other physical sciences. Systems thinking can influence
fundamental explanation for the allure of systems thinking: many of the existing concepts, theories and knowledge in
it offers a model for thinking differently. Despite this each of these fields. Yet, systems thinking can also be
allure, there is disagreement about what constitutes systems ambiguous and amorphous. There are numerous conflict-
thinking, and its meaning is ambiguous. This article seeks ing models and claims about systems thinking that need to
to address and eliminate some of this ambiguity so that the be reconciled, and while attempts have been made in the
reader may gain more insight into what systems thinking is past to reconcile the myriad models in the systems
and, how to apply its main ideas to a particular field or ‘‘universe’’, most of these efforts can best be described as
practical context. methodological pluralism (Gregory, 1996; Jackson, 1991,
2000; Midgley, 2000; White & Taket, 1997). Instead of a
pluralistic approach, in this paper we identify four
Corresponding author at: Cornell University, 87 Olde Towne Road, universal conceptual patterns that apply to all human
Ithaca, NY 14850, USA. Tel.: +1 607 592 4562; fax: +1 607 330 4815.
thinking and thus crosscut systems models and systems
E-mail addresses: derekc@thinkandthrive.com (D. Cabrera), thinking so that it be applied and its great practical promise
lac19@cornell.edu (L. Colosi), cel25@cornell.edu (C. Lobdell). can be realized.

0149-7189/$ - see front matter r 2008 Elsevier Ltd. All rights reserved.
doi:10.1016/j.evalprogplan.2007.12.001
 ARTICLE IN PRESS
300 D. Cabrera et al. / Evaluation and Program Planning 31 (2008) 299–310

2. Systems thinking in evaluation Table 1

Comparison of web of science search results for ‘‘Systems thinking’’ and
‘‘Critical thinking’’
The application of systems thinking concepts to evalua-
tion theory and practice explicates two separate, important Key word searched Systems thinking Critical thinking
ideas: evaluation systems and evaluation of systems. The
idea of systems as entities to be evaluated is nothing new in Number of hits 635 1659
Number with key word in title 270 830
the evaluation field, nor is the idea of designing and
Types of materials 499 articles 1324 articles
implementing an evaluation system. Many concepts found 55 book reviews 111 book reviews
in the systems thinking literature have already been 46 editorials 84 editorials
presented in the evaluation literature, for example, paying 35 ‘‘other’’ 133 ‘‘other’’
attention to multiple perspectives of different stakeholders Date range of materials found 1969–2007 1949–2006
Diversity of authors 44 countries 49 countries
and evaluating a system from multiple levels of scale. Times cited 2376 5979
Attempts to wed the two fields have been made since at Average number of times cited 3.74 3.60
least the late 1980s. Ulrich (1988) applied critical systems
heuristics, a systems methodology, to policy analysis and
evaluation. Gregory and Jackson (1992a, 1992b) applied
systems methods to four broad classes of evaluation but each is drawn to systems thinking because they perceive
methodologies in an attempt to better gauge when different the need to change how they, or others think.
evaluation techniques should be used. Midgley (1996) Changing the way we think does not automatically solve
surveyed the systems field and applied various systems the various problems, issues, or crises we face. However, it
methodologies to evaluation, and in 1998, Eoyang and does reframe how we think about what we view as a
Berkas wrote a paper, ‘‘Evaluation in a Complex Adaptive problem in the first place, and what solutions might look
System’’ that was included in a larger volume on like. Even after a person’s, group’s or organization’s
organizational complexity (Eoyang & Berkas, 1998). In thinking is changed, much hard work remains to solve
November 2003, the EVAL-SYS (Systems in Evaluation) their problems. Systems thinking alone will not solve these
listserv was formed, and, as of this writing, has 288 problems. Whether the problems are local (e.g., organiza-
members (The Evaluation Center, 2006). Shortly there- tional management, life management, parenting) or global
after, a ‘‘Systems in Evaluation’’ topical interest group (e.g., global warming, food security, violence, terrorism,
(TIG) was established by the American Evaluation public health, and even sleep deprivation), it is the vigorous
Association in February 2004. In addition, many of the problem-solving efforts in each of these areas, informed by
sessions sponsored by this TIG at the 2006 AEA a systems thinking perspective, that will uncover a viable
conference in Portland were standing-room-only. Finally, solution to the issue, problem, or crisis at hand.
the book Systems concepts in evaluation: An expert The reasons for the popularity and promise of systems
anthology was released in November 2006 and provides thinking are extensive. Examples of its popularity show
an overview of the application of various systems both that systems thinking as a discipline holds great
approaches to evaluation (Williams & Imam, 2006). promise and that as such, there is an increasing need for a
Movements of thought in systems thinking have also greater understanding of ‘‘systems thinking’’ as a con-
mirrored similar movements in evaluation. For example, struct. In December 2006, a search1 for the term ‘‘systems
‘‘boundary critique’’, ‘‘critical systems thinking’’, and thinking’’ on the Web of Science (ISI Web of Knowledge)
‘‘critical systems heuristics’’ (Ulrich, 2005a, 2005b), read database shows the extent of interest in systems thinking.
like evaluation methodologies and may be very applicable Table 1 shows the breakdown of the results of this search
to evaluation contexts. At this, the evaluator should take and contrasts it with an identical search for the term
heart. Systems thinking is not necessarily a matter of ‘‘critical thinking’’.
drawing an entirely new skill-set out of the intellectual In a similar analysis of scholarly publications, Cabrera
ether; rather, it is a unique perspective that transforms the (2006) found that in contrast to critical thinking, systems
approach taken to evaluate any program, policy, or thinking is interdisciplinary and may act as a bridge
initiative. between the physical, natural, and social sciences.
Whereas 88% of the papers in which the term critical
thinking appeared were in the social sciences, arts, and
3. Popularity and promise of systems thinking humanities literature, systems thinking appeared only 48%
in the literature from those fields, with the remainder
Many types of people are drawn to systems thinking, dispersed across the disciplinary spectrum from business,
including practitioners in evaluation, public health, educa- administration, finance, and economics, to engineering,
tion, and business who attempt to implement systems computer science, and mathematics, to physics, astronomy,
thinking in their organizations, and scholars and research- and planetary science. This interdisciplinary dispersal is
ers who study systems thinking. Each of these people faces
1
different problems and is concerned about different issues, Sources from 1900 to present, English language sources only.
 ARTICLE IN PRESS
D. Cabrera et al. / Evaluation and Program Planning 31 (2008) 299–310 301

increased when systems thinking is combined with its near 4. Thinking about systems
cousin ‘‘systems science’’. Systems thinking may also act as
a bridge between the academic, professional and lay Commonly understood meanings of ‘‘system’’ generally
communities, providing feedback between ‘‘what we know refer to a ‘‘complex whole of related parts’’—whether
about systems’’ (e.g., systems sciences) and ‘‘the conceptual it is biological (e.g. an ecosystem), structural (e.g. a railway
patterns of how we think systemically’’ (e.g., systems system), organized ideas (e.g. the democratic system),
thinking) (Cabrera, 2006).2 or any other assemblage of components comprising a
In spite of its popularity, there is great ambiguity as to whole. As such, when one sees a system, one usually sees
what constitutes systems thinking. For example, systems the whole first, and then its elemental parts (Fuenmayor,
thinking is often thought of as synonymous with systems 1991); that is, our view of the system is content specific. In
sciences, yet there are clear indications that they are not the its broadest sense, everything is a system, and what makes
same in practice. Both public and private organizations something a system is dependent on how each person
seek employees in leadership positions that have some thinks about the system. Thinking about systems is an ad
expertise in systems thinking. Examples include frequent hoc, primarily informal process that each of us does on a
job postings for positions as diverse as the US Army War daily basis.
College Professor of Leadership Transformation (Visiting In contrast, systems thinking is a more formal, abstract,
Professor of Leadership Transformation, 2006) to the and structured cognitive endeavor. While not all systems
President and CEO of the $90 million Casey Family are complex, all thinking is complex, and as such, the
Foundation (President and CEO, 2006). It seems clear that process of thinking in a systemic way is complex. Systems
these job descriptions are not seeking an individual with thinking is also based on contextual patterns of organiza-
expertise in the systems sciences (i.e., nonlinear dynamics, tion rather than specific content. For example, systems
complexity and chaos), but those individuals who possess a thinking balances the focus between the whole and its
particular ability to think in systemic ways. It begs the parts, and takes multiple perspectives into account. Nobel
question: if systems thinking is not the same as systems laureate Richard Feynman (2006) provides a famous
sciences, then what is it? What are the patterns of thought example of the kind of contextual patterns to which we
that are so desired at the Casey Family Foundation and the refer. It makes no difference that Feynman refers to specific
US Army War College, among others? content domains (i.e., chemistry, climatology, physics,
As one ventures into the tangled overgrowth of the cognition, etc.). What makes this famous quote an example
systems thinking literature, it is helpful to remember that of systems thinking is the way he transforms contextual
systems thinking has become increasingly popular because patterns: he transgresses parts and wholes, takes new
people believe it provides a new way to think about, or perspectives, forms new relationships, and makes new
conceptualize the world around us, whether our issues rest distinctions:
within a local or global context. Interestingly, because the
construct of systems thinking is unclear, people who view A poet once said, ‘The whole universe is in a glass of
systems thinking as a kind of solution see its potential even wine.’ We will probably never know in what sense he
while they do not yet entirely understand what it is. We said that, for poets do not write to be understood. But it
suggest that this is true for many evaluators and their is true that if we look at a glass of wine closely enough
clients, funding agencies, program planners, field staff, and we see the entire universe. There are the things of
other stakeholders involved in the evaluation process. physics: the twisting liquid which evaporates depending
There are many ways to think about systems thinking. on the wind and weather, the reflections in the glass, and
Some scholars and evaluation practitioners view it as a our imagination adds the atoms. The glass is a
specific methodology, such as system dynamics, while distillation of the earth’s rocks, and in its composition
others believe it is a ‘‘plurality of methods’’ (Williams & we see the secrets of the universe’s age, and the evolution
Imam, 2006). Others see systems thinking as systems of the stars. What strange arrays of chemicals are in the
science, while others see it as a general systems theory. Still wine? How did they come to be? There are the ferments,
others see systems thinking as a social movement. We the enzymes, the substrates, and the products. There in
propose that systems thinking is conceptual, because wine is found the great generalization: all life is
changing the way we think involves changing the way we fermentation. Nobody can discover the chemistry of
conceptualize. That is, while systems thinking is informed wine without discovering the cause of much disease.
by systems ideas, systems methods, systems theories, the How vivid is the claret, pressing its existence into the
systems sciences, and the systems movement, it is, in the consciousness that watches it! If in our small minds, for
end, different from each of these. some convenience, we divide this glass of wine, this
universe, into parts—physics, biology, geology, astron-
2 omy, psychology, and so on—remember that nature
The distinction between systems science and systems thinking was first
made by Checkland (1981) in his claim that systems thinking is thinking in does not know it! So let us put it all back together, not
terms of systems rather than being about actual systems; this distinction forgetting ultimately what it is for. Let us give one more
remains controversial in the systems science community today. final pleasure: drink it and forget it all!
 ARTICLE IN PRESS
302 D. Cabrera et al. / Evaluation and Program Planning 31 (2008) 299–310

In many ways, viewing an evaluand from a systems To put some workable limits on this mass of systems
thinking perspective would likely reveal the same kind of theories, we have chosen to define the systems thinking
elements Feynman sees in a glass of wine. For example, ‘‘universe’’ as all of the concepts contained in three broad
imagine an educational outreach curriculum designed to and inclusive sources: the International Encyclopedia of
increase school age children’s interest in science and Systems and Cybernetics by Charles Francois; Some
ultimately their propensity to choose a career in the Streams of Systemic Thought, a visual map of systems
sciences. As evaluators, we typically begin our work with thinking compiled by Eric Schwarz and modified by the
an examination of the content the program hopes to deliver International Institute for General Systems Studies; and a
to its participants, the outcomes desired, and a measurable four-volume set of the influential writings by systems
way to assess progress towards those outcomes. One could thinkers, compiled by Gerald Midgley (Francois, 2004;
argue that a more systemic approach (like Feynman’s General Systems Studies, Schwarz, & Durant, 2001;
approach to wine) to evaluating any program would Midgley, 2003).
include: defining what the program is and is not; Franc- ois’ encyclopedia is a two-volume set containing
identifying the components (parts) of the program; and approximately 3800 entries, drawn from approximately
recognition of the relationships among the parts and 1200 cited works.
between each part and the program as a whole. Note that Schwarz, visual map contains about 1000 nodes, each
each component of the program affects the delivery of representing a different idea, theory, or scholar, connected
content and achievement of outcomes. Further, many to the other ideas through a network of colors and
evaluators who advocate approaches that include multiple connecting lines. Each node contains the name of the idea,
stakeholders in the evaluation process (e.g., participatory and most contain the name of one or two key theorists, for
action research) do so because they recognize both the example ‘‘Systemic Selfness’’, by Paul Ryan. The colors
importance of taking multiple perspectives to better inform represent 12 broad groupings of systems concepts: general
the evaluation design and to ensure that an evalautor has a systems, cybernetics, physical sciences, mathematics, com-
comprehensive understanding of the program relative to all puters & informatics, biology & medicine, symbolic
the people who comprise part of the system in which the systems, social systems, ecology, philosophy, systems
program lives. analysis, and engineering.
Ultimately, we would argue that much like Feynman’s In contrast, Midgley’s four-volume set contains 76 papers
glass of wine, any evaluand can and should be viewed which he selected from a list of over 700 papers suggested
in the same way that transforms contextual patterns: as by a panel of experts from across the systems movement.
parts, wholes, and the relationships among them; as well as The volumes in this set are arranged thematically.
the relationships between the program and the larger, These three sources are not infinite, but they represent
external forces with which it rests; distinctions must be three attempts by respected systems theorists and histor-
made to set boundaries on the program’s scope and thus, ians to exhaustively describe the systems thinking universe.
establish criteria as to what can be measured to make There is a large degree of overlap between the three, which,
assessments; and finally, the ability to take varied by the nature of their different formats, necessarily include
perspectives enables evaluators to better understand the or exclude varying degrees of detail.
richness of both a program’s content and the system of
which it is a part. 6. Patterns not taxonomies

5. A bounded universe By defining the ‘‘systems universe’’ one can then begin to
think about what features are essential for membership and
Systems thinking is often considered an unwieldy therefore arrive at a less ambiguous description of systems
agglomeration of ideas from numerous intellectual tradi- thinking. Though Checkland (1981) and Senge (1990),
tions. The precise beginning of the field cannot be amongst others, have proposed influential systems thinking
pinpointed, as the beginning is a matter of perspective. approaches that are more than taxonomies of methods,
To many, the roots of systems thinking reach back to many scholars take a pluralistic approach and offer
ancient Western and Eastern philosophers (and -phies) taxonomic lists of examples of systems methodologies.
including Aristotle and Lao Tsu. To many others, the field We propose that the question ‘‘what is systems thinking?’’
and study of systems began in the early 20th century with cannot be answered by a litany of examples of systems
either Alexander Bogdanov or Ludwig von Bertalanffy thoughts, methods, methodologies, approaches, theories,
(Midgley, 2000, 2006). Debora Hammond has done an ideas, etc. Such a response is analogous to answering the
excellent job of tracing the 20th century history of these biologist’s question ‘‘what is life?’’ with a long list of
theories, and Gerald Midgley has divided them into three kingdoms, phyla, classes, orders, families, genus and
broad ‘‘waves’’ of systems thought (which, he and others species. Taxonomy of the living does not provide an
point out, correspond to movements or waves of evalua- adequate theory of life. Likewise, taxonomy of systems
tion theory) (Bawden, 2006; Hammond, 2003; Imam, ideas, even a pluralistic one, does not provide an adequate
LaGoy, & Williams, 2006; Midgley, 2000, 2006). theory for systems thinking. In our attempt to move away
 ARTICLE IN PRESS
D. Cabrera et al. / Evaluation and Program Planning 31 (2008) 299–310 303

from a taxonomic approach to defining systems thinking, This article draws on an alternative concept theory
we define the boundaries of the systems universe using the comprised of four component rules or patterns: Distinc-
work of Midgley, Franc- ois, and Schwarz. In the end, we tions, Systems, Relationships, and Perspectives (DSRP)
believe that an adequate description of systems thinking (Cabrera, 2006). DSRP provides the mechanism for a view
will be a fundamental conceptual pattern, not a pluralistic of concepts as dynamic, patterned, evolving, adaptive, and
taxonomy. Recognizing that systems thinking is: (a) complex. From this complex view, even a single concept
patterned and (b) conceptual, is essential to understanding can be thought of as a robust, complex system. Complex
systems thinking, especially in light of the considerable adaptive systems (CAS) are systems in which the individual
diversity of propositions about it in the literature. behavior of agents following simple local rules, leads to
If understanding the fundamental patterns that connect complex and emergent properties. Nobel laureate Murray
the many instantiations of systems thinking in the systems Gell–Mann (1995/1996) describes the relationship between
universe is the central process to describing what systems simple rules and complexity:
thinking is, then it is equally informative to give some
What is most exciting about our work is that it
thought to the patterns that do not connect. That is
illuminates the chain of connections between, on the
especially true for those claims that are popular in the
one hand, the simple underlying laws that govern the
systems thinking literature, but can clearly be shown not to
behavior of all matter in the universe and, on the other
be essential to every instantiation in the Midgley–Franc-
hand, the complex fabric that we see around us,
ois–Schwarz systems universe. We have already mentioned
exhibiting diversity, individuality, and evolution. The
a few of these patterns that do not connect: not all
interplay between simplicity and complexity is the heart
instantiations are methodological, systems science, etc.
of our subject. It is interesting to note, therefore, that
Cabrera (2006) writes at length about these patterns that
the two words are related. The Indo-European root
do not connect and includes some of the most common
*plek—gives rise to the Latin verb plicare, to fold, which
violators such as: systems thinking is the same as system
yields simplex, literally once folded, from which our
dynamics; systems thinking is the same as any proprietary,
English word ‘‘simple’’ derives. But *plek-likewise gives
insular field; systems thinking is holistic; and systems
the Latin past participle plexus, braided or entwined,
thinking is biological or ecological thinking. The reasons
from which is derived complexus, literally braided
these claims do not apply across the Midgley–Franc-
together, responsible for the English word ‘‘complex.’’
ois–Schwarz universe are varied and deeper than can be
The Greek equivalent to plexus is plektoB (plektos),
covered herein. Suffice to say however, that understanding
yielding the mathematical term ‘‘symplectic,’’ which also
why these claims (which are made so often in the systems
has the literal meaning braided together, but comes to
thinking literature) are not adequate descriptions of
English from Greek rather than Latin.
systems thinking is as revealing as understanding the
patterns that connect. Complex adaptive conceptual systems (CACS) is a term
Critical review of the theoretical and conceptual ideas invented by the authors to describe a new approach to
underlying the systems thinking construct highlights concepts. CACS explore the pattern of relations between
several ambiguities that must be better understood and concepts and their environment.
eventually resolved in order to properly implement The system of any individual concept, or that concept’s
systems thinking in practice. As practitioners are drawn ‘‘ecology’’, is made up of content and context, where
to the hope and promise of systems thinking, their first content is defined as the set of symbolic or informational
objective is to identify it—that is, to understand what variables in a conceptual space. Alfred Korzybski (1933),
makes systems thinking different from other forms of who developed the theory of general semantics, explained
thinking and to assess where the boundaries of the that the ‘‘map is not the territory’’. A concept is not merely
construct lie. its content (i.e., symbol-labels such as ‘‘dog’’ or ‘‘terrorist’’
or the image-symbol ‘‘’’), but is a function of the context
7. Systems thinking is conceptual it is in. Any given concept is a function of its inter-
relationships and organization with other concepts in the
Concept theorists in the cognitive sciences and philoso- conceptual space.
phy have proposed several theories about the nature of Context is a set of processing rules for content; the
concepts including: classical, prototype, theory–theory, resulting pattern of interaction yields concepts. This is
neo-classical, and conceptual atomism (Laurence & Mar- evident in the underlying contextual patterns in Richard
golis, 1999). Each of these competing theories is weakened Feynman’s thinking above; the contextual patterns, not the
in some way or another by problems3 such as: composi- specific content, are what we recognize as being uniquely
tionality, reference determination, categorization and systemic. This treatment is similar to Guilford’s original
stability. framework for divergent thinking, a key concept in
creativity research. Baer and Kaufman (2006) explain that
3
Laurence and Margolis (1999) provide a thorough review in their Guilford’s divergent thinking was an ‘‘attempt to organize
edited volume covering such theories and problems in greater depth. all of human cognition along three dimensions’’. Guilford’s
 ARTICLE IN PRESS
304 D. Cabrera et al. / Evaluation and Program Planning 31 (2008) 299–310

three dimensions include thought processes, content, and This interplay of ‘‘A’’ and ‘‘not A’’ is the essence of
the products of the interactions between process and distinction making: in order to make a distinction, one must
content. A whole mess of these conceptual patterns is establish an identity and exclude the other. Previous work
referred to as a ‘‘CACS’’—a pattern of content (symbolic in the systems literature reinforces the importance of
variables) and context (processing rules). As a formal set of drawing distinctions. For example, Von Foerster (1984)
processing rules, DSRP offers a mechanism for the pattern offered the idea that a concept has meaning only in its
of interactions among content and context that result in relationships with other concepts, and Bateson (1970)
concepts. It is important to note that while not all systems emphasized the significance of ‘‘difference’’, which is
are complex, systems thinking, because it is based in directly related to ‘‘distinction’’. In addition, Fuenmayor
thinking, is both complex and conceptual. (1991) recognized the importance of distinction in its
relationship with an opposing concept. Finally, Midgley
(2000) and Mingers (2006) refer to Spencer Brown’s (1969)
8. Four fundamental patterns that connect the systems work, indicating that distinction is more than the concept
universe of a number in mathematics.
All distinction making involves a boundary that
What follows is a concise explanation of the four rules of differentiates between what/who is in and what is outside
conceptualization: Distinction, System, Relationship, and the set boundary, between internalities and externalities. As
Perspective. Each of the four rules contains an interaction an example of the universality of distinction making,
between two elements as shown in Table 2. It is shown that consider one of the most common distinctions we make:
the existence and nature of concepts necessitates these the act of giving something a name. When we describe
dynamical rules, and that these rules are also sufficient to something by name, we are creating a boundary between
describe conceptual dynamics. It should be noted that that named thing and everything that it is not, thereby
theoretical, empirical, and practical examples exist for each highlighting or valuing certain patterns over others. So, the
of the individual patterns of D, S, R, and P and that this existence of concept A necessitates the existence of some
work is often transdisciplinary (occurring across different other concept, which will be called B.
fields). The reader may refer to the inventory of such works Consequently, A also necessitates the distinction be-
relating to each pattern in Appendix A as references, but tween A and B. The interrelation of concepts may also be
future work should include evaluative and integrative thought of in terms of a general notion of affect and effect,
reviews of this literature. where ‘‘affect’’ refers to the action taken by an agent and
In cognitive systems such as the human mind, ideas are ‘‘effect’’ refers to the result of that action on or to another
constantly evolving. Concepts are not static; they simulta- entity. For instance, in the case of distinctions, A affects B
neously adapt in response to other concepts, link together to be distinct from A, and B affects A to be distinct from B,
with them, conflict with them, or coexist. How might this etc. Thus, a distinction is comprised of the two concepts in
occur? As is often the case, the essence of the objects in question and four relations or two interrelations: the affect
question (concepts) determines the rules by which they of A’s identity, the effect of A’s identity on B (i.e., if A is an
behave. Consider a simple conceptual system consisting of ‘‘identity’’, B is an ‘‘other’’), the affect of B’s identity and
a concept A. Concepts exist only in context with other the effect of B’s identity on A. This does not imply that A
concepts. For instance, my concept of DOG exists in the affects B in the sense that A ‘‘causes’’ B to exist or vice
context of ANIMAL and FURRY and THING, etc. In versa, but that A affects an A-like-effect on B and vice
general, any concept A has identity only in contrast to versa. Think of this interaction as the effect your boss
some other concept from which it can be distinguished (for might have on you in a meeting. Your boss (or wife,
instance, there must at least be a concept of ‘‘not A’’ or siblings, colleagues) does not cause your identity, but can
‘‘other than A’’). shape it in a particular context. Just as our identity and
behavior is often a function of the people and context in
which we are situated,4 the same is true for concepts.
Table 2 If there is a distinction between A and B, there must be
DSRP rule-set some concept of relationship between them, namely at least
Concepts that relation of being distinct from one another. The
(content+context) relation of being distinct is dependent on the more general
relationship rule. That is, relations are comprised of two
Content Context
relations and four interrelations: the affect of relation from
(8 informational or (processing rules/patterns)
symbolic variables) A to B and from B to A and the effect of relation on B from
(D)(S)(R)(P) ) {DSRP} A and on A from B. Making relationships between
Distinction (D)3{identity (i)3other (o)} otherwise different concepts increases connectivity and
System (S)3{part (p) 3 whole (w)}
Relationship (R)3{cause (c)3effect (e)} 4
Perspective (P)3{subject (s)3object (ö)} (Davis-Blake & Pfeffer, 1989; Granovetter, 1985; Ridgeway & Correll,
2004; Smith-Lovin & McPherson, 1992; Tsui & Oreilly, 1989).
 ARTICLE IN PRESS
D. Cabrera et al. / Evaluation and Program Planning 31 (2008) 299–310 305

expands the within-group distinction; realizing the degree observer from A, the effect of object or observed on B, etc.
to which we are interconnected makes the lines between This can be most easily demonstrated by bringing a third
in/out group increasingly fuzzy and eventually redrawn. concept C into the mix; the BC system can be viewed from
Relationship-making forces our conceptual systems to A’s perspective as A(BC), or alternatively AC can be
expand and become more interconnected and fuzzier, but viewed from B as B(AC), etc.
over time as these relationships mutually reinforce each This conceptual perspective taking—attributing a per-
other, concepts can also crystallize or become more spective to a concept rather than an individual—is an
concrete. essential aspect of human thought processes, creativity,
Any collection of related concepts can naturally be innovation and problem solving. It is the conceptual
viewed as a system, since the simplest definition of a system equivalent to attribution of mind theories in cognitive
is a whole made up of two or more related parts. So A psychology. Also, perspectives may be regarded as distinc-
necessitates a system which can be expressed as the tions between the viewer and the viewed, or as systems of
collection of concepts and the two, two-way relations viewpoint (frames of reference). One might take the
between them: the affect of system membership from A, the perspective of an individual or of a group of individuals
effect of system membership on B, the affect of member- or of a single concept. Of course, when one takes another’s
ship from B and the effect of membership on A. Note also perspective, one is not actually seeing the other’s perspec-
that in addition to parts A and B, the relationships between tive but instead is making a conceptual attribution of one’s
them are also considered ‘‘parts’’ of the system. Here, concept of the other (Gregory, 1992; Midgley, 2000).
membership can be entire or partial, in the sense that A The nature of any concept necessitates the existence of
may be contained in B, B may be contained in A, A and B distinctions, relationships, systems and perspective taking.
may be effectively disjoint, or sub-concepts of A may be Each of these four rules is a special kind of relation
contained in B and vice versa (partial membership). To between two elements: identity–other for distinctions,
visualize this, think of a traditional Venn diagram of affect–effect for relationships, part–whole for systems,
overlapping circles. If one circle represented A and the and subject–object for perspectives.
other B, the places where they crossed would be where A is Furthermore, each of these rules and elements is itself a
contained in B, or vice versa. Of course, at any given time, concept, to which the DSRP rules apply. For instance, a
concept A fully contains A, but the constitution of A will relation R may be viewed as a concept, which is
almost certainly change over time given that systems are distinguished from another concept (such as A and B, or
constantly in flux. We may take a ‘‘snapshot’’ of a system some other relation R0 ). A relation may also be viewed as a
at an instant, but a moment later the system will likely be system or part of many systems, or one can view a
different. conceptual system from the perspective of R. The same
Furthermore, any concept naturally carries with it a analysis can be applied to a system: a system can be
perspective or frame of reference, for instance A from the thought of as, for instance, a relation between other
perspective of B, or vice versa. This conceptual perspective systems (for instance the system of ‘‘science education’’
taking is akin to viewing one concept from the point of might be regarded as a relation between the systems of
view of another, and therefore necessitates a subjective ‘‘curricula’’ and ‘‘educational outreach’’).
viewer (subject) and an objective view (object)—a sub-
ject–object relationship. Each concept has a unique
identity, but can also take a point of view on its 9. An example of DSRP in practice
environment. This point of view is attributional and it
always has a human ‘‘root’’ perspective. That is, any one A practical description may offer insight into the utility
concept (subject) cannot literally ‘‘see’’ another’s point of of DSRP generally, and specifically to the field of
view, but instead interprets and attributes a particular evaluation. The Santa Fe Institute’s Complex Systems
perspective to the other (object). Therefore, reorienting a Summer School (CSSS) provides a framework for scientists
system of concepts by deciding the focal point from which to learn from each other, benefit from methods and
attribution occurs is a central function of all conceptual techniques pioneered in diverse fields of study of complex-
systems. By attributing a conceptual state to a conceptual ity at SFI, and exposes the next generation of scientists to
point in the system, a view of the other objects in the interdisciplinary approaches that might enhance their
system can be established (e.g., a point of view). future success as scientists. When this program sought
This ‘‘perspective taking’’ or ‘‘conceptual attribution’’ out evaluation consultation, the authors applied DSRP
can have a catalytic effect on the conceptual system as a elements to what would be typically thought of as
whole, causing a cascade of interconnections and reor- traditional evaluation practice. It is important to note that
ientations. Perspective has the potential to instantly trans- an evaluation approach that incorporates our proposed
form whole systems, rearrange distinctions, and cause notion of ‘‘systems thinking’’ (informed by DSRP) does not
relationships to appear or disappear. Perspective can require a new set of evaluation tools for an evaluator, but
similarly be characterized by the relevant concepts and rather a shift in their thinking to re-frame components
the four causal interrelations: the affect of subject or essential to any evaluation.
 ARTICLE IN PRESS
306 D. Cabrera et al. / Evaluation and Program Planning 31 (2008) 299–310

The ideas of DSRP are all very useful to any evaluation. anthropomorphize these perspectives. That is, one can view
Many evaluators begin the dialogue with a client by setting the system from multiple perspectives to see or sense things
boundaries on the program, policy, or initiative they are that a human cannot. At each step along the way, we make
working with by determining what the program is and is choices about what to recognize, about what to include and
not. In other words, in order to evaluate any program, we exclude and from which perspective to view a given system.
must know what it actually consists of. This often includes There are various distinctions, inter-relationships, organi-
not only the ‘‘who, what, where, and when’’ of a program, zations of parts and wholes, and perspectives, and some of
but more importantly, the larger context in which an these are visible to the naked eye and some invisible. But
evaluand is situated (otherwise known as ‘‘the bigger there are many more that are invisible to the ‘‘mind’s eye’’,
picture’’). While this may sound obvious to most of us, it is limited by one’s knowledge of the topic, program, or area
clear that setting boundaries on the program itself is a of study itself. Or, humans may purposefully limit
much-needed step to designing an effective evaluation. It is themselves to avoid intellectual gridlock and as a matter
also often that case that the staff or managers of a given of pure functionality. It is not practical nor is it feasible to
program lack a full understanding of their program, and at take every thing into account. This is true for most
times, either overstate or underestimate its scope. This endeavors and certainly holds true for evaluators who are
common problem can be resolved by a dialogue guided by familiar with the many tradeoffs made (to either fit an
the need to draw distinctions, and thus, determine what a evaluation budget, or satisfy a funder, etc.) in the course of
program is and is not. an evaluation. These boundaries are drawn constantly out
After the program is defined, it is important to look at of necessity, and are done so many more times than are
the program in a larger context—or more specifically—the conscious to us.
system of which it is a part. In our work with SFI, it was
evident that the CSSS program was both a self-contained 10. Conclusion
program and a part of a larger whole—the Santa Fe
Institute. The system itself is also a distinction that has an All of the rules of DSRP are interdependent and
identity and interacts with things other than it. The CSSS simultaneously implemented by each concept. At a
program relies on external phenomena to function (as does micro-level it is important to note that an instantiation
SFI). First, the program must have institutional support of: D requires instantiations of SRP; S requires instantia-
from the Santa Fe Institute. Second, it relies on faculty tions of DRP; R requires instantiations of DSP; and P
involvement to teach for the program. Third, it must have requires instantiations of DSR. So, it can be said that each
a reasonably predictable audience of students to remain rule is dependent upon the other rules, that: D is dependent
stable and thus, be delivered annually. So, while an on SRP; S is dependent on DRP; R is dependent on DRP;
evaluator may typically zero in on the specific program and P is dependent on DSR. These micro-interactions
of interest, a more systemic approach to evaluation occur on every concept at every step in time. At a macro-
explores the impact of system membership on the scale, DSRP operates on complexes of content (A, B, AB,
program’s specific content, organizational contribution, etc.). Concepts (content and context) exist in a space of
and impact on its target population. concepts and interact with each other. Each concept is
Another common component of evaluation is the comprised of a system of sub-concepts, all of which are
distillation of program activities and outcomes and the implementing DSRP rules. Concepts interact with each
relationship between them. Some evaluators utilize tools other via the DSRP rules, i.e. forming distinctions,
such as logic models, or ‘‘causal pathway models’’ to do relations, etc., as their sub-concepts interact. The sub-
this. These models are useful when focusing on program concepts also have sub-concepts, which overlap with other
content only, but programs do not exist in vacuums. As a sub-concepts, all of which are simultaneously implement-
result, we believe that an evaluation is strengthened by not ing DSRP. At each step and at each point in the concept
only examining the relationships between activities and ecology, DSRP operates simultaneously. The number of
outcomes, but the relationships (affect and effect) between such associations (sub-concepts and DSRP implementa-
and among all of the components of the program and its tions) is so large that it can be taken to be effectively
larger context or system. infinite, yielding an essentially scale-free DSRP network
It is important to recognize that drawing distinctions (meaning that DSRP is a sort of fractal algorithm).
involves a perspective, and each distinction can also be It is important to note that the DSRP rules are used to
attributed a unique perspective. Not all perspectives are elucidate patterns that underlie all thoughts; in essence, to
from an observer outside of a system looking in. In fact, identify deeper levels of understanding by recognizing
many perspectives involve sentient beings taking attribu- patterns in what one already knows or by ‘‘blindly’’
tional perspectives of non-sentient concepts. So, one might (algorithmically) creating new knowledge by simple altera-
conceptualize the CSSS program from the point of view of tions of contextual pattern. We suggest that because
the Santa Fe Institute, the topic of inter-disciplinarity, the systems thinking is a pattern of thought, it can apply to
scientific community, ideas of complexity science, or any existing body of knowledge. This may appear to be an
students in the program. It is not always necessary to ambitious claim; however, we contend that systems
 ARTICLE IN PRESS
D. Cabrera et al. / Evaluation and Program Planning 31 (2008) 299–310 307

Fig. 1.

thinking is simply a way of reframing one’s thinking in a perspectives, this does not mean that explicit and formal
domain, accomplished by a reconstruction of systems practice in these thinking skills is not important. Indeed, it
thinking based on the elements of DSRP, allowing for a is precisely because we are using these schema implicitly
universal approach to manipulation of concepts relevant to that we must recognize their usage explicitly. For example,
all thinking in both professional practice and intellectual we will draw a distinction between what something is and is
disciplines. Note also that many systems methodologies not (i.e., terrorism), but if we are unaware that these
and methods have been developed over the years, and they boundaries are dynamic and related to the systems and
can be drawn upon in support of DSRP (Midgley, 2003). perspectives we recognize as important, then we will be
Systems thinking is not the same as a pluralistic unaware of our own biases. To make these patterns explicit
taxonomy of systems thoughts. It is the underlying is to know how one thinks and therefore how one might
conceptual pattern that connects all instantiations of alter this thinking to avoid bias, to be more compassionate,
systems thoughts. more creative, or to better understand the structure of
Systems thinking is not content specific and is therefore one’s own thoughts. This all bodes well for practitioners in
not disciplinary in scope. It is a pattern of thinking that evaluation who want to apply systems thinking to their
formally alters context and therefore transforms the daily work because systems thinking is easily learned,
meaning of any kind of content (i.e., subject matter). applicable to the existing knowledge base of evaluation
Systems thinking is not the same as systems science(s). and will lead to transformative results for any endeavor
Each of us already thinks about systems. To become a (Fig. 1).
systems thinker, one need not spend many years learning
new methods or scientific content knowledge such as Acknowledgements
complexity, chaos, or nonlinear dynamics. Instead, we
propose that systems thinking can be readily learned and This material is partially based upon work supported by
can be formally, explicitly, even algorithmically applied. the National Science Foundation under Grant no. EREC-
Not all systems are complex, but all systems thinking is 0535492, and by the NSF Integrative Graduate Education
complex because thinking is, by definition, a complex and Research Training (IGERT) program. Any opinions,
system. It follows then that the ‘‘emergent property’’ that findings, and conclusions or recommendations expressed in
we perceive as systems thinking is based on remarkably this material are those of the authors and do not
simple rules (i.e., DSRP). Therefore, systems thinking is necessarily reflect the views of the National Science
not something one does, but something one gets as a result Foundation.
of applying simple rules based on patterns of thinking.
To become a systems thinker, one need only to under- Appendix A. Additional references for DSRP rules
stand and apply these four conceptual patterns: draw
distinctions between an identity and a non-identity;
Distinction rule
recognize the bi-directional properties (affect and effect)
of relationships; organize parts and wholes into alternative Clark, T. (1994). National boundaries, border zones, and
nested systems; and take new perspectives by transforming marketing strategy: A conceptual framework and
one’s point-and-view. Although we are always making theoretical model of secondary boundary effects. Journal of
distinctions, interrelating, organizing systems, and taking Marketing, 58(3), 67–80.
 ARTICLE IN PRESS
308 D. Cabrera et al. / Evaluation and Program Planning 31 (2008) 299–310

Cook, T. D., & Campbell, D. T. (1979). Quasi- Spencer Brown, G. (1969). Laws of form. London: George
experimentation: Design & analysis issues for field settings. Allen & Unwin Ltd.
Boston: Houghton Mifflin Company. Tajfel, H., & Wilkes, A. L. (1963). Classification and
Coye, D. (1986). The Sneakers/tennis shoes boundary. quantitative judgment. British Journal of Psychology, 54,
American Speech, 61(4), 366–369. 101–114.
Davies, C. (1982). Sexual taboos and social boundaries. Ulrich, W. (1983). Critical heuristics of social planning:
American Journal of Sociology, 87(5), 1032–1063. A new approach to practical philosophy. Haupt, Berne.
Dorfman, D. D. (1967). Recognition of taboo words as a
function of a priori probability. Journal of Personality and
System rule
Social Psychology, 7(1), 1–10.
Durand, R., & Calori, R. (2006). Sameness, otherness? Ackoff, R. L. (1971). Towards a system of systems
Enriching organizational change theories with concepts. Management Science, 17(11), 661–671.
philosophical considerations on the same and the other. Anderson, J. R. (1991). The adaptive nature of human
Academy of Management Review, 31(1), 93–114. categorization. Psychological Review, 98(3), 409–429.
Edwards, A. W. F. (2004). Cogwheels of the mind: The story Bertalanffy, L. v. (1972). The history and status of general
of Venn diagrams. Baltimore, MD: The Johns Hopkins systems theory. In G. J. Klir, (Ed.), Trends in general
University Press. systems theory (pp. 407–426). New York: Wiley-
Franc- ois, C. (2004). Distinction; distinction (primary); Interscience.
distinctions (cinematics of). In C. Franc- ois (Ed.), Davidz, H. L., Nightingale, D. J., & Rhodes, D. H. (2004,
International encyclopedia of systems and cybernetics (2nd September, 2004). Enablers, barriers, and precursors to
ed.) (Vol. 1, pp. 176–177). München: K.G. Saur. systems thinking development: The urgent need for more
Gillette, J. M. (1925). Boundary lines of social phenomena. information. Paper presented at the international
The American Journal of Sociology, 30(5), 585–593. conference on systems engineering/INCOSE, Las Vegas,
Glanville, R. (1990). The self and the other: The purpose of NV.
distinction. In R. Trappl (Ed.), Cybernetics and systems ’90. Lewin, K., Adams, D. K., & Zener, K. E. (1935). A
Singapore: World Scientific. dynamic theory of personality (1st ed.). New York, London:
Grossberg, S. (1997). Cortical dynamics of three- McGraw-Hill Book Company, Inc.
dimensional figure-ground perception of two-dimensional Tversky, A., & Kahneman, D. (1981). The framing of
pictures. Psychological Review, 104(3), 618–658. decisions and the psychology of choice. Science, 211(4481),
Hardin, G. (1968). The tragedy of the commons. Science, 453–458.
162(3859), 1243–1248. Tversky, B., & Hemenway, K. (1984). Objects, parts, and
Herbst, D. P. (1995). What happens when we make a categories. Journal of Experimental Psychology-General,
distinction: An elementary introduction to a co-genetic 113(2), 169–197.
logic. In T. A. Kindermann, J. Valsiner (Eds.), Wertheimer, M. (1923). Laws of organization in perceptual
Development of person–content relations (pp. 67–79). forms (English translation, 1938). In W.D. Ellis (Ed.),
Hillsdale, NJ: Lawrence Erlbaum Associates. A source book of gestalt psychology. New York: Harcourt
Heylighen, F. (1989). Causality as distinction Brace.
conservation—A theory of predictability, reversibility, and
time order. Cybernetics and Systems, 20(5), 361–384.
Relationship rule
Langer, E. J., Bashner, R. S., & Chanowitz, B. (1985).
Decreasing prejudice by increasing discrimination. Journal Cook, T. D., & Campbell, D. T. (1979). Quasi-
of Personality and Social Psychology, 49(1), 113–120. experimentation: Design & analysis issues for field settings.
Leudar, I., Marsland, V., & Nekvapil, J. (2004). On Boston: Houghton Mifflin Company.
membership categorization: ‘us’, ‘them’ and ‘doing Gopnik, A., Glymour, C., Sobel, D., Schulz, L., Kushnir,
violence’ in political discourse. Discourse & Society, T., & Danks, D. (2004). A theory of causal learning in
15(2–3), 243–266. children: Causal maps and Bayes nets. Psychological
McWhorter, J. (2001). The power of Babel: A natural Review, 111(1), 1–31.
history of language. New York: Henry Holt. Grotzer, T. A. (2005). Transferring structural knowledge
Newman, R. S., & Jusczyk, P. W. (1996). The cocktail about the nature of causality to isomorphic and non-
party effect in infants. Perception & Psychophysics, 58(8), isomorphic topics. Paper presented at the conference name|.
1145–1156. Retrieved access date|. from URL|.
Perdue, C. W., Gurtman, M. B., Dovidio, J. F., & Tyler, R. Pearl, J. (2000). Causality: Models, reasoning, and
B. (1990). Us and them—Social categorization and the inference. Cambridge: Cambridge University Press.
process of Intergroup Bias. Journal of Personality and Piaget, J. (1974). Understanding causality (D. Miles & M.
Social Psychology, 59(3), 475–486. Miles, Trans.). New York: The Norton Library.
Rubin, E. (1921). Visuell wahrgenommene figuren. Schulz, L., & Gopnik, A. (2004). Causal learning across
Cogenhagen: Glydendalske. domains. Developmental Psychology, 40(2), 162–176.
 ARTICLE IN PRESS
D. Cabrera et al. / Evaluation and Program Planning 31 (2008) 299–310 309

Perspective rule Bawden, R. J. (2006). A systemic evaluation of an agricultural

development: A focus on the worldview challenge. In B. Williams, &
Batson, C. D., Early, S., & Salvarani, G. (1997). I. Imam (Eds.), Systems concepts in evaluation: An expert anthology
Perspective taking: Imagining how another feels versus (pp. 35–46). Point Reyes, CA: Edge Press.
imagining how you would feel. Personality and Social Cabrera, D. (2006). Systems thinking. Thesis, Cornell University, Ithaca, NY.
Psychology Bulletin, 23(7), 751–758. Checkland, P. (1981). Systems thinking, systems practice. Chichester:
Wiley.
Churchman, C. W. (1968). The systems approach. New Davis-Blake, A., & Pfeffer, J. (1989). Just a mirage: The search for
York: Dell. dispositional effects in organizational research. Academy of Manage-
Churchman, C. W. (1979). The systems approach and its ment Review, 14(3), 385–400.
enemies. New York: Basic Books. Eoyang, G. H., & Berkas, T. H. (1998). Evaluation in a complex adaptive
Davis, M. H., Conklin, L., Smith, A., & Luce, C. (1996). system. In M. Lissack, & H. Gunz (Eds.), Managing complexity in
organizations. Westport, CT: Quorum Books.
Effect of perspective taking on the cognitive representation Feynman, R. (2006). The Feynman lectures on physics [UNABRIDGED].
of persons: A merging of self and other. Journal of Audio CD. Basic Books.
Personality and Social Psychology, 70(4), 713–726. Francois, C. (2004). International encyclopedia of systems and cybernetics
Davis, M. H., Soderlund, T., Cole, J., Gadol, E., Kute, M., (2nd ed.). Mnchen: K.G. Saur.
Myers, M., et al. (2004). Cognitions associated with Fuenmayor, R. L. (1991). The roots of reductionism: A counter-
ontoepistemology for a systems approach. Systems Practice, 4(5),
attempts to empathize: How do we imagine the perspective 419–448.
of another? Personality and Social Psychology Bulletin, General Systems Studies, I. I. for, Schwarz, E., & Durant, W. (2001).
30(12), 1625–1635. Some streams of systemic thought. (This is a copy of the 2001 draft
Duncker, K. (1929). Uber induzierte bewegung. update.)
Psychologishe forschung 12: 180–259. Condensed Granovetter, M. (1985). Economic action and social structure: The
problem of embeddedness. American Journal of Sociology, 91,
translation published as induced motion. In W. D. Ellis 481–510.
(Ed.), A sourcebook of gestalt psychology (1938 ed., pp. Gregory, W. J. (1992). Critical systems thinking and pluralism: A new
161–172). New York: Harcourt Brace. constellation. Thesis, City University, London.
Galinsky, A. D., & Moskowitz, G. B. (2000). Perspective- Gregory, W. J. (1996). Discordant pluralism: A new strategy for critical
taking: Decreasing stereotype expression, stereotype systems thinking? Systems Practice, 9(6), 605–625.
Gregory, A. J., & Jackson, M. C. (1992a). Evaluating organizations—
accessibility, and in-group favoritism. Journal of A systems and contingency approach. Systems Practice, 5(1), 37–60.
Personality and Social Psychology, 78(4), 708–724. Gregory, A. J., & Jackson, M. C. (1992b). Evaluation methodologies—
Marvin, R. S., Greenberg, M. T., & Mossler, D. G. (1976). A system for use. Journal of the Operational Research Society, 43(1),
The early development of conceptual perspective taking: 19–28.
Distinguishing among multiple perspectives. Child Hammond, D. (2003). The science of synthesis: Exploring the social
implications of general systems theory. Boulder, CO: University Press
Development, 47(2), 511–514.
of Colorado.
Neale, M. A., & Bazerman, M. H. (1983). The role of Imam, I., LaGoy, A., & Williams, B. (2006). Introduction. In B. Williams,
perspective-taking ability in negotiating under different & I. Imam (Eds.), Systems concepts in evaluation: An expert anthology
forms of arbitration. Industrial and Labor Relations Review, (pp. 3–10). Point Reyes, CA: Edge Press.
36(3), 378–388. Jackson, M. C. (1991). Systems methodology for the management sciences.
New York: Plenum Press.
Parker, S. K., & Axtell, C. M. (2001). Seeing another
Jackson, M. C. (2000). Systems approaches to management. New York:
viewpoint: Antecedents and outcomes of employee Kluwer, Plenum Press.
perspective taking. Academy of Management Journal, 44(6), Korzybski, A. (1933). A non-Aristotelian system and its necessity for
1085–1100. rigour in mathematics and physics. Science and Sanity, 1, 747–761.
Piaget, J. (1974). Understanding causality (D. Miles & M. Laurence, S., & Margolis, E. (1999). Concepts and cognitive science. In E.
Margolis, & S. Laurence (Eds.), Concepts: Core readings. Cambridge,
Miles, Trans.). New York: The Norton Library.
MA: The MIT Press.
Premack, D., & Woodruff, G. (1978). Does the chimpanzee Midgley, G. (1996). Evaluation and change in service systems for people
have a theory of mind? The Behavioral and Brain Sciences, with disabilities: A critical systems perspective. Evaluation, 2(1), 67–84.
4, 515–526. Midgley, G. (2000). Systemic intervention: Philosophy, methodology, and
Schober, M. F. (1993). Spatial perspective-taking in practice. New York: Kluwer Academic Publishers.
Midgley, G. (Ed.). (2003). Systems thinking. London: Sage Publications.
conversation. Cognition, 47(1), 1–24.
Midgley, G. (2006). Systems thinking for evaluation. In G. Midgley, &
Whitehead, A. N. (1967). Adventures of ideas. New York: I. Imam (Eds.), Systems concepts in evaluation: An expert anthology
Free Press. (pp. 11–34). Point Reyes, CA: Edge Press.
Mingers, J. (2006). Realising systems thinking: Knowledge and action in
management science. New York: Springer.
References President and CEO (2006, March 13, 2006). The chronicle of higher
education, from /http://chronicle.com/jobs/id.php?id=0000449443-
Baer, J., & Kaufman, J. C. (2006). Creativity research in English-speaking 01&pg=s&cc=S.
countries. In J. C. Kaufman, & R. J. Sternberg (Eds.), The Ridgeway, C. L., & Correll, S. J. (2004). Unpacking the gender system—
international handbook of creativity (pp. 10–38). Cambridge, NY: A theoretical perspective on gender beliefs and social relations. Gender
Cambridge University Press. & Society, 18(4), 510–531.
Bateson, G. (1970). Form, substance, and difference. General Semantics Senge, P. (1990). The fifth discipline: The art and practice of the learning
Bulletin. Reprinted in Steps to an Ecology of Mind, Vol. 37. organization. London: Random House.
 ARTICLE IN PRESS
310 D. Cabrera et al. / Evaluation and Program Planning 31 (2008) 299–310

Smith-Lovin, L., & McPherson, J. M. (1992). You are who you know: A ThinkBlocks, a tactile educational tool that teaches systems thinking
network approach to gender. In P. England (Ed.), Theory on gender/ skills. Prior to his current appointments, he was a National Science
feminism on theory (pp. 223–251). New York: Aldine. Foundation Post Doctoral Fellow, Co-Investigator in the College of
Spencer Brown, G. (1969). Laws of form. New York: Julian Press. Human Ecology and a Lecturer in the Department of Education at
The Evaluation Center (2006). List archives and web access to e-mail lists Cornell University where he taught a graduate level course in systems
hosted by the evaluation center (No. December 7), Western Michigan thinking. He was a National Science Foundation IGERT Fellow in
University. Nonlinear Systems at Cornell and a recipient of the Association of
Tsui, A. S., & Oreilly, C. A. (1989). Beyond simple demographic effects— American Colleges and Universities’ K. Patricia Cross Future Leaders
The importance of relational demography in superior–subordinate Award. He is the author of two books, Remedial Genius and Journey to
dyads. Academy of Management Journal, 32(2), 402–423. Planet Knowledge, and numerous refereed journal articles and book
Ulrich, W. (1988). Churchman’s ‘‘process of unfolding’’—its significance chapters.
for policy analysis and evaluation. Systems Practice, 1(4), 415–428.
Ulrich, W. (2005a). A brief introduction to critical systems heuris-
tics(CSH). Retrieved March 21, 2006, from /www.geocities.com/ Laura Colosi is an extension associate and lecturer in the Cornell
csh_home/csh.htmlS. University, Department of Policy Analysis and Management. She is the
Ulrich, W. (2005b). A mini-primer of ‘‘boundary critique’’. CSH Critical COO of ThinkWorks, a company dedicated to teaching thinking skills to
Heuristics. Retrieved March 3, 2006, from /http://www.geocities.com/ children, adults, and corporations. Her research and extension interests
csh_home/boundary_critique.htmlS. are in the area of families and social policy, with a particular interest in
Visiting Professor of Leadership Transformation (February 16, 2006). The low-income families, as well as the evaluation of parenting and other
chronicle of higher education. Retrieved February 17, 2006, from programs at the local, state and national levels. In addition to her work at
/http://chronicle.com/jobs/id.php?id=0000448512-01&pg=eS. Cornell, Laura has conducted research at the National Academy of
Von Foerster, H. (1984). Observing systems (2nd ed.). California: Sciences’ Institute of Medicine, the US Department of Health and Human
Intersystems. Services, and the US Department of Justice in Washington, DC. She holds
White, L., & Taket, A. (1997). Critiquing multimethodology as both a Masters in Public Administration from Cornell’s Institute for
metamethodology: Working towards pragmatic pluralism. In J. Public Affairs and a Ph.D. from the College of Human Ecology at
Mingers, & A. Gill (Eds.), Multimethodology: The theory and practice Cornell. She is the author of Journey to Planet Knowledge, a children’s
of combining management science methodologies. Chichester: Wiley. book that teaches about the importance of learning, thinking, and
Williams, B., & Imam, I. (Eds.). (2006). Systems concepts in evaluation: An interdisciplinary science.
expert anthology. Point Reyes, CA: Edge Press.

Claire Lobdell is a research assistant in the Cornell University,

Derek Cabrera is a Visiting Fellow at Cornell University and a Research Department of Policy Analysis and Management, where she researches
Associate at the Santa Fe Institute for the study of complex systems. He is the integration of systems ideas with program evaluation practice. She is a
the founder and CEO of ThinkWorks, and holds a US patent for graduate of Cornell University’s College of Arts and Sciences.