Assumptions Underlying Evaluation

Models
ERNEST R. HOUSE1
University of Illinois
at Urbana-Champaign

The Major Models

One way of understanding evalua­ sifications of several writers, especially Health, Education, and Welfare since
tion is to compare the numerous evalua­ Stake (1976); also Popham (1975), and about 1965. (McLaughlin, 1975;
tion models with one another. There are Worthen and Sanders (1973). There House, 1977)
many possibilities for comparison but seems to be fairly good agreement on Behavioral Objectives.. The objec­
perhaps the most significant compari­ what are the major models. I believe tives of a program are spelled out in
sons are, those among the underlying that most of the other models are simi­ terms of specific student performances
theoretical assumptions' on which the lar to these, although certainly some that can be reduced to specific student
models are based. In this way, one approaches have been overlooked. behaviors. These behaviors are meas­
might see how logically similar the A very brief_sketch of each evalua­ ured by tests, either norm-referenced or
models are to one another and deter­ tion model is in order. criterion-referenced. Ralph Tyler_was
mine what logical possibilities do and Systems Analysis. In this approach the originator_of this approach.
do not exist. one assumes a few quantitative output Decision Making. The evaluation is
The basic theme is that all the evalua- measures, usually test scores, and tries structured by_the decjsions_to .be, made.
tion models are based on variations in to relate differences in programs to var­ The evaluator is to supply information
the assumptions of liberal ideology, or iations in test scores. The data are often on these particular decisions. Stuf-
if one prefers, the conceptions of liberal survey data and the outcome measures flebeam is the major figure here. The
democracy. The models differ from are related to the programs via correla­ methodology tends to be questionnaire
one another as the basic assumptions tional analyses. Recently, experimen­ and_interview_ surveys. The decision
vary. Assumptions Γtake here in the tal design has been more heavily em­ makers are administrators and mana­
commonsense meaning of "things ployed. One of the main antecedents for gers, although the category has
taken for granted" or "things taken to this approach was systems analysis as broadened somewhat recently.
be true." developed in the Department of De- Goal Free. Scriven has been primar­
Figure 1 presents a _taxonomy of fenśe under Secretary McNamara. It ily concerned with redųcing.the effects
some evaluation models. In selecting hãs¯ served as the major evaluation of bias in evaluation. This model re-
the models I have made use of the clas­ perspective in the Department of duceŝ“the bias of searching only for the

4 Downloaded from http://er.aera.net at Monash University on March 7, 2015 ER

program developers' prespecified in­ moves down the methodology column Subjectivist Ethics
tents by not informing thè evaluator of the more subjective and, less objective All the major evaluation models are
them. Hence the evaluator must.search the research methodology becomes. subjectivist in ethics, although there is
for all outcomes. The consumer js the The more one moves down the out­ no necessity for this. One type of ethi­
primary audience and Consumer's comes column; the less the overall con- cal subjectivism sees the end of ethical
Union procedures are analogs to this cern becomes social efficiency and the conduct as the realization of some type
approach. more it becomes personal uņdersţaņd- of subjective experience. This de­
Art Criticism. Evolving from the iņg. These are oversimplifications, scribes the top four models, which I
traditions of art and literary criticism is since the actual ordering is more com­ have labeled "utilitarian." Properly
the model of an educational critic, one plex. speaking, utilitarianism refers to the
who is attuned by experience and train- The majore_eļements in understanding idea_of maximizing. happiness in soci-
ing to judge the Important facets of the models are their ethics, their_emis ety. Any activity that maximizes hap­
educational programs. Eisner is the temology, and their political ramifica­ piness is the " r i g h t " thing to do
leading figure. tions,. The current models all derive (House, 1976). Usually surrogate
Accreditation. For more than 50 from the philosophy of liberalism, with measures, like gross national product in
years, schools have cooperatively. deviations from the mainstream being economics or mean test scores in educa­
united to evaluate each other. This is responsible for differences in ap­ tion, are used as the indicators for hap­
ordinarily done by a team of out side proaches. The ethics, epistemology, piness.
professionals visiting_oη-site. The local and politics are not entirely separable
people have previously collected in­ from one another. Besides the systems analysis, be­
formation and studied their program Liberalism itself grew out of an at­ havioral objective, and decision mak­
according to a set of external standards. tempt to rationalize and.justify a market ing models, Scriven's goal free model
The reviewers.commend or disapprove socieţy (MacPherson, 1966) which was is also included in this classification.
of the local programs. organized on the principle of freedom All these models try to arrive at a single
Adversary. Quasi-legal procedures of choice. Choice remains a key idea in. judgment of overall social utility. The
have been used by several people to the evaluation models, although whose simplest approach is the homogenous
present the pros and. cons of a program. choice, what choices, and the grounds scaling of the systems analysis ap­
The adversary system ensures ţhe_pre- upon which choices are made are mat­ proach which tries to reduce all vari­
sentation of both sides. These evalua­ ters of difference. Consumer's choice ables into a quantitative model like a
tions often take the form of trial by jury. is the ultimate idea but the concept of regression analysis. Only one or a few
Transaction. This approach concen­ the consumer differs. outcome variables are employed.
trates on the educational processes A second key idea, of liberalism is The most complex is that of
themselves: the classroom, the school, that of an individualist psychology. Scriven s in which many outcome vari­
the program. It uses various informal Each individual mind is presumed to ables are considered, including secon-
methods of investigation and has been exist prior to_society. The individual is • dary and tertiary effects. All these vari­
drawn increasingly to the case study as not conceived initially as a part of a ous scales must be weighted and com­
the major methodology. Stake is the greater collectivity, although he may bined somehow into an overall summa-
leading figure. submit to one later as in a social con- tive judgment. The analogy is to a
tract situation. Liberalism is pro­ Consumer's Union report in which
foundly methodologically individualist many different criteria are summed up
in its intellectual constructions. in an overall ranking of "better" and
Another key idea is the, empiricist "best." As in Consumer's Union, the
In the first column of the taxonomy orientation. Often liberalism is radi­ final scaling will be ordinal rather than
in Figure 1, the common names of the cally empiricist. For example, John interval. It is utilitarian in the sense that
models are listed. In the top row are the Stuart Mill, the apostle of liberalism, there is one final scaling of merit, lead­
critical dimensions of comparison: the believed that even mathematics was in­ ing to best consumer choice and even­
audiences to whom the evaluation is ductively based. All the evaluation tually to social utility.
addressed, what the model assumes models here have such an empiricist
consensus on, the methodology of data flavor. The other four major evaluation
collection, the ultimate outcome ex­ The evaluation models also assume a models ì have labelled intuition-
pected, and the typical question that the free marketplace of ideas in which con­ ist/pluralist. The ethical principles are
approach tries to address. At one level sumers will "buy" the best ideas. They not single in number nor explicitly de­
of abstraction, these are the assump­ assume that competition of ideas fined as in utilitarian ethics. There are
tions underlying the models. strengthens the truth. Ultimately they several principles derived from intui­
In the taxonomy the models are re­ assume that knowledge will make peo­ tion and experience, but no set rules for
lated to one another in a systematic ple happy or better in some way. So the weighting them. This captures another
way. Generally the more one progres­ evaluation models partake of the ideas meaning of ethical subjectivism — that
ses down thę column of major audi­ of a competitive, individualist, market the ultimate criteria of what is good and
ences, the more democratic_or less_ society. But the most fundamental idea right are individual feelings or ap­
elitist the audience becomes. The more is_freedom of choice, for without prehensions.
one moves down the consensus col­ choice, of what use is evaluation? Fig­ In the art criticism model, the critic
umn, the less consensus is assumed on ure 2 presents the evaluation models as has arrived at his invariably plural prin­
goals and other elements. The more one they are related to liberalism. ciples by long training and experience

Downloaded from http://er.aera.net at Monash University on March 7, 2015

March 1978 5
His taste has been refined so that he is on professional experts for judgments Stuart Mill, for whom the mind was a
an expert judge. Much of his expertise about the program being evaluated bundle of impressions. Psychological
lies in how well he can intuitively bal­ while the adversary and transaction laws of association were the foundation
ance the principles of judgment. In the models require participation by both of a science of society. Everything
accreditation model, the principles may the practitioners and the public. came from and had to be tested by ex­
be spelled out in books like Evaluative perience, to the extent that even deduc­
Criteria (1969). But their origin and The Liberal Objectivist tive logic, e.g., mathematics, had an
application remain very much a matter Epistemology inductive basis. Mill saw no fundamen­
of professional intuitive judgment In their orientation toward manage­ tal difference between logical truths
based on long experience. ment, the managerial models look for and factual truths. All knowledge, he
In the adversary model, as in a ways in which management can felt, had to be grounded in and tested by
quasi-trial, the jury is free to employ monitor and improve the programs. experience. Knowledge was the pooled
multiple criteria in an intuitive fashion. Accountability, efficiency, and quality sun of individual observations.
The adversaries are free to introduce control are major management con­ This extreme methodological indi­
anything in evidence that can be con­ cerns. Evaluation has something of a vidualism , .ran_ deep in Mill's
strued as relevant. Finally, in the trans­ watchdog function. The managerial philosophy, To him, society was a col­
action model, pluralistic criteria are in­ school also tends to be based on a lection of individual persons bound by
troduced by soliciting the judgments of common epistemology, which I will common interests. His radical empiri­
various people involved in the pro­ label ''objectivism.'' Evaluation in­ cism led him to formulate rules for the
gram. The data are weighted intuitively formation is considered to be "scienti­ conduct of inductive inquiry in A Sys-
by the evaluator and by the audience of fically objective." This objectivity is tem of Logic- Since the mind consisted
the evaluation. achieved by using "objective" instru­ of sensations, feelings, and their as­
Generally speaking, the utilitarian ments like tests or questionnaires. Pre­ sociations, the categories of the real
models tend to take explicitly defined sumably results produced with these in­ world seemed to Mill to be easily ac­
techniques as the basis of ethics and struments are reproducible. The data cessible. But could these categories be
knowledge, whereas the intuitionists/ . are analyzed by quantitative techniques relied upon? If they were ephemeral,
pluralists take intuitive tacit knowl­ which are also "objective" in the sense one could not know anything. To over­
edge derived from professional experi­ that they can be verified by logical in­ come this problem, Mill assumed the
ence and participation as the basis. spection regardless of who uses t! e world consisted of constant unifor­
Although all the major models are techniques. mities among classes of sensations.
based on subjectivist ethics, there is no In its extreme form, objectivism in Experimentation was essential for
reason why this must be so. It would be epistemology entirely excludes the the scientific testing of theories. Valid
possible to have an objectivist ethic nonquantitative. If it exists, it can be causal connections could be established
within the liberal philosophy. For ex­ measured, the thought goes. Reduction only through the meticulous sorting and
ample, in his Theory_of Justice, John into numbers sometimes becomes independent variation of antecedent
Rawls (1971) attempted to base his so­ equated with objectivity. The origins of conditions and consequences. These
cial ethics on two principles of justice. much of this extreme empiricism and were often concealed from causal ob­
These principles were not justified on objectivism can be found in the earlier servation. "Unwashed" appearances
the presumption that they maximized epistemology of the liberal philosophy. were deceptive. Only techniques of in­
some subjective feeling like happiness, The origins of liberalism as a ductive logic could clean them. It was
nor on the fact that people intuitively philosophy can be traced back, in part, here that techniques like the method of
held these principles. Rather, the prin­ to the first great British empiricist, agreement, method of difference, or
ciples were justified on the basis that Thomas Hobbes. An admirer of method of concomitant variation could
rational analysis, namely the theory_of Galileo's mechanical view of the uni­ be useful. The canons of induction
minimizing losses, would lead any verse, Hobbes developed a mechanistic were to provide a method for sifting
rational person to the same principles. philosophy of mind based on the idea evidence of correlations upon which
They were objectivistic in the sense that that sensations were produced in the scientific laws were based.
they were based on inspectable public mind by the motions of external bodies. The scientific Jaws were what one
logic. Rawls' theory would still fit Words have meaning only when as­ was after. Science should be concerned
within the liberal tradition of free sociated with these sensations, and with causal uniformities which held for
choice and methodological indi­ truth consists of the right ordering of all members of a given class. The as­
vidualism. names with sensations. Clear defini­ sumption of the uniformity of nature
The utilitarian group is differentiated tions are critical. Reasoning is equiva­ was the key on which Mill's whole sys­
internally by" orientation to different lent to calculation and will be correct if tem of logic depended. In Mill's lan­
audiences. The systems, analysis, be­ the signs are attached to consistent im­ guage:
havioral objectives, and decision mak­ ages. Cause-and-effect is. a direct re­
ing models are strongly oriented to flection of the world. Thus the way to We must first observe that there is a
supplying information to managers. truth is through clear definition and sci­ principle in the very statement of
The goal free model is aimed more at entific method. what Induction is; an assumption
with regard to the course of nature
consumers. This elite/mass differentia­ This epistemology was reformulated and the order of the universe; namely
tion is carried through among the in turn by Locke, Berkeley, and Hume. that there are such things in nature as.
intuitionists/pluralists as well. The art Two hundred years later came the last parallel cases; that what happens
criticism and accreditation models rely of the great British empiricists, John once will, under a sufficient degree

6 Downloaded from http://er.aera.net at Monash University on March 7, 2015 ER

 of similarity of circumstances, hap­ Thorndike. Thorndike also had a strong manufacturing as yielding to the same
pen again, and not only again, but as influence on the "scientific manage­ techniques of inquiry. The result would
often as the same circumstances ment" movement in education. The be generalizations (in this case produc­
recur. . . . And if we consult the ac­ scientific management movement in­ tion functions) that would hold in vari­
tual course of nature we find that the
assumption is warranted. The uni­ troduced efficiency engineering into ous circumstances, if one could only
verse, so far as known to us, is so the schools. Tests, rating scales, and find what those circumstances were.
constituted that whatever is true in quantitative techniques stimulated by Such a mechanistic conception goes
any one case is true in all cases of a Thorndike's work were used as devices back to Hobbes, and the uniformity of
certain description; the only diffi­ for achieving efficiency in the schools nature assumptions were straight out of
culty is, to find what description. (Callahan, 1962). J. S. Mill. Results would be consistent
(Mill, 1893) In its emphasis on task analysis, pre- everywhere. Rivlin also believed, as
specification, and many other features, did Mill, that large samples were
scientific management bears a strong necessary to rid oneself of idiosyn-
resemblance to the systems analysis cracies.
Hamilton (1976)_has identified and behavioral objectives models of Mill felt that the chief moral problem
some of Mill's key epistemological ax- evaluation (House, 1978). In fact, was enlightenment, and that men were
ioms: one of the leading advocates of scien­ misled by their institutions. Rivlin be­
- There is a uniformity of nature in tific management in education was lieved that the major problem was ig­
time and space. This lends a pro­ Franking Bobbitt. Later Bobbitt and norance in not knowing how to produce
cedural certainty to inductive logic. Charters adopted a task analysis - job social services more effectively. Both
- Concepts can be defined by direct specification approach to curriculum believed in the use of experts to unravel
reference to empirical categories, and development. One of Charter's these problems. Both also believed in
laws of nature can be_derived induc­ graduate students was Ralph Tyler, using a few measures as surrogates of
tively using the proper procedures. who later originated the behavioral ob­ happiness. These are needed for calcu­
- Large samples can suppress idiosyn- jectives model of evaluation. The man­ lations in order to enable the utilitarian
cracies in the data and reveal "gen­ agerial orientation to evaluation has ethics to work.
eral causes." long historical roots. There is one place where the classic
- The social and natural sciences have With or without the historical con­ liberal J. S. Mill and a modern liberal
the same aim of discovering general nections, the similarity of the manage­ diverge however. To Mill, liberty was
laws for purposes of prediction and rial evaluation models to Mill's epis­ necessary for enlightenment to occur.
explanation. temology is strong. Perhaps the leading Mill was one of the great defenders of
- The social and natural sciences are proponent of the systems .analysis laissez faire. The government should
methodologically similar; social sci­ model has been Alice M. Rivlin. She stay out of men's affairs. For Rivlin, a
ence is just more complex. was in the group that intròďuced_cpst welfare state liberal, the government
This extreme empiricism relied on benefit analysis to education and other woul be deciding and acting. In order
the belief that categories were rather social programs in the Department of to find out what works, the government
easily obtainable, much like categories Health, Education and Welfare in the would need more control to conduct
in the physical sciences. Using the 1960s. Most of the large federal evalua­ experiments. Mill was not for large-
proper techniques one could define tion efforts were based on the systems scale social planning with the exception
variables and establish relationships be­ analysis logic (McLaughlin, 1975). of special areas like education, where
tween them. Since nature was uniform, The key ideas in Rivlin's (1971) ap­ he allowed for planning and even coer­
one could induce laws in social science proach were these (House, 1978): cion. Rivlin is the epitome of large-
just as in natural science. Furthermore scale planning, as discussed later.
this could be done with survey - Key decisions will be .made at higher
methodology. The extreme emphasis governmental .levels.
on empiricism to the exclusion of - The end of evaluation is efficiency in
theory has been inherited by evaluation the production of social services. The objectivism of the managerial,
from liberal epistemology: One can - The only true knowledge is a produc­ evaluation models that tends to equate
find relationships in nature without tion function specifying stable rela­ objectivity with quantification relies on
theoretical constructs. “ tionships between educational inputs intersubjective agreement as the exclu­
Hamilton (1977) traces the introduc­ and outputs. sive indicator of objectivity. Scriven
tion of some of these ideas into evalua­ - The only way to such knowledge is (1972) calls this the quantitative jense
tion. A contemporary of Mill, Francis through experimental methods and of objectivity. One person's perception
Galton, invented corręlationaL.tech- statistical techniques. of something is regarded as being sub­
niques to relate Mill's easily discovera­ - It is impossible to agree on goals and jective — the disposition of one indi­
ble categories to one another. Galton on a few output measures. vidual. Being objective is what a
also provided an important basis for - There is a direct parallel between number of observers experience.
psychometrics in his work on indi­ production in social services and in Common experiencing makes the ob­
vidual differences. Cattell, greatly in­ manufacturing. The same techniques servation public through intersubjec­
fluenced by Galton, introduced mental of analysis will apply. tive agreement.
testing into the United States. Like Mill, Rivlin assumed the physi­ Objectivity has come to be equated
The use of tests in education for sort­ cal science paradigm would be appro- with externalizing all references so that
ing purposes was promoted by E. L. priate. She saw social services and multiple witnessing can be achieved,

March 1978 Downloaded from http://er.aera.net at Monash University on March 7, 2015 7

since by the doctrine of intersubjec- ity. This occurs in both the ethics and Likewise, in the adversary and trans­
t¡vism what cannot be experienced di­ epistemology. In fact, utilitariansinsm action approaches how the jury will
rectly by others cannot be taken seri­ requires that there be a single standard come to a decision or how a case study
ously. Hence, objectivity is often of social utility against which to com­ worker chooses certain material over
equated with being able to specify and pare things. other material is not made totally
explicate completely all data collection Although I have placed Scriven's specific. There is a good deal of subjec­
procedures. Complete externalization goal free evaluation in the utilitarian tivity in how the key actors conduct
permits replication, the hallmark of re­ group, it is somewhat more complex. their investigations.
liability. Scriven employs the qualitative notion, Whereas the objectivists rely on ex-
In describing systems analysis pro­ in which objectivity is equated with plicitness of detail in defining tech­
cedures, the Department of Defense being free from bias or distortion. niques which others can use, the in-
puts it this way: “Perhaps the most Scriven's overriding concern is with tuitionists rely on training_and experi-
important of these is the scientific the control of bias. He may employ ence to insure that truth is served. At
method. This means that the data used experimental design like the manageri­ the extreme in these models, both valid­
in-the analysis must be capable of ver­ al group, but he has evolved an arsenal ity and utility are defined subjectively.
ification, either by observation or by of organizational and social devices for Validity is conceived as being relative
deduction from plausible premises, and controlling biases in the broadest sense to the conditions of the human mind
that the procedures employed in the (Scriven, 1975; 1976). either because of universal limitations
analysis must conform to accepted In Scriven's framework, it is possi­ on the way people think or because of
rules of logic. Systems analysis is ble for a single observer, unaided by personal limitations. (In the objec­
characterized, in the second place, by any psychometric instrumentation, to tivists' framework, validity often is
the use of quantifiable data and be more objective than a battalion of conceived as predicting one observable
mathematical techniques. . . . Explic- observers loaded with reliable instru­ category from another.)
itness is the final common denominator ments — if the single observer is look­ Subjective validity means that truth
of systems analysis. The assumptions ing for the right thing and is sufficiently is relative to human nature and perhaps
and criteria employed in an analysis are protected from numerous biases. Scri­ even to particular humans. What is
specified, and incommensurables and ven does not equate objectivity with valid for one person may not be valid
areas of uncertainty are carefully de­ intersubjective agreement. Neither for another. Likewise the subjective
lineated." (Snyder, 1967) does he assume consensus on goals and utility of something is based on per­
"Reliable instruments" receive top objectives, as dramatically indicated by sonal judgment and personal desires.
priority following this reasoning. In the goal-free approach. Goal-free Each person is the best judge of events
fact, the higher the observer agreement evaluation probably does assume for himself.
(the reliability coefficient), the better agreement on the consequences of the Actually, the subjectivist epistemol­
the instrument or procedure appears to program. (Again the choice at issue is ogy is not quite as anarchic as it ap­
be. Unfortunately reliability is no consumer's choice rather than that of pears. Whereas the objectivist position
guarantee of validity, and the intersub- the manager.) Scriven combines multi­ is based on externalization and total
jective fallacy can lead to the use of ple criteria of consequences with ordi­ specification of techniques, the subjec­
"instruments" on which one achieves nal scaling and relentlessly compara­ tivist position is based on the tradition
high observer agreement as opposed to tive approach. of the "closed_case." Through train­
procedures which may have much ing, experience, and socialization, the
greater validity. Instruments with high The Subjectivist Epistemology evaluator incorporates precedents into
reliability are hallmarks of this epis- Utilitarian evaluation, then, is based his judgments. The law is the most ob­
temological objectivism. Validity often on a subjectivist ethic, such as max­ vious example of a set of cases that
suffers accordingly. imizing a state of mind like satisfac­ guide judgment.
- "There is a sense in which the inter- tion, but employs an objectivistic epis­ The subjectivists are less interested
subjective verification principle is car­ temology in doing so. Once one has in arriving at a proposition that is
ried through in the definition of goals as determined what is to be maximized, "true" (in the generalizable sense)
well as in the measurement of effects. It the methodology can then be employed than in relating the evaluation to the
is assumed in the managerial models of objectivistically to allocate resources to particular experience of the audience.
evaluation that one can reach aconsen satisfy various desires. They attempt to obtain valid insights
sus on the goals.of-a-paιticular project. The intuitionist/pluralist evaluation within the frame of reference of the
This assumption is critical in the sys- models employ a subjectivist ethic of a group for whom they are working. It is
tems analysis and behavioral objectives different sort. Generally speaking, assumed that there is a gap between
models, for the consensus defines what their research methodology also tends language and experience. Tacit rather
the evaluation_will_look for. In addi­ to be subjectivist. Procedures for an than explicit knowledge is what the
tion, the systems analysis model also educational critic or an accreditation evaluator seeks. The evaluation is in­
assumes agreement on team to conduct their investigation are tentionally context-bound and findings
cause-and-effect relationships. Mill as­ general rather than specific. Explicit- are interpreted in context. Since the au­
sumed cause-and-effect to be obvious ness of detail and externalization of dience may well have a firmer grasp of
and discernible in nature. procedures so that others might observe the context, based on greater experi­
Hence, in objectivism the notion of the same thing are not the common de­ ence, the audience's interpretation of
objectivity becomes equated with pro­ nominators. Reproducibility is not the an event may be superior to that of. the
cedures for determining intersubjectiv- major criterion. evaluator. The subjectivist evaluator is

8 Downloaded from http://er.aera.net at Monash University on March 7, 2015 ER

more concerned in his work with Hence, one can study perceptions only Political Assumptions of the Models
specific causal statements (internal va­ by studying particular transactions in — The Utilitarians
lidity) than with general causal state­ which the perceptions car be observed. Liberalism is not only an ethical and
ments (external validity) (Ennis, All parts of the situation enter into the epistemological theory; it is a pro­
1973). Specific causal statements ob­ transaction as "active participants," foundly political theory too. As indi­
tain for a particular time and place. and do not appear as separate already- cated earlier, although Rivlin and the
Generalizations may be left to the audi­ existing entities. modern liberals accept J. S. Mill's
ence. Secondly perceiving is always done utilitarian ethics and extreme empiri­
The subjectivist methodology tends by each person from his own unique cism, they differ dramatically from
to be naturalistic. It aims at naturalistic position, experiences, and needs, in­ Mill over the role of government in
generalization (based on the experience cluding the scientist-observer. He af­ making decisions and taking action.
of the audience); is directed more at fects and is affected by the situation, Rivlin's welfare liberalism holds that
non-technical audiences like teachers thus he is part of the transaction. the government must make the jnajor
or the general public; uses ordinary Third, each person creates hisjown decisions, whereas Mill was the
language and everyday categories of psychological environment by attribut­ champion of laissez-faire, of keeping
events; and is based more on informal ing aspects of his experience to the en­ the government as far removed from
than formal logic (House, 1977). In­ vironment. He believes the environ­ decisions as possible. In this regard, the
formal interviews and observations, ment exists independently of his ex­ intuitionists are closer to_classic
often written up as case studies, are the perience. In other words, he exter­ liberalism than are the utilitarians.
favorite data collection devices. nalizes his experience. Consequently, For Mill, enlightenment was neces­
The evaluator tries to collect multi- the world as it is experienced is the sary for happiness and liberty. Freedom
ple perspectives, although the perspec- product of perception, not the cause of from government interference was
tives may not agree with one another. it. necessary for enlightenment. For Riv­
Historical and longitudinal investiga­ These presuppositions about percep­ lin, enlightenment will occur through
tions also are critical. The historical tion are strikingly different from those government-sponsored research, and
mode of investigation is more appro­ of J. S. Mill and Hobbes. In The on the basis of this knowledge, the gov­
priate than the natural science mode. mechanistic Hpbbesian view, the envi­ ernment will then take steps to alleviate
Emphasis is on the qualitative rather ronment acts on the organism to pro­ the problems. In both cases knowledge
than the quantitative. duce perception. In the transactional will make men happy.
In the subjectivist methodology, util­ view, the organism also works on the For Mill there were two distinct
ity is in terms of the observer's.in­ environmenļin an active manner so that spheres of activities — the internal and
terests. Theory and practice are perception is a product of the two. the external. The internal included the
blended together. Improving the under­ Categories are not easily discernible individual's own mind, body.,, and pri­
standing of particular individuals is the from the environment because they are vate actions. In the internal sphere the
goal. In the objectivist approach, there inextricably entwined with ones own government had absolutely no.rightto
is a rigid separation of observer and perceptions. interfere no matter what. The indi­
facts. Highly, abstract theory is sep­ vidual could amuse or abuse himself as
arated from application. Prediction is In the Hobbesian-Millean point of he saw fit in matters that concerned
the goal. view, the world is taken as given. The only himself.(Wolff, 1968). The scope
The technique of the closed .case and mechanistic uniformity' of nature as­ of government was limited only to that
use of the precedent gives stability to sures that. Perceptions can be isolated which concerned others, the external
the judgments derived from the subjec- and studied. One can communicate sphere.
tivistic epistemology. In fact, tradition statements to another scientist, which
Even in the external sphere in Mill's
is so strong in these evaluation models are reducible to what the other scientist
framework, societal interference was
that this is a criticism of them. This is will experience if he does certain things
justified only on the basis of the princi­
because the subjectivistic models rely — the operational viewpoinţ. In the
ple of utility. That is, governmental'
heavily on practitioner judgment which transactional point of view, reliability
interference was justified only when it
is in turn based on the experience of the is not so easily obtained. What one ob­
would promote the greatest happiness
classroom. The subjectivist models serves is a function of the unique posi­
of the greatest number of people by
treat education_and teaching as a craft tion and. experience of each observer
maximizing satisfactions.
rather than as a set of explicit, exter­ and even of the observer's purposes.
nalized techniques. Perceptions can only be studies in con­
crete real-life situations. . . . . actions are right in proportion
The transaction model of evaluation as they tend to promote happiness;
focuses on events "occurring in and Not all the proponents of the transac­ wrong as they tend to produce the
around the actual program in context. It tion models of evaluation would sub­ reverse of happiness. By happiness is
is based on perception and knowing as a scribe to all these presuppositions. But intended pleasure and the absence of
transactional process. In one version of this description captures the transac­ pain; by unhappiness, pain and the
transactionaī psychology, human per­ tional viewpoint as a dialectical interac­ privation of pleasure. (Mill, 1861)
ception is dependent on three features tion of people and "their environment
(Ittelson & Cantril, 1954) First, the which results in perceptions. Individual Mill's arguments were strictly
facts of perception are always pre­ perceptions are a focus of study, and utilitarian, i.e., based on an estimate of
sented through concrete, individuals active participation is essential to future consequences of various alterna­
dealing with concrete situations. knowing. tives. If the empirical estimates of con-

Downloaded from http://er.aera.net at Monash University on March 7, 2015

March 1978 9
sequences change, so could the conclu­ grams. The one exception is goal-free tices. Individuality and diversity are
sions. The modern welfare state liber­ evaluation which takes the consumer as encouraged rather than suppressed.
als haye .changed the empirical esti­ the major audience. Society is conceived as a market place
mates and hence the conclusions. Ac­ in which each individual pursues his
cording to the modern liberals, the Political Assumptions of the own private goals. Each man is free to
principle of utility requires that the Intuitionists/Piuralists pursue his own goals compatible with
government take strong action. The How is one to conduct an evaluation others pursuing theirs. In" the public
calculation of the utility principle leads if there is no single principle like the sphere, society has a right to impose
to welfare state liberalism eventually principle of utility to use as the criter­ some rules of equity. Otherwise the
(Wolff, 1968). ion? There are two types of answers market will automatically work things
For Mill, every person was the best found among the intuitionist/pluralist out.
judge of his own interest. (It should be models. One is professional authority, The ideal society enyisioned is that
noted that Mill was writing at a time as in the educational critic and accredi­ of a large diverse city like London or
when the masses had no vote. His tation models, and the other is a combi- New York,.Classic liberalism sees man
theories applied only to the more "en­ nation of scientific authority and par­ as a rational calculator.of pain, and
lightened" citizenry.) For modern lib­ ticipation in the evaluation by those in­ pleasure. Rationality is calculating
erals like Rivlin (1971), the govern­ volved in the program. The accredita- prudence. Each individual views others
ment is often considered the best judge. tion model began as a voluntary associ­ as instruments in the pursuit of his pri­
In Mill's framework, complete free­ ation to ward off government interfer­ vate ends. Others are means to some
dom of speech led to a competition of ence. end, not ends in themselves. This is
ideas which strengthened truth. In a The participatory models are at­ sometimes called the "instrumental"
free market of ideas, the consumers tempts to establish more direct partici­ theory of pluralism (Wolff 1968).
would buy the best without government pation of the people who are most Justice is the protection of the inner
interference. For the modern liberal, closely involved in the program. The sphere of the person and a more or less
the government must provide indices of adversary model does this by involving equal opportunity for all to pursue their
effectiveness on which judgments are people in a decision-making operation private goals. In more sophisticated
based and often make the choice in the like a mock trial. The transaction mod­ versions, the pleasure maximizing is
public interest. The classic and modern els involve people through negotiation, softened somewhat and The pursuit of
liberals agree on liberal political prin­ interviewing, and responding to drafts higher goals is emphasized. Nonethe­
ciples but disagree as to whether gov­ of the evaluation, although it is still the less, society is still viewed as a collec­
ernment interference will help or harm. evaluator who writes the report. If peo­ tion of independent "centers of con­
Mill felt that government interfer­ ple are to participate as major audi­ sciousness," each pursuing its own
ence would result in impeding indi­ ences, the evaluation must be im­ private goals and confronting others as
vidual development; in centralization mediate and comprehensible. The par­ alien objects (Wolff, 1968).
and abuse of power; in inefficiency; in ticipatory school is aimed at establish­ Associated with this version of
government doing indifferently and ing a more direct democracy among pluralism is the "referee" theory of
poorly what interested private parties program participants rather than in rel­ government. The government will es­
could do better; and in suppression of egating decision-making to the gov­ tablish ground rules for individual and
individual initiative which would lay ernment, as in utilitarian evaluation. group competition but will not interfere
the groundwork for despotism. How­ Pluralistic principles are derived by the in any other way. Action is to be direct
ever, even Mill felt that if the buyer participation of diverse individuals and action by the individuals and groups
were not the best judge of a commod­ groups. concerned and not action by the gov­
ity, then the competition of the market There are two distinct approaches to ernment. Each person must be free to
did not exist. In cultural and educa- political pluralism. One is derived di­ engage directly in the decision proces­
tional affairs Mill did not think that the rectly from the classic liberal ses and this role cannot be delegated or
uncultured were the best judges. There­ philosophy of J. S. Mill, without his usurped by government or anyone else.
fore, the state might legitimately inter­ epistemology or utilitarian ethics. This version of direct democracy grows
fere, a position close to that of modern Classic liberalism sees society as an out of classic liberalism in which each
liberals. association of self-determining_jndi- individual's choice is maximized.
viduals who cooperate with others for The evaluation model which most
self-interested ends. Following from closely corresponds to this version of
But, generally, the calculation of Mill, it is essential that they have a liberal pluralism is MacD.onald-s
the utiliţy.principle leads to strong gov- direct hand in governing themselves democratic evaluation (1974). Mac-
erņment interference rather than away since they know themselves and their Donald sees the evaluator as "broker in
from it. It is not surprising then that all interests best. This version of pluralism exchanges of information between
the utilitarian models of evaluation —- accepts individual idiosyncracies and groups," representing a range of in­
systems analysis, behavioral objec­ interpersonal conflicts as inevitable terests, presenting information acces­
tives, decision making, and goal-free (Wolff,1968) sible to non-specialists, giving infor­
— are based on strong government ac- The sanctity of the individual against mants confidentiality and control over
tion, usually even government man­ the intrusion of society is paramount the data, having no concept of informa­
dates. Most of these models also take as and the inner sphere of the private man tion misuse, negotiating with sponsors
their major audiences the managers and must be held inviolate. There can be no and participants, and not making re­
administrators of government pro­ interference with his thoughts and prac­ commendations. The evaluator feeds

Downloaded from http://er.aera.net at Monash University on March 7, 2015

10 ER
information to audiences and lets the shifts in pressure. In both types of Note
market work things out. Each person pluralism the capacity to see competing Ί wish to express my thanks for helpful com­
makes use of the information as he sees claims as legitimate is essential to make, ments to Robert Ennis, David Hamilton, and
Rruce Stewart.
fit with the evaluator removed from in­ the system work. References
terpretation. The evaluator operates on Stake's "responsive" evaluation Càllahan, R. E. Education and the cult of effi-
a set of procedural regulations which comes “closest to the_democratic ciency. Chicago, 111.: University of Chicago
control information flow. pluralism version. The evaluator must Press, 1962. '
Ennis, R. Ģn causality. Educational Re­
MacDonald has also been most remain responsive to any legitimate in­ searcher, 1973,2,6.
adamant about protecting the sanctity terests and pressures around the pro­ Hamilton,- Ď. A science of the singular?
of the "inner sphere," giving infor­ gram. But he is not obliged to represent University of Illinois, 1976, mímeo.
mants veto power over what goes into any point of view, nor is he obliged to Hamilton, D. Making sense of curriculum
the report. This is captured by his represent points of view in the evalua­ evaluation. In Lee Shulman (Ed.), Review of re­
search in education, ltasca, 111.: F. E. Peacock,
phrase, "People own the facts of their tion unless an interest group involved in 1977, in press.
lives." Interference by the evaluator in the program is actively promoting that House, E. R. Evaluation as scientific man­
either the private sphere or public point of view. Active involvement is a agement in U.S. school reform. Comparative and
sphere is rejected except to facilitate criterion for representing a group's International Education Review, October 1978,
information flow which will increase viewpoint in the evaluation. in press.
House; E. R. Justice in evaluation. In Gene V
knowledge and presumably happiness Issues are defined by legitimaţe_ Glass (Ed.), Evaluation studies review annual.
by allowing individuals to pursue their groups, and only a few issues are to be Volume 1, Beverly Hills, California: Sage Pub­
goals better. explored. Methods of investigation are lishing Company, 1976.
Ä somewhat different version of House, E. R. The logic of evaluative argu­
to be chosen to fit the issues. Problems ment. Center for the Study of Evaluation, UCLA,
pluralism preyails in the. United States. are best solved directly by local people Monograph 7, 1977. ,
Based on the idea that every person close at hand (Stake, 1975). This is in Ittelson, W. H. and Cantril, H. Perception: A
achieves his identity from primary contrast to utilitarian evaluation in transactional approach. New York: Ooubleday
groups, it does not see man as a totally which the government evaluates, de­ Papers in Psychology, 1954.
detached individual as does classic MacDonald, B., Evaluation and the control of
fines the problem, and takes action. education. Norwich, England: Center for Ap­
liberalism. Since ethnic, religious, The pluralistic evaluations tend to max­ plied Research in Education, 1974.
economic, and cultural groups deter­ imize local and individual choice rather MacPherson, C. B. The real world of democ­
mine what a person is, his interests than social utility. Both forms of racy. Oxford, England: Oxford University Press,
must be expressed through the larger pluralism are based on the concept of a 1966.
McLaughlin, M. W. Evaluation and reform.
groups to which he belongs. This ver­ free market of ideas. Justice tends to be Cambridge, Mass.: Ballinger Publishing, 1975.
sion, called "democratic pluralism" or identified with the free play of oppos­ Mill, J. S. A system of logic. (8th ed.) New
"conservative liberalism," recognizes ing individuals and groups, as the York: Harper Publishing, 1893.
toleranace for established, diverse working out of a free market requires. Mill, J. S. Utilitarianism. Indianapolis, 111.:
groups but not for idiosyncratic, indi­ Bobbs-Merri!lCo., 1861. National Study of Sec­
Although these evaluation models ondary School Evaluation. Evaluative criteria.
viduals. It is considered_ good for_an differ in the ways suggested, what they (4th ed.) Washington, D.C., 1969.
individual to conform to .some, groups, share are some of the basic tenets of the Popham, W. J. Educational evaluation.
but good for the groups to differ. liberal ideology. This includes the Englewood Cliffs, N.J.: Prentice Hall, 1975.
Rawls, J. A theory of justice. Cambridge,
principle of freedom of choice. Al­ Mass.: Belknap Press, 1971.
though who shall make the choice is at Rivlin, A. M. Systematic thinking for social
In this vision of pluralism as the con­ issue, choice is maximized. They are action. Washington, D.C.: The Brookings Insti­
flict of permanent interest groups, based on a strongly individualist tution, 1971.
Scriven, M. Bias control systems in
community is_a much_ stronger value psychology and their intellectual con­ evaluation. Paper presented at the Annual Meet­
than in classical liberalism. Pluralismis structions reflect that. They are also ing of the American Educational Research As­
based on group rather than individual thoroughly empiricist in orientation. sociation, 1976.
diversity. The ideal society is more like All the evaluation models also pre­ Scriven, M. Evaluation bias and its control.
a small town than a large city. Toler­ Occasional Paper 4. Kalamazoo: The Evaluation
sume a free market of ideas from Center, Western Michigan University, 1975.
ance and mutual acceptance are exer­ which, ideally, consumers will select Scriven, M. Objectivity and subjectivity in
cised among groups. the best. Through a competition of educational research. Philosophical Redirec­
Associated with this vision of ideas, truth will be strengthened and tions in Educational Research, National Society
pluralism is the "vector-sum" .or education improved. Finally, the mod­ for the Study of Education, 1972.
Snyder, W. Case studies in military systems
"balance-of-power" theory of gov­ els all assume that increased knowledge analysis. Washington, D.C.: Industrial College
ernment. The purpose is to get the gov­ will make people happy, better, or of the Armed Forces, 1967.
ernment to act in a certain direction. satisfied in some way. Stake, R. E. Evaluating educational pro­
The individual must work through his This paper has been description. of grammes. Organization for Economic Co-
groups in order to do this since only the major evaluation models and an ex­ Operational Development, 1976.
Stake, R. E. Some alternative presumptions.
they have the influence to guide the ploration of their premises. It is part of Urbana, III.: Center for Instructional Research
ship of state. As in classical liberalism, a longer work which attempts to present and Curriculum Evaluation, October 1975,
the individual must have immediacy, a comprehensive analysis of evalua­ mimeo.
effectiveness, involvement, and par­ tion. Each model has strengths and Wolff, R. P. The poverty of liberalism-
Boston, Mass.: Beacon Press, 1968.
ticipation in his group. The government weaknesses, both in theory and in prac­ Worthen, B. R. and Sanders, J. R.
(and evaluator) act as middlemen, ab­ tice. A critique of the models is a neces­ Educational evaluation: Theory and practice.
sorbing directions and responding to sary step to further understanding. Worthington, Ohio: Charles A. Jones, 1973.

Downloaded from http://er.aera.net at Monash University on March 7, 2015

March 1978 11
 FIGURE 1: A TAXONOMY OF MAJOR EVALUATION MODELS

Major Assumes
Model Proponents Audiences Consensus on Methodology Outcome Typical Questions
Systems Rivlin Economists, Goals; known PPBS; linear pro- Efficiency Are the expected effects
Analysis managers cause & effect; gramming; planned achieved? Can the effects be
quantified variables, variation; cost benefit achieved more economically?
analysis. What are the most efficient
programs?
Behavioral, Tyler, Popham Managers, Prespec¡fied Behavioral Objec- Productivity; Are the students achieving the
Objectives psychologists objectives; quantified fives: achievement accountability objectives? Is the teacher
outcome variables tests producing?
Decision Stufflebeam, Decision-makers, General goals; Surveys, question- Effectiveness; Is the program effective?
Making Alkin esp. administrators criteria naires, interviews; quality control. What parts are effective?
natural variation
Goal Free Scríven Consumers Consequences; Bias control; logical Consumer choice; What are all the effects?
criteria analysis; modus social utility,
operandi
Art Criticism Eisner, Kelly Connoisseurs, Critics, standards, Critical review Improved Would a critic approve this
Consumers Standards program?
Accreditation North Central Teachers, public Criteria, panel. Review by panel; Professional How would professionals rate
Association procedures self study acceptance this program?
Adversary Owens, Levine, Jury Procedures and ' Quasi-legal Resolution What are the arguments for and
Wolf judges procedures against the program?
Transaction Stake, Smith, Client, Negotiations; Case studies, Understanding; What does the program look like
MacDonald, Practitioners activities interviews, diversity to different people?
Pariθtt-Ham¡lton observations

Downloaded from http://er.aera.net at Monash University on March 7, 2015

12 ER