Textbook Evaluation for Educators
Textbook Evaluation for Educators
1. Definition
Materials evaluation can refer to either before the program begins or after or both.
Here, materials evaluation refer to a before program evaluation of published textbooks.
This is motivated by the need to choose materials that will be relevant and appropriate for a
particular group of learners and also by the need to identify specific aspects of the materials
that require adaptation.
In materials evaluation the evaluator identifies a set of criteria that are used to reach a decision
regarding which book to adopt and how it needs to be adapted.
Instruments used in evaluating materials. The instruments used in evaluating materials or
textbooks are questionnaires, checklists and guides.
• Approach
• Syllabus
• Techniques (ways of presenting materials)
• Exercises
b. Design:
1. Aims
2. Principles of selection
3. Principles of sequencing
4. Subject matter and focus of subject matter
5. Types of learning/teaching activities: - what they require the learner to do - manner in which they
draw on the learner’s process competence (knowledge, affects, abilities, skills)
6. Participation: who does what with whom
7. Learner roles
8. Teacher roles
9. Role of materials as a whole
Material Evaluation: Principles in Materials Evaluation
It involves making judgments about the effect of the materials on the people using them and it tries to
measure some or all of the following:
1. The appeal of the materials to the learners.
2. The credibility of materials to learners, teachers, and administrators.
3. The validity of the materials.
4. The reliability of the materials.
5. The ability of the materials to interest the learners and the teachers.
6. The ability of the materials to motivate the learners.
7. The value of the materials in terms of short-term learning (important, for example, for performance
on tests and examinations.
8. The value of the materials in terms of long-term learning (of both language and communication
skills)
9. The learners’ perceptions of the value of the materials.
10. The teachers’ perceptions of the value of the materials.
11. The assistance given to the teachers in terms of preparation, delivery, and assessment.
12. The flexibility of the materials.
13. The contribution made by the materials to teacher development.
14. The match with administrative requirements.
EVALUATING MATERIALS
Evaluation of learning materials is usually integrated in the overall learning design and
development plan.
It is considered to be a vital component of a quality assurance strategy and the expectation is
that evaluation activities can contribute significantly to the development of quality learning
materials.
The purpose is often to get an idea of how well the materials are aligned to the intended
learning outcomes and how well they support students in achieving these outcomes.
Additional questions may be:
o how accessible the materials are,
o how the students are using the materials,
o how up to date the content is, whether the learning text is based on sound learning
principles.
Evaluation can be conducted during each phase of the design and development process, during
the post development period when students are using the learning materials for the first time,
and as part of a review of a course or program.
Where the development of the materials has been sponsored, it is normal practice to evaluate
the materials and provide the funder with an evaluation report
Evaluation instruments are usually based on an agreed set of criteria.
Interviews with learners, facilitators, and learning material developers can be used to probe
particular areas.
can be useful in pinpointing learner behavior in specific areas.
e.g. learner interaction with the learning activities:
o how many students complete all the activities,
o which activities are left out,
o which activities present difficulties.
o The purpose of the evaluation determines the kind of methods that are most suitable.
In this part, it presents how the strong surge of interest in the goal, roles, and methods of
evaluation and also attributes this trend primarily to the increase in the influence of mainstream
educational theory, and beside that in this part need to carry out a large scale program
evaluation for external funding according to Ellis speak.
In material evaluation have two parts there are macro evaluation and micro evaluation which to
make distinction between
macro evaluation of such large scale project, typically carried out for accountability and or
developmental purpose and
micro evaluations used in classroom level, especially to know achievement of learners however
this achievement is not only cognitive aspect but include all potential learners have.
By this definition material evaluation, selection and adaptation fall under the purview of micro
evaluation, as does the evaluation of teachers and learners classroom behaviors.
That it is mean the micro evaluation is compatible with many teachers perspective about
evaluation involves, encouraging teacher to adopt a micro rather than a macro perspective to
evaluation will help them undertake evaluation that accords with their own perspective.
Because the material evaluation would involve assessing how effective and useful the material
are found to be in actual use by a specific group of teacher and students or how effective they
may have been in promoting learning, selection of material which is concerned with the
potential that a set of material may have in effectively and efficiently supporting learning, as a
frame for learning and teaching opportunities.
Evaluation, like selection is a matter of judging the fitness of something for a particular purpose.
However, while it is true that the selection of materials inevitably involves, or subsumes, a
process of evaluation, evaluation can be undertaken for a variety of purposes and carried out in
a variety of ways.
Besides that,. selection of material, on the other hand, what assumes primary importance is the
analysis of learner needs, and interest and how these are addressed
Evaluation material has been designed to help teacher make systematic selection of textbooks.
Because these vary in the extent to which they reflect the priorities and constrains that might
characterize specific context of English language teaching.
There some of the earliest attempts to developing teacher-friendly systems for rigorous
assessments include the elaborate questionnaires design by Tucker (1975), Van Lier (1979) and
William (1983).
Breen and Candin’s (1987) interactive step-by-step guide to course book evaluation envisage
two phases, one addressing the overall usefulness of the material and another aiming at a more
searching analysis with a particular group of learners and classroom situation in mind.
Arguing for the need to look below surface features to discover the value system and
assumptions underlying material design, Hutchinson (1987) likewise views evaluation as an
interactive process involving a subjective and objective analysis of materials and the extent to
which they match teacher and students need in a given context.
And more recently, McDonough and Shaw (1983) have also proposed two complementary
stages, beginning with an external evaluation and moving on to in depth internal evaluation of
two or more units in terms of presentation of skills, grading and sequencing task, kinds of text
used and the relationship between exercise and test.
Focusing on materials as pedagogic device, Littjeohn(1998) analysis contains two main
dimensions :
A. publication, which refer to the tangible or physical aspects of the materials and
B. design which relates to the thinking underling its production and use including its aims, how
the task, language and content in the material are selected and sequenced and the nature and
focus of the content.
And according to Sheldon (1998:245) say it correctly, “it is clear that course book assessment is
fundamentally a subjective, rule of thumb activity and not neat formula, grid or system will ever
provide a definitive yardstick.
The survey guides that have been proposed, although of practical use to teachers, raise many
questions relating to how the materials should be considered and, crucially, how one aspect
should be weighted in relation to another.
Universal criteria are those which would apply to any language learning materials anywhere
for any learners.
They derive from principles of language learning and the results of classroom observation
and provide the fundamental basis for any materials evaluation.
The following are the examples of universal criteria would be:
a. Do the materials provide useful opportunities for the learners to think for themselves?
b. Are the target learners likely to be able to follow the instructions?
If the evaluation is going to be used as a basis for revision or adaptation of the materials, or if it
is going to be a formal evaluation and is going to inform important decisions, it is useful to
subdivide some of the criteria into more specific questions.
For example; Are the instructions:
Succinct?
Sufficient?
Such a subdivision can help to pinpoint specific aspects of the materials which could gain from
revision or adaptation.
3. Monitor and revise the list of universal criteria
e. Is each question reliable in the sense that other evaluators would interpret it in the same way?
Some terms and concepts which are commonly used in applied linguistics are amenable to
differing interpretations and are best avoided or glossed when attempting to measure the
effects of materials.
For example, each of the following questions could be interpreted in a number of ways:
Are the materials sufficiently authentic?
Is there an acceptable balance of skills?
There are also a number of ways in which each question could be rewritten to make it
more reliable and useful.
It is very useful to rearrange the random list of universal criteria into categories which facilitate
focus and enable generalizations to be made.
An extra advantage of doing this is that you often think of other criteria related to the category
as you are doing the categorization exercise.
Possible categories for universal criteria would be:
o Learning Principles
o Cultural Perspective
These are criteria which relate to the actual or potential environment of use.
They are questions which are not concerned with establishing the value of the materials per se
but rather with measuring the value of the materials for particular learners in particular
circumstances.
Typical features of the environment which would determine this set of materials are:
o The type (s) of institution (s);
o The resources of the institution (s);
It is im
portant to trial the criteria (even prior to a small, fairly informal evaluation) to ensure that the
criteria are sufficient, answerable, reliable and useful. Revisions can then be made before the
actual evaluation begins.
10. Conducting the evaluation
From experience the writer has found the most effective way of conducting an evaluation is to:
o Make sure that there is more than one evaluator; discuss the criteria to make sure
there is equivalence of interpretation;
Answer the criteria independently and in isolation from the other evaluator(s)
PRE-USE EVALUATION
Pre-use evaluation involves making predictions about the potential value of materials for their
users.
It can be context-free, as in a review of materials for a journal, context influenced as in a review
of draft materials for a publisher with target users in mind or context-dependent, as when a
teacher selects a course book for use with her particular class.
Tomlinson and Masuhara (2004, p. 7) proposed the following criteria for evaluating criteria:
o > Is each question an evaluation question?
o > Does each question only ask one question?
o > Is each question answerable?
o > Is each question free of dogma?
o > Is each question reliable in the sense that other evaluators would interpret it in the
same way?
Tomlinson (2012) reports these criteria and gives examples from the many checklists in the
literature of evaluation criteria which their use exposes as inadequate in terms of specificity,
clarity, answerability, validity and generalizability.
WHILST-USE EVALUATION
Exactly what can be measured in a whilst-use evaluation is controversial but It would be
included the following:
o > Clarity of instructions
o > Clarity of layout
o > Comprehensibility of texts
o > Credibility of tasks
o > Achievability of tasks
o > Achievement of performance objectives
o > Potential for localization
o > Practicality of the materials
o > Teach ability of the materials
o > Flexibility of the materials
o > Appeal of the materials
o > Motivating power of the materials
o > Impact of the materials
o > Effectiveness in facilitating short-term learning
POST-USE EVALUATION
Post-use evaluation is probably the most valuable (but least administered) type of evaluation as
it can measure the actual effects of the materials on the users.
It can measure the short-term effect as regards motivation, impact, achievability, instant
learning, etc., and it can measure the long-term effect as regards durable learning and
application.
WAYS OF MEASURING THE POST-USE EFFECTS OF MATERIALS INCLUDE:
ADAPTING MATERIALS
ADAPTATION
Adaptation is a process that often attracts a great deal of interest, yet remains less used than expected.
Although many organizations consider adapting materials, the cost, copyright implications, and practical
issues of techniques may form a sizeable barrier.
However, the increasing use of multimedia and a number of other forms of electronic publishing is
stimulating both increasing options for and renewed interest in adaptation as a source of learning
materials.
Developing high quality learning materials can be complex, time consuming and costly.
Many organizations are not staffed to produce their own materials having insufficient resources
or expertise.
Adapting materials can itself be a substantial materials development and production task.
The decision to adapt will usually is considered when a set of materials partially meet a specific
need, but could not be used in their current form.
At this point, you will need to identify the features that are absent or inappropriate and specify
what form of adaptation is needed.
Step 1
Specify learning need in as precise terms as possible including a profile of characteristics of
learners and any implications this might have for learning materials e.g. does place of work
restrict options for media
Step 2
Identify material for potential adaptation
Step 3
Identify effectiveness of material/need for adaptation covering the following features:
- learning objectives - approach
- terminology - media used
- fit with your organization's culture - presentation and style
- content level - comprehensiveness of coverage
- depth of coverage
Step 4
Defining the extent of adaptation and the form this might take
Step 5
Produce a specification, budget and schedule for an adaptation to examine feasibility and cost
effectiveness.
Learning objectives
Most learning materials and most planning for a course or programme will include the definition
of learning objectives.
Matching the objectives of your course/programme with those of existing material is a useful
early step.
In most subjects it may give a clear idea of match or variation and will usually indicate a need for
additional material or adaptation. Points to consider are:
o check all aspects of the objectives not just the behavior or knowledge desired. The
standard or conditions may be particularly important in indicating whether the material
is appropriate
o examine how closely and explicitly the materials follow and meet the objectives set
o do not let different forms of wording obscure any similar objectives i.e. two sets of
objectives may be written in different formats and styles, but may have very similar
goals or outcomes.
MEDIA USED
The most rapidly growing area of interest in adaptation is the addition to or re-use of different media.
Stimulated by developments in multimedia, the possibilities created by authoring and production tools
have made media related issues more important. There are several variations:
adding different media to a learning package e.g. supplementing text with video
drawing together different parts of existing material in a multimedia package
revising existing computer based material into new formats.
When you are considering specific material for adaptation, you will need to consider whether
the value of adding or revising media justifies the cost.
In many instances, this cost may be difficult to justify when set against that of an existing new
‘tailor-made’ set of materials or compared with the value gained from the adaptation.
Increasingly, organizations are selecting media and methods to form part of organization-wide delivery
systems.
This can include following a specific house style and approach or delivery using learning technologies.
This type of development is inevitable as organizations move to embedding forms of open and flexible
learning within their provision.
However, if you decide to adapt the media used you should seek to ensure that:
choice of media remains appropriate to the learning needs and user group
media selection or technology selected does not become the dominant factor in your open
learning provision.
The physical appearance of most learning material is often the area of concern.
Most larger organizations have their own house style and may prefer learning materials to be
consistent with their own visual style.
However, the impact and value of simply re-designing materials can be sufficient to justify the
expenditure.
You will need to assess whether the visual appearance does have a major impact on learning.
There groups or sectors where this may be a significant factor, but in many other cases,
presentation alone may not be sufficient to decide to adapt.
Materials adaptation means matching materials with the learner’s needs, the teacher’s
demands and administration’s purpose.
To adapt materials we have to consider five major factors:
(1) Addition:
Addition is an adaptation procedure which involves supplementation of extra linguistic items and
activities to make up for the inadequacy/ insufficiency of materials.
Addition of extra materials is necessary/applicable/appropriate when the following situations are faced:
(2) Deletion/omission:
Deletion is an adaptation procedure which involves removal of some of the linguistic items and
activities which are found to be extra and unnecessary.
So, deletion is a process in which materials are taken out rather than added. Materials should be
reduced through omission when the following situations are faced:
Learners are clear about a language point.
Learners are competent in a skill.
There are too many tasks on a particular area/point.
The item/area concerned is not a priority.
The item/task is not well designed.
The item/task is not well-suited to its aim(s).
The topic is not appropriate for learners.
(3) Modification/changing:
Modification means changes in different aspects of materials, such as linguistic level, exercises,
assessment system and so on.
Modification of materials is applicable/ appropriate in the following situations:
o
o
o Texts are of inappropriate length.
o Materials are inappropriate to the aim.
o Materials are inappropriate to the learners’ age/ experience.
o Materials are unclear, confusing or misleading.
o Tasks are badly designed.
o
(4) Simplification:
(5) Rearrangement/re-ordering:
ADAPTING MATERIALS
A teacher-centred approach to Materials adaptation, is still left to the teachers’ hands, and it is
largely based on their experience.
A learner-centred approach to adaptation
Clarke (1989) distinguishes Negotiated Syllabus and Externally Imposed Syllabus.
Adaptation as critical awareness development.
It supports a much more active learner’s role.
The learner is given the opportunity of sharing the ownership of the classroom and materials.
The learners provide classroom input adapting courses can be used as an awareness
development activity (Tomlinson, 2003)
1. facilitates learner involvement
2. promotes the use of materials adaptation
3. apply them also to teacher development
Aesthetic Experience originated from the theory of Aesthetic Response by Rosenblatt (1995)
It refers to the process of reacting spontaneously when reading literary texts (typically
represents the immediate response to language and literature)
It involves interaction between readers, language and texts , Literature and Aesthetic
Experience, part of a subjective process.
It is created every time the text is read or written
Aesthetic Experience promotes the subjectivity of texts and their various interpretations.
also materials should promote an aesthetic experience, they should, not only be based on
right/wrong testing and practice but, rather, they should also focus on open-ended tasks and
texts.
RELEVANCE
To draw a link between the adaptation process and reading open-ended materials,
Have the potential to become relevant to the learners when they fill those gaps with their ideas,
interpretations and discussions.
It is only at this level that materials become relevant and potentially more effective for learning
development.
UNIVERSALITY
In order to expose the learners to realistic input materials should be based on authentic texts.
In order to draw the learners’ attention to certain linguistic features with activities based on
texts selected from authentic sources a combination of authentic and non-authentic tasks.
In order to make learning more engaging and perhaps also more humanistic materials should
include topics and activities that can potentially provoke a reaction, provide an aesthetic
experience, whether it be positive or negative, that is personal and subjective.
CONCLUSIONS
Developing critical awareness of learning and teaching is the main aim of adapting and evaluating
courses;
* learners can become, the main input providers,
* the teacher’s role is simply that of facilitator, coordinator and
monitor.