Different Levels of Outcomes

Programme Educational Objectives Few years after

Graduation – 4 to 5 years

Programme Outcomes Upon graduation

Course/subject Outcomes Upon subject completion

Weekly/Topic Outcomes Upon weekly/topic

completion
 Institutional
Stakeholders Interest
Mission Statement

Programme Objectives

Programme Outcomes
(Knowledge, skills, attitudes of graduates)

Outcome-Related Course Learning Objectives

(Ability to: explain, calculate, derive, design)

Assessment of Attainment Level

Continual Improvement
A CQI
Programme Course
Outcomes Outcomes

Teaching Plan 1
CQI Implementation
Contents 2 CQI

Levels

Contact Time
3
Learning Time Cohort’s
Evaluation
Assessments
 5
Intervention
3 for the following year
Cohort’s
Evaluation
4
Summative
at year
6
Summative
4 years

A CQI
Programme Course
Outcomes Outcomes

B
Other
Stakeholders
 Programme Objectives
What is expected (3-5 years) upon
graduation (What the programme is
preparing graduates in their career and
professional accomplishments)

Engineering Accreditation Council 5

 CHARACTERISTICS OF GOOD PROGRAMME
OBJECTIVE (PEO) STATEMENTS

➢ Each addresses one or more needs of one or more

stakeholders
➢ Consistent with the mission & vision of the institution
➢ Number of statements should be limited and
manageable
➢ Should not be simply restatement of outcomes
➢ Forward looking and challenging
 CHARACTERISTICS OF GOOD POGRAMME
OBJECTIVE (PEO) STATEMENTS

➢ Should be stated such that a graduate can

demonstrate in their career or professional life
after graduation (long term in nature)
➢ Distinctive/unique features/having own niche
➢ Specific, Measurable, Achievable, Result
oriented, and having a Time frame (SMART)
➢ Has clear link to the programme outcomes &
curriculum design
eg. Programme Educational Objectives
• To provide graduates with sufficient
knowledge in engineering and possess the
necessary skills for work in the industry.
• To produce graduates who are sensitive and
responsible towards the society, culture and
environment.
• To prepare graduates for work in advanced
design and innovation at international level.

Megat Johari Megat Mohd Noor 8

 Programme Outcomes

• What the graduates are expected to know

and able to perform or attain by the time of
graduation (skills, knowledge and
behaviour/attitude)

There must be a clear linkage between

Objectives and Outcomes
Need to distribute the outcomes throughout
the programme, and not one/two courses only
addressing a particular outcome
 PROGRAMME OUTCOME
(i) Engineering Knowledge
Apply knowledge of mathematics, science,
engineering fundamentals and an engineering
specialisation to the solution of complex
engineering problems;
 PROGRAMME OUTCOME
(ii) Problem Analysis
Identify, formulate, research literature and
analyse complex engineering problems reaching
substantiated conclusions using first principles of
mathematics, natural sciences and engineering
sciences
 PROGRAMME OUTCOME
(iii) Design/Development of Solutions
Design solutions for complex engineering
problems and design systems, components or
processes that meet specified needs with
appropriate consideration for public health and
safety, cultural, societal, and environmental
considerations
 PROGRAMME OUTCOME
(iv) Investigation
Conduct investigation into complex problems using
research based knowledge and research methods
including design of experiments, analysis and
interpretation of data, and synthesis of
information to provide valid conclusions
 PROGRAMME OUTCOME
(v) Modern Tool Usage
Create, select and apply appropriate techniques,
resources, and modern engineering and IT tools,
including prediction and modelling, to complex
engineering activities, with an understanding of the
limitations
 PROGRAMME OUTCOME
(vi) The Engineer and Society
Apply reasoning informed by contextual knowledge
to assess societal, health, safety, legal and cultural
issues and the consequent responsibilities relevant
to professional engineering practice
 PROGRAMME OUTCOME
(vii) Environment and Sustainability
Understand the impact of professional engineering
solutions in societal and environmental contexts
and demonstrate knowledge of and need for
sustainable development
 PROGRAMME OUTCOME
(viii) Ethics
Apply ethical principles and commit to professional
ethics and responsibilities and norms of
engineering practice
 PROGRAMME OUTCOME
(ix) Communication
Communicate effectively on complex engineering
activities with the engineering community and with
society at large, such as being able to comprehend
and write effective reports and design
documentation, make effective presentations, and
give and receive clear instructions
 PROGRAMME OUTCOME
(x) Individual and Team Work
Function effectively as an individual, and as a
member or leader in diverse teams and in multi-
disciplinary settings
 PROGRAMME OUTCOME
(xi) Life-long Learning
Recognise the need for, and have the preparation
and ability to engage in independent and life-long
learning in the broadest context of technological
change
 PROGRAMME OUTCOME
(xii) Project Management & Finance
Demonstrate knowledge and understanding of
engineering and management principles and apply
these to one’s own work, as a member and leader
in a team, to manage projects and in
multidisciplinary environments
 Linking topics to Programme
Educational Objectives
• Topics lead to learning objectives
• Group/individual learning objectives lead to
course outcome
• Course outcomes must relate to programme
outcomes
• Programme outcomes address the programme
objectives (What kind of “animal” are we
producing?)

Megat Johari Megat Mohd Noor 22

 Assessment tools
• Exit surveys, Exit interviews (P)
• Alumni surveys and interviews (P)
• Employer surveys and interviews (P)
• Job offers, starting salaries (relative to national
benchmark) (P)
• Admission to graduate schools (P)
• Performance in group and internship assignments
and in PBL situation (P,C)
• Assignments, report and tests in capstone design
course (P,C)
• Standardized tests (P,C) P: Program C: Course
 Assessment tools (cont)

• Student surveys, individual and focus group

interviews (P,C)
• Peer-evaluations, self evaluations (P,C)
• Student portfolios (P,C)
• Behavioral observation (P,C)
• Written tests linked to learning objectives (C)
• Written project reports (C)
• Oral presentation, live or videotape (C)
• Research proposals, student-formulated
problems (C)
• Classrooms assessment Techniques (C)
 Rubric
• It is a working guide for students and
teachers, usually handed out before the
assignment begins in order to get students to
think about the criteria on which their work
will be judged.
• Authentic assessment tool which is designed
to simulate real life activity where students
are engaged in solving real-life problems.
 Rubrics - What are they good for?
• It is a set of categories developed from the
performance criteria that define and describe
progression toward meeting important
components of work being completed, critiqued,
or assessed.
• Each category contains a gradation of levels of
completion or competence with a score assigned
to each level and a description of what
performance criteria need to be met to attain the
score at each level.
 3 common features of rubrics
• focus on measuring a stated objective
(performance, behaviour, or quality).
• use a range to rate performance.
• contain specific performance characteristics
arranged in levels indicating the degree to
which a standard has been met (Pickett and
Dodge).
 Types of Rubrics
• An analytic rubric provides specific information about student
performance on any given performance criterion.
• A holistic rubric is broad in nature and provides information
about the overall, general status of student performance
(instead of creating separate categories for each criterion, the
criteria are grouped under each level of the rubric).
• A generic rubric can be used across a variety of activities where
students get an opportunity to demonstrate their performance
on an outcome (e.g., communication skills, where it could be
used in a writing course or a design course).
• A task-specific rubric is developed with a specific task in mind
(focused and would not be appropriate to use outside of the task
for which it was designed).
 Rubric Scoring
• The use of rubrics when scoring student work
provides the programme with valuable
information about how students are progressing
and also points to specific areas where students
need to improve.
– For example, when a staff member is grading a
student’s paper, he/she can also score the paper for
the student’s writing skills using the rubric provided.
– The scores obtained by each student can be
aggregated and used for programme assessment.
 Levels?
• How many points (levels) should a rubric have?
• It is important to consider both the nature of the
performance (complexity) and the purpose of the scoring.
• If the rubric aims to describe student performance at a
single point in time, then three to five points are
recommended.
• If student performance is to be tracked over time and the
focus is on developmental growth, then more points are
needed.
• Remember, the more points on the scale, the more difficult
it is to get multiple raters to agree on a specific rating.
 Effective Rubrics
• For programme assessment, the most effective
rubrics (generally speaking) are analytic, generic,
and the use of a three- to five-point scale.
• Good websites designed to help with the
development of rubrics.
http://edtech.kennesaw.edu/intech/rubrics.htm.
• Many examples of rubrics on the web, but just
because they are on the web, it doesn’t mean
they’re good examples. Proceed with caution.
 Presenting Assessment Results
• A staff member can represent the data
graphically.
• How many students meet the expected
standard of “meets criterion” , the number
who exceed standard and the number that are
making progress can be determined.
• Staff should think through how the data are
going to be used before developing a rubric.
 Advantages
• Rubrics improve student performance by clearly
showing the student how their work will be
evaluated and what is expected.
• Rubrics help students become better judges of
the quality of their own work.
• Rubrics allow assessment to be more objective
and consistent.
• Rubrics force the teacher to clarify his/her criteria
in specific terms.
• Rubrics reduce the amount of time teachers
spend evaluating student work.
 Advantages (cont)
• Rubrics promote student awareness about the
criteria to use in assessing peer performance.
• Rubrics provide useful feedback to the teacher
regarding the effectiveness of the instruction.
• Rubrics provide students with more informative
feedback about their strengths and areas in need
of improvement.
• Rubrics accommodate heterogeneous classes by
offering a range of quality levels.
• Rubrics are easy to use and easy to explain.
 Outcome-based Assessment
Implementation Assessment Data
Strategy Strategy Sources/Assessment
instruments
Industrial project Exams, interview, Reports, interview
Improve student survey, observe, schedule, survey,
competence in assess skill level, observation records,
communication, monitor grades of exams and
teamwork, and project development of projects, exit skill
management skills checklist
Design course Assessment criteria List of assessment
Address industry from literature, by criteria, observation,
needs industry, and reports, interview,
lecturers students evaluation,
exams, exit skill
checklist
 Some Thoughts

• Provide clear guidelines for all work

– Report writing – nature and structure of the
information required
– Oral presentation – detailed evaluation criteria:
clarity, effective use of visual aids, eye contact
• Use of higher order thinking skills
• Team involvement to be defined
 Unified key outcomes
• Allow lecturer to decide on the
criteria/indicator
• Provide a standard and calibration
• Get definition (perception from lecturer) and
then standardise the definition

Megat Johari Megat Mohd Noor 37

 Performance Criteria/ Indicators -
Good Teamwork
Students are able to demonstrate

1. Positive contribution to the team project (minutes of

meeting)
2. Well prepared and participate in discussion (observation)

3. Volunteer to take responsibility

4. Prompt and sufficient attendance

5. Aplomb and decorum

 Course Outcomes
• Statement … explain, calculate, derive,
design, critique.
• Statement … learn, know, understand,
appreciate – not learning objectives but
may qualify as outcomes (non-observable).
• Understanding cannot be directly
observed, student must do something
observable to demonstrate his/her
understanding.
 Bloom’s Taxonomy
• Knowledge (list)
• Comprehension (explain)
• Application (calculate, solve, determine)
• Analysis (classify, predict, model,derived)
• Synthesis (design, improve)
• Evaluation (judge, select, critique)
lower order Intermediate Higher order
lower order Intermediate Higher order
 Course Outcomes (CO) Contribution to
Programme Outcomes (PO)
Ability to function in multidisciplinary team

◼ Assign multidisciplinary design projects in

engineering courses.
◼ Implement design projects with
multidisciplinary teams

Exercise:
Identify a course and discuss how it can be
implemented
 Course Outcomes (CO)Contribution to
Programme Outcomes (PO)
Broad education necessary to understand the impact of
engineering solutions in a global, environment and
societal context + knowledge of contemporary issues

• Include structured controversies in engineering

course
• Conduct class exercise or homework problems
that involve global/societal issues

Exercise:
Identify a course and discuss how it can be
implemented
 Course Outcomes (CO) Contribution to
Programme Outcomes (PO)
Life Long Learning
• Teach students about learning styles and help them
identify the strength and weakness of their styles and give
them strategies to improve
• Use active learning methods to accustom them to relying
on themselves
• Give assignments that requires library and www searches
• Anything done to fulfil criteria on: (a) understanding
ethical and professional responsibility and (b)
understanding societal and global context of engineering
solutions, will automatically satisfy this criteria
Active (A) Learners Reflective (R) Learners
 Tend to process actively (doing  Tend to process reflectively
something physical with presented (thinking about presented material,
material, then reflecting on it) then doing something with it)
 Think out loud  Work introspectively

 “let’s try it out and see how it  “Let’s think it through and then
goes” try it”
 Tend to jump in prematurely  Tend to delay starting

 Like group work  Like solo or pair work

Sequential (Sq) Learners Global (G) Learners

 Built understanding in logical  Absorb information randomly,

sequential steps then synthesize the big picture
 Function with partial  Need the big pictures
understanding of information (interrelations, connections to other
subjects and personal experience)
in order to function with information
 Make steady progress  Large leaps in understanding with
little progress between them
 Explain easily  Can’t explain easily

 Good at analytical thinking (the  Synthesis, holistic thinking (the

trees) forest)
Student-Centered Learning