Developing Advanced Crew Resource

Management (ACRM) Training:

A Training Manual

BAGASOO

August, 2009

COSCAP-BAG Office of the

Technical Advisor for Human
Factors, Abuja, Nigeria
 This Page

Intentionally Left Blank

Table of Contents

Page
Table of Contents ............................................................................................................ i

Acronyms and Definitions............................................................................................... iii

Summary ......................................................................................................................... ix

Part 1. Introduction to ACRM Training .......................................................................... 1

Background to ACRM Training .............................................................................. 1

Elements of ACRM Training................................................................................... 4

Benefits of ACRM Training .................................................................................... 8

Part 2. Guiding the Organization..................................................................................... 11

Developing Organizational Commitment ................................................................ 12

Integrating ACRM within the Organization ............................................................ 16

Part 3. Developing CRM Procedures .............................................................................. 19

Overview of CRM Procedures Development ......................................................... 20

Procedures Development Requirements .................................................................. 22

Identifying Industry and Own Airline Needs ........................................................... 27

Specifying CRM Procedures.................................................................................... 33

Refining CRM Procedures and Media ..................................................................... 35

CRM Procedures Development Guidelines ............................................................. 40

Part 4. Developing Instructor/Evaluator Training ........................................................... 43

Overview of Instructor/Evaluator Training Development ....................................... 44

Instructor/Evaluator Requirements .......................................................................... 46

Developing Introductory ACRM Modules .............................................................. 48

Developing LOFT/LOE Modules ............................................................................ 51

Developing Assessment Standards and IRR Process............................................... 55

Developing Standardization and Training Modules ................................................ 58

Instructor/Evaluator Training Development Guidelines.......................................... 64

Part 5. Developing ACRM Crew Training...................................................................... 67

Overview of Crew Training Development............................................................... 68

Curriculum Development Requirements ................................................................. 70

Developing CRM Procedures Crew Modules.......................................................... 73

Developing Crew Effectiveness Modules................................................................ 77

Developing Briefing/Debriefing Modules ............................................................... 80

Developing Crew LOS Assessment Modules .......................................................... 82

ACRM Crew Training Development Guidelines..................................................... 86

Part 6. Implementing and Evaluating ACRM ................................................................. 91

Overview of Implementing ACRM within the Organization .................................. 92

Implementation Requirements ................................................................................. 95

Implementing ACRM for the Instructor/Evaluator.................................................. 99

Maintaining Standards and the IRR Process............................................................ 102

Using Performance Data to Improve CRM Procedures and Training ..................... 105

ACRM Implementation Guidelines ......................................................................... 109

References and Resources ............................................................................................... 113

Index................................................................................................................................ 117

ACRM Training Manual Page i

Appendix A. Summary of ACRM Guidelines ............................................................... A-1

Appendix B. Sample Instructor/Evaluator and Organizational Forms........................... B-1

Appendix C. Sample ACRM Instructor/Evaluator Training Manual TOC.................... C-1

Appendix D. Instructions for Facilitating an IRR Training Workshop.......................... D-1

Appendix E. Considerations for Videotaping Simulator Sessions................................. E-1

Appendix F. Sample LOFT/LOE Development Materials............................................. F-1

Appendix G. Sample ASRS Incident Reports................................................................ G-1

Appendix H. Sample ACRM Crew Training Manual TOC........................................... H-1

Appendix I. Sample Quick Reference Handbook Procedures and Briefing Guide....... I-1

ACRM Training Manual Page ii

Acronyms and Definitions

Above Standard Point on a rating scale or criteria for performance that signifies a
level of crew performance that exceeds (in specific ways) the
expected level of individual or crew performance (i.e., the
Standard).

ACRM Advanced Crew Resource Management - a comprehensive

implementation package including CRM procedures, training of
the instructor/evaluators, training of the crews, a standardized
assessment of crew performance, and an ongoing implementation
process providing an integrated form of CRM by incorporating
CRM practices with normal and emergency SOP.

Agreement Index Agreement is the degree to which a group of raters give the same
rating for the same performance or items. Agreement is from 1 to
–1, where 1 means complete agreement and 0 to -1 is no
agreement. A .6 or .7 has been used as a minimum level of
agreement in crew performance ratings.

AQP Advanced Qualification Program - An alternative training and

assessment program based on proficiency-based training where the
Proficiency Objectives are systematically developed, maintained,
and validated.

ASRS Aviation Safety Reporting System - A NASA sponsored reporting

system where reports are submitted voluntarily, deidentified, and
entered into a database.

Assessment Scale A set of points used to measure a targeted behavior.

ATA Air Transport Association.

ATC Air Traffic Control.

Attitude A predisposition to react in a given manner to individuals, objects,

or situations. Attitudes have affective, behavior, and cognitive
components.

Backup Plan Plan to be used in case the Bottom Line has been exceeded.
Backup Plan may be developed into a CRM procedure under
ACRM.

Bottom Line One or more limits, beyond which an alternate or Backup Plan is
initiated. The Bottom Line may be expressed in time, location,

ACRM Training Manual Page iii

 fuel quantity or other flight critical items. Bottom Line may be
developed into a CRM procedure under ACRM.

Brief A specific briefing such as the Takeoff Brief or the Approach

Briefing. Under ACRM, CRM procedures may be added to certain
briefs.

Cognitive Skill Those mental skills that are a prerequisite to performing a task or
subtask. Most skills have both cognitive and non-cognitive
components, but a cognitive skill has a substantial mental
component.

Conditions In Event Sets, conditions are the elements that provide the realism
and operational relevance. Event set conditions include visibility,
wind, turbulence, traffic, and other elements that further constrain
the operational environment.

Congruency Assessment of the distribution of the ratings of each rater as

compared to that of the group. Complete congruency is
represented by the number 1.

Consistency Correlation between rates representing the degree of consistent

shifting of responses across items. Complete correlation is
represented by the number 1.

CQP Continuing Qualification Program - The ongoing program during

which the proficiency objectives are trained and evaluated. A
continuing qualification cycle may last two years and be made up
of two evaluations taking place at 12-month intervals.

CRC Camera-Ready Copy - The final version of the procedure, QRH,

QRC, or guide that is sent to the printers for reproduction.

CRM Crew Resource Management - The effective use of all resources to

include human and other aviation system resources.

CRM Objectives Training objectives related to CRM or the CRM Skills, the crew
and team elements.

CRM Procedure Procedures developed to emphasize specific CRM elements by

incorporating them into SOP for normal as well as abnormal and
emergency flight situations.

CRM Skill Crew performance elements associated with the more cognitive
and management aspects of flying. These skills are contrasted with
Technical Skills associated with the more psychomotor aspects.

ACRM Training Manual Page iv

Distracters Events used in an event set to hide the trigger or to increase crew
workload or distraction for critical tasks.

DOT Department of Transportation.

EO Enabling Objective - A lower level learning or training objective

that is required to achieve higher level objectives such as
Supporting or Terminal Proficiency Objectives.

Event Set A relatively independent segment of a scenario made up of several

events including a Trigger, possible Distracters, and Conditions.

FAA Federal Aviation Administration.

First Look Initial look at crew proficiency, generally specific maneuvers in a

PC or Maneuver Validation. With First Look, individuals are
generally allowed to repeat maneuvers when required.

FOM Flight Operations Manual.

FSM Flight Standards Manual.

GPWS Ground Proximity Warning System.

ICAO International Civil Aviation Organization

I/E Instructor/Evaluator - Those conducting the training, especially in

the simulator, as well as those providing the assessment, such as
checkairmen.

IRR Inter-Rater Reliability - The extent to which rater data would be

replicated in other, similar situations, and thus are descriptive of
consistent phenomena.

ISD Instructional Systems Development - A systematic process for

planning, developing, and evaluating instructional programs with
an emphasis on the required Tasks, Subtasks, Knowledge, and
Skills.

Knowledge The concepts and information required to perform skills such as

recalling facts, identifying concepts, and applying rules or
principles in the appropriate context.

KSA Knowledge, Skills, and Attitudes - (see Attitude, Knowledge, or

Skill for definitions).

LOE Line Operational Evaluation - LOE is an evaluation of individual

and crew performance in a flight training device or flight simulator

ACRM Training Manual Page v

 conducted during real-time LOS under an approved AQP program
as described in SFAR 58. The LOE must be designed by an
approved design methodology described as a part of the AQP.

LOFT Line-Oriented Flight Training - Under AQP, LOFT is categorized

by Qualification LOFT and Recurrent LOFT. Both types of LOFT
are conducted as a line mission, allow for minimum or no input
from the facilitator during the session, and are conducted for
training, not evaluation, purposes.

LOS Line Operational Simulation - A simulator training session

conducted in a “line environment” setting. LOS includes Line
Oriented Flight Training (LOFT), Line Operational Evaluation
(LOE) and Special Purpose Operational Training (SPOT).

Maneuver Validation A simulator evaluation of technical maneuvers, usually a required

part of recurrent training in the U.S.

NASA National Aeronautics and Space Administration.

Needs Questionnaire A questionnaire designed to collect data about own airline needs in
the area of CRM training. This type of questionnaire is more
specific than the Organizational Survey and is usually given to
instructor/evaluators and checkairmen.

NTSB National Transportation Safety Board.

Observable Behavior Individual or crew behavior used in the assessment of crew

performance. Specific Observable Behaviors may be used in the
assessment of crew performance within the context of Event Sets.

Organizational Survey A general type of questionnaire developed in the early stages of

ACRM to collect information on how different departments view
job, safety, and training issues at their airline.

PAD Program Audit Database.

PC Proficiency Check.

PF Pilot Flying - The pilot flying the aircraft, either PIC or SIC.

Phase of Flight The standard stages that occur in most operational flights to
include preflight, taxi, takeoff, climb, cruise, descent, approach,
landing, and after-landing.

PIC Pilot In Command.

PNF Pilot Not Flying.

ACRM Training Manual Page vi

POI Principal Operations Inspector.

PPD Performance/Proficiency Database.

Proficiency Objective A statement of the behavior that must be demonstrated on the job,
including the statement of performance, conditions, and a standard.

QRC Quick Reference Card - Brief set of guidelines and procedures,

often taking the place of memory items, used during abnormals and
emergencies.

QRH Quick Reference Handbook - A document designed to be used in

the cockpit that specifies emergency and abnormal procedures.

Sensitivity Index of how accurately a rater evaluated a crew with different

levels of performance or evaluated different crews representing a
range of behaviors relative to the Standard.

SIC Second In Command.

Skill Goal-directed actions, both cognitive and psychomotor, that are

acquired through practice. A skill is evaluated through
performance; performance characterized by an economy of effort.

SME Subject Matter Expert.

SOP Standard Operating Procedure.

SPO Supporting Proficiency Objective - Training objective created at

the subtask level that may include the knowledge and skills in that
subtask.

SPOT Special Purpose Operational Training - SPOT is a simulator

training session designed to address specific training objectives.
SPOT may consist of full or partial flight segments depending on
the training objectives for the flight.

Standard Parameters or criteria of performance that signifies the expected

level of individual or crew performance. Standard may also be
used as a point in a rating scale to designate the expected level of
performance.

Standard Scale Scale based on a Standard and used throughout most of an airline’s
assessment and rating process.

Statement of Condition CRM procedure used in briefings to maintain situation awareness

and make the briefs relevant to the current operation conditions.

ACRM Training Manual Page vii

 Conditions may include a combination of WX, airspeed, altitude,
fuel, and traffic.

Subtask Unit of work below the Task level representing a required step in
the performance of a task.

Systematic Differences Comparison of the mean score of an individual rater with the mean
score of the group. Significant differences indicate that the
individual rates crew performance are higher or lower than the
group.

Task A basic unit of work with a clear beginning and ending point, a
goal, and one or more products. A task is a high-level unit for
Instructional Systems Development and includes one or more
subtasks.

TCAS Traffic alert and Collision Avoidance System.

Technical Objectives Training objectives related to maneuvers and procedures. These

are related to, but different than, the CRM Objectives.

Technical Skill Maneuvers, procedures, and other crew performance elements

associated with the process of flying. These skills are contrasted
with CRM Skills that represent the more cognitive and crew
components of flying.

Theme A higher-level training objective for a training event such as a

LOFT or LOS session.

TOC Table of Contents.

TPO Terminal Proficiency Objective - The highest level of definition for

a training objective. Successful accomplishment of a terminal
objective (task) includes all of its subtasks.

Trigger Instructor controlled element that defines the start of focused

observation within an Event Set.

ACRM Training Manual Page viii

Summary

CRM and the Need for ACRM Training

U.S. airlines have implemented Crew Resource Management
(CRM) training with an emphasis on principles and concepts that
improve crew performance and flight safety. This has resulted in
crew requirements that have been trained and assessed as
additions to, rather than as part of, Standard Operating Procedure
(SOP). Advanced Crew Resource Management (ACRM)
provides a more integrated form of CRM by incorporating CRM
practices with normal and emergency SOP.

ACRM is a comprehensive implementation package

including the CRM procedures, training of the
instructor/evaluators, training of the crews, a standardized
assessment of crew performance, and an ongoing implementation
process. ACRM has been designed and developed through a
collaborative effort between the airline and research community.
ACRM training is an ongoing development process that provides
airlines with unique CRM solutions tailored to their operational
demands. Design of CRM procedures is based on critical CRM
principles that require emphasis in airline’s specific operational
environment. Procedures were developed to emphasize these
CRM elements by incorporating them into SOPs for normal as
well as abnormal and emergency flight situations.

As can be seen in this Manual, ACRM is an ongoing,

dynamic, development process and should not be confused with a
single set of products. The Manual does present some products
of the ACRM training development process, but these are to be
used as examples only and should not be used as a substitute for
the process. Reproducing a briefing card from another airline
will not, by itself, produce the type of organization change that
the ACRM training development process can.

FAA Evaluation of ACRM Training

The Federal Aviation Administration (FAA) has sponsored a
Grant, Analysis of CRM Procedures in a Regional Air Carrier,
conducted by a team including George Mason University and
Subject Matter Experts (SMEs) from a regional airline, a major
airline, and other research organizations. The Grant is in the
process of evaluating the effectiveness of CRM procedures in a
regional airline environment. Both the airline and the FAA are

ACRM Training Manual Page ix

 interested in determining whether the implementation of CRM
procedures can improve overall crew performance. Under the
Grant, the regional airline’s key CRM principles were translated
into procedures that have been implemented through ACRM
training. The regional airline involved in this Grant was
authorized to develop an innovative approach to crew training
and assessment under the Advanced Qualification Program
(AQP).

The results of this Grant have significant ramifications for

flight crew training, specifically in the area of integrated CRM
and technical skill training. The airlines have not had the
capability to perform detailed assessments of CRM skill training,
nor have they had the ability to assess different forms of CRM
training. The results of this Grant provide guidelines for the
training of CRM procedures (see Appendix A for a complete list
of the guidelines) as well as a framework for the assessment of
skill-based crew performance. With this capability to train and
assess CRM performance, airlines can become proactive and
improve training based on the assessment data rather than having
to rely exclusively on accident and incident information.

Key Elements of ACRM Training

Key elements of an ACRM program are the development of
CRM procedures, training of the instructor/evaluators, training of
the fleet crews, and assessment of crew performance based on the
airline’s operational environment. Supporting elements to the
development of ACRM training include the survey forms,
changes to the Flight Operations Manual (FOM), Flight
Standards Manual (FSM), and Quick Reference Handbook
(QRH), the Line Oriented Flight Training/Line Operational
Environment (LOFT/LOE) development process, and the Inter-
Rater Reliability (IRR) process to standardize crew assessment.
These are important supplements and examples are presented in
the appendices.

Organization of this Manual

This Training Manual is organized around the steps an airline
would follow to develop an ACRM training program. Those
steps are based on the key elements of ACRM training
development. The first two parts present background and
introductory information an airline should consider prior to
starting the development of a training program. The next four

ACRM Training Manual Page x

 parts, Part 3 through Part 6, describe the actual steps, starting
with the development of the CRM procedures to their
implementation. The remainder of the document contains
examples to guide and support the development process.

Acknowledgments
Several individuals and organizations were instrumental to
the progress of this Grant and their strong commitment has made
this work operationally relevant. Captain Kim Schultz remains
an active champion of this work and has given greatly of her time
and expertise to the effort. The individual reviewers of this
Manual also made valuable suggestions on how to improve
content and structure.

FAA AAR-100, the Office of the Chief Scientific and

Technical Advisor for Human Factors, provided guidance and the
funding for this Grant. They were instrumental in establishing
the research team and in providing the direction and resources for
this work. Dr. Eleana Edens provided excellent support as the
FAA’s program manager. FAA AFS-230, Advanced
Qualification Program Branch, also provided sponsorship helping
to keep this work relevant within an AQP environment.

Questions and Additional Help

This Training Manual is designed to provide a complete set
of steps to develop ACRM training. If you should have questions
or require additional help, please contact one of the following:

Deborah A. Boehm-Davis Robert W. Holt

HFAC Program/ARCH Lab Department of Psychology

George Mason University George Mason University

MSN 2E5 Fairfax, VA 22030-4444

Fairfax, VA 22030-4444 Phone: (703) 993-1344

Phone: (703) 993-8735 E-mail: bholt@vms1.gmu.edu

E-mail: dbdavis@gmu.edu

Thomas L. Seamster
Cognitive & Human Factors
104 Vaquero Road
Santa Fe, NM 87505
Phone: (505) 466-1445
E-mail: seamster@unix.nets.com

ACRM Training Manual Page xi

 This Page

Intentionally Left Blank

ACRM Training Manual Page xii

Part 1. Extended Table of Contents

Part 1. Introduction to ACRM Training......................................................................1

Background to ACRM Training ..........................................................................1

Advancing from CRM Principles to CRM Procedures....................................1

How CRM Procedures Work ........................................................................2

Example of a CRM Procedure .......................................................................3

Elements of ACRM Training ...............................................................................4

Overview of CRM Procedures.......................................................................4

Overview of Instructor/Evaluator Training ....................................................4

Overview of Crew Training ...........................................................................5

Overview of Crew Assessment ......................................................................6

ACRM Implementation .................................................................................7

Benefits of ACRM Training.................................................................................8

Promoting a Standard CRM for Crews ..........................................................8

Standardizing CRM Training and Assessment................................................8

Expanding CRM Skill Practice ......................................................................9

Focusing on Carrier-Critical Procedures ........................................................10

Part 1. Introduction to ACRM Training

Part 1 of the Manual explains the background of ACRM

training and describes the main elements for developing an
ACRM training program. The last section in this part outlines
the major benefits of developing and implementing ACRM
training. That section should be particularly useful for those
needing justification for implementing ACRM.

Part 1 was written for those not familiar with ACRM training
and its elements. Those with knowledge of or experience with
ACRM can jump to Part 2, Guiding the Organization, or Part 3,
Developing CRM Procedures.

Background to ACRM Training

Advancing from CRM Principles to CRM
Procedures
Most airlines emphasize CRM principles in the form of topics
� CRM principles lack or markers. These principles include topics such as crew
performance coordination, decision making, and situation awareness. These
standards. principles are in the form of recommended practices, and crews
are encouraged to implement these practices when and how they
see fit. The resulting behavior is not always predictable, and
most airlines have found it difficult to specify standards of
performance for CRM principles.

Some operationally relevant CRM principles can be

� CRM procedures can translated into airline-specific procedures that will benefit crew
be made part of performance in certain situations. These CRM-based procedures
existing or new can be integrated with existing normal or non-normal procedures,
procedures. or they can be designed into new procedures.

The identification of CRM procedures normally starts with

existing principles and moves to forming preliminary procedures.
This process includes:

• Reviewing existing CRM principles.

• Identifying crew performance problem areas.
• Reviewing procedures at other airlines.
• Identifying possible procedural changes or additions.

Part 1. Introduction to ACRM Training Page 1

 The identification of CRM procedures should first address the
most important crew performance problems. Once ACRM
� CRM procedure training has been developed and implemented, additional CRM
identification is an procedures can be developed. CRM procedure identification
ongoing process. should be treated as an ongoing process involving the entire
organization in constantly looking for ways to improve crew
performance.

How CRM Procedures Work

ACRM is directed to the training and assessment of CRM
skills within crew training programs. CRM procedures become a
focal point in CRM training, and those procedures allow crews to
practice specific CRM behaviors both in normal and non-normal
situations. The procedures help crewmembers develop a
consistent pattern of crew coordination allowing crews to know
what to expect from each other. The CRM procedures also serve
as a constant reminder to the importance of CRM within the
operational environment.

CRM procedures are an integral part of SOP. CRM

procedures may be integrated within briefings, checklists, and
emergency or abnormal procedures, such as those found in a
QRH, the FSM, or the FOM. These procedures promote good
CRM in consistent ways during appropriate times for normal and
non-normal flight situations.

For example, crew communication and situation awareness

can be improved by requiring specific items in briefings prior to
takeoff. A takeoff brief that requires the crew to address
situationally relevant items critical to that particular takeoff can
be inserted during times with lower levels of workload. By
� A CRM procedure having the takeoff brief address important conditions related to
can be integrated the airport, weather, and performance, the crew discusses those
with the takeoff brief conditions that affect the takeoff. The brief should include
to help crews specific plans for abnormals that may occur during takeoff.
address relevant Having the briefing scheduled for the lower workload period
conditions. prior to taxi helps improve situation awareness and decision
making during a critical phase of flight.

Part 1. Introduction to ACRM Training Page 2

 Example of a CRM Procedure
CRM procedures may be embedded in a range of crew
activities. Some CRM procedures, as the one shown in this
� CRM procedures example, are inserted into required crew briefings prior to critical
may be embedded in times of flight such as approach/landing.
most critical crew
activities. There are a number of cases where an airline may notice a
pattern or crews are being rushed during the approach that results
in the following type of incident (from ASRS Reports):

Due to the proximity of the airport, the high indicated

airspeed, the excess alt and the flight crew’s anticipation of
the ILS 34 approach, the workload of the flight crew was
quite high...The PF descended from the published segment alt
(3500’ MSL) at the 18 DME position to the published straight
in landing MDA of 2000’ MSL. The FAF for the procedure
was at the 13 DME position and the PF’s premature descent
put the aircraft 1500' below the published segment alt.

The following Arrival Brief was designed to help crews

address the main conditions relevant to each arrival. The brief
was placed at the end of cruise phase when crews have more time
to review and plan for the critical conditions.

Arrival Brief

• ATIS / NOTAMS
• Brief Descent Profile
• Statement of Conditions
Select and Prioritize:
Fuel status/delays
Runway conditions
Low visibility procedures
Terrain / MSA
Convective activity
Crosswinds / windshear
Hydroplaning
Aircraft performance
GPWS/TCAS alerts

• Bottom Lines for arrival, approach & landing

• Backup Plan for arrival, approach & landing

Part 1. Introduction to ACRM Training Page 3

Elements of ACRM Training

Overview of CRM Procedures

CRM procedures, instructor/evaluator training, crew training
and assessment, and the ongoing implementation of ACRM form
� CRM procedures the essential elements of ACRM. CRM procedures are the basis
may be added to or of ACRM training. They are SOPs that contain significant CRM
integrated with elements. These Procedures can be used to add required CRM to
normal or non- an airline’s briefings, calls, and checklists for normal and non-
normal procedures. normal conditions.

It has been demonstrated that CRM procedures can be

� CRM procedures successfully developed and fully implemented within a regional
help to brief and airline’s SOP helping to integrate the technical with the CRM
debrief technical and performance in training, assessment, and, most importantly, in
CRM performance the operation of aircraft. CRM procedures are designed to
on an equal basis. integrate CRM with standard aircraft operation and provide
structure to crew management training and assessment. During
training, the procedures become a major focus in CRM skill
development. These procedures help crews form a set of
beneficial and predictable CRM behaviors that increase crew
coordination, communication, awareness, planning, and decision
making.

During crew assessment, CRM procedures help

instructor/evaluators brief and debrief the technical and CRM
performance more objectively. The assessment of a crew’s
procedural performance is more focused than the traditional
evaluation of general CRM markers. This permits a more
accurate understanding of crew performance leading to the
identification and development of better targeted CRM training.

Part 3 of this Manual provides guidance in developing CRM

procedures, starting with the identification of general industry
and own airline needs and moving on to the development and
finalizing of CRM procedures.

Overview of Instructor/Evaluator Training

Training of the instructor/evaluators is the key to combining
ACRM training and assessment into a well-structured training
system. For airlines implementing this approach to crew training
and assessment for the first time, the new methods can seem

Part 1. Introduction to ACRM Training Page 4

 complicated and difficult to assimilate. Therefore, it is important
to identify a few basic training areas that can serve to organize
instructor/evaluator training. For the first year, the main focus of
the instructor/evaluator training can be the CRM procedures, the
LOE or other forms of assessment, and the use of a gradesheet
(such as the LOE Worksheet).

The first part of instructor/evaluator training should provide

instructor/evaluators with an understanding of ACRM. Next,
training can be developed to provide instructor/evaluators with
� Initial instructor/ the knowledge and preliminary skills required to train the CRM
evaluator training procedures and how to brief, administer, assess, and debrief the
should focus on LOE or other forms of ACRM assessment. Then, instructor/
CRM procedures
evaluators should be given ample practice to build up their skills
and their
in standardizing the assessment process using some form of IRR.
assessment.
This practice can be provided initially during the final part of the
basic instructor/evaluator training. Thereafter, IRR should be
refined on a regular basis throughout standardization training in
order to maintain quality control and encourage ongoing
instructor/evaluator participation in the ACRM process.

The evaluator part of the training can be organized around the

� Instructor/evaluator primary IRR tools allowing the instructors to practice developing
need substantial assessment skills by working with real grading sheets, observing
practice to develop
the actual scenarios they will be using, and rating tapes of real
assessment skills.
crew performance. Assessment skills should be trained in a task-
specific context providing the instructor/ evaluators with multiple
observations of the range of crew performance that they will
likely encounter.

Part 4 of this Manual provides guidelines for developing

instructor/evaluator training, including the development of the
introductory modules, instructional skills development modules,
the ACRM assessment modules, and modules on
instructor/evaluator standardization.

Overview of Crew Training

Crew training under ACRM should be viewed as an extension
of existing CRM training with emphasis on the new CRM
procedures. ACRM can provide a number of training
improvements. First, the CRM procedures provide a new focus
for the training emphasizing the most critical aspects of crew
coordination and communication. By merging CRM with SOP,
ACRM training integrates CRM with the technical training,

Part 1. Introduction to ACRM Training Page 5

 giving both aspects equal importance.

A thorough crew training program should be based on specific

behavioral objectives such as those developed under an AQP.
Objectives are essential for training development and ultimately
direct crew performance assessment. Crew training should
present the new CRM procedures in a clear and compelling
manner, and demonstrate how the procedures improve crew
performance. This part of crew training can be based on the
concept of crew effectiveness where individual pilots improve
their coordination by developing CRM skills that lead to overall
improved crew effectiveness.

Part 5 of this Manual provides guidance in developing crew

training including how to develop the introductory modules,
modules that explain crew effectiveness in relation to CRM skills
and procedures, and modules about how crew performance will
be assessed.

Overview of Crew Assessment

Two forms of crew assessment can provide resource
management performance data within the constraints of an
operational setting. First, an LOE-based assessment allows for
� A combination of
the collection of a wide range of crew performance data within
LOE and Line
the carefully designed and controlled event set environment.
Checks provides a
This precise method of crew assessment should be augmented
complete CRM
with a second type of assessment, the Line Checks. Line Checks
assessment.
provide valuable data about the state of the CRM procedures and
overall crew performance on the line. Line Checks, although not
as controlled as LOEs, provide an efficient method for collecting
more general crew performance data.

Crew assessment techniques are an essential part of

� ACRM evaluations instructor/evaluator training and should be based on the
must be reliable and
collection of reliable data. The IRR analysis tools (discussed in
valid.
Part 4 and in Appendix D) are designed to increase that
reliability. Computer-based IRR analysis tools can be used to
inform one or a group of instructors on how they are rating crew
performance in relation to the other raters. The IRR analysis
tools focus on rater standardization by addressing Agreement,
Congruency, and Consistency. Agreement allows
instructor/evaluators to determine how close the ratings are for
each item being rated. Congruency helps individual raters
understand how their use of the rating scale compares with the
total group of raters, and

Part 1. Introduction to ACRM Training Page 6

 Consistency shows how individual raters correlate with the
group.

Crew assessment is discussed in Part 4, dealing with the

development of instructor/evaluator training. ACRM crew
assessment is based on clear standards and the ongoing process of
collecting reliable crew performance data.

ACRM Implementation
Once CRM procedures and training have been developed,
there are a number of activities that will help ensure that ACRM
� ACRM should be is implemented throughout the organization. From experience at
implemented as an one airline it is evident that ACRM should be implemented as an
ongoing process ongoing process and not as a one-shot training and SOP package.
involving the entire ACRM provides a set of steps for improving crew performance
organization. that can be reused to develop additional CRM procedures and
requires the involvement of the entire organization, not just the
training department.

To help ensure successful ACRM implementation, an airline

should make sure the organization, instructor/evaluators, and
crews are ready for the new training process. Prior to ACRM
� Review sessions can implementation, the organization should be involved with
help to keep key training scheduling an announcement of ACRM, and setting a
personnel informed date when ACRM will become company SOP. From an
about ACRM organizational perspective, key personnel should be kept
development. informed of the development progress. Review sessions with
those key personnel can help to ensure that the organization is
informed and supportive of the effort.

Instructor/evaluators represent the front line of ACRM

� Steps can be taken implementation, and steps should be taken to make sure they are
to ensure instructors informed, practiced, and comfortable with the new training.
are informed and Experience has shown that certain activities will help develop
practiced in the new instructor/evaluators into a training and assessment team. These
training. activities include having training sessions where the
instructor/evaluators establish a good level of agreement in their
assessment of CRM performance. Accurate and timely feedback
should be given to instructors prior to and throughout ACRM
implementation. In addition, standardization meetings should be
planned to allow the instructors to voice problems and to work as
a team to identify solutions to those problems.

Part 1. Introduction to ACRM Training Page 7

 A key to ongoing successful implementation is the reporting
and use of crew performance data. Once an airline has
established that they are collecting reliable and stable data, they
should start reporting crew performance data and trends to
appropriate departments within the organization. Different types
of data and formats should be used when reporting to the crew,
the instructors, fleets, or management. When properly reported,
that data will direct changes or additions to the CRM procedures
and modifications to ACRM training.

Part 6 of this Manual provides guidance on how to implement

ACRM within the organization, discusses the important
implementation requirements, and presents considerations for
keeping a standard assessment and collecting usable performance
data.

Benefits of ACRM Training

Promoting a Standard CRM for Crews
Both training and flight operations should benefit from
ACRM. The crews are a major beneficiary in that they are
provided with a standard, proceduralized form of CRM. CRM
procedures promote a predictable form of crew coordination that
is shared and understood by all crewmembers. This results in a
more standard crew performance that helps crewmembers
participate in planning, decision making, and situation awareness.

Research has shown that predictable patterns of interaction,

especially in the area of crew communication (Kanki, Lozito, and
Foushee, 1989) are associated with better performing flightcrews.
� CRM procedures It has been suggested that when communication is more
promote standard predictable it tends to be more reliable and more likely to
crew communication succeed. CRM procedures promote that standard crew
and coordination, communication and coordination which should result in
which will improve improved crew decision making and situation awareness.
crew performance.
Standardizing CRM Training and Assessment
The airline benefits from ACRM through the development of
a standard CRM training and assessment process. CRM
procedures allow instructors to focus their training on key areas
of the operation and allow the evaluators to concentrate on well-
specified areas of crew performance. This promotes a standard
training and assessment environment.

Part 1. Introduction to ACRM Training Page 8

 One complaint about CRM training has been the lack of
� ACRM provides a set objective standards leading to a range of performance and,
of standards for crew ultimately, to substantial variability in the effectiveness of CRM.
performance and its ACRM addresses this problem on two critical fronts by 1)
assessment. providing clear procedures for the crews to follow, and 2) giving
airlines a set of unambiguous standards for the assessment of
crew performance.

A complaint about CRM assessment has been that instructors

are provided with insufficient training and given too high a
workload during LOFT or LOE sessions. ACRM training
ensures that instructor/evaluators are given ample practice to
build up their skills in standardizing the assessment process.
This can be done initially during the final part of the basic
� ACRM’s standards instructor/evaluator training, and should be done on a regular
can reduce instructor basis thereafter under some form of standardization training.
workload during crew With the standard and focused approach that ACRM gives to the
assessment. assessment process, instructors are able to manage their workload
by concentrating on the essential elements rather than trying to
assess many poorly defined concepts.

Expanding CRM Skill Practice

Once ACRM is implemented, the crews are provided with
focused opportunities to practice CRM procedures under normal,
� Frequent execution non-normal, and training conditions. Crews, through the normal
of CRM procedures CRM procedures, are provided with the opportunity to practice
helps crews to specific CRM behaviors every time they fly. This frequent
develop superior practice of learned behaviors promotes the development of CRM
CRM skills. skills, skills that an airline has identified as essential to good
performance within its operational environment.

In addition, crews are provided with the opportunity to

practice good CRM behaviors under emergency and abnormal
conditions when training in flight simulators. By inserting CRM
procedures into an airline’s Quick Reference Handbook or
emergency procedures, crews are given CRM skill practice every
time they follow one of those procedures. As an example, one
airline inserted a preparation and planning cycle into certain
emergency procedures where the malfunction has a significant
impact on future phases of flight. That preparation and planning
cycle provides crews with practice in discussing the critical
conditions and stating a detailed plan for their specific situation.
This focused CRM skills practice under normal and training
conditions is designed to improve crew performance. The

Part 1. Introduction to ACRM Training Page 9

 objective is for crews to develop that higher level of CRM skill to
help them manage real emergency or other non-normal situations.

Focusing on Airline-Critical Procedures

The ACRM development process is airline-specific, because
there is not a single set of CRM procedures that will address the
crew coordination needs of all airlines. This approach to ACRM
� ACRM promotes the development is aware that one size does not fit all organizations.
development of Airlines have different missions, philosophies, and SOP, and
procedures tailored ACRM development helps an airline refine its procedures to best
to specific airlines. meet its unique operational needs. ACRM development is a
process, and that process helps the entire organization focus on
the procedures that will most improve crew performance in the
airline’s operational environment.

This operational focus should be used to involve training and

flight operations in the process of improving CRM.
� Raising CRM to the Traditionally, CRM training has been the responsibility of a
level of SOP relatively small group, or in some cases an individual, within the
magnifies the training department. ACRM training, by addressing procedures
importance of good and fundamental issues of crew performance, extends
crew communication involvement to the entire flight operations, involving those in
and coordination. standards, training, and operations. Raising CRM to the level of
SOP magnifies the importance of crew coordination and activates
all critical parts of the operation in the development, training, and
assessment process.

Part 1. Introduction to ACRM Training Page 10

Part 2. Extended Table of Contents

Part 2. Guiding the Organization ...............................................................................11

Developing Organizational Commitment .............................................................12

Preliminary Organizational Presentations .......................................................12

Ensuring Ongoing Management, Union, and Inspector Commitment..............12

Developing and Maintaining Organizational Links .........................................14

Involving Instructors in the Ongoing Process.................................................14

Integrating ACRM within the Organization .........................................................16

Working with Management ...........................................................................16

Conducting an Organizational Survey ............................................................17

Reporting Results to Organizational Elements ...............................................18

Part 2. Guiding the Organization

Complete organizational involvement is required in order for

ACRM training to be successful. This involvement is needed
prior to the start of ACRM development and should follow
through to implementation and become an ongoing part of the
program. Often the need for ACRM is identified at the training
department level, so a key challenge is to communicate that need
to the rest of the organization and to establish a strong
commitment from top management on down.

Part 2 has been written for individuals who have not

established a CRM training program at an airline, and is based on
some useful lessons learned through working with an airline over
a four-year period. Those with experience in this area can move
on to Part 3, Developing CRM Procedures.

Guiding the Organization

• Developing Organizational Commitment......page 12
• Integrating ACRM within the Organization...page 16

Developing CRM
Procedures

Developing Developing
I/E Training Crew Training

Implementing
and Evaluating
ACRM

Part 2. Organizational Requirements Page 11

Developing Organizational Commitment

Preliminary Organizational Presentations

The need for an ACRM-type training program often is
identified at the training department level. Sometimes that need
is identified by the individual or group in charge of CRM
training, and on occasion by those developing an AQP. Once the
need has been identified, training objectives and supporting data
should be organized into one or more presentations to gain
organizational support.

Your instructor/evaluator group will ultimately act as the

� Consider having front line for the ACRM training program. Consider using a
instructors help group of these instructor/evaluators to help in preparing the
prepare the organizational presentations. At the very least, have
organizational instructor/evaluators review your presentations before you show
presentations. them to management or training department supervisors.

Organizational presentations should include a set of higher-

level objectives, additional lower-level objectives, primary
development and implementation steps with dates, organizational
requirements, and organizational benefits. The higher-level
objectives might include some of the following:

• Standardized form of CRM crew training.

• Standardized form of CRM assessment.
• Improved CRM skill and performance development.
• Improved integration of CRM with technical training.

Additional, lower-level objectives might include:

• Updating critical briefings.

• Updating the Quick Reference Handbook (QRH).
• Improved CRM assessment training for instructor/evaluators.
• Improved reliability of instructor/evaluator ratings.

Ensuring Ongoing Management, Union, and

Inspector Commitment
Once initial organizational involvement has been established,
steps should be taken to further develop and ensure ongoing
management, union, and Principal Operations Inspector (POI)
commitment. Each airline has a different management structure

Part 2. Organizational Requirements Page 12

 and a unique relation with union and POI, but some common
steps can be taken to build a strong commitment for the ACRM
training program.

On the management side, an important step is to develop an

understanding of what will be achieved under ACRM training,
and then provide regular reports to show the progress and trends.
In developing the preliminary understanding it can be helpful to
provide data on current trends in training, and if CRM data is not
� Management should available, use technical data such as maneuvers to demonstrate
be informed about the type of crew and instructor data that will be collected. At
the types of many airlines this may be the first carefully evaluated CRM
questions that can program linked to crew performance and SOP. In such cases,
be addressed by management should be familiarized with the methods that will be
CRM performance used to collect CRM performance data and the types of questions
that can be answered with that type of data. Use the
data.
commitment-building presentations to identify the types of
reports most useful to management. One important outcome of
this process is to develop strong commitment for the ACRM
program from top management on down.

On the union side, consider getting one or more union

representatives involved in the development as early as possible
in the process. In the preliminary meetings it should be
established that ACRM offers a much more complete form of
CRM training that is well integrated with the technical. Further,
crew assessments under ACRM are objective, based on SOP, and
well specified based on observable crew behaviors. Work closely
with at least one representative, and invite them to fly or ride the
new LOE or LOFT. Also invite them to be present at the ACRM
crew training course and at crew assessment sessions such as the
LOE or line check. Invite union feedback and keep them
informed as the program develops.

The working relationship with the FAA’s POI is also very

important. Some airlines have developed ambitious new training
programs only to find that the POI either does not understand the
program or does not see the need for change. The POI should be
included, informed, and consulted as soon as is practical within
the specific organization. If possible, make the POI, as with
union representatives, a part of the process. Take the time early
in the process to explain the need for the new training and make
explicit the approach that will be taken.

Part 2. Organizational Requirements Page 13

 Developing and Maintaining Organizational
Links
Although the need for ACRM training may originate from the
training department, links should be established with other
departments to ensure the ongoing viability of the program. The
� Both formal and main links that should be established and maintained are between
informal the ACRM training development group, the customer fleets, and
communications are the standards department as well. Whether formal or informal, it
essential to is important to have cooperation from the different departments
maintaining good that will be affected by the ACRM training.
organizational links.
The preliminary organizational presentations, mentioned at
the beginning of Part 2, are one of the preliminary ways to
establish links. Once the link has been established, open
communication needs to be developed and maintained. This
should be encouraged by the ACRM development team, both
formally and informally. The formal communication should
include several scheduled reviews allowing the fleet and
standards representatives to see and comment on intermediate
versions of the new CRM procedures as well as the training
materials. Other formal communications may include scheduled
briefings and other meetings to ensure that other departments are
informed of progress and issues in the development process.

Informal communication links are also important, especially

in the maintenance of flexible and strong links. During the
formal communication process, identify individuals from the
fleets and standards who are interested in the ACRM training and
who understand the advantages of cooperation between
departments. Develop these informal links by inviting interested
individuals to the informal discussions where decisions that
affect fleet policy or standards are made. These informal links
should not require additional time or meetings on the part of the
ACRM development team, and they can be essential to getting
good input and cooperation from the different departments. In
addition, identify a champion within the higher levels of the
organization who can lend strong support to the program at key
periods such as initial funding and ACRM implementation.

Involving Instructors in the Ongoing Process

Early and continuous involvement of instructor/evaluators is
the key to a well-implemented and well-received ACRM training
program. The instructors can be very helpful throughout the

Part 2. Organizational Requirements Page 14

 development and implementation process, and they are the main
role models of ACRM for the crews. Ultimately, ACRM training
works best if the program becomes their idea and they are its
main architects.

A key to good instructor/evaluator involvement is to start

early and involve several of the more experienced instructors in
the initial development steps. Consider having one of these early
adopters present the ACRM training program at an instructor
meeting so the audience can concentrate on the benefits of the
� Keep as many new program rather than on the difficulties and issues of change.
instructors as is Keep as many instructors as possible actively involved in both
practical involved in the development and implementation steps.
the ACRM process.
Instructor/evaluators can be very helpful in many of the
ACRM development steps including:

• Identifying own airline CRM training needs.

• Specifying gaps in existing procedures and documents.
• Providing feedback on the prototype CRM procedures.
• Providing input when identifying instructor/evaluator training
requirements.
• Providing SME input to the instructor/evaluator training
course development and LOFT/LOE development.
• Standardizing inter-rater reliability.
• Developing and refining the gradesheets.

Once the instructors have received their ACRM training and

the program has been implemented, there are a number of other
areas where instructors can help:

• Refining instructor/evaluator training/assessment tools.

• Providing ongoing feedback.
• Planning and establishing standardization meetings.
• Identifying instructor and crew performance problems.
• Identifying training areas that need improvement.
• Specifying new ACRM procedures.

Part 4 explains the steps in developing instructor/evaluator

training, and shows how instructors should be involved in many
of those steps. Also, Part 6 presents the implementation and
evaluation process. Again, instructors should remain active, not
only in training and assessing ACRM, but also in being role
models and support-persons throughout the organization.

Part 2. Organizational Requirements Page 15

Integrating ACRM within the Organization

Working with Management

ACRM offers the opportunity to change not only the crew’s
acceptance of CRM, but also the instructor/evaluator and
organizational understanding and involvement. Preliminary
acceptance of ACRM training should be developed into an
ongoing appreciation for the substantive changes being made in
crew performance and the meaningful data that is being reported
back to the different levels within the organization.

The previous section discusses the use of commitment-

building presentations to identify the types of reports best
understood by, and most useful to, management. The reports are
important, but they are just one part of developing an ongoing
working relationship with management. In order to establish a
vital working relationship, an understanding of the benefits of
ACRM training is important, and should be combined with a
strong level of communication between the training department
and airline management.

Airline management realizes the importance of CRM

training, especially at a general level. That is, they understand
that many accidents in commercial aviation have a human factors
or CRM component. What is more difficult to understand is the
relationship between own airline incidents and human factors.
Most airlines do not report possible or actual human factors
causes of incidents, so there can be a tendency to concentrate on a
� ACRM can help crew’s technical problems. ACRM, over time, should bridge this
management move gap in understanding to help management understand the CRM
from a general problems that impact daily operations and the training and
understanding of assessment steps that should be taken to remove those problems.
CRM to an The precise methods used to develop this understanding will vary
appreciation of what from airline to airline, but there are two important elements.
CRM can do for their First, the training department should establish and clearly
operations. represent to management the link between ACRM training and a
crew’s development of CRM skills. Second, a clear
understanding of the ongoing, feedback-driven nature of the
ACRM program should be developed. These two elements are
essential in establishing a good working environment with
management.

Formal reports, discussed in the last section of this part, are

just one dimension of the required communication process.

Part 2. Organizational Requirements Page 16

 Informal communications and frequent updates must also be
worked into the ACRM process. The training department can
start by providing management with preliminary crew
performance data before a quarter of trend analysis data is
available. It should be made clear that the data are incomplete
and that any interpretation is preliminary. The training
department can use these update sessions to identify the types of
questions that are important to management and refine their
understanding of the ACRM training process.

Conducting an Organizational Survey

Conducting an organizational survey during the early stages
of ACRM development can help identify organizational elements
� The organizational that need to be involved in the process. The survey can show
survey can be used which elements understand and want to be involved with the
to introduce ACRM. ACRM program and which will require more communication and
information. In addition, the organizational survey can be used to
increase organizational awareness of ACRM and its benefits in
improved safety, communications, and cooperation.

Consider using the survey development and administration

process as a way to get essential parts of the organization
introduced to the new program. You can ask key individuals
from different departments to provide material for the survey or
have them review specific survey items. Make sure that the
survey addresses issues and concerns of the flight attendants,
dispatchers, maintenance, and other key parts of the operation.

Some of the items that should be considered for inclusion in

an organizational survey are:

• Airline safety climate.

• Communication and cooperation.
• Departmental management and structure.
• Job responsibilities and standards.
• Organizational management.
• Professionalism and job performance.
• Quality and frequency of training.

Part 3 presents the steps to identify own airline needs that are
related to the conduct of the organizational survey. The
organizational survey may be planned and administered during
the identification of own airline needs. Appendix B provides
sample questions that can be used to develop the survey.

Part 2. Organizational Requirements Page 17

 Reporting Results to Organizational Elements
Timely, accurate, and relevant data reporting is one of the
most visible contributions an ACRM program can make to the
various organizational elements. Under ACRM it is possible to
� Different collect a large amount of CRM performance data. In addition,
organizational the IRR tools allow airlines to analyze different dimensions of
elements require instructor/evaluator reliability. The large amount of data
different data and combined with the different forms of analysis can result in an
formatting to fully overwhelming amount of information, so the challenge is to
benefit from ACRM report the essential data in a usable format to the appropriate
reports. organizational elements.

The four main organizational elements to consider are:

• Crews and union representatives.

• Instructor/evaluators.
• Training department management.
• Fleet managers.

Consider reporting on some of the following within fleet data:

• Distribution of overall performance by position (Capt., F/O,

S/O).
• Distribution of technical performance by position.
• Distribution of CRM performance by position.
• Maneuver validation ratings by type of maneuver.
• Distribution of event set performance where there are
problems.
• Overall crew performance trends (quarterly and annually).

Consider some of the following across fleet data:

• Distribution of overall performance by fleet.

• Distribution of technical performance by fleet.
• Distribution of CRM performance by fleet.
• Maneuver validation ratings by type of maneuver and by
fleet.
• Event set performance by fleet where there are crew
performance problems.
• Overall crew performance trends by fleet.

Sections in Part 6 discuss the crew and instructor

performance data reporting process.

Part 2. Organizational Requirements Page 18

Part 3. Extended Table of Contents

Part 3. Developing CRM Procedures............................................................................19

Overview of CRM Procedures Development .........................................................20

Planning for the Main Development Activities .................................................20

Establishing a Broad-Based Development Team ..............................................21

Ensuring Organizational Communication and Coordination..............................22

Procedures Development Requirements.................................................................22

Outlining the Personnel Requirements..............................................................22

Establishing a Development/Implementation Timeline......................................23

Budget Considerations ....................................................................................26

Reporting to the Organization and Union.........................................................26

Identifying Industry and Own Airline Needs ..........................................................27

Defining the Scope of the Needs Analysis ........................................................27

Reviewing NASA and NTSB Data, Reports, and Studies.................................28

Reviewing Airline Safety and Training Reports ................................................29

Collecting Additional Needs Data from the Organization .................................30

Identifying Primary Airline CRM Procedure Needs ..........................................31

Example a CRM Procedure Needs...................................................................32

Specifying CRM Procedures..................................................................................33

Specifying Gaps in Existing Procedures and Documents ..................................33

Linking Needs to Procedures and Documents ..................................................33

Identify Points of Lower Workload for Normal and Abnormal Conditions .......34

Developing Preliminary Procedures .................................................................34

Refining CRM Procedures and Media....................................................................35

Preparing Prototypes of Procedures.................................................................35

Sample Prototype of Arrival Brief Procedure...................................................36

Eliciting Instructor/Evaluator and Other User Feedback...................................37

Presenting Prototype CRM Procedures to the Organization .............................38

Preparing Final Version of Procedures.............................................................38

Getting Fleet-Level Approval ..........................................................................39

CRM Procedures Development Guidelines ............................................................40

Guidelines for Identifying Own Airline Needs ..................................................40

Guidelines for Specifying CRM Procedures .....................................................40

Guidelines for Refining CRM Procedures and Media .......................................41

Part 3. Developing CRM Procedures

Part 3 of the Manual presents the steps for developing CRM

procedures. This part has been written for those who need to
develop a set of CRM procedures at an airline. The development
process starts with identifying industry trends and own airline
needs and goes through to finalizing the actual procedures and
the media that will be used to disseminate those procedures. Two
keys to a well-managed development effort are careful planning
and good communications with the rest of the organization. The
last section in this part summarizes the CRM procedure
development guidelines.

Guiding the
Organization

Developing CRM Procedures

• Overview of CRM Procedures Development..page 20

• Procedures Development Requirements........ page 22
• Identifying Industry & Own Airline Needs.... page 27
• Specifying CRM Procedures.................... page 33
• Refining CRM Procedures and Media.......... page 35
• CRM Procedures Development Guidelines.... page 40

Developing Developing
I/E Training Crew Training

Implementing
and Evaluating
ACRM

Part 3. Developing CRM Procedures Page 19

Overview of CRM Procedures Development Activities

Planning for the Main Development Activities

� All airlines should plan The development of CRM procedures is based on airline-
these three steps: 1) specific operations, but there are several basic steps that should
identifying CRM be considered by all airlines. Those steps are presented in this
needs, 2) specifying part, and include the identification of CRM needs, the
procedures, and 3) specification of the procedures, and the refinement of those
refining procedures. procedures.

Identification of CRM needs is based on the most important

crew performance problems at an airline. Training departments
may not know how to start looking for these problems, so it can
be useful to first review industry-wide reports to learn where
other airlines have the greatest crew performance problems.
Industry sources of information can help locate general problem
areas that are more likely to be accepted by an airline which
might otherwise be defensive about its own pilot population.
When several general areas have been identified, the airline can
� It is vital for an airline then look at its own safety and training reports to start
to identify a few pinpointing its own specific problems in those areas. The
operationally identification of CRM problem areas is ongoing, as initial
significant CRM- problems are resolved and new problems surface. To ensure
related performance initial success and long-term acceptance, it is vital for the airline
problems. to identify a few operationally significant CRM-related
performance problems for procedure development. Once the
problems have been isolated, they are reviewed to determine if
they point to operating standards, training, or a combination of
those needs.

Specification of CRM procedures uses the airline

performance problems and CRM needs to determine the
procedures that will address them. This is the first pass at
specifying the new procedures, so the development team should
be encouraged to examine a range of possible procedures and
consider a number of different areas where these procedures
could be implemented. This is a major step for the development
team, and planning considerations include making sure the team
members have sufficient time and access to airline resources so
they can develop a credible set of initial CRM procedures.

The refinement step is based on a comprehensive review of

the CRM procedures. Individual procedures, along with other
options, must be reviewed by all departments affected by the

Part 3. Developing CRM Procedures Page 20

 changes including training, flight operations, and fleet

management departments. The refinement process is essential
for improving the CRM procedures and for gaining acceptance
across departments. These three steps require forming a broad-
based development team and continued communications with the
organization; elements are that discussed next.

Establishing a Broad-Based Development

Team
The training department, in cooperation with flight standards,
will likely perform most CRM development, but additional
members should be considered in forming the development team.
In smaller airlines, there may be only one or two individuals who
have the time to work on CRM procedure development, but it can
be better to have more individuals working as a team rather than
only one person doing all the work.

There are several reasons to consider a team approach to

developing CRM procedures and the ACRM training as well.
� Consider forming a First, with just one person you risk losing the entire effort if that
CRM procedures person moves to another department or airline. Also, when the
development team development is performed by just one person, it can be more
rather than having difficult to establish the links to other departments and ACRM
just one developer. can be perceived as just a one-person effort rather than an
organizational effort.

The functions of the CRM procedures development team

include developing the CRM procedures, building links
throughout the organization, and providing input for the ACRM
training development. At smaller airlines it is more likely that
the CRM procedures development team will go on to develop
much of the ACRM program. At larger organizations, where
functions are more specialized, the procedures development team
will become a resource for the individuals who develop instructor
and crew ACRM training.

Working within a team can facilitate this development

process. A successful team can help to distribute the effort
beyond the training department to other areas of the organization.
Also, a team can ensure continuity in the development effort,
even when some of the members are not present. Finally, a
successful team can achieve more than just the sum of its
individual members, thus providing the airline with a more
efficient way of developing the CRM procedures.

Part 3. Developing CRM Procedures Page 21

 Ensuring Organizational Communication and

Coordination
Good communication with key elements of the organization
ensures a well-coordinated effort. Communication starts at the
beginning of the process with activities such as the preliminary
presentations (see Part 2) and continues throughout the ACRM
program. The CRM procedures development team should
establish formal as well as informal communications to achieve
ongoing, open communication with the main elements of the
organization.

Formal communication should include several scheduled

reviews, allowing the fleet and standards representatives to see
and comment on intermediate versions of the new CRM
procedures. Other formal communications may include
scheduled briefings and meetings to ensure that other relevant
departments are informed of progress and issues in the
development process.

Informal communications are important to maintaining

flexible and strong links. Inviting interested individuals to attend
the working sessions where decisions affecting fleet policy or
standards are made enhances communication. Those individuals
should participate in the process and be encouraged to contribute
their ideas to the development of CRM procedures. This broader
audience will ensure that the CRM development team gets input
from the rest of the organization and identifies potential problems
during the design phase rather than later in development.

Procedures Development Requirements

Outlining the Personnel Requirements
CRM procedure development has personnel, time, and budget
requirements. Lack of planning in any one of these areas can
delay or terminate the development process.

As indicated in the previous section, there can be a substantial

range in personnel requirements from just a single individual to a
team with members from key organizational departments. Even
small operations should consider the team approach that team
personnel requirements will be considered first. Developing the
first set of CRM procedures can best be managed as a small and
efficient project, and the team composition should match that

Part 3. Developing CRM Procedures Page 22

 objective. A CRM procedures development team can be formed

with three to eight members; larger airlines can work with a
larger team. However, even in the major airlines the team should
be kept manageable by not growing too large.

The composition of the team will depend on the

� In addition to having organizational structure and the requirements of the development
one or more people plan, but certain departments should be able to provide valuable
from the training team members. In addition to the training department with its
department, consider instructors, consider someone from flight standards and someone
having someone from the safety department. If you are considering concentrating
from flight standards the development effort with one fleet, an approach that can make
and flight safety. procedure development more manageable, have a representative
from that fleet on the team.

Additional team formation considerations include selecting

individuals who work well together. Concentrate on those
personnel who have show a strong interest in the project and have
experience in procedures development. Finally, have a limited
timeline (see the next subsection) and clear plan to help in your
recruitment. It is much easier for individuals to join a team when
its purpose is clear and its product will enhance the organization’s
performance.

Establishing a Development/Implementation
Timeline
A realistic and efficient timeline can help in team formation
and in getting the project off to a good start. Preliminary
timelines can be based on the development team composition and
planned activities. That preliminary timeline for those early
presentations can be used to elicit team participation. Once the
team membership has been finalized, the timeline should be
reviewed to ensure that it is compatible with the member
schedules.

Time is a limited commodity in aviation, so restricting the

amount of time needed for the development process is an
� When estimating the important aspect of planning. At least two areas should be
timeline, limit considered when making the best use of development time. First,
planning to essential
planning should be limited to essential activities. Second, actions
activities and do not
should be assigned to team members in such a way that no one
overload the
becomes overloaded and slows down the team.
development team
members.

Part 3. Developing CRM Procedures Page 23

 In planning activities, consider how much time it will take the

development team to identify the needs, specify the procedures,
and refine them. If an airline already is collecting crew
performance data and has trend data for the last few years, then
that airline will not need to spend much time identifying crew
performance problem and needs. For needs identification, you
may be able to budget as little as 40 hours or as much as several
hundred hours. The second activity, specifying the CRM
procedures, has a narrower range, especially if an airline limits
the number of procedures to introduce at the start. With two to
six procedures, a good number to start with, an airline should be
able to develop the initial procedures taking about 20 hours per
procedure. The refinement process is very important and,
because it requires organizational input, may require several
hundred hours, especially in larger airlines.

In addition to estimating the amount of time required for

each step, there are a number of other considerations. Once you
have an estimate of total time, consider assigning and scheduling
the activities in such a way that some items can be performed in
parallel. Also, schedule the key products and milestones in a way
that the team can meet them at the same time so the organization
will have a sense that progress is being made.

Depending on team size and the number of hours required for

each activity, CRM procedures development should take between
one and three months. Time less than a month risks not
completing some of the essential activities, and taking much
longer than three months risks losing the interest and
commitment of the organization. A sample ACRM development
timeline might look like the Sample Timeline on the next page.
Each organization will establish a timeline to best fit the pace of
its individual departments, but a 12-month timeline is a
reasonable starting point.

Part 3. Developing CRM Procedures Page 24

 SAMPLE ACRM DEVELOPMENT TIMELINE

Month 1 Present to Organizational Elements

GUIDING THE Conduct Organizational Survey

ORGANIZATION
Month 2 Establish CRM Procedures
Development Team

Month 3 Determine Carrier Needs

DEVELOPING
Develop CRM Procedures
CRM
Month 4 PROCEDURES
Review CRM Procedures

Month 5 Establish Training Development

Team
DEVELOPING
I/E Develop Instructor/Evaluator
Month 6 TRAINING Training

Develop LOFT/LOEs and

Month 7 Gradesheets
Train Instructor/ Evaluators

Month 8 Develop Crew Training

DEVELOPING
CREW TRAINING
Train and Refine Gradesheets
Month 9
Announce ACRM Program

Month 10 Train Crews

IMPLEMENTING
& Implement CRM Procedures
Month 11 EVALUATING
ACRM
Hold Instructor/Evaluator
Month 12 Standardization Meetings

Evaluate Crew Performance

Part 3. Developing CRM Procedures Page 25

 Budget Considerations
The procedures development process itself can be
accomplished with a relatively small budget, but associated
elements such as changes to operations documents and training
can result in substantial expenses. With careful planning,
development and establishing the CRM procedures can be
achieved efficiently by reducing or eliminating some of the larger
expenses.

For major airlines, one change in a procedure can result in the

need to reprint one or more pages of 40,000 manuals. That could
become a very large cost, and those who do not fully support or
understand the need for CRM may use it as an argument to limit
or terminate the program. However, a strong case can be made
that procedural changes can be integrated into a scheduled
manual update for little or no additional cost.

Major airlines have scheduled Flight Operations Manuals

(FOM) updates, and if the changes proposed by the CRM
procedures are coordinated with those updates, the cost can be
� Coordinating the
greatly reduced. In addition, documentation that is normally kept
introduction of new
in the cockpit deteriorates over time and may have other reissue
CRM procedures
requirements. Coordinating checklist and Quick Reference
with scheduled
document updates Handbook (QRH) changes with the update cycle will further
can greatly reduce reduce costs.
your costs.
Additional considerations for managing the budget include
integrating some of the team members’ development time with
their normal responsibilities. New activities should not be
assigned without providing the hours, but ways often can be
found to replace some of the existing activities with the CRM
procedures development work. Also, the development process
should be kept on schedule; delays that can increase costs in time
and resources should be anticipated. CRM procedures
development can be the least expensive set of activities (as
compared with ACRM training). It is important to demonstrate
fiscal responsibility as the first step in the ACRM process.

Reporting to the Organization and Union

Part of a successful development process includes good two-
way communication between the CRM procedures development
team and the rest of the organization. If the development team

Part 3. Developing CRM Procedures Page 26

� The organization includes representatives from key parts of the organization, the
must understand the need for additional communication is substantially reduced. In
objectives of the all cases it is important that the organization understand the
ACRM program ACRM objectives and the team’s plans before or as work begins,
before or as the and it is equally important that proposed CRM procedures are
development work presented to all key departments during the refinement step.
begins.
On the management side, it is important to communicate
what the CRM procedures development team will accomplish
and then schedule reviews to show management what has been
developed. On the union side, it is important to consider
involving one or more of the union representatives in the
development of the procedures. At the very least, union
representatives should participate in the review process. In
general, union feedback should be solicited, and representatives
should be kept informed as the procedures are developed.

Identifying Industry and Own Airline Needs

Defining the Scope of the Needs Analysis

The CRM needs analysis is used to identify the most
important crew performance problems, and then determine what
form of intervention will best correct those problems. For
training departments that do not have data pointing to specific
problems, it is useful to review industry-wide reports to learn
where other airlines have crew performance problems. Industry
sources of information can help locate those general problem
areas more likely to be accepted by an airline that might
otherwise be defensive about its own pilot population. The next
subsection presents some sources that can be used for this
purpose. Then, when several general areas have been identified,
the airline can look at its own safety and training reports and start
pinpointing its own specific problems in those areas.

If an airline has crew performance and trend data for the last
few quarters or years or has good safety and incidents
information, it does not have to spend much time at the industry
level. Such airlines can focus on their own documented problem
areas. Once the problem areas have been identified at the
industry or own airline level, the airline can perform a
preliminary analysis to better identify the source or sources of the
problem. Once the problem or problems have been pinpointed,
possible solutions, in the form of CRM procedure and/or training,
should be evaluated.

Part 3. Developing CRM Procedures Page 27

 The needs analysis can be planned based on an understanding

of the airline’s background, data, and endorsement of CRM
performance. If an airline has good data and a strong CRM
program, the needs analysis should require a smaller level of
effort. If, on the other hand, the airline has no data and/or little
experience with CRM, the needs analysis may require a
substantial effort so that the results can be understood and
accepted by the organization.

It should be remembered that the identification of CRM

problem areas is ongoing, and it is vital that the airline identifies
operationally significant problems at the start of the process.
Once procedures to address those problems have been
implemented, the airline should be alert to new crew performance
problems that point to the need for additional procedures.

Reviewing NASA and NTSB Data, Reports, and

Studies
Airlines may want to obtain a snapshot of industry performance
problems before concentrating on their own airline. A number of
sources provide general accident trends as well as specific
incident information. Airlines that do not have detailed CRM
performance data can use accident analysis reports to identify
general problem areas, and then use specific aircraft accident and
incident reports to obtain the details.

In 1994, the National Transportation Safety Board (NTSB)

published a Safety Study that reviewed major U.S. airline
accidents from 1978 through 1990 (see NTSB, 1994). The study
was based on an analysis of performance and operational
environment characteristics associated with 36 accidents and 1
incident, and their analysis showed patterns that can help an
airline identify some crew performance problem areas. For
example, in about 80% of the accidents the Captain was the pilot
flying. In addition, where data was available, Captains had been
awake on average 10.5 hours, and First Officers about 10 hours.
The safety study analyzed both priory errors, such as aircraft
handling, communication, and situational awareness, and the
secondary error of monitoring/challenging. A review of the
Safety Study can help airlines identify a number of problem areas
related to their own operations.

Part 3. Developing CRM Procedures Page 28

 Additional NTSB aircraft accident reports, such as the 1993

Uncontrolled Collision with Terrain at Guantanamo Bay,
highlight some specific crew performance problems. Also, a
large number of Aviation Safety Reporting System (ASRS)
incident reports can be searched for specific operational
characteristics such as type of aircraft, flight conditions, and type
of anomaly. Once an airline has identified several problem areas,
an ASRS database search is useful to determine the reported
incidents in those areas (see Appendix G for some Sample ASRS
incident reports).

Reviewing Airline Safety and Training Reports

With the problem areas identified, either from the analysis of
industry or own airline trends, it is now time to take a closer look
at the safety and training data. The purpose of this review is to
pinpoint specific incidents or crew performance trends that will
help the CRM procedures development team better understand
the nature of the performance problem at their own airline.

Crew performance problem areas may be in several forms.

Some problems can be grouped by phase of flight, such as
runway incidents during taxi to takeoff. A review of this type of
incident should provide the main causes for such incidents. For
example, some type of distraction may be taking place, or there
may be a lack of monitoring and backup. Other problem areas
may be associated with a specific aircraft system or subsystem,
and the review will help pinpoint causal factors. For airlines
collecting more detailed CRM data, the problem area may be in
decision making or situation awareness. This review should
identify probable causes both proximal to the problem such as
“task overload,” and possibly more distal causes such as “failure
to divide flying duties.”

When existing data or reports do not point to a clear cause,

the development team should consider interviewing one or more
individuals from the department collecting the data or producing
the report. The structured interview can be a more efficient way
of collecting causal data and, if successful, should reduce or
eliminate the need to conduct a survey such as that described in
the next subsection.

Part 3. Developing CRM Procedures Page 29

 The safety and training review process can be used to

familiarize the training and safety departments with the ACRM
program and the performance problem identification process. If
structured interviews are conducted, good introductory material
should be provided to inform the interviewee about the ACRM
project and the identification of CRM procedures. The interview
process should be designed to be a positive and informative
experience, leaving the interviewee inclined to help in future
activities.

Collecting Additional Needs Data From the

Organization
After a review of what has been collected to this point, a
decision should be made about the evidence for CRM
performance problems. If the data and information can be used
to establish several patterns of crew performance problems and
there are likely causal factors, no additional data collection is
required. If, on the other hand, the data do not show a pattern, or
if there are a number of performance problems with no particular
areas of concentration, it may be useful to collect additional data
via some type of needs questionnaire.

This is a more focused effort than the organizational survey,

but results from the organizational survey can be used in the
design of the needs questionnaire. Those results can be used to
determine which departments or groups of respondents were best
able to provide CRM performance data. The organizational
survey items also should be reviewed to see if any of them could
be used in the needs questionnaire. If a questionnaire is required,
there is a good chance that the airline does not have very detailed
or conclusive CRM data. The needs questionnaire should be
designed primarily for instructors, evaluators, and check airmen,
and should investigate at least two areas: CRM performance
problems by phase or sub-phase of flight, and performance
problems by CRM topic or element area (see Appendix B for a
sample Instructor/Evaluator Questionnaire).

The phase of flight portion of the survey can ask for crew
performance problems and possible causes by phase or flight or
sub-phase. Respondents can be asked to rank or rate each phase
of flight based on frequency or severity of CRM performance
problems. The CRM topics portion of the survey should ask
about performance problems with crew decision making,
inquiries and assertiveness, leadership/followership, and
preparation/
planning. Again, respondents can be asked to rank, rate, and/or

Part 3. Developing CRM Procedures Page 30

 provide specific information for each topic (see Appendix B). As

with the data review, use this additional data collection effort to
further involve the organization. Consider asking individuals
from different departments to provide material for the
questionnaire or have them review a draft. This helps to ensure
that the survey will address a broader set of issues, including
those that concern flight attendants, dispatchers, maintenance,
and other key parts of the operation.

Identifying Primary Airline CRM Procedure

Needs
The preceding activities will provide you with sufficient data to
pinpoint the CRM problem areas and provide possible causes.
This last step is used to choose those problems that point to the
need for one or more CRM procedures. The primary
determination to be made is whether a performance problem
would be resolved best by training, equipment redesign, or
through new or modified procedures. Working with airline needs
is at a higher level than SOP; it involves understanding an
airline’s philosophy and policy to determine whether a procedural
solution is the right one for that specific operation and
organizational climate.

Determining whether a CRM procedure will alleviate a

specific performance problem in a specific organizational
environment requires reviewing all available data and
information about the CRM problem. Industry data, whether
from NTSB, NASA, the ATA, etc., can confirm the key issues,
such as phase of flight, pilot flying, operational conditions, and
general types of errors. Then airline-specific data can be used to
pinpoint where a CRM procedure might be implemented. At this
point it may be helpful to review procedures from other airlines
to determine if they have integrated CRM at those key points in
the SOP.

The exact nature of a procedure is not specified at this point,

but the development team needs to specify where a procedure
should be considered for insertion into SOP and be clear how a
CRM procedure would substantially improve crew performance.
The CRM procedures development team should be aware of the
benefits and possible liabilities of each procedure they plan to
add. Their role of identifying areas for new or modified
procedures should be tempered with the realization that too many
procedures, or ones in the wrong place, can be as dangerous as no
procedures at all.

Part 3. Developing CRM Procedures Page 31

 Example of a CRM Procedure Needs

In order to pinpoint the types of briefing and workload
problems that crews were having, one carrier sent out an
� CRM procedures instructor/evaluator questionnaire similar to that in Appendix B.
may be embedded in They received back valuable information about the specific types
most critical crew of problems observed in recurrent training. For example,
activities. "Briefing - sets tone, calls for questions, participation
encouraged, states how SOP deviations will be handled," was
ranked as the number one or two problem by a number of
instructors who provide the following remarks:

INSTRUCTOR ONE REMARKS: BRIEFINGS

Briefings I have seen are typically too "general" &

often fail to specifically address how a problem should
be handled. Also, I think the briefings to flight
attendants are usually cursory and often totally
inadequate.

INSTRUCTOR TWO REMARKS: BRIEFINGS

It is rare that any crew gave good briefings, therefore

there were many unanswered questions.

This same group of instructor/evaluators identified

"Workload/distractions avoided - Overload in self and others
reports. Task prioritized to deal with primary flight duties first,
recognize distractions," as another important problem. Their
specific comments read:

INSTRUCTOR THREE REMARKS: WORKLOAD

Crews can often be distracted by an ATC call or some

other non-related distraction. I think the main reason
for this is failure to distribute and delegate specific
duties.

INSTRUCTOR FOUR REMARKS: WORKLOAD

Crew usually performs too fast. PNF usually over

loaded.

Part 3. Developing CRM Procedures Page 32

Specifying CRM Procedures

Specifying Gaps in Existing Procedures and

Documents
With the problems, their location, and probable cause
specified, the next step is to propose one or more procedures to
address the problems. Working with the location and probable
cause, it is possible to look at the existing briefs, documents, and
procedures used in that immediate environment and determine if
a change or addition would address the performance problem.

If nothing in the current SOP addresses the specific CRM

performance problem, then there is a gap. It is also possible that
an inconsistency in procedures is actually causing the problem.
A problem with an existing procedure, such as distracting or
confusing crews in certain conditions, may also be causing a
problem. When reviewing existing SOP and documentation,
look for problems with existing procedures and lack of
consistency, as well as gaps, where the performance problem is
not being addressed. Identify the procedural causes or obstacles
to better crew performance.

Linking Needs to Procedures and Documents

For a specific performance problem, there may be several
possible systemic causes, whether gaps or inconsistencies with
existing SOP. The objective is to link the performance problem
to the most likely cause in the operational system. That linkage
should be based primarily on the need identified, based on the
organizational environment. Although ACRM emphasizes the
implementation and training of CRM procedures, developers
should keep in mind that CRM procedures are not always the best
solution. CRM procedures do offer a good solution when they
match an organizations needs based on its philosophy and
policies.

An example of modifying procedures is the case where crews

may have problems with workload management when executing
specific emergency procedures. The CRM procedures
development team might consider a range of strategies to address
that problem. First, the procedure in question could be reviewed

Part 3. Developing CRM Procedures Page 33

 to determine if it could be simplified. The team could also look

at the division of crew responsibilities. It may be that assignment
of specific duties at the start of the procedure would help in
managing workload. If such an approach is compatible or can be
made compatible through policy modification, then that CRM
procedure links well with the organization and has a good chance
of improving the workload management problem.

Identify Points of Lower Workload for Normal

and Abnormal Conditions
Looking beyond the exact location of the problem is key to
� Look prior to the finding the optimum solution. The point where the CRM
location of the performance problem surfaces may not be the best place to insert
performance a procedure. In many cases, crew performance problems become
problem when apparent at times of relatively high workload, and adding a
considering where to procedure at that point would only increase the crew’s workload.
insert the procedure. The CRM procedures development team should look at points in
time prior to the buildup in workload to identify periods of lower
workload where a procedure would be more effective.

At the very least, the appropriate phase of flight should be

examined to determine the points of lower workload. In a few
situations the team may have to look at the previous phase of
flight, especially if the performance problem tends to surface at
the start of the landing phase. In such cases, the approach phase
should be reviewed to determine if there are points of low
workload where a CRM procedure could be inserted.

Developing Preliminary Procedures

With the location for the procedure identified, the next step is
to consider the appropriate form of procedure. The main forms
include briefs, calls, checklist items, guides, flows, non-normal
procedures, and other forms of quick reference items. One of
these forms should be sufficient to address most problems, but
there may be cases where the integration of two forms is less
intrusive and provides a better fit with the airline’s SOP.

In selecting the form of the procedure, consider the airline’s

overall policy and approach to standardization. For example,
some airlines place a greater emphasis on specific checklist
items, and, consequently, that may be a good format to consider.
Another airline may place greater emphasis on the flows and, for
that airline, consider introducing a CRM element into the flows.

Part 3. Developing CRM Procedures Page 34

 In a few cases, departing from an airline’s policy will actually

improve crew performance. When an airline places an emphasis
on a specific format but that format is heavily used at the needed
point in time, then consider integrating the CRM procedure into
another form.

Once the form and location are set, the content of the
procedure needs to be developed. For the first pass, all that is
required is a verbal description of the procedure with examples of
the type of wording that should be considered. In the next step,
actual prototypes are developed.

Refining CRM Procedures and Media

Preparing Prototypes of Procedures
At this point, the CRM procedure development team has a
written description of the preliminary CRM procedures.
However, clear examples or prototypes are needed to
communicate those procedures to other parts of the organization.
The prototypes are more than just a written description of the
procedures; the development team should prepare mockups of the
sample checklists, guides, or parts of manuals to clearly show
what these procedures will look like.

Prototype development is an important part of the refining

process, where the development team interacts with a range of
users to determine the best form and content for the CRM
procedures. This last step should be iterative, with the feedback
from each review being incorporated into the design to achieve
one or more CRM procedures that will be adopted by the users to
improve performance.

What is required for this step is a paper prototype of the

actual checklist, brief guide, QRH, or other form of CRM
procedure (see Appendix I for Sample QRH and Briefing Guide).
Even if the final form will be an electronic display, such as the
electronic checklists, it is not necessary to develop the prototype
in the final media unless it can be done relatively easily and
efficiently. The prototype should reflect the content and format
of the proposed CRM procedure and, in some cases may include
alternative representations. Documentation for the prototype
should also include an explanation of why the CRM procedure is
being proposed and what performance problem is being
addressed.

Part 3. Developing CRM Procedures Page 35

 Sample Prototype of Arrival Brief Procedure

The sample prototype does not have to be the finished
product, but it does need to be presented in sufficient detail so
that the development team can obtain meaningful feedback from
instructor/evaluators and other important groups within the
organization.

The following prototype was an earlier version of the Arrival

Brief presented on page 3 of this Manual. In comparing the two,
it can be see that a number of items were eventually added and
� When reviewing a the order of items to prioritize were changed. All of those
prototype, ask changes were due to comments from those who had carefully
reviewers to give reviewed the prototype. Encourage reviewers to mark directly on
feedback in whatever the prototype or generate a new example. In other words, give
form they are the reviewers freedom to provide feedback in whatever form they
comfortable with are most comfortable with.

PROTOTYPE
Arrival Brief

∼Statement of Condition
Select and Prioritize:
Runway conditions
Low visibility procedures
Hydroplaning
Crosswinds/windshear
Terrain/MSA
Aircraft performance
Convective activity
GPWS/TCAS alerts
Fuel status/delays

∼Bottom Lines
∼Backup Plan

Part 3. Developing CRM Procedures Page 36

 Eliciting Instructor/Evaluator and Other User

Feedback
Two forms of feedback should be collected during this
prototype refinement stage. The first form is feedback from the
users, both pilots and instructors. This is detailed feedback, and
is generally best collected first. Once the prototype has been
refined through user feedback, the second form of feedback to be
collected is the more general organizational feedback discussed
in the next subsection.

Prototype presentations to the users allow the development

team to collect informal comments during the presentation, and
more formal, quantitative feedback through a form that can be
handed out or administered during the user feedback sessions.
The user feedback sessions should be designed and scheduled so
that the development team can collect good data in an efficient
manner. Ideally, airlines would be able to perform usability
testing, where the procedures are evaluated in some operational
or simulated context. That is not possible in most cases, so
working with small groups provides a good alternative. If
possible, the sessions should be scheduled for five to ten
individuals who are likely to work well together. Working with
too small a group (less than five) is less efficient, and the
individuals are less likely to be stimulated by a wider range of
comments. Working with too large a group (substantially more
than ten) is more difficult to manage, and the feedback will likely
cover a broad range of topics but not in depth.

In scheduling these sessions, there are a number of additional

considerations. If instructors and crews are very busy and it is
difficult to schedule a meeting, consider scheduling the prototype
feedback sessions as part of other planned meetings. If resistance
is likely from a specific group, consider having a separate session
with that group addressing their issues as soon as possible within
the feedback process.

The feedback sessions should have two parts. The first part is
a more general presentation of the CRM procedures and their
rationale, followed by encouraging users to ask questions and
make general comments. Once the group has a good
understanding of the procedures and their purpose, a form can be
administered to ask for comments or ratings about the different
aspects of the procedures. Items may include questions about the
effects of a proposed procedure on workload, ease of

Part 3. Developing CRM Procedures Page 37

 understanding the procedure, how the procedure interfaces with

the rest of the SOP, possible problems, and how the procedure
might be improved. In most cases the development team is less
interested in consensus and more interested in individual
feedback, so the forms should be completed individually and not
as part of the group discussion.

If formal user feedback sessions are not possible, consider

holding informal small group or individual sessions as pilots and
instructors are available in the training center or flight operations.
Obtaining meaningful feedback from a cross section of pilots and
instructors, either formally or informally, is critical.

Presenting Prototype CRM Procedures to the

Organization
Similar to the initial presentations, the development team
should present the refined prototype CRM procedures to the
essential elements of the organization. These prototype
presentations serves three primary purposes: 1) familiarizing the
organization with the procedures while collecting general
comments during the presentation; 2) involving the key players in
the organization in the development and feedback process; and 3)
checking on other necessary changes in training, management or
documents that should be made to fit the CRM procedure.

The primary objective is to explain the CRM procedures and

encourage organizational feedback, which tends to be more
general than the user feedback discussed in the previous
subsection. The presentations are similar to those made to the
users, but with fewer operational details and more process details.
Emphasis should be on the need for the procedures, the
development process, and the feedback process. Again, the
development team should be prepared to collect the general
comments made throughout the presentation. If difficulty in
obtaining organization support is anticipated, the development
team should consider designing a feedback form that can be
distributed during the organizational presentations. The form
should be designed to collect data in areas that may be at issue
within the organization. The results of such a form should prove
helpful in resolving specific organizational controversies or
concerns.

Preparing Final Version of Procedures

Both the general and specific feedback should be considered
in preparing the final version of the CRM procedure. With most

Part 3. Developing CRM Procedures Page 38

 of the user comments and issues having already been made, this
activity involves reviewing the general comments at the
organizational level to determine whether additional changes are
required. If there are a large number of general comments,
consider grouping them into categories and then attending first to
those categories with the greatest number of comments.

This can be used as an opportunity for final revisions and, in

the case where general comments conflict with the more specific
user data, to give more weight to the user feedback. Finally, if
some issues cannot be resolved by an examination of the
development process and feedback, consider presenting those
issues to a small group of procedure design experts. This can be
done through a letter and/or form showing the specific CRM
procedure and asking for comments about the issues.

The final version of the procedure should be prepared in

camera-ready format so they are in the precise form that will be
sent to the printers or publishers. In addition to the Camera-
Ready Copy (CRC), prepare a specification sheet stating on what
material the CRC will be printed, what type of additional
punching, tabbing, or binding will be used, and related
specifications such as color and size of printing job.

Getting Fleet-Level Approval

The fleet-level approval process is an essential step that will
vary depending on organizational requirements. If the
organization has been included in the CRM procedures
development process, fleet approval should be a routine matter of
following the submittal process. This is one reason why it is
important to have the organization and affected fleets involved
from the beginning. Some considerations include whether it
would be helpful to prepare a fleet-specific presentation, what
additional materials should be included to help the fleet approval
process, and who from the CRM procedures development team
will be the point of contact for a specific fleet. In larger
organizations, different approaches may be required for different
fleets.

Informal contacts with fleet personnel can be very useful

during this final activity in the development process. Supporters
within the fleet can ensure that the approval process is kept on
track and can help answer fleet-specific concerns or issues.

Part 3. Developing CRM Procedures Page 39

CRM Procedures Development Guidelines

Guidelines for Identifying Own Airline Needs

1) The identification of CRM problem areas is ongoing, and to
ensure the long-term success of the ACRM program, an airline
should identify a few operationally significant CRM problems at
the beginning of the development process.

2) Airlines that do not have detailed CRM performance data

should use industry accident reports and incident summaries to
identify general problem areas, and then the airline should use
specific aircraft accident and incident reports to obtain the details.

3) When existing airline data or reports do not point to a clear

CRM cause, the development team should consider interviewing
one or more individuals from the department that collected the
data or produced the report.

4) The needs survey should be designed primarily for instructors,

evaluators, and checkairmen, and should investigate at least two
areas: 1) CRM performance problems by phase or sub-phase of
flight, and 2) performance problems by CRM topic or element.

5) The CRM procedures development team should be aware of

the benefits and possible liabilities of each procedure they plan to
add. The team’s mandate to identify areas for new or modified
procedures should be tempered with the realization that too many
procedures, or ones in the wrong place, can be as problematic as
not adding any new procedures.

Guidelines for Specifying CRM Procedures

1) For initial procedure development, an airline should first
identify weaknesses in existing procedures and then review
airline philosophy and policy to clarify those needs. Based on
that information, an airline should then identify possible locations
for the new procedure as different forms (e.g., briefs, checklists,
etc.) are being considered. Once that is done, the airline should
work on the procedure’s content.

2) When reviewing existing SOP and documentation, look for

problems with existing procedures, lack of consistency, and gaps,
where the performance problem is not being addressed.

Part 3. Developing CRM Procedures Page 40

 3) In many cases, crew performance problems become apparent

at times of relatively high workload, and adding a procedure at
that point could further increase workload. The CRM procedures
development team should consider times prior to the buildup in
workload to identify periods of lower workload where a
procedure would be more effective.

4) The main forms of CRM procedures include briefs, calls,

checklist items, guides, flows, non-normal procedures, and quick
reference items. One of these forms should be sufficient to
address most problems, but there may be cases where the
integration of two forms is less intrusive and provides a better fit
with the airline’s SOP.

Guidelines for Refining CRM Procedures and

Media
1) The procedure prototype development (working with a mock-
up of the procedure) is an important part of the refining process
where the development team interacts with a range of uses to
determine the best form and content for the CRM procedures.
The refinement step should be iterative with the feedback from
each review being incorporated into the design to achieve one or
more CRM procedures that will be adopted by the users and will
contribute to performance improvements.

2) User feedback sessions should include five to ten individuals

who work well together. Working with too small a group (less
than five) is less efficient and the individuals are less likely to be
stimulated by a wider range of comments. Working with too
large a group (substantially more than ten) is more difficult to
manage, and the feedback will likely cover a broad range of
topics but not in depth.

3) If formal user feedback sessions are not possible, consider

holding informal small group or individual sessions as pilots and
instructors are available in the training center or flight operations.
Meaningful feedback should be obtained from a cross section of
pilots and instructors, either formally or informally.

4) Organizational presentations, generally made after user

feedback sessions, are similar to those made to the users but with
fewer operational details and more information about the ACRM
program. Emphasis should be on the need for the CRM
procedures, the development process, and the feedback process.

Part 3. Developing CRM Procedures Page 41

 This Page

Intentionally Left Blank

Part 3. Developing CRM Procedures Page 42

Part 4. Extended Table of Contents•

Part 4. Developing Instructor/Evaluator Training .........................................................43

Overview of Instructor/Evaluator Training Development.......................................44

Planning Development Activities .....................................................................44

Establishing an Instructor/Evaluator Development Team .................................45

Coordinating with Instructor and Training Department Schedules....................46

Instructor/Evaluator Requirements ........................................................................46

Selecting Subject Matter Experts.....................................................................46

Scheduling Instructor/Evaluator Training.........................................................47

Ongoing Standardization Requirements ...........................................................47

Developing Introductory ACRM Modules.............................................................48

Background and Theory of CRM Procedures ..................................................48

Explaining Characteristics of New Procedures .................................................48

Training Implications of CRM Procedures .......................................................50

Assessment Implications of CRM Procedures ..................................................50

Developing LOFT/LOE Modules ..........................................................................51

Briefing the LOFT/LOE and ACRM................................................................51

Administering the LOFT/LOE .........................................................................52

Assessing Crew Performance by Event Set ......................................................53

Debriefing the LOFT/LOE...............................................................................54

Developing Assessment Standards and IRR Process ..............................................55

Standardizing Crew Assessment and Inter-Rater Reliability..............................55

Developing and Maintaining General and Specific Standards............................55

Understanding the Components of Inter-Rater Reliability (IRR).......................57

Working with IRR Data under AQP ................................................................58

Developing Standardization and Training Modules ................................................58

Working with Videotapes of Real Crew Performance ......................................58

Developing and Refining Gradesheets..............................................................60

Providing Instructor/Evaluators with Immediate Feedback...............................61

Planning and Establishing Standardization Sessions .........................................63

Instructor/Evaluator Training Development Guidelines..........................................64

Guidelines for Planning and Developing Introductory ACRM Modules............64

Guidelines for Developing LOFT/LOE Modules ..............................................64

Guidelines for Establishing Assessment Standards............................................65

Guidelines for Developing Standardization and Training Modules....................65

Part 4. Developing Instructor/Evaluator

Training
Part 4 of the Manual presents the steps for developing ACRM
instructor/evaluator training. This development process starts
with planning, and goes through the development of the modules
that are required to familiarize instructors with the CRM
procedures and the assessment process that can provide a more
systematic assessment of CRM performance.

Guiding the
Organization

Developing CRM
Procedures

Developing
Instructor/Evaluator Training
• Overview of I/E Training Development......... page 44
• Instructor/Evaluator Requirements.............. page 46
• Developing Introductory ACRM Modules..... page 48
• Developing LOFT/LOE Modules............... page 51
• Developing Assessment Standards & IRR..... page 55
• Developing Standardization & Training........ page 58
• I/E Training Development Guidelines.......... page 64

Developing
Crew Training

Implementing &
Evaluating ACRM

Part 4. Developing I/E Training Page 43

Overview of Instructor/Evaluator Training Development
Instructor/evaluator training development may start ahead of
or in parallel with the development of crew training. Instructor
training generally requires a lower investment in time and cost
compared to crew training with its larger number of trainees, so it
is a good idea to develop instructor training first and use the
lessons learned from that process in the development of the crew
training. In addition, developing and conducting
instructor/evaluator training can be a source of instructor
feedback that can help tailor ACRM crew training.

This part has been written for training developers and those
managing the development of instructor/evaluator training. It
emphasizes the development of the CRM procedures and their
assessment, pointing to some of the issues about developing a
standard and reliable assessment program. Developing reliable
and valid CRM assessment is an essential part of a successful
ACRM program, but it is beyond the scope of this Manual.
Therefore, references and additional information about Inter-
Rater Reliability (IRR) is provided, but not discussed in detail in
this part (see Appendix D for information on facilitating an IRR
training workshop). Those requiring detailed information about
the IRR process are urged to use Improving Crew Assessment, a
workshop manual developed by George Mason University and
the FAA in 1996 and available from:

Deborah A. Boehm-Davis
HFAC Program/ARCH Lab
George Mason University
MSN 2E5
Fairfax, VA 22030-4444

Planning Development Activities

When planning for the development of instructor/evaluator
training, consider not only the development process but also the
actual implementation of the training. Try to plan so that the
ACRM training will be ready at the point when the instructors
� In planning, try to receive some part of their regular recurrent training. ACRM
coordinate the instructor training should be as integrated as possible with the
training completion rest of their training, and in many cases can be treated as a minor
to coincide with expansion of existing training rather than a substantial addition.
scheduled instructor
recurrent training.

Part 4. Developing I/E Training Page 44

 In planning activities, consider how much time it will take the
new or existing development team to develop the range of
instructor training modules. If an airline is already using
LOFTs/LOEs based on event sets and/or some form of IRR, the
instructor training process can concentrate on the CRM
procedures and maintaining (rather than establishing and
maintaining) reliable crew performance assessment (see
Appendix C for a sample instructor/evaluator training outline).

Several additional considerations can help in planning the

development of instructor training. This development process
can be viewed as a preliminary to developing the ACRM crew
training, and as such, the instructor training does not need to be
as highly refined as the crew training. Thus, parts of instructor
modules may have options that can be presented for instructor
feedback and outlines for instructor activities that can be refined
through instructor input during the process of the training.
Further, if the instructors are brought in from the beginning and
encouraged to be involved in this "work in progress," they will
start to take ownership, not only in their own training but also in
the ACRM crew training they will ultimately need to conduct.
Finally, some of the modules for instructor/evaluator training can
be conducted in parallel, an approach that can expedite the
development process.

Establishing an Instructor/Evaluator
Development Team
The training department should work with the
instructor/evaluator group or its representatives to develop the
ACRM instructor/evaluator. In smaller airlines there may be only
one or two people working as training developers, but it is
generally better to have more individuals working as a team,
especially if the airline needs to develop some of the modules in
parallel.

As with the development of the CRM procedures, there are

reasons for using a team approach to developing ACRM training.
First, with a single person the entire effort is at risk if that person
becomes unavailable. Further, it is unlikely that any single
person has the full scope of expertise to develop the required
modules. Consequently, with a small or one-person development
team there is a greater need for outside expertise as well as a
stronger possibility that the effort will encounter bottlenecks
through the process.

Part 4. Developing I/E Training Page 45

 For a complete package, the instructor/evaluator training
development team should develop introductory ACRM modules,
LOE/LOFT briefing/debriefing modules, assessment modules,
and standardization and recurrent training modules. Having
someone from the CRM procedure development team can be
most helpful in preparing the introductory modules; while
experienced instructors and evaluators can help with the LOFT
and assessment modules. Working with a team of different
experts can facilitate this development process and allow for
parallel development efforts. A coordinated development and
review process can ensure that the different individuals stay
informed about the issues and progress across all modules.

Coordinating with Instructor and Training

Department Schedules
The development of instructor/evaluator training should be
coordinated with the implementation of the training. ACRM
training should be completed in time for scheduled instructor
recurrent training. ACRM instructor training should be
integrated with recurrent training and should be designed to have
as small an increase in required hours as possible. With careful
planning, ACRM modules can replace or augment existing CRM
recurrent training.

A realistic timeline is required to ensure good coordination.

A preliminary timeline should be developed based on instructor/
evaluator training schedules. The development team members
can then be selected so they can meet planned deadlines. Once
the team membership has been finalized, the timeline should be
reviewed to ensure it is compatible with member schedules and,
if necessary, add team members. The instructor/evaluator
training development team should establish and maintain a good
relationship with the person in charge of scheduling instructor
recurrent training to ensure a high level of coordination.

Instructor/Evaluator Requirements
� SMEs should Selecting Subject Matter Experts
represent all
stakeholders in the The Subject Matter Experts (SMEs) may come in part from
development the instructor/evaluator group as well as from the fleets and the
process. standards group. SMEs with extensive involvement in the
training development should be part of the development team.

Part 4. Developing I/E Training Page 46

 This approach can help insure both their greater involvement and
their availability throughout the process.

The number of SMEs should be as small as possible while

making sure the SMEs represent all of the topics that will be
covered under the ACRM training. If this is the first time an
airline has developed training in areas such as LOFT/LOE design
or IRR modules, consider looking for an SME from another
airline or developing those modules jointly with another
organization. Joint development is far preferable to borrowing or
using a module developed for another organization. Borrowed
modules rarely address the operational needs of an airline and the
development team may not understand the material sufficiently
well to modify it to meet own airline needs.

Scheduling Instructor/Evaluator Training

Training of the instructor/evaluators is the key to combining
all ACRM elements into a successful training and assessment
process. For smaller airlines implementing this approach to crew
training and assessment, the new methods can seem complicated
and difficult to implement. Therefore, it is important to identify a
few basic training areas and integrate them with existing CRM
and technical training. For the first year, the main focus of the
instructor/evaluator training can be the new CRM procedures, the
LOE, and the use of an LOE gradesheet or worksheet.

The first part of the training can be designed to provide

instructor/evaluators with an understanding of the CRM
procedures and how those procedures should contribute to CRM
skill development. The next part can concentrate on the
LOFT/LOE, and how to brief, administer, assess, and debrief it.
The final part of the training should help instructors standardize
the assessment process. As much as possible of these three areas
should be incorporated into existing recurrent training. Further,
where possible, instructor/evaluators should be scheduled for this
training as part of their normal recurrent training. The first year
may require additional and special training, but a substantial part
of ACRM training can be integrated with existing training.

Ongoing Standardization Requirements

A major departure from traditional instructor/evaluator
training is the requirement for ongoing training and
standardization. In order that the airline has confidence in the
crew performance data, instructor/evaluators need to have

Part 4. Developing I/E Training Page 47

 periodic checks to determine whether they are within the airline’s
training and assessment standards. This involves group meetings
or short workshops, with the instructors assessing crew
performance, often videotaped LOE or LOFT sessions. These
standardization sessions can include instructor discussions of
issues they have faced in making assessment.

Consider turning the standardization requirement into a

positive attribute of ACRM. This can be done by encouraging
instructor/evaluators to take a team approach to crew
performance assessment. The evaluators or check airmen now
have a support team to help them develop and maintain the
standards. Instructor/evaluators must control their
standardization meetings and turn these sessions into positive
experiences where meaningful tips are exchanged and standards
are further clarified. Finally, encourage the development of
group consensus when clarifying standards or settling specific
assessment issues.

Developing Introductory ACRM Modules

Background and Theory of CRM Procedures
This first set of modules, normally one or two modules,
should introduce the CRM procedures and how they work to
enhance crew. The initial module can explain how airlines have
been very good at training technical flight skills for some time.
However, they are only now in the process of identifying and
developing training for the CRM skills. This module can start by
outlining the status of CRM training and its evolution toward
skill-based training.

Traditionally, CRM has emphasized an awareness of pilot

attitudes and personality factors. More recently there has been a
shift toward trainable skills that help crews in the management of
resources. Currently, there is no complete listing of required
CRM skills, but the CRM procedures development is a first step
in identifying the key CRM skills. Over the next few years,
airlines will introduce specific procedures that will help crews to
practice and perfect specific CRM behaviors. At present, the
instructor/evaluator training makes general references to CRM
skill training, but over the next few years that training will
become much more precise as the specific CRM skills required
for CRM procedures are identified.

Part 4. Developing I/E Training Page 48

� General CRM Another of the initial instructor/evaluator training modules
principles have been should explain that the procedure development process has
translated into translated CRM principles, something very familiar to the
operational instructors, into operational procedures, providing the airline with
procedures. an opportunity to emphasize some important CRM actions that
should be taken by all crews. This explanation should include
these points: 1) The development of these initial CRM
procedures is the first step in an ongoing process that will result
in a complete set of CRM procedures for all crews. 2) The
current CRM procedures were developed by a team using a set of
systematic steps including the identification of airline needs,
specification of the procedures, and the refinement of the
procedures. 3) Instructor/evaluators were involved in the
procedure development process and will be involved in the
ongoing identification and specification of additional procedures.

Explaining Characteristics of New Procedures

The specific form and rationale for each CRM procedure
should be explained in the introductory modules. This should
include a presentation of why the specific procedure was
� When presenting a
developed based on airline needs incident data. The procedure,
procedure, include
in its actual form, should then be presented, highlighting each of
its rationale and an
its main features. Finally, a set of questions or other form of
instructor activity to
instructor activity should be included to ensure the instructors
insure their
understanding. understand all the main elements of the new CRM procedure.

For example, if the normal checklist has been expanded to

include one or more briefings, the nature of the briefings and
their effects on crew performance should be presented. One
might explain that briefings were shifted toward lower workload
times during the takeoff and approach phases of flight, and that
all briefings were restructured to contain effective information
relevant to the phase of flight.

At some point in the module consider repeating the main

features of the new CRM procedures. For example, in the case of
new briefings one might itemize the following characteristics:

• The new briefings take place at lower workload times

during the course of a flight.
• They are structured to contain relevant information
specific to the relevant phase of flight.
• These briefing have become part of SOP by being
included in the normal checklist.

Part 4. Developing I/E Training Page 49

 Training Implications of CRM Procedures
Each of the new CRM procedures has implications for crew
� Each new CRM training because it places an emphasis on the crew’s development
procedure has of specific CRM behaviors in the operational context. Because
implications for CRM of this emphasis it is important for instructor/evaluators to have
skill training. an understanding of the nature of skill performance and its
implications for the training of these new CRM procedures.

CRM skill training can be a complex topic, and many of its

issues have yet to be resolved by the aviation research
community. Still, there are some basic guidelines for CRM
procedures and skill training that should be imparted to
instructor/evaluators:

• Skills should be trained in a task-specific context.

• Skills training requires some degree of practice that is
followed by specific feedback.
• Skill training can positively affect both technical and
CRM performance.
• CRM skill training and assessment should be emphasized
throughout ACRM crew training

� The new CRM Assessment Implications of CRM Procedures

procedures have
profound implications In addition to implications on training, the new CRM
on how crew procedures have profound implications on the way that crew
performance is performance is assessed. To provide a complete picture of crew
assessed. performance across an airline's operations, two different forms of
crew assessment can be used.

First, a detailed simulator-based method of assessment can be

developed to collect crew performance data both before and after
the ACRM crew training has been implemented. This form of
crew assessment, based on the LOE, allows for the collection of
substantial crew performance data within a carefully designed
and controlled environment. This comprehensive method of
crew assessment should be augmented with a second type of
assessment; the less structured Line Check. Line Checks are not
as controlled as LOE sessions, but they provide an efficient
method for collecting more general crew performance data.

Part 4. Developing I/E Training Page 50

 New standards will have to be developed for the assessment,
and instructor/evaluators should be told about their involvement
in setting up this more systematic form of crew assessment.
Instructor/evaluators should also be reminded of the shift from
the stand-alone, individual check airmen to the team approach to
evaluation. The instructors should work as a team to support
each other in the development and maintenance of this more
reliable form of assessment.

Developing LOFT/LOE Modules

Briefing the LOFT/LOE and ACRM
ACRM training emphasizes the use of LOFT and LOE in the
training and assessment of crew performance. These modules
about the instructor/evaluator conduct of the LOFT/LOE should
provide the knowledge and some of the preliminary skills
required to:

• Brief the LOFT/LOE.

• Administer the LOFT/LOE.
• Assessing crew performance.
• Debrief the LOE.

Instructor/evaluators should be provided the specific information

to brief the LOFT and/or LOE that will be used for that year in
� Crews should be training and assessing the effects of the new CRM procedures. A
briefed to act as they standard briefing and standard administration of the LOFT/LOE
would in line are very important because lack of standardization can affect
operations. crew performance. The role of the instructor and crews should be
established during this briefing. Inadequate briefings often set
the stage for problems that later interfere with operational
realism.

The most common difficulty is a failure to convince the crew

that the instructor/evaluator is not present as an instructor during
the simulation. Rather, his or her role is to provide
communication as ATC, company dispatch, flight attendant,
maintenance, etc., as needed. Instructor/evaluators should brief
crews to act as they would in a line operation, which includes
dealing with everyone, including the Flight Attendants, as if they
were actually present throughout the session.

Part 4. Developing I/E Training Page 51

 In addition, consider providing instructor/evaluators with
training on how to brief crews about some of the following
topics:

• Overall objective of LOFT/LOE.

• The role of the new CRM procedures in the LOFT/LOE.
• The role of the instructor in the LOFT/LOE session.
• What is expected from the crew.
• The nature of crew performance and its assessment.

Administering the LOFT/LOE

The effectiveness of a LOFT/LOE relies on script detail and
proper administration of that script during the simulator session.
To accomplish this, the scenario should be carefully scripted,
including ensuring that all ATC communications use correct
� Careful scripting of terminology, timing, and routing. Precise communication
the scenario details scripting will enhance the realism of the session. Providing
lead to LOFT/ LOE instructor/evaluators with one document to use in the simulator,
session and a more detailed document, such as a guide, to help them learn
effectiveness and the new scrip and the rationale behind its key elements, is very
realism. useful.

If an airline decides to develop a guide, instructors should be

provided with and trained on how to use it. The LOFT/LOE
guide is an essential training and reference document that can
provide detailed information on:

• Events and conditions for that phase of flight.

• Alternate crew decisions besides the “expected” response.

• Event set number, phase of flight, communications.

(including frequency and radio call), key events, and

expected actions).

• Detailed success criteria such as CRM performance

criteria.

Other topics could be included in this module. One important

topic is how to establish and maintain operational realism
throughout the session. A primary goal of LOFT/LOE
administration is to allow the crew to perform as they would in an
actual line flight given the same set of circumstances as those
developed in the scenario. Flight realism is supported through
routine activities such as flight paperwork, manuals, and
communications.

Part 4. Developing I/E Training Page 52

 Another topic could address the need to be ready to accept
and manage alternate courses of action that the crew may wish
to follow while remaining as unobtrusive as possible within
the physical limitations of the simulator. Some airlines have
included appendices either in the guide or the LOFT/LOE
script that present alternative crew actions and administrator
considerations.

Assessing Crew Performance by Event Set

An important element of LOFT/LOE assessment under
ACRM is the use of event sets to help instructor/evaluators
pinpoint key aspects of crew performance (see Appendix F for
� Event sets help some specific event set topics). Instructor/evaluators should be
evaluators pinpoint trained in the role of the event set as a tool to administer the
key aspects of crew LOFT/LOE session and as the primary unit of crew performance
performance. assessment in an LOE (Hamman, Seamster, Smith, & Lofaro,
1993). The event set is a refinement of the concept of event and
is an integral part of training as well as assessment. The event set
is made up of one or more events, including an event trigger,
supporting conditions, and distracters. The event trigger is used
to fully activate the event set. Supporting conditions are other
events used primarily to maintain event set realism. Finally, a
distracter may be inserted within the event set time in order to
divert the crew’s attention or to increase their workload.

If instructors are not experienced in the use of event sets, they

should be provided with information about the role of event sets
in assessing crew performance. Event sets divide a LOFT/LOE
session into a set of manageable elements leading to a structured
evaluation process based on systematic and reliable observations
and ratings. Each event set is designed to concentrate on specific
CRM and technical training objectives and allows the
instructor/evaluator to concentrate on a limited range of
observable behaviors. There is evidence (Seamster, Edens, &
Holt, 1995) that assessments made at the event set level result in
more reliable assessments compared with the overall assessments
that have been used at airlines.

Event sets are carefully design to support the instructor and

evaluator functions. Although a LOFT/LOE session is more than
just the sum of its event sets, a clear understanding of a scenario’s
event sets is essential to a standard assessment of crew
performance.

Part 4. Developing I/E Training Page 53

 Debriefing the LOFT/LOE
Debriefing is an essential part of the LOFT/LOE and very
important in the training of CRM procedures. The modules
addressing LOFT/LOE debriefing should cover the following
topics:

• How an instructor should state the debriefing agenda,

soliciting agenda topics from the crew on items they
would like to cover.
• How an instructor can set time limits and ask the crew for
their overall self-appraisal of the flight.
• How to guard against making the crew defensive and
exercise patience in directing the crew to the main points
of the session.
• How to integrate technical and CRM feedback into the
debrief.
• How to ensure that all crewmembers participate in the
discussion and effectively draw out quiet or hostile
crewmembers.
• How to provide a clear summary and recap of key
learning points.

Additional tips and techniques to help instructor/evaluators in

the debriefing process can be included. For example, explaining
how the instructor/evaluator should operate as a resource to
crewmembers by highlighting different portions of the LOE that
may be suitable for review, critique, and discussion can be useful.
Tips on how to ensure that the discussion is led by the
crewmembers can also be helpful.

LOFT/LOE debriefing has been the topic of some recent

NASA and FAA funded research. One of the products of that
research has been a training manual, Facilitating LOS
Debriefings (McDonnell, Jobe, & Dismukes, 1997), available
from NASA Ames Research Center:

Key Dismukes

NASA Ames Research Center

MS-262-4

Moffett, Field, CA 94035-1000

Part 4. Developing I/E Training Page 54

Developing Assessment Standards and IRR Process

Standardizing Crew Assessment and Inter-

Rater Reliability
Another important element of ACRM instructor/evaluator
training is developing and maintaining superior assessment
standards. The module that addresses this set of topics should
explain that LOFT/LOE crew performance assessment is based
on observations by a trained instructor/evaluator (see Appendix D
for facilitating an IRR training workshop). The reliability of
those observations is critical to the airline in determining the
performance level of its crews as well as the organizational
training needs. Reliable ratings mean that all raters provide the
same scores or grades to the same performance, both technical
and CRM. Instructor/evaluators need to constantly maintain a
high standard of reliability, something that can best be developed
in a group setting. Without this high level of reliability, the crew
performance score or ratings may say more about the instructor’s
biases than about the crew performance being assessed.

There is a high probability of some rater bias when making

crew performance assessments. The most common forms of bias
� There is a high that can be addressed through instructor/evaluator training are
probability of rater central tendency, the halo error, and leniency error. One of the
bias, and that bias most common biases in LOFT/LOE performance rating is that of
should be addressed central tendency, where the rater tends to rate most performance
through training. toward the middle, average, or standard point on a rating scale.
The halo error occurs when raters are biased by an overall
positive or negative impression of the crew rather than specific
performance elements. Without training in assessment standards
and IRR, raters may develop a bias toward leniency where they
unintentionally rate crew performance as being higher than it
really is. The instructor/evaluators should be informed of these
biases and provided with training to avoid them.

Developing and Maintaining General and

Specific Standards
The ACRM process supports the use of both general and
specific standards for the assessment of crew performance.
General standards include a standard rating scale used throughout
the airline and guidelines for how to assign overall ratings to the

Part 4. Developing I/E Training Page 55

 individual and crew performance. Specific standards are
associated with observable behaviors and technical procedures.
Instructor/evaluators should be informed of and trained in the use
of both sets of standards.

A standard rating scale can be used across the full range of

� General standards evaluation environments, from simulator sessions to line checks.
can help improve a This is important because using a standard rating scale can reduce
group’s use of the the training time required to familiarize instructor/evaluators with
basic assessment different assessment instruments. Its use can also increase the
parameters. amount of practice instructor/evaluators have with that scale,
resulting in better instructor/evaluator assessment skills. The
specific points of the standard rating scale should be presented,
along with examples of how to rate a range of performance using
that scale.

A second general set of standards includes how to formulate

overall individual and crew ratings based on the ratings of
specific technical and CRM items. Some airlines have general
standards that address what to do in such cases as when an
individual gets one unsatisfactory rating. Other airlines develop
general standards for event set failure. Such standards (e.g.,
“three strikes and you’re out”) may apply to the LOFT/LOE
sessions rather than specific event sets. When an
instructor/evaluator group shows low rater reliability, in addition
to more training there may be a need for better general standards
to help the group work together on the basic steps of the
assessment process.

Specific standards should also be established for all rated

elements that may cause problems or may be new to the
instructor/evaluators. In most cases this will include standards
for the new CRM procedures. The standards are presented to the
instructor/evaluators as criteria of performance that signifies the
expected level of individual or crew performance.
Instructor/evaluator training may start with the standard for
certain elements anticipated to cause rater problems. In this
process the development team should not overlook the technical
items, which may have standards that for some reason are
difficult to interpret or rate. In providing specific standards,
developers should concentrate on the criteria for “standard
performance” (note that standard performance is not the average
crew performance, rather, it is the expected crew performance).
If instructor/evaluators are still having problems rating a specific
element, the criteria for unsatisfactory performance should be
added. In most cases, criteria for Above and Below Standard can
be inferred from the Standard criteria.

Part 4. Developing I/E Training Page 56

 Understanding the Components of Inter-Rater
Reliability (IRR)
If the airline has not implemented some form of IRR training,
� If the airline has not IRR should be presented as a group process. This should begin
implemented some with an overview of the IRR process, followed by the critical
form of IRR training, nature of crew assessment, the IRR measures, the assessment
it should develop a form or worksheet, and rating scales and examples of the criteria
module that for each point on the scale. The module should explain that IRR
introduces IRR and training needs to take into account key organizational elements.
its components. This should include the individual instructor or rater, the pilots
being assessed, the instruments and scales being used, the
specific values and implications of the rating scale, the types of
analyses used on the data, and the organizational use of the
results.

Consideration must also be given to organizational and

regulatory environment. Airline philosophy and policy must
support, or may have to be revised to support, developing a high
level of instructor standardization. IRR training was developed
to provide a more reliable technical and CRM assessment, and
consideration needs to be given to the entire system and
organization when planning for IRR training.

IRR training may include the use of one or more measures to

determine the key elements of rater reliability. The main IRR
measures and their functions are:

• Systematic Differences: Shows whether a rater is

substantially or significantly above or below the group.
• Congruency: Shows the general relationship between an
individual's ratings and group ratings by comparing the
score distribution of individual rates with that of the
entire group.
• Consistency: Shows the degree to which rater scores
correlate or shift in unison.
• Sensitivity: Shows the degree to which raters can
discriminate small differences in crew performance.
• Agreement Index: Shows the degree to which raters have
the same rating for a specific item or performance.

Additional IRR information may be found at the end of the

first page of this part and in Appendix D.

Part 4. Developing I/E Training Page 57

 Working with IRR Data Under AQP
With the advent of AQP and its requirement for a more
systematic LOFT/LOE design and assessment process, the
aviation industry needed a more robust form of reliability
training. IRR training has been developed to meet the data
collection requirements under AQP that reduce or eliminate the
subjective nature of CRM rating while providing airlines with
consistent data that can be compared over time. Consistently
reliable data are very important because they allow organizations
to conduct historical analyses of crew performance.

IRR training has been developed to provide airlines with a

more precise, longer-term form of crew performance assessment.
The IRR approach compares ratings to organizational standards
based on the development of benchmark data. This allows
airlines to maintain a systematic rating of crew performance over
time, even as instructor/evaluators are replaced, new LOFT/LOEs
are developed, and new CRM procedures are added to the SOP.

A high level of reliability is required for airlines to be able to

make sound decisions about the state of crew performance. AQP
allows airlines to modify their training programs based on an
accurate assessment of crew performance at specific points in
time. If a particular form and cycle of training produces Above
Standard results, it may be possible for that airline to extend the
training cycle for that content. Conversely, if the crew
performance data show specific deficits, the airline will be
required to strengthen those training areas. In order to make such
strategic decisions, the airlines and the FAA require highly
accurate data.

Developing Standardization and Training Modules

Working with Videotapes of Real Crew
Performance
A key to developing and maintaining standard assessment of
crew performance is to train and evaluate assessors using
videotaped crew performance. Videotapes allow a group of
instructors to observe and rate the same crew performance, but
require that the tapes be prepared using some specific guidelines.

Part 4. Developing I/E Training Page 58

 Traditionally, crew performance videotapes have been
developed with pilots acting a part rather than performing
naturally. This results in tapes of a stereotypic good crew or bad
crew, what some call, "The Good, the Bad, and the Ugly." What
is required under ACRM is a substantial departure from that kind
of tape. Standardization tapes should represent the types of
� ACRM assessment behaviors evaluators will see during real simulator sessions.
training requires From experience, one of the best ways to prepare such videotapes
tapes of actual crews is to work with real crews, flying the actual LOFT/LOE for that
flying the actual year in a representative simulator without the benefit of coaching
LOFT/LOE. or preparation (see Appendix E for videotaping considerations).

There are a number of reasons for these guidelines. First,

consider working with real crews to capture actual crew
performance, the type that instructor/evaluators are likely to
observe in the simulator. This means avoiding working with
other instructors because they are likely to model a training
department form of behavior rather than realistic line behavior.
Second, if scripted videotapes are used, instructors are likely to
notice the acting and not pay attention to the actual crew
performance. Third, by working with the upcoming LOFT/LOE,
instructors learn about the new script as well as the specific
standards that they will be using. Following these guidelines
makes the training or standardization session substantially more
relevant for the instructor/evaluators.

Finally, when using these tapes in the standardization process,

the instructor/evaluators should be informed that the crew
performances they will observe are real crew responses to the
LOFT/LOE session, and that no coaching was provided to these
crews to act either as superior crews or below standard crews.
The crews were instructed to act as they would on the line.
Further, it should be explained that the tapes were made this way
to capture the types of behaviors that evaluators are most likely to
see during actual LOE sessions.

For good IRR results, both the audio and video quality of the
videotape must be excellent. Ensuring audio quality may require
auxiliary microphones and control of extraneous noise. Ensuring
video quality may require a special light-sensitive camera,
auxiliary lighting and careful adjustment of panel lights and other
ambient light sources (for more details, see Appendix E). The
objective is to capture as much detail of the crew performance as
possible on the video tape while not interfering with the
simulator environment.

Part 4. Developing I/E Training Page 59

 Developing and Refining Gradesheets
A LOFT/LOE gradesheet or worksheet can be an important
tool in achieving standard crew performance assessment. A
number of guidelines should be considered in the design of these
gradesheets. The gradesheet should be organized around event
sets; that unit of assessment which helps instructors to
standardize the implementation as well as the assessment of LOE
sessions. The gradesheet should use a standard rating scale and
should depict that scale on every page. The gradesheet should
include observable behaviors or topics that have been carefully
identified and validated as being central to successful
performance on a specific event set. The well-designed
gradesheet simplifies what could be a relatively complex process:
providing the instructor with debriefing areas, both the CRM and
technical elements for each event set.

After ACRM instructor/evaluator training, the LOFT/LOE is

the main tool used to guide the assessment process. The
development of that worksheet should be based on the above
guidelines as well as a number of working sessions with the
instructor/evaluators. The training development team may want
to prepare the first draft of the gradesheet based on event set
samples from other airlines or some of its own efforts. Once the
draft is prepared, it should be given to the instructor/evaluator
group for comments and feedback. Some of the instructor
comments will point to the need for a particular form of training,
while other comments may point to needed changes in the
gradesheet. This form of instructor/evaluator feedback is
essential to the development of a viable gradesheet. The
following two examples show the refinement process moving
from a draft set to more targeted observable behaviors.

DRAFT CRM OBSERVABLE BEHAVIORS

SITUATION AWARENESS
OBSERVABLE BEHAVIORS
����������������������������������������������������������������
����������������������������������������
��������������������������������������
�������������������������������������������
OTHER
OBSERVABLE BEHAVIORS
������������������������������������������������������������������
���������������������������������������������

Part 4. Developing I/E Training Page 60

 FINAL CRM OBSERVABLE BEHAVIORS
FOR GRADESHEET
SITUATION AWARENESS
OBSERVABLE BEHAVIORS
����������������������������������������������������������������
����������������������������������������

BRIEFING
OBSERVABLE BEHAVIORS
������������������������������������������������������������������
���������������������������������������������
���������������������������������������������

Providing Instructor/Evaluators with Immediate

Feedback
Instructors/evaluators should be provided with accurate and
immediate rating feedback from the start of their assessment
training through the standardization sessions. The first rating
sessions may take place in a larger group using taped pilot
performance to present the crew behavior that must be rated. The
group results may be presented via a spreadsheet and charts
showing both individual and group data along with appropriate
benchmarks or standards the group is trying to meet. This
feedback is an essential part of IRR training and should take
place as soon as possible after the ratings are made.

As assessment training progresses, it is advisable to work in

smaller groups with fleet-specific performance as the object of
the ratings. One reason for the smaller group is that it allows the
individuals to start working as a team. This is something that can
be done with 5 to 15 instructor/evaluators, but is much more
difficult to achieve with substantially larger groups. Small group
feedback should be provided on the agreement for each item
being rated along with the average agreement for all items. In
addition, small group feedback should include the systematic
differences, average congruency, and average consistency.

When working with small groups and assessment teams,

individual feedback is also important. A smaller group permits
greater attention to be paid to individual feedback. Individual
feedback includes systematic individual differences, individual
congruency with the group’s distribution, individual consistency
with other raters, and sensitivity to small performance

Part 4. Developing I/E Training Page 61

 differences. Discussion should focus on the individual and group
problems for each of these measures.

Using this feedback approach over several rating sessions and

providing feedback before going on to the next session can
reduce the divergence found in many raters, and the measures
allow the instructors to concentrate on the quantitative
components of this process. After individual and group feedback
is provided and explained, instructor/evaluators should be
encouraged to develop new rating rules and strategies. This cycle
� ACRM assessment of practice, feedback, and discussion allows the participants to
requires ongoing improve their reliability, and should continue until corporate IRR
training and benchmarks have been met. After that, the methods are used
standardization. periodically in the standardization sessions.

One type of feedback that can be helpful when some of the

instructor/evaluators depart from group norms is to provide
individuals or a small group with Congruency graphs. In the
following two examples, the first graph shows a rater who closely
mirrors the group averages. With a Congruency Index of .97, this
instructor has rated CRM the way the rest of the group is rating
with about 80% of the ratings as "3" and about 15% as "4."

PIN 80486 vs GROUP on CRM Ratings

Congruency Index = .97

0.8

0.6 80486
0.4 GROUP
0.2

0
1 2 3 4

This second graph shows the Congruency of a rater who tends

to be more generous than the group. This rater has given close to
55% of ratings as "3" and about 35% as "4." With "4"
representing "Above Standard," this rater is giving higher marks
in CRM. If these results were based on a large number of
observations, then this rater is departing from the group, and the
team should work to help this rater understand the criteria for

Part 4. Developing I/E Training Page 62

 "Standard" and "Above Standard." Immediate feedback allows
individuals and the group to spot these differences and to address
them as a team before they become established rating practices.

PIN 40817 vs GROUP on CRM Ratings

Congruency Index = .59

0.8

0.6
40817
GROUP
0.4

0.2
0
1 2 3 4

Planning and Establishing Standardization

Sessions
ACRM assessment requires ongoing standardization.
Standardization is required so the airline will have confidence in
the crew performance data and its indications about CRM
procedures. During training development, plans should be made
for scheduling at least two standardization sessions. As a rule of
thumb, if it is anticipated that instructor/evaluators will have few
difficulties with the new approach to assessment, two sessions,
spaced about three months apart, should be planned. If
substantial difficulties are anticipated it would be better to plan to
hold the two sessions one month apart. Once an airline has
collected data from the two sessions it can determine whether the
frequency of the standardization sessions should be increased or
decreased. Those few airlines that presently collect
standardization data tend to work with an annual cycle. Twelve
months is far too long to go without collecting standardization
data, especially when introducing a new system.

Instructor/evaluators should be encouraged to take an active,

team approach to these standardization sessions. The
instructor/evaluators should see these sessions as an essential part
of maintaining their assessment standards. One way to ensure
this is to encourage the instructor/evaluators to control the
standardization sessions and ultimately to determine their own
schedule and length of cycle between sessions.

Part 4. Developing I/E Training Page 63

Instructor/Evaluator Training Development Guidelines
Guidelines for Planning and Developing
Introductory ACRM Modules
1) When planning for the development of instructor/evaluator
training, consider not only the development process but also the
actual implementation of the training. Try to plan so that the
ACRM training will be ready at a point where instructors receive
scheduled recurrent training. ACRM instructor training should
be as integrated as possible with the rest of their training and
should be treated as a minor expansion of existing training rather
than a substantial addition.

2) One initial instructor/evaluator training module should

explain that the procedure development process has translated
CRM principles into operational procedures, providing the airline
with an opportunity to emphasize important CRM actions that
should be practiced by all crews.

3) When presenting CRM procedures to instructors it should be

explained how each procedure was developed based on airline
needs and incident data. The procedure, in its actual form,
should then be presented, highlighting each of its main features.
Finally, consider including a set of questions or other form of
instructor activity to ensure that the instructors understand the
main elements of the new CRM procedure.

4) Each new CRM procedure has implications for crew training

because it places an emphasis on the crew’s development of
specific CRM behaviors in the operational context. Because of
this emphasis on CRM skills, instructor/evaluators should have
an understanding of the nature of skill development and its
implications for the training of these new CRM procedures.

Guidelines for Developing LOFT/LOE Modules

1) The effectiveness of LOFT/LOE sessions depends in good

part on script detail and proper administration of that script
during the simulator session. The scenario should be carefully
scripted with ATC communications using correct terminology,
timing, and routing. Precise ATC communication scripting will
also enhance session realism.

Part 4. Developing I/E Training Page 64

 2) LOFT/LOE briefings are an important part of the session, and
instructor/ evaluators should brief crews to act as they would in
line operations dealing with everyone, including the Flight
Attendant, as if they were actually present throughout the
LOFT/LOE.

3) Event sets should be used in the development of LOFT/LOS

scenarios under an ACRM program to help instructor/evaluators
pinpoint key aspects of crew performance for each segment of the
flight. Instructor/evaluators should be trained in the functions
and use of event sets.

Guidelines for Establishing Assessment

Standards
1) When an instructor/evaluator group shows low inter-rater
reliability, in addition to more training there may be a need for a
better rating form or clearer rater standards to help the group
work together on the basic parameters of the assessment process.

2) The instructor/evaluator group should establish specific

standards for elements to be rated. This is especially true for
elements that may cause problems or are new to the
instructor/evaluators. In most cases the new CRM procedures
should have explicit standards to reduce rating difficulties.

3) When making crew performance assessments there is a high

probability of rater bias, and the common forms of bias that
should be addressed through instructor/evaluator training include
central tendency, halo error, and leniency error.

4) If the airline has not already implemented some form of IRR

training, IRR should be presented as a group process beginning
with an overview of IRR, followed by the critical nature of crew
assessment, the IRR measures, the gradesheet, rating scales, and
examples of the criteria for each point on the scale.

Guidelines for Developing Standardization and

Training Modules
1) Under ACRM, a substantial departure from a scripted and
acted videotape is required. Consideration should be given to
preparing standardization tapes working with real crews flying
the actual LOFT/LOE for that year in a representative simulator
without the benefit of coaching or preparation.

Part 4. Developing I/E Training Page 65

 2) Instructor/evaluators should be provided with accurate and
immediate rating feedback from the start of their assessment
training through standardization sessions. The first rating
sessions may take place in a larger group using spreadsheets and
charts showing individual and group data along with appropriate
benchmarks that the group is trying to meet.

3) After individual and group feedback is provided and

explained, instructor/evaluators should be encouraged to develop
new rating rules and strategies. This cycle of practice, feedback,
and discussion allows the participants to improve their reliability,
and should continue until group benchmarks have been met.

4) Under ACRM assessment there should be ongoing training

and standardization to establish confidence in the crew
performance data, its indications about CRM procedures, and the
procedures’effects on overall crew performance.

5) Instructor/evaluators should be encouraged to take an active

team approach to standardization sessions. Instructor/evaluators
should see these sessions as an essential part of maintaining their
assessment standards. One way to ensure team involvement is to
encourage instructor/evaluators to control the standardization
sessions, and ultimately to determine their own schedule and
length of cycle between sessions based on their rating
performance.

Part 4. Developing I/E Training Page 66

Part 5. Extended Table of Contents

Part 5. Developing ACRM Crew Training ....................................................................67

Overview of Crew Training Development..............................................................68
Planning for Crew Training Development Activities.........................................68
Establishing the Development Team ................................................................68
Coordinating with Crew Training Cycles and Requirements .............................69
Curriculum Development Requirements ................................................................70
Developing ACRM Crew Training Footprint ...................................................70
Establishing Development Timeline .................................................................71
Coordinating Development of Crew Training with Other ACRM Elements ......72
Establishing a Curriculum Review Process.......................................................73
Developing CRM Procedures Crew Modules ........................................................73
Establishing the Need for ACRM.....................................................................73
Explaining the Move from CRM Principles to CRM Procedures ......................74
Reviewing the Components of ACRM .............................................................75
Explaining Crew Training Improvements .........................................................76
Developing Crew Effectiveness Modules...............................................................77
Understanding the Role of ACRM in Developing CRM Skills..........................77
Integrating CRM with Technical Performance .................................................77
Training and Assessing Crew Performance ......................................................78
Using CRM Procedures to Improve Crew Effectiveness ..................................79
Developing Briefing/Debriefing Modules...............................................................80
Briefing the LOFT/LOE ..................................................................................80
Explaining the Role of Crew During LOFT/LOE .............................................81
Debriefing the LOFT/LOE Session ..................................................................81
Developing Crew LOS Assessment Modules .........................................................82
Primary Technical and CRM Training Objectives .............................................82
Incidents that Support Training Objectives ......................................................83
Event Sets and the LOFT/LOE Scenario .........................................................85
ACRM Crew Training Development Guidelines ....................................................86
Guidelines for Planning ACRM Crew Training ................................................85
Guidelines for Developing the CRM Procedures Crew Modules ......................87
Guidelines for Developing Crew Effectiveness Modules...................................87
Guidelines for Developing Briefing/Debriefing Modules...................................88
Guidelines for Developing Crew LOS Assessment Modules.............................88
Part 5. Developing ACRM Crew Training
Part 5 of the Manual presents the steps for developing ACRM
crew training. This development process may parallel the
development of instructor/evaluator training especially where
similar module content can be used for both types of training (see
Appendices C and H to compare two sample outlines). The
development of crew training starts with the planning process and
works through the development of the modules that are to be
used to train crews in ACRM.

Guiding the
Organization

Developing CRM
Procedures

Developing
I/E Training

Developing ACRM Crew

Training
• Overview of Crew Training Development........ page 68
• Curriculum Development Requirements.......... page 70
• Developing CRM Procedures Crew Modules.... page 73
• Developing Crew Effectiveness Modules......... page 77
• Developing Briefing/Debriefing Modules......... page 80
• Developing Crew LOS Assessment Modules.... page 82
• ACRM Crew Training Development Guidelines.page 86

Implementing &
Evaluating ACRM

Part 5. Developing ACRM Crew Training Page 67

Overview of Crew Training Development

Planning for Crew Training Development

Activities
Planning for ACRM crew training should consider the
development process in the context of ACRM implementation
within the organization. Crew training should be completed and
� Crew training can be ready for delivery at a point when crews receive their recurrent or
a significant cost of
other form of scheduled training. ACRM crew training should be
the ACRM program,
integrated with the rest of the training as much as possible. Crew
so determine ways to
training can be the largest cost of the ACRM program, so
reduce its cost.
determine ways for reducing the cost of that training. For
example, ACRM crew training may be implemented as an
extension to existing crew training rather than a new addition.

In planning, consider how much of the instructor/evaluator

training material can be adapted to crew training. Much of the
CRM procedure training material for I/Es should transfer to crew
training. That material can then be expanded to include incidents
and examples that support the need for the new procedures. The
instructor/evaluator material on crew assessment will have to be
changed from representing the evaluator’s perspective to a crew
perspective. From experience at one airline, most of the
instructor/evaluator training material does transfer to crew
training; this is one of the advantages of developing most of the
instructor training before starting work on the crew training.

Several additional considerations can help in planning the

development of crew training. It is helpful to have the instructors
involved in the development process. As with the development
of the instructor/evaluator training (see Part 4), if the instructors
are brought in at the beginning and encouraged to be involved,
they will take ownership in the ACRM crew training. Finally,
some of the crew training modules can be developed in parallel.
This is a strategy to consider if crew training needs to be
developed in a relatively short period of time.

Establishing the Development Team

ACRM crew training is generally developed by the training
department with the cooperation of the instructor/evaluator
group. In smaller airlines there may be only one person available
to develop the training material, but even in that situation it is

Part 5. Developing ACRM Crew Training Page 68

 better to have several individuals working as a team. A
development team provides more flexibility, especially if there is
a need to develop some of the modules in parallel. With a single
developer the entire effort is at risk if that person becomes
unavailable. In addition, it is less likely that just one person has
� Consider the full scope of expertise to develop the required modules.
establishing a Establishing a development team whose members have most of
development team
the expertise required to design and refine the ACRM crew
whose members
training materials should be considered.
have most of the
expertise required to
The functions of the ACRM crew training development team
design the ACRM
include developing the CRM procedures modules, the crew
crew training.
effectiveness modules, the evaluation modules, and the briefing
and debriefing modules. Members from the instructor/evaluator
development team can be very helpful in preparing many of these
crew modules because much of the content is similar to that for
the instructors. Therefore, consider keeping as many developers
from the instructor/evaluator team as possible.

The materials for crew training may require additional design

considerations such as strategies for getting the trainee’s attention
and ways to maintain crew interest. If this is the case, a
professional writer and artist may be required to develop a set of
compelling materials. As with the development of the CRM
procedures and instructor/evaluator training, a coordinated
development and review process ensures that team members stay
informed about the issues and progress taking place across the
different modules being developed.

Coordinating with Crew Training Cycles and

Requirements
During planning, consider a thorough review of all crew
training with special attention to existing CRM training. A
number of existing training areas can be extended or modified to
meet ACRM training needs. Much of the existing CRM
� There are existing recurrent training can be modified from an emphasis on general
training areas that CRM principles to a concentration on the specific CRM
can be extended to procedures. Depending on how the airline conducts CRM
meet ACRM training training, that can provide between 8 and 24 hours of training
needs. time that can go toward ACRM crew training. ACRM crew
training

Part 5. Developing ACRM Crew Training Page 69

 should be integrated with recurrent training and should be
designed to have as small an increase in required hours as
possible. Because the crew training component of ACRM can be
its single largest cost, time should be taken during the planning
stage to keep the number of additional training hours to a
minimum while providing comprehensive ACRM training to all
the affected crews.

Good coordination should be directed by a development

timeline that ensures that the crew training will be available at the
appropriate time in the training cycle (see the next section). A
preliminary timeline based on the crew training cycle needs to be
developed. The size of the development team can then be scaled
to meet the schedule. Once the team has been formed, the
timeline should be reviewed to ensure that it is compatible with
team member schedules. If necessary, team members can be
added or the schedule adjusted so that the training material can be
developed within the time specified. Maintaining a good
relationship with the person in charge of scheduling crew training
can help ensure a high level of coordination.

Curriculum Development Requirements

Developing ACRM Crew Training Footprint
� Consider what of the In planning for the ACRM crew training, the training
existing CRM training footprint can be developed so that it will result in a minimal
can be replaced or increase in training hours. Working with existing CRM training
expanded to include modules and times, consider what can be replaced or expanded to
the ACRM crew include the ACRM crew training material. Some of the ACRM
training material. training can be coordinated with crew classroom training.
Additional means for training include the release of a video
showing the new CRM procedures being used in operational
settings and guides or other written material supporting elements
of ACRM. In addition, give consideration to simulator-based
training and evaluation, two important components of ACRM
crew training.

Classroom and simulator training can be used for the more

interactive portions of ACRM crew training while the other
forms of media can be used to introduce the key concepts of
ACRM and the details of the CRM procedures. The ACRM crew
training footprint can be from 4 to 16 hours depending on the
number of new CRM procedures and the type of CRM

Part 5. Developing ACRM Crew Training Page 70

 training that crews have been given (see Appendix H for an
ACRM crew training outline). Consider the need for each of the
following modules:

• Advanced Crew Resource Management

• CRM Procedures.

• Examples of CRM procedure performance.

• Crew Effectiveness.

• Implementing LOEs and Line Checks.

• LOFT/LOE Sessions.

Establishing Development Timeline

� The development The development timeline should be established during the
timeline has to serve planning stage to ensure good coordination between the
the organization and development team and the actual crew training. Consider
the ACRM crew developing a preliminary timeline based on the crew training
training has to be cycle. Once the development team has been formed, the timeline
completed and ready should be reviewed to ensure that it is compatible with team
to meet established member schedules. Ultimately the development timeline has to
training cycles. serve the organization and the ACRM crew training has to be
completed and ready to meet established training cycles.

The development timeline should be planned to meet the

airline’s crew training needs in the context of its available
resources. With a substantial part of the content developed for
instructor training, crew training development can concentrate on
refinement of that material and appropriate media selection. If
the development team decides on extensive use of different
media, the media production considerations, especially those
involving video, computer-based training, and high quality
manuals, will strongly influence the timeline. Advanced forms of
media can take one to several months to produce and require
scripts and storyboards before being ready to go into production.

In establishing the crew training development timeline, there

are a number of other considerations. Once the total
development time has been determined, consider assigning and
scheduling the activities in such a way that some items can be
performed in parallel. Parallel development is particularly suited
for crew training because of all the available material from
instructor training. Usually, the LOFT/LOE scripts and the CRM
procedures materials have been developed for instructor training,
so different team members can proceed with the process of
tailoring the content to crews. If possible, consider developing

Part 5. Developing ACRM Crew Training Page 71

 those modules needed first, and wait for actual crew trainee
feedback on them before completing those modules that are
needed later.

Coordinating Development of Crew Training

with Other ACRM Elements
The development of ACRM crew training should be
coordinated not only with crew training cycles but also with the
instructor/evaluator training and the implementation of the new
CRM procedures. There are several points of coordination.
First, consider the LOFT/LOE instructor/evaluator training,
specifically the administration and assessment of the Line
Operational Simulation (LOS) sessions. For an airline that has
not made extensive use of this form of simulator training, it may
� Crew training should take some time to ensure that there are sufficient instructors to
be coordinated with give and assess the LOFT/LOE sessions. With the increased
crew training cycles, reliance that ACRM places on LOFT/LOE, it is important to have
instructor/evaluator that part of the instructor/evaluator training completed prior to
training, and the the start of crew simulator sessions. Some airlines have
implementation of increased instructor/evaluator training in this area, making sure
the new CRM that the instructors actually fly the new LOFT/LOE and then
procedures. administer one under the supervision of a qualified individual
prior to working directly with the crews. This can take additional
time that needs to be accounted for in scheduling the crew
training.

A second consideration is coordinating the development of

crew training with the training of classroom instructors. For best
coordination in this area, consider having the instructors trained
in time, but not too early, for the scheduled classroom training.
The classroom training can be a crew’s first exposure to ACRM,
and it is important that the instructors are current in the ACRM
concepts and training strategies.

A final consideration is the coordination between crew

training and the implementation of the CRM procedures. Each
organization may take a slightly different approach to timing
these two critical events, but should consider a sequence where
the CRM procedures are announced, crews are trained, and then
the procedures are implemented. This should be planned as a
short sequence, where the crew training and implementation
follow closely. Because crews should receive training prior to
implementation, consider scheduling ACRM crew initial training
to occur within a narrow window. This will reduce situations

Part 5. Developing ACRM Crew Training Page 72

 where crews have ACRM training eight or more months prior to
implementation and tend to forget some of the key elements.

Establishing a Curriculum Review Process

� Successful training Successful training development and implementation includes
development and a good review process. Consider scheduling ACRM crew
implementation training curriculum reviews for the key organizational elements
includes a good that control the future of the ACRM program. At most
review process. organizations this will include instructor/evaluator
representatives, the training department, airline management, and
union representatives. If there are groups that work well
together, they can be scheduled as one review, but in cases where
there are very different agendas, consider scheduling those
reviews as separate events.

On the management side it is important to communicate the

progress being made, the efficiency and effectiveness of the crew
training, and the coordination taking place between crew training
and the rest of the training system. The management-level
review concentrates on an overview, plan, and schedule. On the
union and instructor side, consider a more detailed review of the
curriculum content and trainee requirements. Union
representatives should participate in the review process. It may
also make sense to include instructors in this review. Having
instructor and union participation on the crew training team can
help with those reviews and ensure a higher degree of acceptance.

Developing CRM Procedures Crew Modules

Establishing the Need for ACRM
The first modules in ACRM crew training should introduce
ACRM and establish the need for this program within the
organization. ACRM may be introduced as a comprehensive
implementation package including the CRM procedures, training
of the instructor/evaluators, and the crew training that they are
about to receive. The point should be emphasized that ACRM
has been designed and developed by the airline to address
specific problems that exist both in industry and at the airline.
For example, it might be explained that this ACRM training was
based in part on the results from the NTSB Safety Study along
with information provided by its instructor/evaluators to identify
the types of training and procedural changes that could improve
CRM performance.

Part 5. Developing ACRM Crew Training Page 73

 If more general industry trends, such as those found in the
NTSB Safety Study, were used in developing the CRM
� Emphasize that the procedures, then the specific results should be presented in some
crews are an detail. The NTSB study has some clear trends that can help
essential part of this crews understand some of the crew conditions present in recent
development U.S. commercial accidents. If general industry trends were a
process. main source of the needs justification process, consider
presenting several documented incidents, such as those in ASRS
incident reports, that support the type of problems the new CRM
procedures will address (see Appendix G for sample reports).

In explaining the airline data or reports that established the

need for ACRM training, provide sufficient detail to fully
establish what the problem area is. In other words, do not
provide just one piece of data or a statistic, because that type of
data can usually be subjected to multiple interpretations. Provide
several pieces of supporting data that more firmly establish and
describe the problem area. Specific incidents can be very helpful
in elaborating the operational relevance of the problem.

� The new CRM Once the problem area or areas have been established,
procedures are introduce how ACRM and its CRM procedures will address
designed to support those areas. In this part of the training, consider introducing
CRM skill ACRM as an ongoing development process that provides airlines
development. with unique CRM solutions tailored to their operational demands.
ACRM is ongoing and dynamic, and it should not be confused
with a single set of products. For example, reproducing a
product, such as a briefing card, from another airline will not, by
itself, produce the type of organizational change that the ACRM
training development process can produce. It should be
emphasized that the crews are an essential part of this
development process, so crew representatives should be involved
in developing the current training. Further, individual
crewmembers should stay involved and help identify new CRM
procedural needs as they become apparent.

Explaining the Move from CRM Principles to

CRM Procedures
One of the early modules in crew training should explain the
role of CRM principles as well as that of CRM procedures. One
way of introducing these concepts is to explain that the design of
the CRM procedures is based on critical CRM principles that are
needed in an airline’s specific operational environment. General

Part 5. Developing ACRM Crew Training Page 74

 CRM principles, such as good decision making and maintaining
situation awareness, dominate traditional CRM training.
Recently, there has been a shift toward trainable skills that help
crews in the management of resources. Although there is no
complete listing of required CRM skills, the new CRM
procedures developed under ACRM are designed to support the
development of CRM skills. It should be explained that over the
next few years the airline will be introducing specific procedures
which will help crews to practice and perfect specific CRM
skills.

Compared with the traditional CRM principles, the CRM

procedures are more specific, more operationally relevant, and
they emphasize CRM actions that should be taken by all crews.
Consider explaining that the development of these initial CRM
procedures is the first step in an ongoing process which will
result in a more complete set of CRM procedures. It should be
explained that these new CRM procedures were developed by a
team that identified airline needs, specified the procedures, and
then refined the procedures.

Reviewing the Components of ACRM

The main components of an ACRM program include the
CRM procedures, training of the instructor/evaluators, crew
training, a standardized assessment of crew performance, and an
ongoing implementation process. An explanation of each of
� Introduce the main these components helps trainees understand the entire program
components of the rather than just one or two of its key elements.
ACRM program.
The specific content and format for each CRM procedure
should be presented. The procedure, in its actual form, should be
presented, highlighting each of its main features. Consider
including an incident that highlights the need for one or more
procedures or some other form of interactive activity to ensure
that the crews understand all the main elements of the new CRM
procedure.

Instructor/evaluator and crew training can be presented in a

parallel manner, and this can help crews understand the elements
in common between the two forms of training. For any given
airline, ACRM can present a new direction in specific areas such
as SOP, the use of LOFT/LOE, and crew assessment. These
relevant new directions should be emphasized so that the crews
understand the need for the new training.

Part 5. Developing ACRM Crew Training Page 75

 Emphasis on crew performance assessment should be
introduced at this point in crew training. With the
implementation of the new CRM procedures, total crew
performance takes center stage both in how crews coordinate
their tasks and in how evaluators assess those crews. CRM is no
� Crews should expect longer assessed as a group of general principles; rather, it is
a more systematic systematically evaluated based on specific actions and the
assessment under intermediate and final crew outcomes. This part of the
ACRM. presentation can introduce the concept of crew effectiveness,
which is covered in more detail in the next section.

Explaining Crew Training Improvements

ACRM crew training has some noticeable improvements over
traditional ACRM training. Although individual
implementations of ACRM training may highlight some different
improvements, there are general elements of most ACRM
training that should be brought to the crews’attention.

A major improvement, referred to in several of the previous

subsections, is the specificity and operational relevance that
CRM procedures bring to what has often been a set of vague
concepts. ACRM crew training presents specific CRM steps for
crews to follow under certain flight conditions. Encourage
crewmembers to ask questions at any point through the
presentation of the CRM procedures to ensure that those
procedures are clear to the entire group.

Another improvement the crews should expect under ACRM

is a more systematic assessment of their performance, both CRM
and technical performance. It should be explained to crews that
precise measures of crew performance were required in the
research stages of ACRM development. That requirement has
resulted in a more detailed and accurate measurement of crew
performance, a form of measurement that has helped evaluators
provide more standard crew assessments. Crews should expect a
fair and even assessment from all instructor/evaluators, and the
evaluators are consistently working to improve the accuracy of
crew performance assessment. The ACRM training will include
one or more detailed modules on this new form of assessment,
and the crews will have substantial opportunity to ask questions
and to fully understand the improvements in this area.

Part 5. Developing ACRM Crew Training Page 76

Developing Crew Effectiveness Modules

Understanding the Role of ACRM in

Developing CRM Skills
Crew effectiveness is a core concept of any ACRM program,
and crew training should present its main characteristics. Crew
� Crew effectiveness is effectiveness is based on a set of CRM skills that are reinforced
by the new CRM procedures. The CRM procedures help
a core concept of
any ACRM program. individuals and crews integrate CRM with technical performance
by improving overall crew coordination and communication.
The purpose of these new CRM procedures is to improve overall
crew effectiveness.

The relationship between the CRM procedures and CRM

skills is complex, and many issues have yet to be resolved by the
aviation research community. However, there are a number of
guidelines related to the CRM skills and procedures that crews
should understand. First, CRM skills should be trained in a task-
specific context providing crews with practice and feedback.
Because CRM procedures require crews to perform specific
CRM actions under certain conditions, they lead to the
development of particular CRM skills when those procedures are
properly trained through practice and feedback. The basic form
of a CRM procedure, such as a required statement of the pertinent
takeoff conditions, can be fairly simple. The skill comes into
play when a crew can perform the important parts of the briefing
quickly and effectively under a variety of conditions and in a way
that improves crew effectiveness.

The ACRM program provides crews with an operational

environment in which they are encouraged to improve their CRM
and technical performance. Further, ACRM, with its emphasis
on continual development, allows the crews as well as the airline
to further identify CRM procedures that can improve crew
effectiveness. Thus, the new CRM procedures should improve
crews’ CRM skills, and the ongoing development of these
procedures will further support CRM skill development.

Integrating CRM with Technical Performance

Crew effectiveness includes both the CRM and technical
elements of performance, and the new CRM procedures are
designed to support technical performance. The ACRM

Part 5. Developing ACRM Crew Training Page 77

� ACRM allows airlines program provides a framework that allows airlines to integrate
to integrate CRM the CRM with the technical at the SOP level. ACRM crew
with the technical at training should emphasize this integration at the operational level
the SOP level. as well as at the crew performance assessment level.

Crews should be informed that the CRM procedures

were carefully developed to support the technical side of the
operation. The content of these procedures should help crews to
communicate about and coordinate the technical tasks, especially
at points where crews are working on multiple tasks. If relevant,
crews should be shown how the new procedures are placed at
points of lower workload and precede points in time when crews
will have to be aware of certain conditions or will require certain
information. Therefore, the new CRM procedures help crews
better perform their technical tasks, and those procedures should
improve overall crew effectiveness.

With ACRM’s emphasis on an integration of CRM with

technical performance, there is a corresponding push for a
balanced assessment. ACRM provides instructor/evaluators with
specific behaviors to assess, and this helps to place CRM
evaluation at a similar level of objectivity with the technical and
maneuver validations. The crews should be encouraged to join
with instructor/evaluators to look beyond the technical reasons
for a specific difficulty and consider what CRM actions could
have prevented or averted the problem. If crews start to see areas
where the SOP needs to be changed or could be improved
through a CRM procedure, they should make specific
recommendations to the ACRM program manager.

Training and Assessing Crew Performance

Each of the new CRM procedures has implications for crew
training and assessment because it places an emphasis on the
crew’s development of specific CRM behaviors in the operational
context. It is important for crews to have an understanding of the
nature of skill performance and its implications for the training of
these new CRM procedures.

� The new CRM In addition to implications on training, the new CRM

procedures have procedures have profound implications on the way that crew
profound implications performance is assessed. A detailed simulator-based assessment
on the assessment has been developed to collect crew performance data after the
of crew performance. ACRM crew training has been implemented. This form of crew
assessment, based on the LOE, allows for the collection of

Part 5. Developing ACRM Crew Training Page 78

 performance data within a carefully designed and controlled set
of flight conditions. This comprehensive method of crew
assessment is augmented with a second type of assessment, the
Line Check. Line Checks provide an efficient method for
collecting more general crew performance data.

Assessment under ACRM involves a set of standards, and

crews should be presented with the standards along with an
explanation of how the standards will lead to a more systematic
form of crew assessment. With the team approach to crew
performance, all members should work together to improve crew
effectiveness. Individual pilots are now expected to work as a
team to support each other in working with the new CRM
procedures and in developing a set of CRM skills.

Using CRM Procedures to Improve Crew

Effectiveness
The new CRM procedures are part of SOP, but they should
not be viewed as just some additional procedures to worry about.
These procedures have been developed in order to increase CRM
awareness and skill development, and the procedures should lead
directly to improved crew effectiveness.

The new procedures should help individuals and crews in

several areas. First, they should help crews integrate CRM with
technical performance by providing direct support to some of the
critical tasks in areas such as takeoff and landings. Once these
procedures are mastered, crews should see better-coordinated
task execution because crews are more aware of each other’s
duties and decisions. Second, the new CRM procedures should
help crews in very specific CRM areas such as being aware of the
conditions affecting takeoff, the need for a backup plan, or the
assignment of duties in the case of non-normal situations. Third,
ACRM helps CRM skill development based on continual practice
of relevant skills and feedback on the quality of skilled
performance.

The entire organization should stay involved with the ACRM

program and crews should be recognized as the core of this
program. If the new CRM procedures are not improving crew
effectiveness, the crews should help determine whether better
training or modifications to the procedures is required. It is the
crews that will be the first to see the effects of these procedures,
and the crews should be encouraged to provide feedback about

Part 5. Developing ACRM Crew Training Page 79

 the new procedures as well as additional CRM procedures that
can improve crew effectiveness.

Developing Briefing/Debriefing Modules

Briefing the LOFT/LOE
With ACRM’s emphasis on the use of LOFT and LOE in the
training and assessment of crew performance, there should be one
or more crew training modules that explain the LOFT/LOE
briefing and debriefing process. These modules differ from the
instructor/evaluator modules on the same topic in that the crew
modules should be more general in nature and they will not
address specific LOFT/LOE scenarios. These modules are
particularly important in organizations that have not conducted
LOFT/LOE sessions in the past.

The LOFT/LOE briefing should prepare the crew for an

� The LOFT/LOE effective training or assessment experience. One of the first
briefing should
things a briefing should do is establish the role of the instructor
prepare the crew for
and crew throughout the session. Generally, the crew should be
an effective training
informed that the instructor/evaluator is not present as an
or assessment
instructor during the simulation; rather, his or her role is to
experience.
provide communication as ATC, company, flight attendant,
maintenance, etc. The crew’s role is to act as they would in a line
flight and deal with everyone, including the flight attendants, as if
they were actually present throughout the LOFT/LOE. A good
briefing is operationally thorough and interesting, and will
provide an overview of the overall LOE.

When developing the briefing modules, consider including

some of the following topics:

• The briefing’s primary function of establishing an environment

for open, interactive communication between crewmembers.
• The LOFT/LOE's joint emphasis on technical and CRM
performance.
• The role of the instructor in providing all communications and
resources normally available under the LOE scenario
conditions.

Part 5. Developing ACRM Crew Training Page 80

 • The fact that crews will have access to all the resources they
would have on an actual line flight.
• Crewmember responsibilities during debriefing.

Explaining the Role of Crew During LOFT/LOE

The part of the LOFT/LOE session that takes place in the

simulator should mirror line operations as closely as possible.
This part of the ACRM crew training should emphasize what
� During a LOFT/LOE, the crew should do to make the session both realistic and a
crews should act as valuable training experience. The crews should act as they
they would under would under similar situations on the line and not try to
similar situations on operate in a manner calculated to provide the ideal training
the line. department solution. This is a very important point for crews
who have not been exposed to substantial LOFT/LOE training
and assessment.

Some of the following actions that crews can take to

ensure a realistic LOFT/LOE flight could be considered:

• Perform their normal flight preparation duties.

• Use radios as they would normally during flight with
frequencies changed as required.
• Perform all normal communications, such as final weight
checks, departure reports, and in-range reports.
• Act and operate as if on the line and interact with all resources
such as maintenance, dispatch, and flight attendants.
• Plan the flight as one would a real line flight, with any service
the Company or ATC normally provides available to the crew.

Debriefing the LOFT/LOE Session

After the simulator segment of the LOFT/LOE is completed,
the debriefing is of key importance in reinforcing the good
behaviors and identifying areas that could be improved. ACRM
crew training should introduce or reinforce good crew practices
� Crews should in conducting a constructive debriefing. Crews should
understand that understand that debriefing is an essential part of the LOFT/LOE
debriefing is an and the main way of providing crew performance feedback.
essential part of the
LOFT/LOE and main
way of providing
feedback.

Part 5. Developing ACRM Crew Training Page 81

 The modules addressing LOFT/LOE debriefing should cover
all of the following topics that are directly relevant to how the
airline will be conducting debriefings:

• Understanding the debriefing agenda and time limits.

• Stating overall appraisal of the flight.
• Leading the discussion using the instructor as a resource.
• Avoiding the pitfall of making another crewmember defensive.
• Integrating technical and CRM feedback into the debrief.
• Ensuring all crewmembers participate in the discussion and
encouraging quiet crewmembers.
• Taking away a good summary and list of key learning points.
• Developing debriefing skills that can be used on the line in
cases where crew critique and review is appropriate.

For further details, see McDonnell, Jobe, and Dismukes (1997)

for guidelines for effective crew-focused debriefs of LOFT/LOE
sessions.

Developing Crew LOS Assessment Modules

Primary Technical and CRM Training
Objectives
With ACRM’s reliance on the LOFT/LOE as an integral part
of crew training and assessment, it is important that crews
understand the objectives, development cycle, and event set
framework of LOFT/LOE scenarios. This module or set of
modules should be developed to give crews confidence in the
fairness and standardization of the LOFT/LOE assessment
process. In the past there have been incidents where LOS has
been used in non-standard ways, with instructors inserting
additional elements when the crew seem to be doing well with
the events of the regular scenario. Also, there have been
questions about the operational relevance of some of the older
scenarios. Crews should be assured that LOS assessment under
ACRM is based on specific objectives and the scenarios are
operationally relevant and systematically developed. They
should also be informed that all crews will be trained and
assessed using the same objectives, and this will be the case no
matter which instructor/evaluator they will be working with.

Part 5. Developing ACRM Crew Training Page 82

 ACRM crew training is based on specific training objectives
used to develop the crew training and to direct crew performance
� It can be helpful to assessment. The high-level objectives of ACRM crew training
present crews with should be presented so that crews have a clear sense of the
the CRM objectives in purpose of the training session. Some of the higher-level CRM
the context of objectives will include learning the content and context of the
new CRM procedures. At some point there will be further
relevant technical
objectives such as developing skills in performing the new CRM
objectives because procedures. ACRM crew training may also have some technical
they may be easier to objectives. In some cases these objectives will be associated with
understand and more the tasks being performed either during or after the CRM
familiar. procedure. For example, an improved takeoff briefing will have
as one of its objectives a better-executed takeoff (possibly under
certain specific conditions). It can be helpful to present crews
with the CRM objectives in the context of relevant technical
objectives because crews may initially be more familiar with and
better able to understand the technical objectives.

Incidents that Support Training Objectives

A structured development process is used in preparing the
LOS assessment scenarios. ACRM crew training should explain
that this process starts with the analysis of the airline’s training
needs and follows through to the development of the LOS and its
� Crews should validation. A major step in this development process is the
understand that the identification of incidents that highlight the need for specific
LOS is based on CRM procedure and that can be used to support the training
operationally objectives. These are incidents that may have happened at the
relevant incidents. airline or at other airlines with a similar type of operation. Crews
should understand that the LOS is based on actual incidents that
have happened to pilots flying in conditions similar to theirs.

Crews are usually interested in such incidents and in the

incident identification process. Consider explaining that
incidents can be identified by searching ASRS reports, talking
with one’s own airline flight safety manager or searching the
airline database. The search usually starts based on a set of
categories identified by the airline as being primary issues for the
LOS scenario. Then either the NASA/ASRS or airline staff can
perform a search using these categories as keywords, or the
airline can access the report database on CD-ROM and perform
their own searches.

Part 5. Developing ACRM Crew Training Page 83

 Examples of topics that have been searched to identify
supporting incidents include rerouting/amended clearance
incidents, low fuel during excessive vectoring, delayed
approaches, similar call signs, auto-flight incidents, flight toward
terrain, and runway incursion. The specific training objectives
and the new CRM procedures are used to refine the topics and
conduct the search for incidents that are to be used in the
development of LOFT /LOE event sets.

Consider planning a crew training activity that can take place

within this module which allows crews to work with several
incidents used to develop the new CRM procedures. Additional
sources for incidents include:

• Incident reports from industry databases such as the ICAO

incident database.
• Maintenance problems identified through operational reports.
• Poor crew performance areas identified in line and simulator
proficiency checks and training.
• Airline safety reporting system.

SAMPLE INCIDENT REPORT

(see full report in Appendix G, Report 56)
BGM information 800’scattered, 1200 broken, 2300’ overcast, 2
mi. light snow, temp 27, dew point 21, winds 270 at 8 kts,
altimeter 29.97, breaks in the overcast, NDB 34 approach in use,
localizer 16/34 out of service, runway 34 plowed and sanded full
width and length, braking action good reported by a vehicle. PF
and PNF discussed possibility of using runway 28 due to surface
wind component. Further inquiry with BGM approach confirmed
runway 28 plowed and sanded full width and length and previous
inbound company reported braking action as poor. Surface wind
was also reported as unchanged. PF (PIC) requested and was
cleared for VOR DME approach runway 28... The PF descended
from the published segment alt (3500’ MSL) at the 18 DME
position to the published straight in landing MDA of 2000’MSL.
The FAF for the procedure was at the 13 DME position and the
PF’s premature descent put the aircraft 1500' below the published
segment alt...

Part 5. Developing ACRM Crew Training Page 84

 SAMPLE INCIDENT ANALYSIS

During the approach the crew focused on the runway choice but
lost situation awareness of the aircraft’s altitude with respect to
the Minimum Safe Altitude. Crew was rushed due to the slam
dunk approach into the airport which increased workload. Not
setting and monitoring a bottom line for the minimum safe
altitude allowed the PF to execute a premature descent to an
inappropriately low altitude for that segment.

SAMPLE LESSONS LEARNED

Crews can benefit from this incident by recognizing the role of

good workload management and clear communication of critical
aspects of the approach. Good workload management would
suggest briefing of the overall plan for the approach and relevant
situational details such as the MSA prior to the high workload
period of the approach or as early in the approach phase as
possible. Prior briefing of the overall plan would enhance crew
situation awareness during approach and reduce the number of
necessary communications during approach. Setting explicit
bottom lines for critical aspects of the approach such as the MSA
and assigning the monitoring of the bottom line to a crewmember
would help maintain situation awareness of these safety-critical
conditions despite the heavy approach workload.

Event Sets and the LOFT/LOE Scenario

Crews should be given an explanation of how the event set is
used under ACRM to help focus the specific aspects of crew
performance in the LOFT/LOE assessment. An event set divides
a LOFT/LOE session into sections, with each section or event set
� Crews should be
having specific training or assessment objectives. Event sets are
given an explanation
designed to concentrate on specific CRM and technical training
of how the event set
is used under objectives allowing the crews to concentrate on a narrow range of
ACRM. performance.

Crews do not need to be concerned with the start and end

point of each event set, but they should understand that
instructor/evaluators have been trained in the role of the event set
as a tool to administer the LOFT/LOE session and as the primary
unit of crew performance assessment. An event set is made up of
one or more events, including an event trigger, supporting
conditions, and distracters. The event trigger is used to fully

Part 5. Developing ACRM Crew Training Page 85

 activate the event set and can be the introduction of a malfunction
or the activation of one or more environmental or operational
conditions. Supporting conditions are other events used
primarily to maintain realism. Finally, a distracter may be
inserted within the event set time in order to divert the crew’s
attention or to increase their workload.

Event sets are carefully designed to support the training and

assessment objectives and the consistent administration of the
LOFT/LOE scenario. In addition, event sets can be designed to
an equivalent level of difficulty to make sure that different
scenarios present a fair and equal group of problems. Difficulty
levels initially can be judged by SMEs, and later checked by data
analysis of average crew performance across the event sets.

ACRM Crew Training Development Guidelines

Guidelines for Planning ACRM Crew Training
1) Crew training can be the largest cost of the ACRM program,
so the development team should use the planning stage to
determine ways for reducing the cost of that training. The team
should consider integrating ACRM with existing CRM training
(see Guideline 2 below) and coordinating ACRM training with
other training cycles (see Guideline 3 below).

2) The development team should consider extending or

modifying existing crew training to meet ACRM training needs.
Much of the existing CRM recurrent training can be modified
from an emphasis on general CRM principles to a concentration
on the specific CRM procedures.

3) The development of ACRM crew training should be

coordinated not only with crew training cycles but also with
instructor/evaluator training and the implementation of the new
CRM procedures.

4) ACRM crew training development and implementation

should include a review process by scheduling ACRM crew
training curriculum reviews that involve key organizational
personnel who control the future of the ACRM program.

Part 5. Developing ACRM Crew Training Page 86

 Guidelines for Developing CRM Procedures
Crew Modules
1) When training crews about the need for CRM procedures, it
should be emphasized that the crews have been an essential part
of the development process with crew representatives involved in
developing the current training. In the future, crewmembers
should stay involved and help identify new CRM procedural
needs as they become apparent.

2) A key component of new CRM procedures is the degree to

which they support the development of CRM skills. To help
crews practice and master the appropriate skills, crews should be
informed of the relationship between the CRM procedures and
their CRM skill development.

3) Early in crew training it should be explained that the main

components of an ACRM program include the CRM procedures,
training of the instructor/evaluators, crew training, a standardized
assessment of crew performance, and an ongoing implementation
process.

4) It should be explained that detailed measures of crew

performance were developed in the research stages of ACRM
specification and analysis. These more accurate crew
performance measures will help evaluators provide more
standard crew assessments, and crews should expect a fair
assessment from all instructor/evaluators.

Guidelines for Developing Crew Effectiveness

Modules
1) Crew effectiveness is a core concept of any ACRM program,
and crew training should present its main characteristics. Crew
effectiveness should be enhanced by using the new CRM
procedures that will help crews develop better CRM skills.

2) The ACRM program provides a framework that allows

airlines to integrate CRM with technical at the level of SOP.
ACRM crew training should emphasize this integration at the
operational level as well as at the crew performance assessment
level.

Part 5. Developing ACRM Crew Training Page 87

 3) It should be explained that the new CRM procedures have
profound implications on the way crew performance is assessed.
For example, if it is the case, crews should understand that a
detailed simulator-based assessment has been developed to
collect crew performance data after the ACRM crew training has
been implemented.

4) It should be emphasized that crews form the core of the

ACRM program, and crews should be encouraged to provide
feedback about the new procedures as well as additional CRM
procedures that can improve crew effectiveness.

Guidelines for Developing Briefing/Debriefing

Modules
1) The LOFT/LOE briefing should prepare the crew for an
effective training or assessment experience. One of the first
things that the LOFT/LOE briefing should do is establish the role
of the instructor as well as the role of the crew throughout the
session.

2) The LOFT/LOE briefing should emphasize specific things the

crew can do to make the session realistic and a valuable training
experience. The crews should act as they would under similar
situations on the line, and they should not try to operate in a
manner calculated to provide the ideal training department
solution.

3) ACRM crew training should reinforce good crew practices in

conducting a constructive debriefing. Crews should understand
that debriefing is an essential part of the LOFT/LOE and the main
way of providing crew performance feedback.

4) ACRM crew training should introduce the concept that

briefings can also be extended to line operations and used by
crews to reinforce positive performance and learn from any
problems experienced during the flight.

Guidelines for Developing Crew LOS

Assessment Modules
1) Crews should be presented with the LOS CRM training
objectives in the context of relevant technical objectives because
crews are generally more comfortable and familiar with the
technical objectives.

Part 5. Developing ACRM Crew Training Page 88

 2) A major step in the LOS development process is the
identification of incidents that highlight the need for specific
CRM procedure, and crews should understand that the LOS is
related to actual incidents that have happened to pilots flying in
conditions similar to theirs.

3) Crew training should include one or more activities that allow

crews to analyze incidents used to develop the new CRM
procedures. One activity that has provided valuable crew training
is having crews identify the CRM causes for each incident.

4) Crews should be given an explanation of how the event set is

used under ACRM to help focus the specific elements of crew
performance in the LOFT/LOE assessment. The event set helps
training and evaluation move from general CRM markers to
specific crew behaviors.

Part 5. Developing ACRM Crew Training Page 89

 This Page

Intentionally Left Blank

Part 5. Developing ACRM Crew Training Page 90

Part 6. Extended Table of Contents

Part 6. Implementing and Evaluating ACRM ................................................................91

Overview of Implementing ACRM within the Organization ...................................92

Preparing Organizational Announcement of ACRM.........................................92

Ensuring Ongoing Management and Union Support.........................................93

Reporting Meaningful Crew Performance Data................................................94

Implementation Requirements ...............................................................................95

Announcing the New Policies and Procedures..................................................95

Implementing the New Documentation ............................................................96

Scheduling Crew Training ...............................................................................97

Selecting Strategies to Reinforce Crew Awareness and Feedback ....................97

Developing Additional CRM Procedures as Needed ........................................98

Implementing ACRM for the Instructor/Evaluator.................................................99

Refining Instructor/Evaluator Assessment Tools..............................................99

Ensuring Evaluator Reliability Prior to LOFT/LOE Assessment.......................100

Providing Instructor/Evaluators with Ongoing Feedback .................................101

Planning and Establishing Standardization Meetings ........................................102

Maintaining Standards and the IRR Process...........................................................102

Identifying LOE/LOFT Event Set Problems.....................................................102

Identifying Individual and Group Instructor/Evaluator Problems ......................103

Identifying Individual or Crew Performance Problems .....................................104

Identifying Trends in Crew Performance..........................................................105

Using Performance Data to Improve CRM Procedures and Training......................105

Ensuring Reliable Data ....................................................................................105

Linking Performance Problems to CRM Skills .................................................106

Identifying Training Areas that Need Improvement..........................................107

Specifying Additional CRM Procedures...........................................................108

ACRM Implementation Guidelines ........................................................................109

Guidelines for Organizational Implementation..................................................109

Guidelines for Implementing ACRM for Instructor/Evaluator ..........................109

Guidelines for Maintaining Standards and IRR.................................................110

Guidelines for Improving CRM Procedures and Training .................................111

Part 6. Implementing and Evaluating ACRM

Part 6 of the Manual presents the steps for implementing and

establishing a solid ACRM evaluation. The implementation
process is ongoing and generally follows the development of the
ACRM crew training. Implementation starts with organizational
announcements and continues by working with the
instructor/evaluators to maintain evaluation standards. The
ongoing implementation and evaluation cycle concludes by using
the performance data to improve the training, with the cycle
starting over again.

Guiding the
Organization

Developing
CRM
Procedures

Developing Developing
I/E Training Crew Training

Implementing and Evaluating ACRM

• Overview of Implementing ACRM.............. page 92
• Implementation Requirements.................... page 95
• Implementing ACRM for the I/E................. page 99
• Maintaning Standards and the IRR Process..... page 102
• Using Performace Data to Improve Training...page 105
• ACRM Implementation Guidelines.............. page 109

Part 6. Implementing and Evaluating ACRM Page 91

Overview of Implementing ACRM within the Organization

Preparing Organizational Announcement of

ACRM
To ensure successful ACRM implementation, the content and
timing of the organizational announcement should be carefully
planned. It is assumed that key personnel have been kept
informed of the ACRM development progress and they should be
very helpful in designing and scheduling the announcement. The
organizational announcement sets the stage throughout the airline
for the start of crew training and the implementation of ACRM.
Throughout the development process, representatives from
different departments and groups have been kept informed, and
the announcement is a way to put out a strong, unified message
that ACRM and its CRM procedures are about to become part of
SOP.

The content of the announcement should be relatively brief

and yet sufficiently complete to convey critical information. The
announcement should provide a brief summary of most of the
following items:

• Organizational endorsement of the program.

• Why the airline needs an ACRM program.

• The key features of ACRM.

• The main dates through ACRM implementation.

• Departments and/or individuals who supported the

development effort.

Timing of the organizational announcement is important.

Instructor training, crew training, and the implementation of the
CRM procedures will all affect that timing. Each organization
� The organizational may take a slightly different approach to the timing of these
announcement critical events, but a sequence where the CRM procedures are
should be timed announced prior to the start of crew training and possibly before
based on instructor or during instructor/evaluator training should be considered.
training, crew Ideally this should be planned as a relatively short sequence,
training, and the where the crew training and implementation follow closely to
implementation of avoid the situation in which some crews have initial ACRM
CRM procedures. training eight or more months prior to implementation. When

Part 6. Implementing and Evaluating ACRM Page 92

 that happens there is a tendency on the part of crews to forget
some of the key ACRM elements.

Ensuring Ongoing Management and Union

Support
Just as management and union representatives should have
been involved in the development process, they should be
encouraged to maintain that involvement once the ACRM
program has been implemented. This ongoing involvement can
provide valuable input to the program and should result in
continued support.

It is important that management be kept informed of the

� For management, it
general trends, both crew performance and instructor
is important that they
standardization. Throughout the development cycle,
be kept informed of
management should have been familiarized with data on current
the general trends,
trends in training and the method used to collect CRM
both crew
performance and performance data and instructor standardization data, and the
instructor types of questions that can be answered with these types of data.
standardization. During and following ACRM implementation, work closely with
management to refine the types of reports most useful to them.
Have management discuss the types of questions they would like
answered through the data and decisions they need to make that
could be informed by the data. Then consider presenting a range
of data at an informal meeting, and ask management which data
and formats are most useful for the decisions that they need to
make. Work to provide them with a report that presents the data
they have asked for in the formats they have requested. Initially
provide some preliminary analyses that address their most
important questions, and ask management to provide feedback
about additional data, questions, and frequency of reports.

With union representatives, continue offering data that

establish the reliable and accurate qualities of the ACRM crew
assessments. Continue working closely with one or more union
representatives in developing new CRM procedures as well as
new LOFT/LOE scenarios. Historically, pilot unions have had
some serious objections to the assessment of CRM. Other union
representatives have supported the use of CRM procedures;
however, they may still have a number of reservations. Union
representatives who are kept informed about the specific CRM
procedures and the objective CRM assessment are more likely to
support the ACRM program and want to see it further established
within the airline.

Part 6. Implementing and Evaluating ACRM Page 93

 Reporting Meaningful Crew Performance Data
The reporting of meaningful crew performance data is one of
� The reporting of the most important ways to ensure ongoing success with the
meaningful crew ACRM program. It can be difficult to understand the relationship
performance data is between own airline incidents and CRM, in part because airlines
one of the most do not tend to report CRM causes of incidents. The reporting of
important ways to ACRM data should be used to develop an understanding of CRM
ensure the ongoing problems. ACRM goes beyond reporting CRM attitudes or
success of the opinions, and it looks for solid links between crew performance
ACRM program. and the CRM procedures. No two airlines will approach this in
exactly the same way, but all airlines should be looking at 4 to
12-month trends that show a positive relationship or correlation
between performance on CRM procedures and overall crew
performance. In addition, it may be possible to identify an
inverse relationship, or negative correlation, between the
performance on CRM procedures and number of incidents.

Care should be taken not to report too much data. There may
be and inclination to report everything, but do not overload those
who need the information. It is better to hold off presenting an
initial set of results with multiple interpretations until addition
data can be collected and the interpretation refined. Consider
reporting some of the following types of data:

• Distribution of overall performance by position and fleet.

• Distribution of CRM performance by position and by fleet.
• Distribution of problem event set performance.
• Distribution of overall performance by fleet.

Data on instructor or evaluator standardization can come

from the recurrent inter-rater reliability sessions. Further, the
LOFT/LOE performance information can be used to isolate
problems with systematic differences among instructors, low
congruency, or low consistency of evaluation profiles.

LOE quality can also be evaluated during initial development

of the LOE and later from rating data. The internal relationship
of ratings gives an indication of how well each event set is
working as part of crew evaluation. LOE performance can also
be used to target areas that need increased training emphasis and
those areas that require less. The pilot performance profiles can
target the necessary re-allocation to optimize the effectiveness of
training.

Part 6. Implementing and Evaluating ACRM Page 94

Implementation Requirements
Announcing the New Policies and Procedures

Although implementation of ACRM affects the airline’s

philosophy, the announcement of the CRM procedures affects
airline policy and procedures. The organizational announcement
sets the stage throughout the airline for the start of crew training,
while the announcement of the procedures is a more detailed
specification of the new policies and procedures. The content
and format of the procedures announcement is of paramount
importance and should be carefully coordinated with flight
standards and documents departments.

The content of the procedures announcement should include

the rationale and format of the new procedures. The policy
segment of this announcement presents the rationale or need for
these procedures, along with any required additions or changes to
airline policy. Generally, the policy segment is fairly brief, and it
can be included in its entirety in the announcement. In addition
to the text of the new policy, references should be provided to all
applicable operating documents. The procedures segment is
more detailed, and consideration should be given to whether the
new procedures will be printed along with the announcement or
whether they will just be referenced with their location in the
operating documents. If there are only a few new CRM
procedures and they can be displayed on a few pages, consider
including the full procedure with the announcement. If, on the
other hand, there are many new procedures or they occupy many
pages, then consider just having references in the announcement
with one or two key examples.

The procedures announcement should be made close to the

implementation time. An airline can consider including
supporting material. For example, one airline included a
videotape showing examples of crews using the new procedures.
This can be particularly helpful for procedures where crews have
had many questions or difficulty performing the procedure during
ACRM crew training or where time has elapsed since initial
training.

Part 6. Implementing and Evaluating ACRM Page 95

 Implementing the New Documentation
Implementing the new operating documents can become a
substantial expense of the ACRM program, but there are ways to
minimize or eliminate some of those costs. By coordinating
closely with the document update cycle, the implementation can
be achieved at little or no expense.

Before presenting the coordination considerations it should be

noted that in some cases a new document is essential to the
success of the CRM procedure. For example, an airline may
identify a need for a QRH or guide that they are currently not
using. In such cases, the new document will be an additional
cost. However, these costs can be moderated by carefully using
more expensive resources, such as color. In many aviation
situations, only one or two additional colors are needed. In such
cases, it can be more cost-effective to go with black and white
plus one or two specific colors rather than going with full color.
Also, the type of lamination, drilling, and tabs can add substantial
expense. Use such features only when they substantially improve
the usability of the document.

The need for coordination with the update of existing

documents is most evident at larger airlines where even one
change in a procedure can result in the need to reprint one or
more pages for 20,000 manuals. That can be a substantial cost
� Ensure a formal
and should be reduced or limited through planning and
review of the camera- coordination. Major airlines have scheduled document updates,
ready material and if the changes required by the new CRM procedures are
because there will be coordinated with those updates, the cost can be greatly reduced.
errors, and the more Depending on the type of document, it may have a quarterly or
informed people that annual update cycle. As is suggested in Part 3, most or all of the
review the material, costs can be included as part of an update by working out the
the greater the coordination details during the ACRM planning stage. As part of
chance of noting this coordination, ensure a formal review of the camera-ready
them. material because there will be errors. The more informed the
people who review the material are, the greater the chance of
noting those mistakes.

Part 6. Implementing and Evaluating ACRM Page 96

 Scheduling Crew Training
Scheduling ACRM crew training should be coordinated not
only with crew training cycles but also with the instructor/
evaluator training and the implementation of the new CRM
procedures. Often, crew training starts with the classroom
segment which may then be followed at a later date with
simulator sessions. Make sure that the instructors needed for
classroom training have completed that portion of their training
prior to the start of crew training. For the simulator portion,
make sure that trained instructor/evaluators are available in
sufficient numbers prior to the start of crew simulator sessions.

As discussed in Part 5, ACRM crew training should be

integrated with the rest of the training to reduce the additional
required training time. Consider integrating ACRM crew
training with recurrent training with a minimum increase in
required hours. Follow through with scheduling to make sure
that all crews are scheduled with minimum disruption to training
and flight operations. Be aware that poor or disorganized
scheduling reflects poorly on the ACRM program, so ensure that
this training scheduling runs as smoothly as possible.

Selecting Strategies to Reinforce Crew

Awareness and Feedback
� In order to promote
an ongoing ACRM In the past, CRM training has been implemented as a one-
program, crew shot deal. In order to promote an ongoing ACRM program, crew
awareness should be awareness should be reinforced and crew feedback should be
reinforced and crew systematically elicited and collected. These two requirements are
feedback should be particularly important during the first year of the program as the
elicited and organization becomes familiar with it. Consider having one or
collected. more individuals from the development team tasked with keeping
ACRM visible and eliciting crew feedback.

The entire organization should stay involved, but crews form

the core of an ACRM program. It is their acceptance and use of
the CRM procedures that will produce a successful program, so
crews should be reminded of the procedures and training at
appropriate times. Crews have many things competing for their
attention, and ACRM crew training can be perceived as just one
of a number of training routines that need to be completed. The
challenge is to select the proper form and time to remind crews of

Part 6. Implementing and Evaluating ACRM Page 97

 the new procedures and their role in improving crew
effectiveness. Some organizations may write up positive crew
reactions or operational stories that underscore the value of the
procedures, and place one or more articles in an existing
organizational publication with good crew readership and
acceptance. In smaller organizations, consider having a respected
crewmember speak up about the new procedures at a scheduled
crew meeting. Instructor/evaluators can also be influential in
sustaining crew awareness, and they may be asked to make an
announcement or comment on improved crew performance in
specific ACRM areas.

The crews are often the first to see the effects of the new
CRM procedures and the training, and they should be encouraged
to provide feedback about all aspects of the program. Consider
doing this not only during ACRM training, but also during line
checks and other points of crew interaction with
instructor/evaluators. In addition to feedback on the new CRM
procedures, ask crews for ideas about additional procedures that
should be considered for implementation. Crews may come up
with their own ideas or they may have heard about or used a
useful CRM procedure at another airline. Consider a range of
ways of collecting information on additional procedures to
include an item on a feedback form, a specific question during
LOFT/LOE debriefing, or request for information at the end of a
ACRM awareness article or reminder.

Developing Additional CRM Procedures as

Needed
Crew feedback and performance data provide essential
information to determine the need for additional procedures.
Another important source is instructor/evaluator feedback about
what they are seeing in the simulator and on the line. Finally,
airline safety reports may show a rise in a particular type of
incident whose frequency could be reduced through some
combination of procedure and training. If an airline actively
collects information about possible CRM problem areas, need
identification and development of additional procedures is
facilitated.

Airlines should treat the development of the first set of

procedures as a valuable learning process, and they should
maintain key elements of that team to help with additional
development. By utilizing the experience gained from the first

Part 6. Implementing and Evaluating ACRM Page 98

 round, airlines can simplify the development process and work
with a more efficient team. Therefore, one of the keys to the
successful development of additional CRM procedures is
planning for that need during the initial procedures development.
At that point an airline should evaluate who the long-term players
will be and take steps to ensure that those individuals can stay
involved in the ongoing elements of the ACRM program.

Implementing ACRM for the Instructor/Evaluator

Refining Instructor/Evaluator Assessment
Tools
The instructors should consider the refinement of the
� Refinement of the LOFT/LOE gradesheet as one of the most important steps in an
LOFT/LOE organization’s drive to advance a reliable and standard crew
gradesheet is one of performance assessment team. As discussed in Part 4, a number
the most important of guidelines should be considered in the design of the
steps in maintaining gradesheets, including organizing them around event sets using a
a reliable and standard rating scale and representing that scale on every page.
standard crew Once the basic elements of the gradesheet are in place, the
performance instructors should be encouraged to refine any of the elements
assessment team. that may be causing assessment problems.

Early indicators of problems with the gradesheet include

finding that required fields are not being completed or are being
completed with out-of-range values. These types of problems
can easily be addressed at an instructor/evaluator meeting, and
normally the refinement involves correcting a data field heading
or adding more information or instructions. More subtle
problems may emerge when lack of agreement or a large amount
of variance is observed for specific items on the gradesheet. This
lack of agreement often is associated with one or more
observable behaviors that have not been well worded or require
additional standards. One of the benefits of ACRM’s reliance on
good data collection is that these problem items can be readily
identified. Again, instructor/evaluator meetings can be used as a
forum for refining the wording or further specifying the standard.

The rating scale is another area that may require refinement.

When airlines start to use a more precise rating scale, as is
required under ACRM, difficulties can arise in defining what is
“Standard” and what is “Not Acceptable” in specific situations.
One point of confusion can develop around the difference

Part 6. Implementing and Evaluating ACRM Page 99

 between “Standard” performance and “Average” performance.
Some evaluators consider what they normally see on the line or in
the simulator as “Standard” performance. Under ACRM, that is
defined as “Average” performance; “Standard” performance is
the behavior that meets specific criteria. Instructor/evaluators can
benefit from the further specification of “Standard” and/or “Not
Acceptable” criteria for specific items. Specification of these
criteria can be time consuming, so evaluators must work to select
those items causing the greatest amount of difficulty.

Ensuring Evaluator Reliability Prior to

LOFT/LOE Assessment
ACRM promotes the use of videotapes of crew performance
� ACRM promotes the in conjunction with the IRR process to establish evaluator
use of videotapes of reliability. Videotapes of real crew performance, as opposed to
crew performance in tapes of crews acting a part, allow a group of instructors to
conjunction with the observe and rate the type of performance they are likely to see in
IRR process to LOFT/LOE sessions. The IRR process is used to provide those
establish evaluator crews with the results of their ratings along with measurements of
reliability. their agreement, congruency, consistency, and sensitivity.

An airline should strive for a specified level of reliability

prior to conducting the LOFT/LOE portion of crew training. One
way to ensure that the instructor/evaluators reach that level is to
pick a realistic set of benchmarks. If an airline does not have
substantial experience in the standardization of assessment, it is
reasonable to establish a more moderate set of benchmarks. For
example, Williams, Holt, & Boehm-Davis (1997) propose a
range between .7 and .8 for an Agreement Index benchmark. For
airline with little or no experience in this area it would prudent to
select a goal of .6 to .7 across all items being rated and work
diligently to establish that level. From experience with a range of
airlines, it is difficult to establish a .8 to .9 level of agreement,
especially when working with a complex LOE and real crew
behaviors. Over time, an airline can raise its benchmarks as the
assessment team develops strategies to maintain good reliability.
The proposed benchmarks for the IRR components (Williams et
al., 1997) are:

Agreement .70 to .80

Congruency .50 to .80
Consistency .65 to .80
Sensitivity .40 to .60

Part 6. Implementing and Evaluating ACRM Page 100

 An airline should also consider several other strategies in
meeting a level of reliability prior to the start of LOFT/LOE
� Other strategies for assessment. If a particular item on the gradesheet is causing
meeting an IRR problems, and if, after trying to refine the items, that problem
benchmark include persists, consider the importance of that item and whether it
removing some could be removed. Sometimes, the refinement or removal of just
items that are a few items allows an airline to meet its Agreement Index
causing rating benchmark. An airline may also have to look at its assessment
problems. team and determine whether all the members can be trained to the
benchmark within the time required. Additional training may be
an option for some team members and, in other cases, some
individuals may decide they do not want to be part of the
assessment team. Benchmarks give airlines a concrete way of
knowing whether they have established the reliability goals.

Providing Instructor/Evaluators with Ongoing

Feedback
The training department should plan on providing
instructors/evaluators with ongoing rating feedback at appropriate
times. By maintaining current instructor data, the training
department and instructor/evaluators can determine the frequency
and exact form of that feedback. In general, if an assessment
team continues to meet its benchmarks and there are no
indications of rater bias, the feedback can be on a monthly or
quarterly basis depending on instructor preference and airline
resources to provide the feedback. If the assessment team cannot
maintain the benchmarks, then more frequent feedback is
necessary.

Several forms of feedback should be considered.

Instructor/evaluators should be provided with both individual and
group measures of IRR. Consider emphasizing the group
measures by presenting the results that emphasize group averages
rather than individual scores. This should be done to promote the
team approach to assessment where the goal is to have all
individuals meet a standard. When there are problem areas it can
be helpful to provide some members with individual feedback
that shows the areas where they depart from the group norm.
Encourage those individuals with specific problems to work with
the group in resolving the issues. ACRM assessment should
always be approached as a group effort, where the team, and not
just one individual, needs to solve any outstanding problems.

Part 6. Implementing and Evaluating ACRM Page 101

 Planning and Establishing Standardization
Meetings
The airline and training department should help the
instructor/evaluators plan and hold ongoing standardization
meetings. To maintain confidence in the crew performance data,
instructor/evaluators need to meet periodically and develop a
procedure that will allow them to determine whether they are still
keeping their IRR benchmarks. An assessment team may have
achieved a good level of reliability, but with the change in team
members and shifts in individual rating standards, it is important
to get the team together and determine their level of reliability at
regular intervals. The standardization meetings can take the form
of group meetings, short workshops, or part of an instructor/
evaluator recurrent training session. These meetings usually
include discussions of group and individual problems, refinement
of the gradesheets and standards, and rating of new crew
performance tapes based on two or more event sets from current
LOEs.

Consider ways to involve instructors in the planning and

implementation of the standardization meetings. This fits in with
� Assessment
the team approach to crew performance assessment, where the
standardization is a
group looks out for the individuals and takes responsibility for
long-term maintaining its standards. Turn each session into a positive
development process experience even if the group is having substantial difficulties
which instructor/ maintaining the desired level of reliability. This can be done by
evaluators should emphasizing several positive data trends, and by making sure that
direct with everyone leaves with some good tips about how to manage or
organizational support. assess the LOFT/LOE session, use the gradesheet, or provide
better debriefings. The airline should treat this as a long-term
development process, which the instructor/evaluators should be
able to direct with organizational support.

Maintaining Standards and the IRR Process

Identifying LOE/LOFT Event Set Problems

� The IRR process When LOFT/LOE crew performance data shows a drop in
permits identification ratings, there can be a range of causes from properties of the LOE
of probable causes scenario to lack of crew training in specific areas. The IRR
process allows the airlines to isolate probable causes with a
with a greater degree
greater degree of certainty than has been possible up to this point.
of certainty.

Part 6. Implementing and Evaluating ACRM Page 102

 Some early warnings about problems with one or more event
sets can surface during instructor/evaluator training. Whether in
large or small group assessment sessions, if one or two event sets
show less agreement and follow up analysis shows lower
consistency for those event sets compared with all or just the
remaining event sets, then there is likely a problem with the items
being rated. If there is no pattern of problems during the training
phase but a pattern arises after the instructor/evaluators start to
administer a number of LOFT/LOE sessions, then the problem
may be caused by some element within the event set or by a
limitation with the simulator. When analyzing the crew
performance data, if crew performance is lower for just one or
two event sets, look for problems in administering a particular
item in the event set. For example, the event set may require
simulating a certain malfunction, but the simulator fails to
display or simulate all required cues. These types of problems
should surface early in the new LOFT/LOE cycle and are some of
the easier to solve. Once the cause has been isolated within a
specific element of an event set, the solution is usually a minor
change to the scenario, script, or gradesheet. If that does not
solve the problem, then the data could be pointing to crew
performance difficulties.

Identifying Individual and Group

Instructor/Evaluator Problems
One point of this causal analysis is that lower ratings do not
automatically mean poor crew performance. Once an airline has
established and maintains a standard assessment, they can use the
resulting data to more quickly and accurately pinpoint the
problems. In addition to problems with the scenario and the
gradesheet, there may be either individual or group problems
with the instructor evaluators.

Under ACRM, a substantial amount of instructor/evaluator

training is provided, but it is never possible to cover all items that
are to be graded anticipating every possible crew behavior. Once
� Lower ratings on instructor/evaluators start administering LOFT/LOE sessions, the
several items may data may show a pattern of lower ratings in certain areas and that
point to areas where some of the group IRR benchmarks are not being met. This
instructors do not could indicate a problem where the instructor/evaluators do not
have the same
have the same interpretation of an event set or the standards of
interpretation of an
performance. This type of group problem can arise when what
event set or
was thought to be a clear criterion or standard was actually
standard.
ambiguous or did not account for the observed crew responses.

Part 6. Implementing and Evaluating ACRM Page 103

 In such cases, instructor/evaluators cannot give a standard
assessment and better LOE specification or additional group
training is required.

In some cases, the IRR measures will show specific

individual problems. For example, one or two instructors will
not meet the individual level consistency benchmark, while the
group measure does. In such a case, those individuals may also
fail to meet the sensitivity benchmark, and it is likely that they
are having a problem discriminating standard vs. non-standard
performance. This problem may arise with rater bias or not using
the group criteria when applying the rating scale. An airline may
decide to work directly with those individuals, but should
consider having the assessment team work as a group to resolve
the problems. Any lack of standardization, whether group or
individual, is a team problem, and members should feel
comfortable turning to the team for help.

Identifying Individual or Crew Performance

Problems
Once the event set, gradesheet, and instructor/evaluator
causes have been eliminated, then the assessment team and the
organization can have confidence in identifying either individual
or larger group performance problems. Of the two, individual
crew performance problems are most frequent and show up as
� Monitoring for group below average ratings in one or more performance areas.
crew performance Statistical methods can be used to objectively identify these
problems allows an below-standard patterns of performance. With ACRM, the
airline to be training department can draw better inferences from such
proactive by patterns, going beyond the old conclusion that the must be a weak
addressing a minor individual or crew. First, the training department can verify that
problem before it this is a bona fide problem with the individual by making sure the
turns into a set of instructor/evaluator has good congruency and consistency with
incidents or an the rating team. This is a particularly important step to take if the
accident. case involves a possible failure. All involved need to be
confident that this is not a case of one instructor placing
unreasonable demands on a particular crew or individual.

A careful analysis of all the gradesheet results, along with the

debriefing summary, can help the instructor/evaluator or training
department analyst pinpoint the problem areas that will require
additional training. The old approach of just giving the crew
more CRM training does not apply. Under ACRM, remedial
training addresses specific behavioral and skill grouping aspects
of CRM where the crew was below standard.

Part 6. Implementing and Evaluating ACRM Page 104

 If over time a pattern of crew problems becomes evident in a
specific area, it is likely that crew training in that area needs to be
modified. This is one reason the "Below Standard" rating is so
important, because it allows you to identify marginal problems
before they turn into "Unsatisfactory" performance. Monitoring
for group crew performance problems allows an airline to be
proactive in its training development where it can address a
minor problem before it turns into a set of incidents or an
accident.

Identifying Trends in Crew Performance

� A major payoff to
establishing and A major payoff to establishing and maintaining a standard
maintaining a assessment is the ability to identify long-term trends in crew
standard performance. Without the team approach to assessment and the
assessment is the data establishing that the IRR benchmark have been met, there is
ability to identify too much variation in the data to make meaningful interpretations
long-term trends in of performance trends. In such cases, a significant increase or
crew performance. decrease in crew ratings may be due to many factors not directly
related to crew performance.

With a standard assessment established over a period of time,

significant improvements or decrements in crew performance can
be analyzed with confidence. For example, if there is a
consistent trend over a year or more showing improvement in
some aspect of crew performance, the training department can
conclude that aspect of the training is working as designed. If
that positive trend is in the ’Above Standard’ area, it may be
possible to move some of the training time from that area to
another area of crew performance. In cases of significant
negative trends, a detailed analysis should help the airline to
pinpoint the problem area and improve or redesign that part of
the training.

Using Performance Data to Improve CRM Procedures and

Training
Ensuring Reliable Data
With the large amount of data that can be produced under
ACRM it is easy to lose sight of the big picture. The ACRM
program, coupled with the use of the IRR process, is designed to

Part 6. Implementing and Evaluating ACRM Page 105

 Provide an airline with ongoing improvements in CRM
performance. Once the CRM procedures have been developed
and the ACRM training implemented, an important short-term
goal is to establish and maintain the collection of reliable crew
performance data. This may sound like a relatively easy task, but
it should be noted that very few airlines to date have been using
appropriate procedures or collecting the proper type of data to
assure reliability.

� Reliable crew Reliable crew performance data are a minimum requirement

performance data for airlines to successfully improve the ACRM program, and
are a minimum airlines should be prepared to follow certain steps to ensure that
requirement for reliability. Reliable crew performance data allow airlines to
airlines to pinpoint the problems and their causes. With confidence in
successfully improve specific causes, an airline can then invest the needed resources to
the ACRM program. correct the problem. Within the ACRM framework, airlines do
not solve crew performance problems by increasing or changing
their CRM training based on what other airlines or organizations
are doing. Rather, an ACRM airline develops airline-specific
CRM procedures or provides focused training to address locally
defined CRM skills.

In order to ensure reliable crew performance data, an airline

should start with the instructor/evaluators empowering them to
develop standard assessment teams. In addition, airlines should
ensure that the instructor/evaluators have standardization
meetings to strengthen the teams and collect assessment
performance data. Along these lines, the organization should
provide the training department and instructors with all the
support needed to maintain that standard assessment. Finally, the
organization should place a high value on reliable data
throughout the organization and act on that data when it points to
the need for additional procedures or training.

Linking Performance Problems to CRM Skills

The discussion of crew performance problems up to this point
� Knowing a specific has concentrated on the identification of specific observable
performance behaviors or CRM areas based on LOFT/LOE gradesheets or
problem is a first checks. Recognizing a specific performance problem is an
step. The next step important first step. The next step is to link that performance
is to link that problem to one or more CRM skills; this step is needed to
performance determine the exact training and/or CRM procedural needs.
problem to one or
more CRM skills. Research work under ACRM has focused on the development
of CRM procedures and a structured assessment based on the

Part 6. Implementing and Evaluating ACRM Page 106

 IRR process. Other airlines have started work on developing
CRM skills lists. Both of these efforts should be combined so
that airlines can link their crew performance problems to CRM
skills. Presently there are no accepted standards for developing
CRM skill listings, but there are a number of guidelines that
should help airlines determine whether they have a usable listing.

Like the CRM procedures, a CRM skill listing should have

been developed by a team rather than by one or two developers.
The team should recognize the organizational needs, and should
have a clear definition of CRM skills. For example, one airline
specified that CRM skills have to be trainable, related to one or
more CRM topics, can be evaluated by instructor/evaluators, and
have to improve individual performance in a crew setting. From
experience with aviation task analyses it is evident that there are
at least 30 CRM skills, and that an airline should be concerned
with training at least half of those skills. Airlines that list just a
few CRM skills are probably talking about skill clusters which
are at too high a level to help in specifying required training or
additional CRM procedures.

Once the airline has a CRM skill list, they can work with
instructor/evaluators or other forms of SMEs to link the crew
performance problems to the most likely skills. This linkage
should not be limited only to CRM performance problems.
When making that linkage, the instructors and SMEs should also
review the technical problems to see if there may be a CRM
cause. The first attempt at this linkage may be tentative, but,
over time, an airline should establish a tested set of links between
what they are assessing and what they need to train.

Identifying Training Areas that Need

Improvement
ACRM is not limited to the development and assessment of
CRM procedures. It is strongly linked to the improvement of
� ACRM is not limited crew effectiveness throughout the organization. Therefore, crew
to the development performance problems do not automatically lead to the
of CRM procedures, development of additional CRM procedures. Once the
rather, It is strongly
performance problems have been linked to CRM skills, the
linked to all CRM
training department can determine whether just training or a new
development and
procedure with training is required based on the type of skills,
training.
SOP, and training currently in place.

An airline under ACRM has a number of things to consider

Part 6. Implementing and Evaluating ACRM Page 107

 when analyzing possible training needs, and most of them are
related to the nature of the CRM skill that is linked to the
performance problem. If it is the type of CRM skill that does not
require substantial practice in order to master, then it may be a
good candidate for a limited training intervention. Some
strategies or advanced skills fall into this category, and all that
may be needed is an additional module and a small amount of
practice in LOFT sessions. Other types of skills require a lot of
practice, making them good candidates for a CRM procedure. It
is the repetitive nature of SOP that provides crews with much
practice and the opportunity to master specific procedures along
with their related CRM skills.

Specifying Additional CRM Procedures

The ongoing nature of ACRM does not mandate a constant
stream of additional CRM procedures, but when crew
performance problems and skill types point to the need for an
additional CRM procedure, the ACRM framework is there to
facilitate its specification and implementation. It should be
emphasized that the success of an ACRM program is not
measured by the number of CRM procedures it has produced, but
by the improvements to crew effectiveness. It is possible to have
an excellent ACRM program with relatively few CRM
procedures.

An organization should consider the development of the first

set of CRM procedures as a learning process that can be stored
and reused in the case where additional procedures are needed.
The organization should maintain key elements of the original
� The organization team to help with additional development by capitalizing on the
should maintain key experience gained from the first round. This can be difficult at
elements of the many airlines where individuals tend to cycle in and out of
development team to departments, especially the training department. If an airline
help with additional places a premium on making key members available, it can
CRM procedure simplify any additional development and have a much more
development.
efficient team. A key to the successful development of additional
CRM procedures is planning for that eventuality during the initial
procedures development. An airline should identify the long-
term players and ensure that those individuals stay involved in
the ongoing ACRM program.

Part 6. Implementing and Evaluating ACRM Page 108

ACRM Implementation Guidelines
Guidelines for Organizational Implementation
1) Management should be kept informed of the general trends,
both crew performance and instructor standardization, and union
representatives should be provided with data that establishes the
reliable and accurate qualities of the ACRM crew assessments.

2) Under ACRM it is possible to collect a large amount of CRM

performance data which can result in an overwhelming amount of
information. There is a tendency to try and report everything, so
care should be taken not to report too much data and overloading
those who need the information.

3) In planning for the organizational announcement,

consideration should be given to a sequence where the CRM
procedures are announced prior to the start of crew training and
possibly before or during instructor/evaluator training. This
should be planned as a relatively short sequence, where the crew
training and implementation follow closely.

4) The announcement of CRM procedures should include a

detailed specification of the new policies and procedures. With
content and format of paramount importance, the announcement
should be carefully coordinated with flight standards and
documents departments.

5) As part of coordinating the release of new CRM procedures

with document updates, a formal review of the camera-ready
material should be scheduled because there will be errors, and
having a number of informed people review the material
increases the chances of catching those mistakes.

Guidelines for Implementing ACRM for

Instructor/Evaluator
1) The gradesheet should be used as a focal point in establishing
reliable crew assessment. Instructor/evaluators should be
encouraged to work on the refinement of the LOFT/LOE
gradesheet as an effective way to develop an assessment team.

Part 6. Implementing and Evaluating ACRM Page 109

 2) The airline should work with the instructor/evaluators to
establish a specified level of rater reliability prior to conducting
the LOFT/LOE portion of crew training. One way to ensure that
the instructor/evaluators reach that level is to work with a
realistic set of benchmarks.

3) Individual instructors with specific assessment problems

should be encouraged to work with the group in resolving the
issues. ACRM assessment should be approached as a group
effort where the team and not just one individual needs to resolve
any outstanding problems.

4) The airline should treat assessment standardization as a long-

term development process, giving the instructor/evaluators the
organizational support that will allow them to direct the process.

Guidelines for Maintaining Standards and IRR

1) When LOFT/LOE crew performance data shows a drop in
ratings, the airline should consider a range of causes from
properties of the LOE scenario to lack of crew training in specific
areas. The IRR process allows the airlines to isolate probable
causes with a greater degree of accuracy than has been possible
up to this point.

2) Once instructor/evaluators start administering LOFT/LOE

sessions, the data may show a pattern of lower ratings for certain
items, with some of the instructor-group IRR benchmarks not
being met. In such cases the airline should consider the
possibility that some instructor/evaluators do not have the same
interpretation of an event set or the standards of performance.

3) Airlines should monitor for crew performance problems in

order to address minor problems before they turn into incidents or
accidents. Minor problems are often identified by the rating of
acceptable but "below standard."

4) Airlines should understand that a major payoff to establishing

and maintaining a standard assessment is the ability to identify
long-term trends in crew performance. Without data establishing
that benchmarks have been met, airlines will find it difficult to
make meaningful interpretations of performance trends because
of the unknown reliability and accuracy of the data.

Part 6. Implementing and Evaluating ACRM Page 110

 Guidelines for Improving CRM Procedures and
Training
1) Once an airline has identified a specific performance problem,
it should work to link that performance problem to one or more
CRM skills. This step is needed to determine the exact training
and/or CRM procedure needs to address the problem.

2) Airlines should recognize that the collection of reliable crew

performance data is a minimum requirement for improving crew
performance. Airlines should be prepared support the training
department and instructor/evaluators in their efforts to establish
and maintain a reliable assessment system.

3) Airlines should understand that ACRM is not limited to the

development and assessment of CRM procedures. ACRM is
strongly linked to improving overall crew effectiveness at all
levels.

4) The organization should maintain key members of the original

ACRM development teams to help with additional development
by capitalizing on the members’ experience gained from the
initial program effort.

Part 6. Implementing and Evaluating ACRM Page 111

 This Page

Intentionally Left Blank

Part 6. Implementing and Evaluating ACRM Page 112

References and Resources

CHECKLIST AND PROCEDURE DESIGN

Adamski, A. J., & Stahl, A. F. (1997). “Principles of Design and Display for Aviation Technical
Messages.” Flight Safety Digest, Volume 16: 1-29.

Degani, A. (1992). On the typography of flight-deck documentation (NASA Contractor Report 177605).
Moffett Field, CA: NASA Ames Research Center.

Degani, A., & Wiener, E. L. (1993). Cockpit checklists: Concepts, design, and use. Human
Factors, 35, 345-359.

Degani, A., & Wiener, E. L. (1994). On the design of flight-deck procedures. (NASA
Contractor Report 177642). Moffett Field, CA: NASA Ames Research Center.

FAA (1995). Human performance considerations in the use and design of aircraft checklists.
Associate Administrator for Aviation Safety, Human Factors Analysis Division.
Washington, DC: Author.

AIRCRAFT ACCIDENTS AND INCIDENTS

NTSB (1994). Safety Study: A review of flightcrew-involved, major accidents of U.S. air
carriers, 1978 through 1990 (PB94-917001 NTSB.SS-94/01). Washington DC: National
Transportation Safety Board.

INTER-RATER RELIABILITY ASSESSMENT AND TRAINING

ATA (1998). Data Management Guide. Data Management Focus Group, AQP Subcommittee
Report, Air Transport Association, May, 1998.

Boehm-Davis, D. A., Hansberger, J. H., & Holt, R. W. (1997) Pilot abilities and performance.
Proceedings of the Ninth International Symposium on Aviation Psychology. Columbus, OH:
Ohio State University.

Holt, R. W. (1997) Data Warehousing for AQP databases. Proceedings of the Ninth
International Symposium on Aviation Psychology. Columbus, OH: Ohio State University.

Holt, R. W., Johnson, P. J., & Goldsmith, T. E., (1997) Application of psychometrics to the
calibration of air carrier evaluators. Proceedings of the Human Factors and Ergonomics
Society 41st Annual Meeting (p 916-920).

Holt, R. W., Meiman, E., & Seamster, T. L. (1996) Evaluation of aircraft pilot team performance.
Proceedings of the Human Factors and Ergonomics Society 40th Annual Meeting (p 44-48).

Index Page 113

Schultz, K., Seamster, T. L., & Edens, E. S. (1997). Inter-rater reliability tool development and
validation. Proceedings of the Ninth International Symposium on Aviation Psychology.
Columbus, OH: Ohio State University.

Williams, D. M., Holt, R. W., & Boehm-Davis, D. A. (1997). Training for inter-rater reliability:
Baseline and benchmarks. Proceedings of the Ninth International Symposium on Aviation
Psychology. Columbus, OH: Ohio State University.

INSTRUCTOR/EVALUATOR TRAINING

Dunlap, J. H., & Mangold, S. J. (1998). Leadership/Followership Recurrent Training Instructor

Manual. Washington, DC: Federal Aviation Administration, AAR-100.

McDonnell, L. K., Jobe, K. K., & Dismukes, R. K. (1997). Facilitating LOS Debriefings: A
Training Manual. (NASA Technical Memorandum 112192.) Moffett Field, CA: NASA
Ames Research Center.

Prince, C. (1997). Guidelines for Situation Awareness Training. Orlando, FL: Naval Air
Warfare Center, Training Systems Division.

LOE/LOFT/LOS DESIGN

ATA (1994). Line Operational Simulations: LOFT Scenario Design, Conduct and Validation.
LOFT Design Focus Group, AQP Subcommittee Report, Air Transport Association,
November 2, 1994.

Hamman, W. R., Seamster, T. L., Smith, K. M., & Lofaro, R. J. (1993). The future of LOFT
scenario design and validation. Proceedings of the Seventh International Symposium on
Aviation Psychology. Columbus, OH: Ohio State University.

Prince, C., Oser, R., Salas, E., & Woodruff, W. (1993). Increasing hits and reducing misses in
CRM/LOS scenarios: Guidelines for simulator scenario development. International Journal
of Aviation Psychology, 3(1), 69-82.

Seamster, T. L., Edens, E. S., & Holt, R. W. (1995). Scenario event sets and the reliability of
CRM assessment. Proceedings of the Eighth International Symposium on Aviation
Psychology. Columbus, OH: Ohio State University.

Seamster, T. L., Hamman, W. R., & Edens, E. S. (1995). Specification of observable behaviors
within LOE/LOFT event sets. Proceedings of the Eighth International Symposium on
Aviation Psychology. Columbus, OH: Ohio State University.

Index Page 114

STUDIES ON CRM PROCEDURES AND CREW COORDINATION

Johnston, A. N. (1992). The development and use of generic nonnormal checklist with
applications in ab initio and introductory advanced qualification programs. The International
Journal of Aviation Psychology, 2, 323-337.

Kanki, B. G., Lozito, S., & Foushee, H. C. (1989). Communication indices of crew coordination.
Aviation, Space, and Environmental Medicine, 60, 56-60.

Index Page 115

 This Page

Intentionally Left Blank

Index Page 116

 Index

abnormal conditions, iv, vii, ix, 2, 9

75, 76, 77, 78, 80, 81, 82, 83, 84, 86, 87,

abnormal procedures, vii, 2

88, 89, 91, 92, 95, 97, 100, 105, 109, 110

ACRM development team, 14, 111

CRM principles, ix, x, 1, 49, 64, 69, 74,

ACRM development timeline, 24

75, 86

ACRM elements, x
CRM procedures additional, 2, 7, 28, 49,

ACRM implementation, 7, 14, 68, 92, 93

79, 80, 88, 98, 99, 106, 107, 108

ACRM training footprint, 70

CRM procedures development, 1, 11,

additional help, xi
20, 25, 40, 73, 87

agreement index, iii, 6, 57, 100, 101

CRM procedures overview, 4, 20

airline safety reports, 84, 98

CRM procedures specification, 20, 49,

AQP, iii, vi, x, xi, 6, 12, 58, 113, 114

95, 109

ASRS, iii, 29, 74, 83

CRM skill, x, 2, 4, 6, 9, 10, 12, 16, 47,

ASRS incident reports, 29, 74

48, 50, 64, 74, 75, 77, 79, 87, 106, 107,

assessing crew performance, 48, 53

108, 111

assessment standards, 48, 55, 63, 66

CRM skill practice, 9

attitude, iii, v, 48, 94

CRM training objectives, 82

backup plan, iii, 79

distracters, 53

bottom line, iii, 36, 61, 85

brief, 2, 3, 4, 5, 35, 47, 51, 52, 65, 92, 95

event set, v, 6, 18, 45, 53, 56, 60, 65, 82,

84, 85, 86, 89, 94, 99, 102, 103, 104,

cognitive skill, iv
110, 114

congruency, iv, 6, 57, 61, 62, 94, 100,

104 FAA, v, ix, xi, 13, 44, 54, 58, 113

consistency, iv, 6, 7, 33, 57, 61, 94, 100,

First Look, v

103, 104
flight operations manual, v, x, 2, 26

crew effectiveness, 6, 69, 76, 77, 78, 79,

80, 88, 98, 107, 108, 111

gradesheets, 15, 60, 99, 102

crew performance data, 6, 7, 8, 17, 24,

guidelines crew training, 86

50, 58, 63, 66, 78, 79, 88, 94, 102, 103,
guidelines CRM procedures, 40, 41, 111

106, 110, 111

guidelines instructor/evaluator, 109

crew performance problems, 2, 15, 18,

guidelines IRR, 110

20, 27, 28, 29, 30, 34, 41, 104, 105, 106,
guidelines organizational

107, 108, 110

implementation, 109

crew training, x, 2, 4, 6, 12, 13, 44, 45,

guidelines own airline needs, 40

47, 50, 64, 67, 68, 69, 70, 71, 72, 73, 74,

Index Page 117

instructor/evaluator assessment tools, 15,
NTSB, vi, 28, 29, 31, 73, 74, 113

99

instructor/evaluator feedback, 61
observable behavior, 53, 56, 60, 99, 106,

instructor/evaluator training, 4, 5, 6, 7, 9,
114

15, 43, 44, 45, 46, 47, 48, 49, 55, 60, 64,
organizational announcement, 91, 92,

65, 67, 68, 69, 72, 86, 92, 103, 109

95, 109

instructor/evaluator training overview, 4,

organizational links, 14

44
organizational requirements, 12, 39

IRR, v, x, 5, 6, 18, 44, 45, 47, 55, 57, 58,

organizational survey, vi, 17, 25, 30

59, 61, 62, 65, 100, 101, 102, 103, 104,

own airline needs, vi, 4, 17, 19, 47

105, 107, 110

IRR components, 100

personnel requirements, 22

ISD, v
phase of flight, 29, 30, 31, 34, 40, 49, 52

preliminary organizational presentations,

Key Elements of ACRM Training, x

14

knowledge, vii, 1, 5, 51
proficiency objective, iv

KSA, v
Providing Instructor/Evaluators with

Immediate Feedback, 61

Linking Performance Problems to CRM

Providing Instructor/Evaluators with

Skills, 106
Ongoing Feedback, 101

LOE, v, vi, x, 5, 6, 9, 13, 15, 46, 47, 48,

50, 51, 52, 53, 54, 55, 56, 58, 59, 60, 64,
QRC, iv, vii

65, 71, 72, 75, 80, 81, 82, 84, 85, 86, 88,
QRH, iv, vii, x, 2, 12, 26, 35, 96

89, 93, 94, 98, 99, 100, 101, 102, 103,

questions, xi, 13, 17, 32, 37, 49, 64, 76,

104, 110, 114

82, 93, 95

LOFT/LOE, x, 15, 25, 47, 51, 52, 53, 54,

55, 56, 58, 59, 60, 64, 65, 71, 72, 75, 80,
rating scale, iii, vii, 55, 56, 57, 60, 65,

81, 82, 85, 86, 88, 89, 93, 94, 98, 99,
99, 104

100, 101, 102, 103, 110

reliable data, 6, 58, 106

LOFT/LOE administration, 103, 110

LOS, vi, viii, 54, 65, 72, 82, 83, 88, 89,
Scheduling Crew Training, 97

114
Selecting Strategies to Reinforce Crew

Awareness and Feedback, 97

Maintaining Standards and the IRR

sensitivity, 100, 104

Process, 102
skill, iv, vii, x, 4, 9, 10, 12, 47, 48, 50,

maneuver validation, 78
64, 74, 77, 78, 79, 87, 107, 108

SOP, iii, iv, vii, ix, 2, 4, 7, 10, 13, 31,

NASA, iii, vi, 28, 31, 54, 83, 113, 114

32, 33, 34, 38, 40, 41, 58, 75, 78, 79, 87,

needs analysis, 27, 28

92, 108

normal conditions, 4
SPOT, vi, vii

Index Page 118

standard, vi, vii, 4, 8, 9, 44, 51, 55, 56,

58, 59, 60, 76, 82, 87, 99, 101, 103, 104,

105, 106, 110

standard CRM, 8

standardization meetings, 15, 48, 102,

106

standardization sessions, 48, 61, 62, 63,

66

subtask, iv, vii

systematic differences, 61, 94

task, iv, viii, 5, 29, 50, 77, 79, 106, 107

technical objectives, 83

technical objectives primary, 82

technical performance, 18, 76, 77, 78, 79

technical skill, x

trends in crew performance, 105, 110

user feedback, 37, 38, 39, 41

workload, v, 2, 3, 9, 32, 33, 34, 41, 49,

53, 60, 78, 85, 86

Index Page 119

 This Page

Intentionally Left Blank

Index Page 120

 Appendix A

Summary of ACRM Guidelines

Appendix A Page A-1

 This Page

Intentionally Left Blank

Appendix A Page A-2

CRM Procedures Development Guidelines

Guidelines for Identifying Own Carrier Needs

1) The identification of CRM problem areas is ongoing, and to ensure the long-term success of
the ACRM program, an airline should identify a few operationally significant CRM problems
at the beginning of the development process.

2) Airlines that do not have detailed CRM performance data should use industry accident reports
and incident summaries to identify general problem areas, and then the airline should use
specific aircraft accident and incident reports to obtain the details.

3) When existing airline data or reports do not point to a clear CRM cause, the development
team should consider interviewing one or more individuals from the department that collected
the data or produced the report.

4) The needs survey should be designed primarily for instructors, evaluators, and checkairmen
and should investigate at least two areas: 1) CRM performance problems by phase or
subphase of flight, and 2) performance problems by CRM topic or element.

5) The CRM procedures development team should be aware of the benefits and possible liabilities
of each procedure they plan to add. The team's mandate to identify areas for new or modified
procedures should be tempered with the realization that too many procedures, or ones in the
wrong place, can be as problematic as not adding any new procedures.

Guidelines for Specifying CRM Procedures

1) For initial procedure development, an airline should first identify weaknesses in existing
procedures and then review airline philosophy and policy to clarify those needs. Based on that
information, an airline should then identify possible locations for the new procedure as
different forms (e.g., briefs, checklists, etc.) are being considered. Once that is done, the
airline should work on the procedure's content.

2) When reviewing existing SOP and documentation, look for problems with existing procedures,
lack of consistency, as well as gaps, where the performance problem is not being addressed.

3) In many cases, crew performance problems become apparent at times of relatively high
workload, and adding a procedure at that point could further increase workload. The CRM
procedures development team should consider times prior to the buildup in workload to
identify periods of lower workload where a procedure would be more effective.

4) The main forms of CRM procedures include briefs, calls, checklist items, guides, flows, non-
normal procedures, and quick reference items. One of these forms should be sufficient to
address most problems, but there may be cases where the integration of two forms is less
intrusive and provides a better fit with the airline's SOP.

Appendix A Page A-3

Guidelines for Refining CRM Procedures and Media
1) The procedure prototype development (working with a mock-up of the procedure) is an
important part of the refining process where the development team interacts with a range of
uses to determine the best form and content for the CRM procedures. The refinement step
should be iterative with the feedback from each review being incorporated into the design to
achieve one or more CRM procedures that will be adopted by the users and will contribute to
performance improvements.

2) User feedback sessions should include five to ten individuals who work well together.
Working with too small a group (less than five) is less efficient and the individuals are less
likely to be stimulated by a wider range of comments. Working with too large a group
(substantially more than ten) is more difficult to manage, and the feedback will likely cover a
broad range of topics but not in depth.

3) If formal user feedback sessions are not possible, consider holding informal small group or
individual sessions as pilots and instructors are available in the training center or flight
operations. Meaningful feedback should be obtained from a cross section of pilots and
instructors either formally or informally.

4) Organizational presentations, generally made after user feedback sessions, are similar to those
made to the users, but with fewer operational details and more information about the ACRM
program. Emphasis should be on the need for the CRM procedures, the development process,
and the feedback process.

Instructor/Evaluator Training Development Guidelines

Guidelines for Planning and Developing Introductory ACRM Modules
1) When planning for the development of instructor/evaluator training, consider not only the
development process but also the actual implementation of the training. Try to plan so that
the ACRM training will be ready at a point where instructors receive scheduled recurrent
training. ACRM instructor training should be as integrated as possible with the rest of their
training and should be treated as a minor expansion of existing training rather than a
substantial addition.

2) One initial instructor/evaluator training module should explain that the procedure development
process has translated CRM principles into operational procedures, providing the airline with
an opportunity to emphasize important CRM actions that should be practiced by all crews.

Appendix A Page A-4

3) When presenting CRM procedures to instructors it should be explained how each procedure
was developed based on airline needs and incident data. The procedure, in its actual form,
should then be presented, highlighting each of its main features. Finally, consider including a
set of questions or other form of instructor activity to ensure that the instructors understand
the main elements of the new CRM procedure.

4) Each new CRM procedure has implications for crew training because it places an emphasis on
the crew's development of specific CRM behaviors in the operational context. Because of this
emphasis on CRM skills, instructor/evaluators should have an understanding of the nature of
skill development and its implications for the training of these new CRM procedures.

Guidelines for Developing LOFT/LOE Modules

1) The effectiveness of LOFT/LOE sessions depends in good part on script detail and proper
administration of that script during the simulator session. The scenario should be carefully
scripted with ATC communications using correct terminology, timing, and routing. Precise
ATC communication scripting will also enhance session realism.

2) LOFT/LOE briefings are an important part of the session, and instructor/ evaluators should
brief crews to act as they would in line operations dealing with everyone, including the Flight
Attendant, as if they were actually present throughout the LOFT/LOE.

3) Event sets should be used in the development of LOFT/LOS scenarios under an ACRM
program to help instructor/evaluators pinpoint key aspects of crew performance for each
segment of the flight. Instructor/evaluators should be trained in the functions and use of event
sets.

Guidelines for Establishing Assessment Standards

1) When an instructor/evaluator group shows low inter-rater reliability, in addition to more
training, there may be a need for a better rating form or clearer rater standards to help the
group work together on the basic parameters of the assessment process.

2) The instructor/evaluator group should establish specific standards for elements to be rated.
This is especially true for elements that may cause problems or are new to the
instructor/evaluators. In most cases the new CRM procedures should have explicit standards
to reduce rating difficulties.

3) When making crew performance assessments there is a high probability of rater bias, and the
common forms of bias that should be addressed through instructor/evaluator training include
central tendency, halo error, and leniency error.

Appendix A Page A-5

4) If the airline has not already implemented some form of IRR training, IRR should be presented
as a group process beginning with an overview of IRR, followed by the critical nature of crew
assessment, the IRR measures, the gradesheet, rating scales, and examples of the criteria for
each point on the scale.

Guidelines for Developing Standardization and Training Modules

1) Under ACRM, a substantial departure from a scripted and acted videotape is required.
Consideration should be given to preparing standardization tapes working with real crews
flying the actual LOFT/LOE for that year in a representative simulator without the benefit of
coaching or preparation.

2) Instructor/evaluators should be provided with accurate and immediate rating feedback from
the start of their assessment training through standardization sessions. The first rating
sessions may take place in a larger group using spreadsheets and charts showing individual
and group data along with appropriate benchmarks that the group is trying to meet.

3) After individual and group feedback is provided and explained, instructor/evaluators should be
encouraged to develop new rating rules and strategies. This cycle of practice, feedback, and
discussion allows the participants to improve their reliability, and should continue until group
benchmarks have been met.

4) Under ACRM assessment there should be ongoing training and standardization to establish
confidence in the crew performance data, its indications about CRM procedures, and the
procedures' effects on overall crew performance.

5) Instructor/evaluators should be encouraged to take an active team approach to standardization

sessions. Instructor/evaluators should see these sessions as an essential part of maintaining
their assessment standards. One way to ensure team involvement is to encourage
instructor/evaluators to control the standardization sessions, and ultimately to determine their
own schedule and length of cycle between sessions based on their rating performance.

ACRM Crew Training Development Guidelines

Guidelines for Planning ACRM Crew Training
1) Crew training can be the largest cost of the ACRM program, so the development team should
use the planning stage to determine ways for reducing the cost of that training. The team
should consider integrating ACRM with existing CRM training (see Guideline 2 below) and
coordinating ACRM training with other training cycles (see Guideline 3 below).

2) The development team should consider extending or modifying existing crew training to meet
ACRM training needs. Much of the existing CRM recurrent training can be modified from an
emphasis on general CRM principles to a concentration on the specific CRM procedures.

Appendix A Page A-6

3) The development of ACRM crew training should be coordinated not only with crew training
cycles but also with instructor/evaluator training and the implementation of the new CRM
procedures.

4) ACRM crew training development and implementation should include a review process by
scheduling ACRM crew training curriculum reviews that involve key organizational personnel
who control the future of the ACRM program.

Guidelines for Developing CRM Procedures Crew Modules

1) When training crews about the need for CRM procedures, it should be emphasized that the
crews have been an essential part of the development process with crew representatives
involved in developing the current training. In the future, crewmembers should stay involved
and help identify new CRM procedural needs as they become apparent.

2) A key component of new CRM procedures is the degree to which they support the
development of CRM skills. To help crews practice and master the appropriate skills, crews
should be informed of the relationship between the CRM procedures and their CRM skill
development.

3) It should be explained that the new CRM procedures have profound implications on the way
crew performance is assessed. For example, if it is the case, crews should understand that a
detailed simulator-based assessment has been developed to collect crew performance data
after the ACRM crew training has been implemented.

4) It should be explained that detailed measures of crew performance were developed in the
research stages of ACRM specification and analysis. These more accurate crew performance
measures will help evaluators provide more standard crew assessments, and crews should
expect a fair assessment from all instructor/evaluators.

Guidelines for Developing Crew Effectiveness Modules

1) Crew effectiveness is a core concept of any ACRM program, and crew training should present
its main characteristics. Crew effectiveness should be enhanced by using the new CRM
procedures that will help crews develop better CRM skills.

2) The ACRM program provides a framework that allows airlines to integrate CRM with
technical at the level of SOP. ACRM crew training should emphasize this integration at the
operational level as well as at the crew performance assessment level.

3) It should be explained that the new CRM procedures have profound implications on the way
that crew performance is assessed. For example, if it is the case, crews should understand that
a detailed simulator-based assessment has been developed to collect crew performance data
after the ACRM crew training has been implemented.

Appendix A Page A-7

4) It should be emphasized that crews form the core of the ACRM program, and crews should be
encouraged to provide feedback about the new procedures as well as additional CRM
procedures that can improve crew effectiveness.

Guidelines for Developing Briefing/Debriefing Modules

1) The LOFT/LOE briefing should prepare the crew for an effective training or assessment
experience. One of the first things that the LOFT/LOE briefing should do is establish the role
of the instructor as well as the role of the crew throughout the session.

2) The LOFT/LOE briefing should emphasize specific things the crew can do to make the session
realistic and a valuable training experience. The crews should act as they would under similar
situations on the line, and they should not try to operate in a manner calculated to provide the
ideal training department solution.

3) ACRM crew training should reinforce good crew practices in conducting a constructive
debriefing. Crews should understand that debriefing is an essential part of the LOFT/LOE and
the main way of providing crew performance feedback.

4) ACRM crew training should introduce the concept that briefings can also be extended to line
operations and used by crews to reinforce positive performance and learn from any problems
experienced during the flight.

Guidelines for Developing Crew LOS Assessment Modules

1) Crews should be presented with the LOS CRM training objectives in the context of relevant
technical objectives because crews are generally more comfortable and familiar with the
technical objectives.

2) A major step in the LOS development process is the identification of incidents that highlight
the need for specific CRM procedure, and crews should understand that the LOS is related to
actual incidents that have happened to pilots flying in conditions similar to theirs.

3) Crew training should include one or more activities that allow crews to analyze incidents used
to develop the new CRM procedures. One activity that has provided valuable crew training is
having crews identify the CRM causes for each incident.

4) Crews should be given an explanation of how the event set is used under ACRM to help focus
the specific elements of crew performance in the LOFT/LOE assessment. The event set helps
training and evaluation move from general CRM markers to specific crew behaviors.

Appendix A Page A-8

ACRM Implementation Guidelines

Guidelines for Organizational Implementation

1) Management should be kept informed of the general trends, both crew performance and
instructor standardization, and union representatives should be provided with data that
establishes the reliable and accurate qualities of the ACRM crew assessments.

2) Under ACRM, it is possible to collect a large amount of CRM performance data, which can
result in an overwhelming amount of information. There is a tendency to try and report
everything, so care should be taken not to report too much data and overloading those who
need the information.

3) In planning for the organizational announcement, consideration should be given to a sequence

where the CRM procedures are announced prior to the start of crew training and possibly
before or during instructor/evaluator training. This should be planned as a relatively short
sequence, where the crew training and implementation follow closely.

4) The announcement of CRM procedures should include a detailed specification of the new
policies and procedures. With content and format of paramount importance, the
announcement should be carefully coordinated with flight standards and documents
departments.

5) As part of coordinating the release of new CRM procedures with document updates, a formal
review of the camera-ready material should be scheduled because there will be errors, and
having a number of informed people review the material increases the chances of catching
those mistakes.

Guidelines for Implementing ACRM for Instructor/Evaluator

1) The gradesheet should be used as a focal point in establishing reliable crew assessment.
Instructor/evaluators should be encouraged to work on the refinement of the LOFT/LOE
gradesheet as an effective way to develop an assessment team.

2) The airline should work with the instructor/evaluators to establish a specified level of rater
reliability prior to conducting the LOFT/LOE portion of crew training. One way to ensure
that the instructor/evaluators reach that level is to work with a realistic set of benchmarks.

3) Individual instructors with specific assessment problems should be encouraged to work with
the group in resolving the issues. ACRM assessment should be approached as a group effort
where the team, and not just one individual, needs to resolve any outstanding problems.

4) The airline should treat assessment standardization as a long-term development process giving
the instructor/evaluators the organizational support that will allow them to direct the process.

Appendix A Page A-9

Guidelines for Maintaining Standards and IRR
1) When LOFT/LOE crew performance data shows a drop in ratings, the airline should consider
a range of causes from properties of the LOE scenario to lack of crew training in specific
areas. The IRR process allows the airlines to isolate probable causes with a greater degree of
accuracy than has been possible up to this point.

2) Once instructor/evaluators start administering LOFT/LOE sessions, the data may show a
pattern of lower ratings for certain items with some of the instructor-group IRR benchmarks
not being met. In such cases, the airline should consider the possibility that some
instructor/evaluators do not have the same interpretation of an event set or the standards of
performance.

3) Airlines should monitor for crew performance problems in order to address minor problems
before they turn into incidents or accidents. Minor problems are often identified by the rating
of acceptable but "below standard."

4) Airlines should understand that a major payoff to establishing and maintaining a standard
assessment is the ability to identify long-term trends in crew performance. Without data
establishing that benchmarks have been met, airlines will find it difficult to make meaningful
interpretations of performance trends because of the unknown reliability and accuracy of the
data.

Guidelines for Improving CRM Procedures and Training

1) Once airlines have identified a specific performance problem, they should work to link that
performance problem to one or more CRM skills. This step is needed to determine the exact
training and/or CRM procedure needs to address the problem.

2) Airlines should recognize that the collection of reliable crew performance data is a minimum
requirement for improving crew performance. Airlines should be prepared support the
training department and instructor-evaluators in their efforts to establish and maintain a
reliable assessment system.

3) Airlines should understand that ACRM is not limited to the development and assessment of
CRM procedures. ACRM is strongly linked to improving overall crew effectiveness at all
levels.

4) The organization should maintain key members of the original ACRM development teams to
help with additional development by capitalizing on the members' experience gained from the
initial program effort.

Appendix A Page A-10

 Appendix B

Sample Instructor/Evaluator and

Organizational Forms

Appendix B Page B-1

 This Page

Intentionally Left Blank

Appendix B Page B-2

 Sample Instructor/Evaluator Questionnaire

1) Rank order the following CRM items giving number 1 to the item where crews had the most
problems and number 10 to the item that crews had the least problems (in your 199-
recurrent training). Use all the numbers from 1 to 10 with no duplicates.

___Big Picture shared with team F/A, F/O, others.

___Bottom Lines established, communicated and maintained.

___Briefing - sets tone, calls for questions, participation encouraged, states how SOP
deviations will be handled.
___Conflicts - Acknowledged and resolved. Atmosphere conducive to open communications.

___Crew self critique - crew provides information to self correct given at appropriate times
with whole crew. Covers positive and negative performance (What's right not who's right)
___Inquiries and assertiveness - crewmembers speak up with appropriate persistence until
agreement achieved. All are encouraged to state recommendations.
___Leadership/followership - Balance between authority and assertions. Climate appropriate
to situation. Adheres to SOP, uses all resources, manages time for task accomplishment.
___Participation in decision process encouraged - operational decision are clearly stated.
Crewmembers acknowledge understanding of decisions.
___Preparation/planning - crew stays ahead of curve, monitors developments, maintains S.A.,
Big Picture shared. Red flags recognized and resolved. Cross checking A.C. status.
___Workload/distractions avoided - Overload in self and others reported. Tasks prioritized to
deal with primary flight duties first, recognizes distractions.

Specify one or two typical CRM problems for the items with the most crew difficulty (those you
have ranked 1, 2, and 3) and provide a brief reason for the CRM problem.

2) Typical Crew Problems and Reasons for CRM item ranked 1: _______________________
_________________________________________________________________________
_________________________________________________________________________

3) Typical Crew Problems and Reasons for CRM item ranked 2: _______________________
_________________________________________________________________________
_________________________________________________________________________

4) Typical Crew Problems and Reasons for CRM item ranked 3: _______________________
_________________________________________________________________________
_________________________________________________________________________

Appendix B Page B-3

6) Of the 11 ACA CRM items listed on the previous page, please specify the three for which
ACA crews would benefit from additional training (you may use the CRM item numbers or
leading word in the description to specify the items).
CRM item requiring additional training _________________________________________
CRM item requiring additional training _________________________________________
CRM item requiring additional training _________________________________________

7) Rank the following Line Check items giving number 1 to the item where you most
frequently observed CRM problems and 14 where you least frequently saw CRM problems.
Use all the numbers from 1 to 14. Then, for items ranked 1 to 5, write a typical problem to
the right.
Flight Planning ___________________________________________________________
Weight and Balance Procedures ______________________________________________
Aircraft Doc. & MEL ______________________________________________________
Preflight ________________________________________________________________
Engine Starts ____________________________________________________________
Taxing _________________________________________________________________
Crew Briefing ___________________________________________________________
V-Speed Compliance ______________________________________________________
Departure / SID Compliance ________________________________________________
Radar Use ______________________________________________________________
Clearance Compliance _____________________________________________________
Navigation ______________________________________________________________
Stabilized Approach _______________________________________________________
Approach Clearance Compliance _____________________________________________

8) Briefly summarize an LOE/LOFT session where the crew had substantial CRM problems.
_________________________________________________________________________
_________________________________________________________________________
_________________________________________________________________________
_________________________________________________________________________

9) Briefly summarize a recent Line Check where the crew had substantial CRM problems.
_________________________________________________________________________
_________________________________________________________________________
_________________________________________________________________________
_________________________________________________________________________

10) What are the 3 most significant incidents that should be addressed in recurrent training?
Incident 1 _________________________________________________________________
Incident 1 should be trained (check one) __ In Sim __ In Class __ By Home Study
Incident 2 _________________________________________________________________
Incident 2 should be trained (check one) __ In Sim __ In Class __ By Home Study
Incident 3 _________________________________________________________________
Incident 3 should be trained (check one) __ In Sim __ In Class __ By Home Study

Appendix B Page B-4

 Sample Organizational Climate Survey
This survey is designed to help develop effective ACRM training. The survey deals with your job and your
interaction with the training department at this carrier. Please take 15 minutes to answer the questions on
the front and back of each page and return it to xxx in the envelope provided. Your answers will be
completely confidential--only summary data analyses will be provided to the appropriate departments. If
you have any questions, please call xxx or xxx.
Job Satisfaction•
ON MY PRESENT JOB, THIS IS HOW I FEEL ABOUT [EACH STATEMENT BELOW].

Please circle one of the following for each statement:

Very Dissatisfied Dissatisfied Neutral Satisfied Very Satisfied

VD D N S VS Being able to keep busy all the time.

VD D N S VS The chance to do different things from time to time.
VD D N S VS The chance to be “somebody” in the aviation industry.
VD D N S VS The way my supervisor handles his or her employees.
VD D N S VS The way my director of operations handles his employees.
VD D N S VS The competence of my supervisor in making decisions.
VD D N S VS The competence of my director of operations in making decisions.
VD D N S VS The way my job provides steady employment.
VD D N S VS Being able to do things that don’t go against my conscience.
VD D N S VS The chance to do things for other people.
VD D N S VS The chance to tell people what to do.
VD D N S VS The chance to do something that makes use of my abilities.
VD D N S VS The way policies are put into practice at this carrier.
VD D N S VS My pay and the amount of work I do.
VD D N S VS The chance for advancement on this job.
VD D N S VS The freedom to use my own judgment.
VD D N S VS The chance to try my own methods of doing the job.
VD D N S VS The physical working conditions of the job (e.g. temperature, noise, safety).
VD D N S VS The way my co-workers get along with each other.
VD D N S VS The praise I get for doing a good job.
VD D N S VS The feeling of accomplishment I get from the job.
VD D N S VS The degree of responsibility I have in this job.
VD D N S VS� The amount of time, personnel, and training this carrier provides for me to complete
the job.

Appendix B Page B-5

Please circle one of the following for each statement:
Strongly Agree Agree NeutralDisagree Strongly Disagree

Job Description•
SA A N D SD This carrier sets very high job performance standards.

SA A N D SD My job requires strict adherence to exacting standards of quality and accuracy.

SA A N D SD There is pressure to continually improve my personal job performance.

SA A N D SD This carrier rewards a person for doing a good job.

SA A N D SD Job training resources at this carrier are very good.

SA A N D SD Equipment and personnel necessary to do the job are almost always available.

SA A N D SD Some people don’t take much pride in their work at this carrier.

SA A N D SD My job gives me the chance to use my skills and abilities.•

SA A N D SD Rewards and encouragements on the job usually outweigh the criticisms.

SA A N D SD Most employees are strongly motivated to achieve this carrier’s work goals.

SA A N D SD My job makes a meaningful contribution and is important at this carrier.

SA A N D SD I get enough feedback so that I always know how well I am performing.

SA A N D SD This carrier’s management emphasizes safety.

SA A N D SD Safety training programs are important.

SA A N D SD Safety is a strong priority in scheduling operations.

SA A N D SD Safety training is frequent enough to ensure safety.

SA A N D SD The Safety Officer position is a very important position at this carrier.

SA A N D SD Time pressures on the job can cause a decreased emphasis on safety.

SA A N D SD I wish I had more safety training.

SA A N D SD Some of my co-workers should have more safety training.

SA A N D SD I get the training I need from the Training Center.

SA A N D SD Training center personnel emphasize “what’s right” rather than “who’s right”.

SA A N D SD The Training Center provides high quality training.

Appendix B Page B-6

I work as a (check one): __Check Pilot for 19 PAX A/C __Line Pilot for 19 PAX A/C
__Check Pilot for 29 PAX A/C __Line Pilot for 29 PAX A/C
__ATS Flight Attendant __Line Flight Attendant
__Dispatcher __Ramp personnel __Management __Other

Which of the following statements best describes your contact with the Training Center:
(Please check the appropriate space)

___ I work currently at the Training Center

___ I have worked in the past at the Training Center

___ I have had repeated contact with the Training Center

___ I have had no contact or only occasional contact with the Training Center (skip to the
“COMMENTS” section)

Please circle one of the following for each statement:

Strongly Agree Agree NeutralDisagree Strongly Disagree

Training Center
SA A N D SD Training center job responsibilities are clearly defined and logically structured.
SA A N D SD The training center concentrates on getting the right people to get the job done..
SA A N D SD Who has the formal authority to make a training decision is sometimes unclear.
SA A N D SD Training productivity sometimes suffers from lack of organization and planning.
SA A N D SD The training cadres cooperate effectively to achieve this carrier’s training goals.
SA A N D SD Communication from trainers to training management is easy and effective.
SA A N D SD Communication from training management to trainers is easy and effective.
SA A N D SD Interactions in the training department are friendly.
SA A N D SD Coordination in the training department can be a problem.
SA A N D SD Training center personnel are aloof and distant toward each other.
SA A N D SD The training department has close working relationships.
SA A N D SD Training supervisors help develop the careers of the trainers.
SA A N D SD� Effective scheduling, coordinating, and planning comes from the top of the training
department.
SA A N D SD� Training management is concerned with how the trainers feel as well as how they
perform.

Appendix B Page B-7

Please circle one of the following for each statement:
Strongly Agree Agree NeutralDisagree Strongly Disagree

SA A N D SD� For a particularly difficult training job, trainers can count on assistance from the top of
the training department.
SA A N D SD All levels of the Training Center participate fully in setting job standards and goals.
SA A N D SD Training supervisors have trust and confidence in the trainers working for them.
SA A N D SD Trainers have trust and confidence in the training supervisors.
SA A N D SD� Training supervisors try to enhance feelings of personal worth and importance among
the trainers.

COMMENTS:

Thank you for your cooperation in this survey.

Appendix B Page B-8

 Appendix C

Sample ACRM Instructor/Evaluator Training

Manual

Table of Contents (TOC)

Appendix C Page C-1

 This Page

Intentionally Left Blank

Appendix C Page C-2

 Table of Contents

Table of Contents.........................................................................................................i

MODULE 1: Recurrent Training Under AQP..............................................................1

1.0 Overview ...........................................................................................................1

1.1 Objectives of Recurrent Instructor/Evaluator Training ........................................1

1.2 Outline of the Training .......................................................................................1

1.3 Advanced Qualification Program ........................................................................2

1.4 Advanced Crew Resource Management (CRM)..................................................3

1.5 Line Operational Evaluation (LOE) ....................................................................6

MODULE 2: Performing Administrative Functions .....................................................7

2.0 Overview ...........................................................................................................7

2.1 Training Forms...................................................................................................7

2.2 Completion of Recurrent Training Forms............................................................12

MODULE 3: Presenting and Assessing Recurrent Training .........................................13

3.0 Overview ...........................................................................................................13

3.1 CRM and Skill Performance/Procedures .............................................................13

3.2 Teaching Skill Performance to Students..............................................................14

3.3 List Common Student Errors..............................................................................15

3.4 Develop Student Practice Exercises....................................................................17

MODULE 4: Design of LOE Scenarios and Event Sets ...............................................19

4.0 Overview ...........................................................................................................19

4.1 Developing the LOE Scenario ............................................................................19

4.2 Scenario Event Set .............................................................................................20

4.3 Event Set Conditions and Triggers .....................................................................21

4.4 Identification of Incidents ...................................................................................21

4.5 Assessment and Validation of Training Materials ................................................23

4.6 Event Set Outcomes...........................................................................................24

4.7 Event Set Repeats ..............................................................................................26

4.8 Identify Incidents and Issues for Event Sets ........................................................27

4.9 Identify Event Sets and Event Triggers...............................................................29

MODULE 5: ACA Recurrent 95 LOE Scenario ..........................................................31

5.0 Overview ...........................................................................................................31

5.1 LOE Guide.........................................................................................................31

5.2 ACA Recurrent 95 LOE Overview .....................................................................32

5.3 Event Set One Conditions and Trigger................................................................34

5.4 Event Set Two Conditions and Trigger...............................................................34

5.5 Event Set Three Conditions and Trigger .............................................................35

5.6 Event Set Four Conditions and Trigger...............................................................35

5.7 Identify CRM Elements for Event Sets ...............................................................37

5.8 Identify Event Set Conditions and Triggers.........................................................39

MODULE 6: Briefing the LOE ...................................................................................41

6.0 Overview ...........................................................................................................41

6.1 Function of the LOE Briefing .............................................................................41

6.2 Role of Facilitator ..............................................................................................41

6.3 Reverse Briefing Techniques ..............................................................................43

Appendix C Page C-3

6.4 Relationship Between Technical and CRM Performance .....................................44

6.5 Additional Facilitation and Motivation Techniques..............................................44

6.6 Identify Elements that Make LOE Briefing Compelling.......................................47

MODULE 7: Administering the 1995 Recurrent LOE .................................................49

7.0 Overview ...........................................................................................................49

7.1 Standardization of the LOE Guide and Administration........................................49

7.2 Flight Realism Requirements ..............................................................................50

7.3 Normal Flight Communications ..........................................................................50

7.4 Crewmember Responsibilities .............................................................................51

7.5 Repeats and the Event Set ..................................................................................51

7.6 Identify Event Triggers for Event Sets One to Five.............................................53

MODULE 8: Observing Crew Behaviors.....................................................................55

8.0 Overview ...........................................................................................................55

8.1 Observing Total Crew Performance....................................................................55

8.2 Observable Crew Behaviors (Interpersonal)........................................................55

8.3 Inferred Crew Behaviors (Mental) ......................................................................56

8.4 Observed/Not Observed .....................................................................................57

8.5 Success Criteria..................................................................................................57

8.6 LOE Worksheet .................................................................................................59

8.7 Identify Statements of Observable Crew Behaviors.............................................61

MODULE 9: Assessing LOEs and Line Checks...........................................................63

9.0 Overview ...........................................................................................................63

9.1 Unsatisfactory Performance................................................................................63

9.2 Assessment Scale ...............................................................................................64

9.3 Identify Behaviors to be Observed/Not Observed ...............................................65

MODULE 10: Debriefing LOEs and Line Checks........................................................67

10.0 Overview ...........................................................................................................67

10.1 Debriefing Function and Guidelines ....................................................................67

10.2 Role of Facilitator in Debriefing..........................................................................68

10.3 Integrating Technical With CRM ........................................................................69

10.4 Integrate the CRM With Technical Performance Issues.......................................71

MODULE 11: Developing Assessment Skills ..............................................................73

11.0 Overview ...........................................................................................................73

11.1 Taped LOEs.......................................................................................................73

11.2 Real Crew Performance......................................................................................74

11.3 LOE Worksheet as Assessment Tool ..................................................................74

11.4 Identify Event Set Observable Behaviors ............................................................75

11.5 Assess Crew Performance ..................................................................................77

MODULE 12: Standardizing the Assessment Process..................................................79

12.0 Overview ...........................................................................................................79

12.1 Standardization of Crew Assessment ..................................................................81

12.2 Working With the LOE Worksheet Data ............................................................83

12.3 Make Consistent Ratings Across Crews..............................................................85

12.4 Discriminate Between Behaviors Across Event Sets ...........................................87

Index ..........................................................................................................................89

Appendix C Page C-4

 Appendix D

Instructions for

Facilitating an IRR Training Workshop

Appendix D Page D-1

 This Page

Intentionally Left Blank

Appendix D Page D-2

 INTRODUCTION

Initially, facilitating an Inter-Rater Reliability (IRR) workshop can seem overwhelming.

Yet, proper planning and attention to detail make this task much easier. This document was
designed to walk you through an entire IRR training session. Using this facilitation guide, your
airline carrier will have the knowledge and tools to be successful at conducting and benefiting
from periodic IRR training. There are 6 major sections that proceed in chronological order.
These sections describe each of the major steps in preparing for and conducting IRR training
sessions, as well as interpreting and using IRR data. We caution you to pay close attention to
detail in each step of the process in order to avoid problems down the line. The 6 Sections are:

• Section 1: Preparation for the IRR Workshop

• Section 2: Viewing and Rating the Video Segment
• Section 3: Data Analysis
• Section 4: Feedback and Discussion
• Section 5: Optional Post-Training IRR
• Section 6: Dissemination of Summary Document

It is important that as an IRR facilitator you are familiar with the mechanics of how to run
an IRR workshop. In addition, you should be knowledgeable about the purpose and reasoning
behind why IRR training workshops are conducted. The main goal of IRR training is to improve
crew assessments. Good quality pilot and crew assessment is important for (1) providing fair
evaluations of pilot and crew performance, (2) providing accurate information for fleet managers,
(3) providing meaningful information for management, and (4) ensuring flight safety. The IRR
rating exercises followed by detailed feedback are helpful in calling attention to rating
discrepancies. Identification of rating discrepancies followed by IE input on how to resolve rating
problems will improve the rating system.
The frequency with which you conduct IRR training is ultimately up to your organization.
However, based on past experience training IEs using the IRR training principles and procedures,
we recommend that IRR training take place at least on a semi-annual basis. In addition, we
recommend that IRR training be part of the indoctrination training for new IEs. Of course, these
recommendations are subject to change based on various factors such as frequent turnover in IE
personnel, changes in the evaluation forms, or the results of previous IRR training sessions
compared to corporate benchmarks.

Appendix D Page D-3

Section 1: Preparation for the IRR Workshop

Preparation should begin approximately 2-3 weeks before the training workshop. In this
section, the necessary training materials are prepared and compiled.
1. Subject Matter Experts (SMEs) select the videotape segments for use in the
workshop (1 segment for each Event Set). These segments should cover a range of performance
categories. For each video segment, the SMEs should agree on the level of performance
exhibited. This is extremely important, as the IEs’(Instructor/Evaluators’) ratings will be
compared to the SMEs’ratings. In general, 6 segments are sufficient for a 2 hour training
session. If you have time, you may also wish to develop 2-6 additional segments for a post-
training test (optional, see Section 5, p. 19).
2. SMEs prepare the necessary pre-briefing information for each video segment. This
should include contextual information, such as a description of what the crew has performed prior
to this point in the video. In addition, critical information such as “gear speed exceeded” (which
is not available on the tape) should also be prepared.
3. Copies of the LOE guide should be prepared for each Event Set that will be used
in the training.
4. Copies of the LOE worksheets should be prepared. The worksheet pages should
appear in the same order as the video segments. To make the rating sheets more user friendly,
each sheet should be clearly labeled regarding the crew that is being evaluated and the Event Set
that is taking place (e.g., “Crew 1, Event Set 1”).
5. Immediately prior to the training, two computers should be prepared. These
computers will be used to analyze the data. The optimal choice would be two Pentium computers
with Microsoft Excel (Version 5.0) and two laser printers. To test the computers, load
Microsoft Excel. From the “Tools” menu, choose “Add-Ins...”. Select all of the “Analysis
Toolpak” add-ins and all of the “MS Excel 4.0” add-ins. Test the IRR macros on both machines.
Lastly, select a workspace area for the data entry task that can accommodate two people.

Appendix D Page D-4

Section 2: Viewing and Rating the Video Segments

This section should begin on the day of the training workshop, as close to 8:00 am as
possible. First, the IEs are given a brief orientation. Following this, the IEs individually rate the
video segments. During the orientation and rating process, the facilitator should:
1. Provide the IEs with the day’s schedule. An example schedule is:
8:00 - 10:00 Video viewing and rating by IEs
10:00 - 1:00 IEs discuss other issues and eat lunch while data
are analyzed by the training facilitator(s).
1:00 - 3:00 Feedback session
3:00 - 5:00 Post-training video viewing and rating
2. Briefly acquaint the IEs with the rating forms, the rating guidelines, and the rating
process. IEs should quickly examine the rating forms so that they know what is expected of
them. The facilitator should then clarify ambiguities and answer any questions that the IEs may
have about the rating process. (This step should take about 20 minutes to complete.)
3. Emphasize the necessity of no interaction among the IEs during the rating cycle.
Laughter, emotional outbursts, and non-verbal communication can contaminate the IEs’ratings.
4. Have the IEs provide either their PIN numbers, or 3- letter initials on the cover
page of their rating packet. We recommend using 3-letter initials because, on occasion, IEs have
had the same 2-letter initials, which created confusion during the feedback/discussion process.
5. Provide necessary contextual information to the IEs before each video segment.
However, the facilitator should be very careful to avoid telling the IEs how to make their ratings.
The video segment should then be played. If necessary, the facilitator should provide necessary
critical information while the tape is running (e.g., “the airspeed exceeded the gear speed while
the gear was down”).
6. Pause at the end of each segment, so that the IEs have time to finish their ratings.
All IEs must finish their ratings before moving on to the next video segment.

Appendix D Page D-5

Section 3: Analysis Phase

The data analyses should begin as soon as possible. The analyses typically take 2-3 hours
to complete. The analysis phase begins with data entry into the computers and ends with the
facilitator preparing for the feedback session.

Getting the Data Ready

1. Before the IEs leave the room, be sure that they have written their 3-letter initials
(or PIN Numbers) on the cover sheet of their rating packet. Then, collect all of the rating
packets.

2. Have a 2-person team enter the data (one person reads off the data, while the other
enters the data into Excel.) Each IE’s ratings should appear as a single column of numbers in
Excel. It may be helpful to highlight the rows of observable behavior data in one color, and the
task/skill data in another color. (This makes it easier to use the “Arrange Data” macro in the
next step).

Appendix D Page D-6

 3. Use the “Arrange Data” macro to separate the 3-point observable behavior data
from the 4-point task/skill data. (All macros are found under the “macros” command in the
“Tools” menu). Follow the on screen prompts and the end result will be two data worksheets,
one with observable behavior data (Sheet 1), and the other with task/skill data (Sheet 2).

4. Save the data file onto the C: drive for faster access while running the macros.
5. Save a copy of the data onto floppy disk (A: or B: drive) for transfer to the other
computer. Once the file is loaded on the other computer, be sure to save it to the C: drive for
faster operation.
6. Divide up the labor. Have one person analyze the 3-point observable behavior
data, while the other person analyzes the 4-point task/skill data. Each person should make sure
that he/she has only the assigned type of data (i.e. there is no task/skill data mixed in with the
observable behavior data).
7. Make a visual inspection of the data. Does missing data cluster by IE or by
question? If 30% or more data is missing for a given IE, delete that IE. If 30% or more data is
missing for a given question, delete that question. If less than 30% of the data are missing, replace
the missing data. You can use the modal (most frequently occurring) value for a given question,
or you can use the “Clean Data” macro to substitute the mean value.

Appendix D Page D-7

Agreement Analyses
1. Run the “Agreement” macro. Follow the on-screen prompts. You will be asked
to enter a benchmark value (GMU recommends using .80). Print the block of all ratings and
agreement indexes, sized to 1 page. (This can be done by selecting “print 1 page high by 1 page
wide” from the “page setup” command in the “File” menu.)

2. Next, examine each of the individual graphs for items that exhibit low agreement.
If time permits, it may be helpful to format all of the graphs so that they are on a common metric.
To do this, first note which graph has the largest maximum value on the vertical axis. For each
graph, change the vertical axis so that they are all on this scale. This can be semi-automated by
changing the format of the first graph and then using the F4 key [repeat function] to change the
format on the remaining graphs.
3. Individually select and print each graph on a separate page.
4. When all the graphs are printed, hand-label each graph with the event set, item
name, and the crew name.

Appendix D Page D-8

Systematic Differences Analyses
1. From the “Tools” menu, choose “Data Analysis”. Select “2 Factor Analysis of
Variance Without Replication”. Block out the raw data, starting with the @ symbol. Click the
“Labels” box. Click the “Output” button, click on the output address box, and click on the
desired location on the worksheet. Click “OK”.

2. After the analysis is complete, find the value for the probability tested (it is the “p­
value” in the row labeled “Columns” in the ANOVA source/summary table). If this value is
below .05, there are systematic differences among the raters.

3. Run the “ttests” macro. Follow the on-screen prompts. Examine the results to
determine which raters are significantly higher or lower than the group mean. Then, use this same
information to change the colors of the bars on the graph to indicate these raters. Next, make
sure that the horizontal axis of the graph starts with “1”, and ends with the appropriate value (“3”

Appendix D Page D-9

for observable behaviors, “4” for task/skill). Print the t-test graph on a single page. It may be
helpful to increase the font size of the title and the legend, so that it is easier to read.

Congruency Analyses
1. Run the “Congruency” macro. Follow the on-screen prompts. Examine the
vertical axes on both sides of each graph. Note the maximum value (on either side). For each
graph, change the left and right vertical axes, so that they are all on the same scale. Print each
congruency graph on a single page. (see next page for example)

Appendix D Page D-10

 2. Lastly, print the block of congruency results, sized to 1 page.

Appendix D Page D-11

Consistency Analyses
1. From the “Tools” menu, choose “Data Analysis”. Choose the “Correlation”
option and follow the on-screen prompts. For the “Input Range” highlight the instructor ID labels
and the raw data. Be sure to indicate that the data is grouped by columns and that the labels are
located in the first row.

2. Run the “Consistency” macro. Follow the on-screen prompts. You will use the
output from the “Correlation” command (previous step) as your input for the consistency macro.
You will be asked to enter a benchmark value in two different steps, one for RATERS and one
for the GROUP, in both cases GMU recommends using zero. Consistency indexes range from –1
(perfect negative correlation: individual goes up, group goes down) to +1 (perfect positive
correlation: individual goes up, group goes up – or – individual goes down, group goes down)
with zero indicating no relationship between individual and group ratings. A benchmark of zero
will indicate those raters whose individual ratings have no significant relationship to the groups’
ratings as being “NS”.

Appendix D Page D-12

DO NOT ask for the graphs in this macro. Click “cancel” to stop the “Consistency” macro
after you enter the GROUP benchmark, when you come to this prompt:

3. Print the overall block of consistency results, sized to 1 page.

Appendix D Page D-13

 4. Run the “Profiles” macro. Follow the on-screen prompts. Examine each IE’s
profile graph. If the vertical axis does not begin with “1” and end with the appropriate maximum
value (“3” for observable behaviors, “4” for task/skill), change the scale of the vertical axis. Print
each IE’s graph on a single page.

Sensitivity Analyses

1. Run the “Sensitivity” macro. Answer the on-screen prompts using the SMEs’
pre-determined decisions on how to label the performance segments (e.g., “3 crew”, “2 crew”).
The final data set should look something like this:

2. Examine each IE’s sensitivity profile. If the vertical axis does not begin with “1” and end
with the appropriate maximum value, change the scale of the vertical axis. Print each IE’s graph
on a single page.

Appendix D Page D-14

3. Lastly, print the overall block of sensitivity results, sized to 1 page.

Prepare group and individual level feedback

1. Prepare the group level feedback. This will include:
• 2 pages with blocks of group data for congruency (one page for the 3 pt data and
another page for the 4 pt data)
• 2 pages of t-test graphs for systematic differences (3 pt data and 4 pt data)
• 2 pages with blocks of group data for consistency (3 pt data and 4 pt data)
• 2 pages with blocks of group data for sensitivity (3 pt data, and 4 pt data)
• 1 page block of all ratings and agreement indexes for 3 pt data
• All graphs of the LOW-agreement 3-pt items in order
• 1 page block of all ratings and agreement indexes for 4 pt data
• All graphs of the LOW-agreement 4-pt items in order
MAKE OVERHEAD TRANSPARENCIES OF ALL THESE PAGES IN ORDER

Appendix D Page D-15

2. Prepare individual level feedback packets for each IE. These will include:
• pages with his or her Congruency graphs (one page for the 3 pt data and another
page for the 4 pt data)
• pages of t-test graphs for 3 pt data and 4 pt data (same graphs as in the facilitator’s
packet)
• pages with his or her Consistency profile graphs (3 pt graph and 4 pt graph)
• pages with his or her Sensitivity graphs (3 pt graph and 4 pt graph)

WRITE EACH PERSON’S CONGRUENCY, CONSISTENCY AND

SENSITIVITY INDEX VALUES ON TOP OF EACH GRAPH (3 PT AND 4 PT).

When writing the values on the top of each graph, we have found it helpful to have one
person read aloud the index values from the block of group data while the second person writes
the index value on top of the IE’s graph. Alternatively, these index values can be added to the
title of each graph before the graphs are printed by editing the title of the graph after they have
been created by their respective macros. The individual packets are now ready to be stapled and
distributed.
OPTIONAL: if facilitators do not have good and bad examples of congruency,
consistency, and sensitivity already prepared, then they should choose the best and worst
examples from this group of IEs and make overheads of these examples.

Appendix D Page D-16

Section 4: Feedback and Discussion
Feedback and discussion should begin when the facilitator is ready to present the results to
the group. This includes describing what the statistical results mean, as well as facilitating group
discussion about the individual items. The facilitator should be familiar with the following most
common types of rater errors so that this information can be available and integrated into group
discussions:

Leniency and Severity: Raters who on a consistent basis are either inordinately lenient
(“Santa Claus”) or inordinately severe (“Ax man”). Systematic differences
analyses will tell you if raters are rating consistently lenient or harsh.

Central Tendency: Raters who avoid using high and low extremes and tend to cluster all of
their ratings about the center of the rating scale. Low congruency indices may
indicate the central tendency rating error.

Halo: Raters who assign ratings on the basis of a global impression of the ratee. An
individual is rated either high or low on many factors because the rater knows that
the individual is high or low on one specific factor. Low consistency indices may
indicate halo rating error.

Contrast Effect: When raters evaluate more than one candidate at a time, they tend to use
other candidates as a standard. Whom they rate favorably, then, is determined
partly by others against whom the candidate is compared. (E.g. If a rater evaluates
a candidate who is just average after evaluating three or four very unfavorable
candidates in a row, the average candidate tends to be evaluated very favorably,
instead of just average.) If two or more crews are taped for each event set, then
contrast errors can occur. Extremely high sensitivity scores may indicate a
contrast effect.

Congruency, Systematic Differences, Consistency, and Sensitivity Data (30-45 minutes)

1. The facilitator should begin with an overview of the discussion procedure. Then, a
discussion about congruency, systematic differences, consistency, and sensitivity results should
occur. Although these components can be discussed in any order, we recommend that your first
discuss congruency, then systematic differences followed by consistency and sensitivity.

Appendix D Page D-17

 2. For the congruency, consistency, and sensitivity measures, the facilitator should
begin with a 1-sentence definition of the measure, and then explain why it is important. This
should be followed with examples of good and bad performance (use overheads to help get the
point across). Next, the IEs should refer to their individual feedback packets to see how they
compare to the average group index (using both 3 pt and 4 pt data). This should end with a
discussion of the implications for IEs who are incongruent, inconsistent, or insensitive in their
ratings.
3. For the systematic difference data, the facilitator should begin with a 1-sentence
definition and then explain why it is important. The facilitator should then present the t-test graph
(for both the 3 pt and 4 pt data) and point out the “Ax-Men” and the “Santa Clauses”. The IEs
will have a copy of these graphs in their feedback packet. This should end with a discussion of the
implications for IEs who are significantly high or low in their average ratings. Appropriate
judgment standards and fairness to pilots can be discussed as focal points.

Agreement on Individual Items (1-2 hours)

1. For the agreement data, the facilitator should begin with a 1-sentence definition of
agreement and then explain why it is important. Then, examples of good and poor agreement
should be presented to the group.
2. Following this, the low agreement (below benchmark) items for each tape segment
or event set should be discussed and resolved. For each tape segment or event set, low agreement
items may include observable behaviors, skill or topic ratings, overall ratings of CRM or technical
performance, and/or ratings of the pilots.
For each low-agreement item, an overhead graph showing the disagreement should be
presented. Then, the following questions should be asked:
Did the raters SEE the same thing?
Did the raters INTERPRET the same behaviors the same way?
Did the raters JUDGE the behavior the same way using the scale?
Possible sources of divergent judgments must be explored. These could include: poor videotape
quality; ambiguous wording of judgment items; unclear judgment criteria; or different judgment
processes by the IEs.
In general, the IEs should work to achieve consensus on each item. Company
policies, flight standards, or other issues may surface during this discussion. One team member
should document the IEs’comments and input using the Feedback and Discussion Worksheet so
that problems and ideas that are generated during the discussion session can be used in the future.
At the end, thank the IEs for their efforts.

Appendix D Page D-18

Section 5: Optional Post-Training IRR Check
Section 5 should begin when the group discussion has ended. This section includes rating
new video segments, and meeting (the next day) to discuss them.
1. IEs view new tape segments in the same fashion as before.
2. Team analyzes these new data, and provides individual and group-level feedback
as before (overnight or next day).
3. Individual feedback packets are distributed to the IEs the next day.
4. Facilitator presents pre- and post-training IRR results (see sample pre/post form
below). As a group, the facilitator and the IEs decide on priorities for the next IRR session.

Average Average Average Average Average

Consistency Congruency Systematic Agreement Sensitivity
Rating Rating Differences Rating Rating

Day 1

Day 2

To demonstrate the effect of training, the numbers in the “Day 2” row should be (statistically)
larger than the numbers in the “Day 1” row.

Appendix D Page D-19

Section 6: Dissemination of Summary Document
Immediately after the IRR training session, one team member needs to revise and
summarize the IE comments/suggestions solicited during the Feedback and Discussion session.
This information should then be distributed to all of the IEs. This will serve to improve the clarity
of judgment criteria and judgment processes among IEs. In addition, all key training and
management personnel should receive this information so that they can address unresolved rating
issues that need to be managed under their authority (e.g. rewording of LOE items, clarification of
policy)

Appendix D Page D-20

 Appendix E

Guidelines for Taping Simulator Sessions

for CRM Assessment Data Collection

Appendix E Page E-1

 This Page

Intentionally Left Blank

Appendix E Page E-2

 Guidelines for Taping Simulator Sessions

for CRM Assessment Data Collection
Purpose of the Guidelines

The following preliminary set of guidelines is designed for those planning to produce videotapes
for collecting CRM assessment data. The guidelines are presented in five groups related to the
following LOFT/LOE session elements:

Crew-related
Instructor-related
Scenario-related
Simulator-related
Editing-related
Equipment-related

In this document, the collection of CRM assessment data refers to having aircrew
instructors/assessors view tapes and rate CRM performance using one or more rating schemes.
These guidelines assume that when videotapes are used to collect that type of data, the production
of those tapes should conform to guidelines that are more restrictive than general guidelines for
producing CRM assessment training tapes. When videotapes are used to collect CRM assessment
data, the video taping process should follow most of the crew-related, instructor-related, and
editing related guidelines presented below.

Crew-Related Guidelines

During the taping of the LOFT or LOE sessions, crews should not be interrupted by the instructor
or by other avoidable disturbances. The natural flow of the scenario is important for realistic
simulation and elicitation of natural crew behavior. Do not provide the crewmembers with scripts
on how they should act and make sure they are not familiar with the session's scenario. They
should be briefed to perform as they would normally under similar conditions

If the goal is to have a video tape which is representative of an air carrier's crews, the crews
selected to make the video tape should have average crew performance, typical crew composition,
normal crew attitudes to CRM and training and other observable crew characteristics that could
affect their behavior.

When videotapes will be used to collect CRM assessment data:

• Use real crews that are qualified for the equipment and position being flown in the
simulator.
• Provide the crew with a regular briefing of the scenario and session (similar to that
provided prior to a recurrent LOFT) to include a discussion of the confidentiality of
the data collected and then specify how the tapes will be used.
• Schedule the taping session and provide for sufficient briefing time to allow the crew
to act as though they were in an actual training or assessment session.

Appendix E Page E-3

 • Have crews should utilize the simulator equipment related to communications as is
normally used in the simulator environment.
• the camera or cameras should be placed so that they do not interfere with pilot tasks
and so that they are not directly visible by either of the crew members.
• If additional microphones are needed for improved sound quality, the microphones
should be positioned before the start of the session with any testing taking place prior
to the start of the session.

Instructor-Related Guidelines

• Use an instructor that is qualified in the type of session being taped (e.g., if you are
taping a LOFT, use instructors qualified to manage LOFT sessions).
• Use an instructor trained and experienced in the scenario or scenarios being used.
• Make sure that the instructor does not interact with the crew except when providing
ATC and other required communication.

Scenario-Related Guidelines

• Have the LOFT or LOE based on a scenario that is clearly structured and that
specifies the training or assessment objectives for each of the scenario elements or
event sets.
• Provide the instructor managing the scenario and researchers collecting data with a
complete script of the scenario.

Editing-Related Guidelines

• Tapes should contain a minimum of edits, and edits should be evident such as through
the used of a "fade-to-black."
• The individual shots of segments or events should be of sufficient duration to allow a
viewer to understand the context of the events taking place.
• The sound track should be that sound that was recorded simultaneously with the video
image, and any narratives or other additions should be clearly identified as such.
• Similar sequences or unique shots should be placed in context either through an
accompanying document or by providing context information prior to the actual
sequence or unique shot when used for rater training.

Simulator-Related Guidelines

Set the simulator motion, sound, visual system, and lighting to best enhance the image and sound
quality of the tape while having a minimum effect on crew performance (see Technical section).
Monitor this quality continuously during the taping.

• Place camera and microphone in the simulator so as to collect the best sound and
picture with the least effect on crew performance.
• Specify any changes in simulator setting that deviate from normal settings for the type
of training or assessment being represented.

Appendix E Page E-4

Equipment-Related Guidelines

These guidelines review the necessary equipment and procedures for videotaping two-person
crews in an aircraft simulator with sufficient quality to use in training applications.

Camera
Videotaping in aircraft simulators requires a professional-quality video camera with
interchangeable lenses and adjustable apertures and shutter speeds. Aircraft simulators are
cramped and extremely low-light conditions (2-3 lux) for videotaping. Cramped cockpit quarters
require a compact camera with a wide-angle lens. The extreme low-light conditions (particularly
for a nighttime simulation) require a camera with adjustable aperture and shutter speeds. Even
with the widest possible aperture, shutter speeds must be adjustable down to 1/15 (day sim) and
1/10 (night sim) of a second. Briefing sessions before or after the simulator session require
normal to mild telephoto lenses. The camera we use is a Canon L-2 Hi-8 camera with
interchangeable lenses and shutter speeds adjustable to 1/8 of a second. The lens we use ranges
from wide-angle (used in cockpit) to mild telephoto (used for briefings).
Use special Hi-8 grade 8mm videocassettes to obtain effective low-light videotaping in the
simulator--these tapes are NOT the consumer-grade 8mm tapes and will cost $15-$20 apiece (e.g.
Sony Hi8 tape E6-120HME). We used one 2-hour tape for the briefing before the simulator
session and the debrief afterwards, and two 2-hour tapes for the four-hour simulator session. We
recommend winding and rewinding the tapes in the camera before use to check if they are faulty-­
we experienced about 1 out of 15 faulty tapes.
To transfer the video from the Hi-8 tapes, use the Hi-8 camera or obtain a Hi-8 VCR for
playback. All components in the taping and playback for transfer (camera, tape, VCR) must be
Hi-8 grade to preserve the resolution of the Hi-8 system. Consider using a professional video
service to transfer the Hi-8 video to VHS format for instructional use.

Other Equipment
Microphones. For brief and debrief sessions, a lavaliere microphone that clips onto the tie
is least obtrusive. For taping in the simulator, the microphone must be small and attachable to the
microphone boom on the pilot’s headset. (Any other mounting position will give unacceptable
levels of background noise from the simulator— simulated engine noise should be set as low as
possible). We used a SIMA camcorder Lapel Microphone(TM) miniature clip-on lavaliere
microphone (around $30). An equivalent alternative is the Audio-Technica U.S. ATR35s
lavaliere microphone. These lavaliere microphones are detachable from the tie clip and small and
light enough to be used on the headset boom. This is a battery-powered microphone with an on-
off switch for the battery pack, which is in-line on the microphone cord. Each battery is good for
5-10 hours of use, so have spare batteries for microphones and camera.
Connectors. An adapter that takes 2 mini headphone jacks and combines the signal into
one output is used to combine the PIC and SIC audio output into one combined signal. A 6” mini
“Y” adapter, which connects 2 mini headphone, jacks to a single mini stereo output jack is used to
connect the combined PIC and SIC microphones on one channel with the instructor pilot’s
microphone on the other channel. This stereo output is plugged into the camera’s microphone
jack.

Appendix E Page E-5

 Auxiliary lighting. Auxiliary lighting is important to increase effective illumination of
specific areas such as the control panel. Small, battery-powered fluorescent lights are
recommended as the batteries will last for a four-hour simulator session and they do not heat up
the cockpit or potentially set off the halon fire control. Arrangement of these lights to avoid
reflections in the visual displays or interfering with pilot movements and vision is critical (see
diagram). We suggest a variety of sizes of these lights with larger lights used for general
increases in illumination while smaller lights increase illumination of specific areas.
Tape. Duct tape is used to tape down the microphone lines to avoid tripping and to
secure lighting in overhead or on-the-wall positions. Black electrical tape is used to secure the
lavaliere microphone to the pilot’s headphone microphone boom. It is important that the lavaliere
microphone be taped to be on the OUTSIDE of the boom microphone. This will avoid unwanted
breathing sounds and the possibility of electric shock from the lavaliere microphone.

Brief and Debrief Procedures

Pre-simulator briefs and post-simulator debriefs are normal light-intensity situations.
Normal shutter speeds and sufficiently narrow apertures to ensure an adequate depth-of-field can
be used. To be unobtrusive, position the camera as far from the instructor pilot, PIC, and SIC as
possible and use the telephoto lens to frame the three persons appropriately. The lavaliere
microphones are mounted on the ties of each person. For soft-spoken persons, mount the
microphone just below the tie knot. Route the microphone cords down and along the floor to be
as unobtrusive as possible. Avoid microphone cords or any apparatus on the table, as this will
interfere with use of the weather pack and the completion of other normal preflight activities.
Make certain video consent forms are read and signed before activating any of the equipment.
Switch the microphones “on” and do a sound check to check audio recording levels before
beginning the brief or debrief.

Simulator Procedures
Before session. Microphone cords for PIC and SIC should be routed around their seats
before they enter the cockpit. PIC and SIC can then enter and be seated. We take the headsets
and tape the lavaliere microphones to the microphone booms. We request the preflight actions be
done with headsets on to improve the audio quality. We also request that the pilots leave the
panel lights on the highest possible intensity that will not interfere with their normal routine.
Auxiliary lighting is then installed as necessary to obtain the best possible video picture. Low
wattage bulbs can be placed on the simulator floor behind PIC and SIC seats or can be taped part
of the way up the bulkhead. The simulator instructor is also miked, and final video and sound
checks are made.
During session. During the simulator session, the video operator must continue to
monitor picture and sound to ensure quality. If the sim is in motion, the operator must be buckled
in, but he or she can still wear the headphones to monitor sound quality. We have encountered
problems with microphones gradually having more static as batteries wear out and the video
camera turning itself off unexpectedly. The operator will notice the latter immediately as the
sound in the headphones cuts out when the camera stops recording, at which point the operator
must restart the camera and re-check sound and video quality.
After session. After the simulator session, disconnect and remove microphones, lighting,
and cables quickly but carefully to move the operation back to the briefing room. After the
debrief session, make sure to switch the microphones “off” to avoid draining the batteries.

Appendix E Page E-6

 Appendix F

Sample LOFT/LOE Development Materials

Appendix F Page F-1

 This Page

Intentionally Left Blank

Appendix F Page F-2

LOFT/LOE SCENARIO DEVELOPMENT PROCESS

A structured development process is needed for LOFT/LOE that specifies the objectives and
related technical and CRM skills that make up LOFT scenarios. This process should start with an
analysis of the carrier's training needs and follow through to the development of the LOFT
scenario, its validation, and the training of instructors that will implement and assess crew
performance (see Table 1 for a listing of LOFT/LOE development steps). This LOFT/LOE
scenario development process is based on the concept of event set, a group of related events that
are part of the scenario and are inserted into the session for specific CRM and technical training
objectives. Event sets may be used in isolation, as in a SPOT, or they may be used in groups, as
in the development of a LOFT or LOE scenario. The LOFT/LOE development process makes
the design of a new LOFTs easier and more systematic resulting in training that is more
manageable and easier to assess by allowing instructors to concentrate on key technical and CRM
objectives within any given event set.

The primary unit of both LOFT design and CRM assessment is the event set, a group of related
events that comprise the scenario and are inserted into a LOFT session for specific training
purposes. The event set is made up of one or more events, including an event trigger, a distracter,
and conditions, the supporting events. The event trigger is the condition or conditions under
which the event is fully activated specifying the focal problem the pilot/crew must deal with in
the event set. The event trigger also functions as the focal point for the assessment of crew
performance. The supporting events are other events taking place within the event set designed
to further training objectives and add to overall scenario realism. Finally, the distracter is a
condition inserted within the event set designed to divert the crew's attention from other events
that are occurring, are about to occur, or to increase crew workload. Although distracters may
not always add to the realism of the event set, they should not significantly detract from that
realism.

Event sets allows the design team to present the appropriate degree of realism in the LOFT.
Instead of focusing on a specific technical issue, the event set integrates the entire complex line
environment (e.g., terrain, ATC, weather issues, etc.) to provide realism and relevance and evoke
crew's performance in response to specified CRM and technical issues. The event set helps the
LOFT designer to introduce operational realism through the specification of conditions within and
across event sets. In addition, the event set helps the instructor to manage and assess the LOFT
by making explicit the purpose of conditions such as terrain, ATC, and weather as they interact
with other LOFT elements. With the LOFT scenario made up of one or more event sets, scenario
validation and crew assessment is performed at the event set level rather than validating the
overall LOFT scenario.

Appendix F Page F-3

Table 1: LOFT/LOE Development Steps

STEP/SUBSTEP
1) Identification of Incidents and Primary CRM/Technical Objectives

1.1 Develop list of relevant incidents and events

1.2 Identify primary technical and CRM TPOs and SPOs

1.3 Identify related observable behaviors for the CRM categories

1.4 Specify observable behaviors, TPOs, and SPOs in a LOFT Outline

2) Development of LOFT Scenario Event Sets

2.1 Translate incidents into a LOFT Outline by identifying the event set(s) to include

event trigger, distracter, and supporting events.

2.2 Review and update the LOFT Outline to ensure a complete scenario

2.3 Prepare and administer the LOFT Check Form to ensure event sets are consistent

with the technical/CRM objectives and observable behaviors

2.4 Modify LOFT Outline based on LOFT Check Form results.

3) Validation of the LOFT Scenario

3.1 Expand the LOFT Outline into the LOFT Guide by adding the operational details to

include ATC, weather, airport information, etc.

3.2 Fly the LOFT Guide using at least two different crews taping the sessions for use in

developing instructor training materials

3.3 Administer the LOFT Validation Form to individuals that have flown or seen the

tapes of the LOS

3.4 Develop the LOFT Worksheet based on the LOFT Validation Form results

4) Instructor Training and Standardization

4.1 Review and make changes to the LOFT Guide and LOFT Worksheet based on

instructor input

4.2 Conduct instructor calibration session using IRR

4.3 Review IRR results and modify training and retrain as necessary

4.4 Conduct periodic Instructor Standardization Sessions

Appendix F Page F-4

 LOE Observable Behaviors
Validation Form

Instructions: Thank you for your help in providing ratings of crew actions or observable
behaviors central to the assessment of CRM. Each of the event sets in this form was designed to
assess a primary CRM element. Each scenario event set also has one or more secondary CRM
elements. The four CRM elements used in this form have been divided into two groups. The first
group of two CRM elements is related to individual mental factors that crew members utilize to
identify and solve the problems presented in the scenario event set. The two Individual Mental
Factors are:
Decision Making
Situational Awareness

The second group of CRM elements is related to crew or team factors. The CRM elements
related to the Team Factors are:
Crew Communication
Team Management

Please work through the following pages of this form by first rating the Primary CRM Element
from 1 to 5 based on the probability that the CRM Element is the primary element or objective of
the CRM assessment. For example, if there is a "High " probability that the element is the
Primary CRM Element, then you would enter a "4" as follows:

PRIMARY CRM ELEMENT: COMMUNICATION 4

Then, please rate the key observable behaviors for that CRM element based on the degree to
which you think that the individual behavior is a key behavior for the assessment of the tasks
related to that scenario event set. Rate these behaviors by reviewing the event set concentrating
on the "CONDITIONS" which specify the tasks that a crew should perform during that event set.
Then, determine the probability or likelihood that each observable behavior is an important
behavior to observe in the assessment of the tasks being performed.

Please use the following scale for all of your ratings:

1 2 3 4 5
Very Low Low Medium High Very High

If there is an additional CRM element that should be considered for the assessment of the event
set, please include it in the ADDITIONAL CRM ELEMENT at the bottom of each page.

Appendix F Page F-5

 EVENT SET 1

PRE DEPARTURE:
Event Set #1 is the pre-departure through the beginning of takeoff. The event trigger is the consideration
and ramifications of summer operations, low visibility and wind shear. Other event distracters will
include:

Conditions for departure include:

POSSIBILITY OF WIND SHEAR

LOW VISIBILITY TAXI

THUNDERSTORMS

TURBULENCE ON DEPARTURE

ENGINE START:
ABORTED ENGINE START.
TAXI OUT:
Congested ramps and taxiways in low visibility.

CONDITIONS FOR EVENT SET ONE:

DISPATCH -
PREFLIGHT - WITH MALFUNCTIONS
START AND PRE-TAXI - WITH ENGINE START PROBLEM

RATE CRM ELEMENTS from 1 to 5 where RATE Obs. Behaviors from 1 to 5 where
1 = Very Low Probability and 5 = Very High 1=Very Low Degree and 5=Very High
PRIMARY CRM ELEMENT: TEAM MANAGEMENT
Obs. Behavior: Encourages crewmember interaction for complex departure
Obs. Behavior: PF inform crewmembers of intentions before acting
Obs. Behavior: Crew discussed summer operations SOP
Additional Obs. Behavior:

SECONDARY CRM ELEMENT: DECISION MAKING

Obs. Behavior: PF selected correct action for aborted engine start
Obs. Behavior: PF analyzed takeoff WX and requested takeoff alternate
Obs. Behavior: Captain made timely decisions after problems were identified
Additional Obs. Behavior:

ADDITIONAL SECONDARY CRM ELEMENT:

Obs. Behavior:
Obs. Behavior:
Obs. Behavior:

Appendix F Page F-6

 EVENT SET 2

Event Set #2 will be as the aircraft approaches the end of the runway for departure. The trigger will be
the stop of departures because of thunderstorms. Conditions include:

TAKEOFF FROM CONTAMINATED RUNWAY NEAR RUNWAY LIMIT WEIGHT

LOW VISIBILITY TAXI

PLANNING FOR WINDSHEAR DEPARTURE.

REROUTE AROUND THUNDERSTORMS

DIFFERENT INTERSECTION FOR DEPARTURE

RATE CRM ELEMENTS from 1 to 5 where RATE Obs. Behaviors from 1 to 5 where
1 = Very Low Probability and 5 = Very High 1=Very Low Degree and 5=Very High
PRIMARY CRM ELEMENT: SITUATIONAL AWARENESS

Obs. Behavior: Crew set clear priorities for tasks and their order
Obs. Behavior: Crew plans for delay and intersection departure
Obs. Behavior: Red flags recognized and resolved
Additional Obs. Behavior:

SECONDARY CRM ELEMENT: COMMUNICATION

Obs. Behavior: PNF advocated if necessary
Obs. Behavior: PNF advised company and at appropriate time
Obs. Behavior: PF asked questions to establish understanding of aircraft condition
Additional Obs. Behavior:

ADDITIONAL SECONDARY CRM ELEMENT:

Obs. Behavior:
Obs. Behavior:
Obs. Behavior:

Appendix F Page F-7

 EVENT SET 3

Event Set #3 is the takeoff and area departure. The event trigger will be a minor system problem after to
V1. The conditions include:

WINDSHEAR PLANNING REQUIRED DURING THE TAKEOFF

THUNDERSTORMS IN THE DEPARTURE AREA

HEAVY WEIGHT, RUNWAY LIMIT TAKEOFF WITH HIGH SURFACE TEMPERATURE

GUSTY SURFACE WINDS ON DEPARTURE

RATE CRM ELEMENTS from 1 to 5 where RATE Obs. Behaviors from 1 to 5 where
1 = Very Low Probability and 5 = Very High 1=Very Low Degree and 5=Very High
PRIMARY CRM ELEMENT: DECISION MAKING

Obs. Behavior: Bottom lines established for abort

Obs. Behavior: crewmember acknowledge operational decisions
Obs. Behavior: PIC made decision in timely manner
Additional Obs. Behavior:

SECONDARY CRM ELEMENT: COMMUNICATION

Obs. Behavior: PNF advocated if necessary
Obs. Behavior: PNF advised ATC and at appropriate time
Obs. Behavior: PF asked questions to establish understanding of aircraft condition
Additional Obs. Behavior:

ADDITIONAL SECONDARY CRM ELEMENT:

Obs. Behavior:
Obs. Behavior:
Obs. Behavior:

Appendix F Page F-8

 Appendix G

Sample ASRS Incident Reports

Appendix G Page G-1

 This Page

Intentionally Left Blank

Appendix G Page G-2

Report 01

We were already in a descent from 10000 to 8000' when approach recleared us to continue the
descent to 6000'. The Captain, who was flying the aircraft on autopilot, did not descend past
8000'. As we began this flight the Captain was in a not so good a mood. Early morning, late
arrival, arguments with the station manager and a very abrasive and intimidating attitude towards
myself. At this point in the flight the Captain was flying the aircraft via the autopilot in a descent. I
told the Captain I was going off 1 to call company on #2. At the same time approach called and
recleared us to a lower alt of 6000'. As I scanned the flight instrument I dialed in 6000' on the alt
preselect and noticed the autopilot attempting to level off at 8000' with a 250' fpm rate of descent.
I looked at the Captain and said, "O.K., now I’m off 1 to call company on 2 again." He
acknowledged with a statement, "fine, knock yourself out." Now as I was on #2 radio I noticed
the Captain off #1 and on the P/A. Next I noticed, as I came back on #1, that the Captain had not
descended to 6000'. We were level on autopilot at 8000' and approach was calling for us to
descend immediately to 6000' for conflicting traffic. The Captain also came back on and heard the
calls. We descended to 6000' and the Captain yelled at me to keep my ears open and so on. I
decided not make an argument with him sensing it would further endanger the flight as we entered
the terminal area and work load was increasing. I wanted to punch out the Captain at first for the
way he treated me and the flight. I calmed my self down and concentrated on doing my job, even
though I knew there would be a lesser degree of safety due to the Captain’s attitude. I learned
again not to relax my scan, even when flying with a guy who was as intimidating as this one. I'll
always speak up.

Appendix G Page G-3

Report 02

BGM information 800' scattered, 1200 broken, 2300' overcast, 2 mi. light snow, temp 27, dew
point 21, winds 270 at 8 kts, altimeter 29.97, breaks in the overcast, NDB 34 approach in use,
localizer 16/34 out of service, runway 34 plowed and sanded full width and length, braking action
good reported by a vehicle. PF and PNF discussed possibility of using runway 28 due to surface
wind component. Further inquiry with BGM approach confirmed runway 28 plowed and sanded
full width and length and previous inbound company reported braking action as poor. Surface
wind was also reported as unchanged. PF (PIC) requested and was cleared for VOR DME
approach runway 28. PNF briefly studied the procedure, and then gave the approach plate to the
PF. Due to the proximity of the airport, the high indicated airspeed, the excess alt and the flight
crew's anticipation of the ILS 34 approach, the workload of the flight crew was quite high. The
PNF went off frequency to make the range call to company. The PF descended from the published
segment alt (3500' MSL) at the 18 DME position to the published straight in landing MDA of
2000' MSL. The FAF for the procedure was at the 13 DME position and the PF’s premature
descent put the aircraft 1500' below the published segment alt. The airport was sighted prior to
the visual descent point and the approach terminated uneventfully. Supplemental information from
ACN 104150
it was discovered by approach control just as we passed the final approach fix. I was pushing the
aircraft to maintain schedule.

Appendix G Page G-4

Report 03

Received taxi instructions to runway 22 for first flight of the day. I set heading but for 220
degrees and was about to taxi when ground reported wind as 290 degrees at 8 kts (ATIS reported
at 240 degrees at 9 kts). F/O and I were doing preflight checklist as we were taxiing to runway
(standard procedure), and even though I had runway 22 in mind, I actually visualized runway 28
and started that way. We checked for traffic on the long runway as we crossed it on the way to
28. Just about the time we finished the checklist I realized that we had just crossed the active

runway 22. At this point we had just switched to tower frequency and he just noticed that we

were on the wrong side of 22. No traffic conflict occurred, but even though we had looked for

traffic, a conflict was possible. Factors were

first flight of the day; at familiar airport, but hadn't been there recently; performing checklist while

taxiing; and not checking HSI reading before crossing runway.

Appendix G Page G-5

Report 04

We were cleared for takeoff when our left-hand pitot heater failed. Our MEL states we can fly
w/o the left-hand pitot heater if we stay clear clouds below +4 sat and/or known icing conditions.
I was not sure we could comply with those restrictions, but I decided to go anyway. Enroute we
checked weather at our destination and alternates. All had high ceilings (6000'), 15+ nm plus
visibility and were well above freezing (55 degrees f). We also received reports of heavy icing in
clouds at 11000' MSL. Still, I elected to press on. We talked over the possible implications and
possibility of losing the left-hand ASI and altimeter. We discussed the procedures for having the
pilot in the right seat assume flying duties if I lost my airspeed and altimeter. We were incredibly
stupid. In our descent at 9000' MSL due encountered not only very heavy icing, but moderate to
severe turbulence. I almost immediately lost both my ASI and altimeter. The right seat pilot
assumed flying duties. His ASI showed a warning flag so was suspect and could not be trusted.
Because of the turbulence he needed both hands to control the aircraft. I had to control the
throttles, monitor his flight ins from my side, assess the situation and make decisions, and handle
the radios. The PF also stated he felt he wasn't turning when he was, and vice versa, obviously he
was experiencing incipient vertigo due to the turbulence and loss of ins. Even though we had
good gyros the loss of other primary flight ins interrupted our normal scan and confused our
normal senses. To further complicate the situation, Center couldn't give us a lower alt. I finally
communicated the urgency of the situation and they gave us a lower alt. At that point we couldn't
maintain alt because of the turbulence anyway. Once we had a clearance to a lower alt we
descended by reference to attitude only. We broke out at 5000' MSL in warmer air and almost
immediately got all ins back. We made a normal approach and landing. There are many lessons to
be learned here. My motivation for continuing the flight with a known deficiency was the result of
pressure of a very heavy schedule with upper management aboard, and my desire not to
inconvenience them. I also now realize I was getting complacent and felt I could do anything
(arrogant). I am appalled by my lack of professionalism. The worst part is I know better, but I still
succumbed to outside pressures. Never again. Callback conversation with reporter revealed the
following: reporter reiterated self-displeasure for his actions. He said, "I did what the company
pays me not to do." The reporter also said how shocked he was at the great speed with which the
situation got out of control. One factor not initially related by the reporter is that at the time of the
incident, the flight crew had been on duty 14 hours, and that fatigue probably affected judgment.
The reporter is also the chief pilot for his company. After this incident, he called all company
pilots to a meeting to discuss this incident, its causes and effects. He felt that his flight department
could benefit from a discussion of the human factors issues.

Appendix G Page G-6

Report 05

The problem arose when the Captain (PNF) called "1000' to go" as we passed 8000'. I estimate
20-30 seconds later the Captain suddenly remember to call in our times (out/off) to company. The
Captain was unsure of our out/off times and asked me. We both looked at our watches and
agreed on the times to be used. Next, the Captain was not sure what company frequency was and
asked me. I looked down to retrieve my chart binder to verify the freq. When I looked back up to
the panel, we were passing 9300'. I immediately lowered the nose and acknowledged my alt
deviation to Captain I began an immediate descent to assigned alt, 9000'. The Captain, who also
was looking for frequency to call company and not monitoring the panel either, tried to find the
right knob to turn mode c off on transponder, which I had to help him find. I estimate we were at
9300-9400' for 10-15 seconds and never actually level at either. I returned the plane to assigned
alt (9000') in about 25-30 seconds. Problem corrected. Being on ready reserve from mid-morning
till late evening. The Captain had flown since early morning, and had totaled 7 hours of flying that
day. He was tired and appeared to be unfamiliar with aircraft. This made me feel uneasy. Both
Captain and I were tired and Captain distracted me and himself when within 1000' of assigned alt.
Having been off duty 3 days before the flight caused both scan and judgment to be slowed. I
would recommend a strict tolerance to a sterile cockpit anytime climbing or descending within
1000' of an assigned alt; i.e., no paperwork, no calls to company or passengers. Both pilots should
be watching the plane and the panel in order to x-check and verify what should be done is being
done.

Appendix G Page G-7

Report 06

Ground control had cleared me to taxi to runway 29. Another company aircraft had been cleared
to taxi to runway 34. I misunderstood the controller’s instructions and taxied across runway
11/29. My crew and I arrived at the aircraft 1 hr, 30 minutes late due to crew rest requirements.
We were hurrying to get back on schedule. Neither my F/O nor myself are that familiar with the
Westchester airport. Contributing factors: rushing to get back on schedule, operating on a new
airport, and confusing my taxi instructions with those given to another aircraft. I was advised by
ground control that I had taxied across runway 11/29. Lessons learned: 1) don't rush, especially
when operating on a new and unfamiliar airport. 2) if you are not sure of instructions, stop and
ask. Supplemental information from ACN 140815: as the Captain taxied the aircraft I proceeded
with the checklist and setting up the radios. Although not mandated, this is usually accomplished
prior to taxiing, but because of the behind schedule situation, we elected to set them up while
taxiing. I repeated the taxi instructions to the Captain, he acknowledged. I periodically cleared the
area during taxi while accomplishing the checklist and setting of the radios. I went back to my
duties prior to the intersection of the taxiway and runway 29. I cleared the area 7-10 seconds later
and noticed we had crossed runway 29 and were headed for runway 34.

Appendix G Page G-8

Report 07

Shortly after departure from Johnstown, Pa, after having made contact with Center and radar
contact, Center asked us where we were going. I was the F/O and PNF. On taxiout during the
before takeoff checklist, the Captain quickly briefed the SID (one of several) for this non-tower
controlled airport. This SID climbed us out NE of JST on the 048 deg radial. I was not at all clear
about this departure or how it would feed us onto the route (V12 out of JST to AGC). We were
IFR in blowing snow and rain, climbing to 8000' MSL. The Captain who did not appear to
understand about the filed rte seemed puzzled about Center’s inquiry. I told the Captain that the
route (V12) was west on the JST 272 deg radial, and we were NE of the vortac. The Captain got
very upset and gave me no directions to respond to Center and seemed confused. I took the
initiative because I knew the rte to respond to ATC. I said, "aircraft ident is off course, please
give us a vector back to course." Center turned us to 270 deg heading, which I assumed was to
intercept the course outbound. During this time the Captain told me to fly while he fished through
the IFR charts to see what the course was--his first course familiarization. He didn't assume the
PNF duties FNAV and COM, so when Center called again there was ambiguity over what anyone
was supposed to do. My HSI was set up for the 274 deg radial. I wanted to compare that
situation to the RMI needles, which I thought were set to VOR mode for JST. The RMI needles
were pointing 180 degrees, so I took it upon myself, since no one seemed to be in charge, to turn
to the station on that information and speed up the intercept. However, to my dismay, I had the
RMI set to ADF mode. Center called in again as I erringly navigated toward JST on the ADF
display and again asked where we were going. The Captain, already fit to be tied, got even more
upset and erratic seeing what was taking place. Finally Center told us to forget the intercept and
radar vectored us to pit approach controller’s airspace and handed us off. I realized my NAV
error and my erroneous assumptions--the Captain accepted blame for not doing his job right, and
we progressed into pit on the ILS runway 32 w/o further incident.

Appendix G Page G-9

Report 08

We were taxiing onto the runway (r21) at Hilton Head airport when my F/O said there is an small
aircraft y turning final. The aircraft on final initiated a go-around when he noticed that we were
taxiing onto the runway. There was nothing we could do to evade the aircraft if he had landed
since it was not possible to maneuver light transport x off the runway in time. I did not hear a
radio call by small aircraft y before or while taxiing onto the runway, or while taxiing out.
However we did not initiate a call, either. There are several contributing factors. One is a fatigue
factor of a 15 hr duty day the day before. The other is the F/O announcement to passengers while
taxiing. The F/O was still making an passengers announcement while I was calling departure
control for clearance. I believe a prerecorded passengers announcement on aircraft w/o a F/A
would be of great benefit at airports where there is a short taxi time. This is, however, also a case
of not enough diligence on our part and or not making the announcement of our intentions.

Appendix G Page G-10

Report 09

The deviation from assigned alt occurred because of a nose gear light. Shortly after takeoff from
reading airport, we experienced a nose wheel warning light indicating that the nose wheel had
failed to lock in the up pos. After cycling the gear twice, the light failed to go out. I (copilot) read
the checklist and it was decided to leave the gear down and proceed to the destination. Shortly
after, within 10-20 seconds of our decision, I requested to cycle the gear one last time. The
Captain agreed and the gear was cycled. On this last attempt the nose wheel locked into the up
pos. During this time the Captain failed to level off at 4000' MSL. At 4300' MSL the mistake was
brought to our attention by the arrival alt warning system. During our level off and subsequent
descent, ATC advised us of our error, and firmly reconfirmed 4000' MSL. The rest of the flight
was normal. Captain (PF) and copilot both became preoccupied with light. The deviation from
assigned alt (4000') was discovered upon passing 4300', which set off our alt alert system's aural
warning. Captain confirmed assigned alt, leveled off and descended back to 4000' (level off was
accomplished by 4600' MSL). Human performance factors: 1) the PF diverted too much attention
to the problem and not enough attention to his primary function--flying the aircraft. 2) although
the primary function of the PNF was to address the abnormality, he should have been aware of the
Captain’s failure to level off. Also may have been untimely to ask for one last cycle of the gear.

Appendix G Page G-11

 This Page

Intentionally Left Blank

Appendix G Page G-12

 Appendix H

Sample ACRM Crew Training Manual

Table of Contents (TOC)

Appendix H Page H-1

 This Page

Intentionally Left Blank

Appendix H Page H-2

 Table of Contents

Table of Contents.......................................................................................................i

List of Acronyms........................................................................................................iii

MODULE 1 Recurrent Training Under AQP..............................................................1

1.0 Overview ............................................................................................................1

1.1 Advanced Qualification Program (AQP)..............................................................1

1.2 Single Visit Requirements ...................................................................................2

MODULE 2 Advanced Crew Resource Management (ACRM) ..................................3

2.0 Overview ............................................................................................................3

2.1 Need for ACRM Based on NTSB Safety Study...................................................3

2.2 Primary NTSB Safety Study Results....................................................................3

2.3 Types of Crew Errors..........................................................................................4

2.4 ACA Training Data .............................................................................................6

2.5 ACRM from CRM Principals to Procedures ........................................................8

2.6 ACRM and the FAA Grant..................................................................................9

MODULE 3 ACRM Briefings ....................................................................................11

3.0 Overview ............................................................................................................11

3.1 ACRM Development Process..............................................................................11

3.2 Crew Briefings and Workload .............................................................................11

3.3 The Normal Checklist..........................................................................................12

3.4 The Briefings ......................................................................................................13

3.5 Group Activity with ASRS Report ......................................................................19

MODULE 4 ACRM QRH..........................................................................................21

4.0 Overview ............................................................................................................21

4.1 ACRM Improved QRH ......................................................................................21

4.2 ASRS Example Showing Need for ACRM Procedures ........................................23

4.3 Before and After Snap-Shot of QRH ...................................................................24

MODULE 5 Crew Effectiveness Under AQP .............................................................29

5.0 Overview ............................................................................................................29

5.1 Developing the LOE Scenario and ASRS Reports ...............................................29

5.2 Scenario Event Sets and Total Crew Performance ...............................................30

5.3 Evaluation of LOE Performance..........................................................................30

5.4 Assessment Scale and Evaluation of Training System ..........................................31

MODULE 6 Implementing LOEs and Line Checks.....................................................33

6.0 Overview ............................................................................................................33

6.1 Briefing the LOE.................................................................................................33

6.2 Crew Role During the LOE.................................................................................33

6.3 Debriefing the LOE .............................................................................................34

6.4 Group Activity with ASRS Report ......................................................................35

APPENDIX ...............................................................................................................A-1

Appendix H Page H-3

 This Page

Intentionally Left Blank

Appendix H Page H-4

 Appendix I

Sample Quick Reference Handbook

Procedures

and Briefing Guide

Appendix I Page I-1

 This Page

Intentionally Left Blank

Appendix I Page I-2

 OLD QRH PROCEDURE

PROPELLER HEATER FAILURE

AFM 4-10-5

__PROP ICE

PROP ICE PROT...............................

OTHER CYCLE
DOES THE [PROP] CAPTION GO OFF?

YES,CONTINUE FLIGHT
///END

NO

AVOID FLIGHT IN ICING CONDITIONS

IF PROPELLER VIBRATION OCCURS

POWER LEVER..............MINIMUM PRACTICABLE
CONDITION LEVER.......MINIMUM PRACTICABLE

///END

NEW QRH PROCEDURE

Propeller Heater Failure

Page 1 of 1

Condition: [ _ PROP] caption illuminated on overhead

Ice Protection Panel

1. Assign PF/PNF duties.

2. Select the opposite cycle (LONG CYCLE or SHORT CYCLE) on the
affected prop heat.

�Does the [ _ PROP] caption extinguish?

YES - [ _ PROP] caption is out.
1. Continue flight using normal procedures
/// End

NO - [ _ PROP] caption is not extinguished.

1. Avoid icing conditions.
2. If a propellor vibration occurs, proceed as follows:

Power lever ................................ ....................... Minimum practical

Condition lever ................................ .................. Minimum practical

Preparation and Planning:

Develop plan
Consider:
• If a diversion is required to avoid icing conditions crew should
review: weather / notams / field conditions / alternates / and
fuel-time remaining.
Establish Bottom Lines
Establish Backup Plan
Brief Plan

/// End
7-6 (rev orig)

Appendix I Page I-3

Consistent Features of the

New QRH
Condition [ _ PROP] caption illuminated on overhead
Ice Protection Panel

1. Assign PF/PNF duties.

�Does the [ _ PROP] caption extinguish?

Preparation and Planning :
Develop plan
Consider:
• If a diversion is required to avoid icing conditions crew should
review: weather / notams / field conditions / alternates / and
fuel-time remaining.
Establish Bottom Lines
Establish Backup Plan
Brief Plan

Appendix I Page I-4

 Sample Briefing Guide

Preflight Brief

Tone - try to follow SOP, new items for today

Crewmember Roles - back each other up with decisions

Crew Communication - keep all crew members in the loop

Teamwork - call switch movements, both visually identify

traffic/airports

Assertion - speak up with questions, doubts or concerns

Operational Issues - low time minimums, DMIs

Clearance Brief

ATIS/NOTAMS

Routing/SID/terrain

Runway/taxi conditions

Assign PF/PNF

Plan for abnormals after takeoff

Performance

Takeoff Brief

Statement of Condition
Select and Prioritize:
� Runway conditions
� Low visibility procedures
� Hydroplaning
� Crosswinds/windshear
� Terrain/MSA
� Aircraft performance
� Convective activity
� GPWS/TCAS alerts
� Fuel status/delays
Bottom Lines for takeoff
Backup Plan for takeoff
Initial heading and altitude

Appendix I Page I-5

 Sample Briefing Guide

Arrival Brief

Descent profile
ATIS/NOTAMS
Statement of Condition
Select and Prioritize:
� Fuel status/delays
� Runway conditions
� Low visibility procedures
� Terrain/MSA
� Aircraft performance
� Convective activity
� Crosswinds/windshear
� Hydroplaning
� GPWS/TCAS alerts
Bottom Lines for arrival
Backup Plan for arrival

Approach Brief

Approach plate information

Required calls/profile

Crew coordination

Debrief Items

Deviations from SOP

Crew coordination - CRM and technical aspects

Unusual situations - positive and negative

Workload management - rushed, overloaded or confusion

Conflict - differences in expectations

Maintenance discrepancies

Planning for aircraft servicing

Appendix I Page I-6

 Advanced CRM Briefing Guidelines

The Captain remains the final authority. These procedures empower all
crew members to speak up on any issue before, during or after a flight.
They encourage discussion to prioritize, make good decisions, and
resolve problems.

Preflight Brief - Completed on the first flight of the day and any crew change. It
should focus on crew coordination and aircraft operational issues.

Clearance Brief - Items discussed prior to engine start to aid decision making
and review current operational conditions as well as planning for the
taxi/takeoff. Planning for items included on the Takeoff Brief is encouraged to
reduce workload and enhance situation awareness during taxi.

Takeoff Brief - Helps crews manage workload before potential distractions

occur during a critical phase of flight such as the takeoff roll or initial climbout.
Crews should prioritize all relevant conditions that exist for that particular
departure and establish Bottom Lines and a Backup Plan.

Arrival Brief - Should be accomplished early in the descent to aid in arrival

planning. Crews should prioritize all relevant conditions that exist for that
particular arrival, approach, and landing and establish Bottom Lines and a
Backup Plan.

Approach Brief - Contains approach plate information and any unusual profile
or crew coordination issues. Crews should emphasize the critical elements of
the approach, missed approach, and landing

Debrief Items - Allows feedback of crew performance (both positive and

negative) and allows planning of non-critical items such as aircraft servicing and
scheduling issues. Crew debriefs should be conducted discretely.

Use this model to make a decision during an event:

1. Captain: assign PF/PNF duties

2. Clearly identify the problem
3. Consult the QRH if appropriate
4. Crew members propose solutions and choose the best plan
5. Prioritize, sequence, and assign crew duties
6. Monitor the plan’s progress (bottom lines)
7. Reevaluate and plan again as necessary (backup plan)

Appendix I Page I-7

 This Page

Intentionally Left Blank

Appendix I Page I-8