This article was downloaded by: [University of Glasgow]

On: 21 December 2014, At: 10:05

Publisher: Taylor & Francis
Informa Ltd Registered in England and Wales Registered Number: 1072954
Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH,
UK

The International Journal of

Aviation Psychology
Publication details, including instructions for
authors and subscription information:
http://www.tandfonline.com/loi/hiap20

Pilot Interaction With Cockpit

Automation II: An Experimental
Study of Pilots' Model and
Awareness of the Flight
Management System
Nadine B. Sarter & David D. Woods
Published online: 13 Nov 2009.

To cite this article: Nadine B. Sarter & David D. Woods (1994) Pilot Interaction With
Cockpit Automation II: An Experimental Study of Pilots' Model and Awareness of the
Flight Management System, The International Journal of Aviation Psychology, 4:1,
1-28, DOI: 10.1207/s15327108ijap0401_1

To link to this article: http://dx.doi.org/10.1207/s15327108ijap0401_1

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the
information (the “Content”) contained in the publications on our platform.
However, Taylor & Francis, our agents, and our licensors make no
representations or warranties whatsoever as to the accuracy, completeness,
or suitability for any purpose of the Content. Any opinions and views
expressed in this publication are the opinions and views of the authors, and
are not the views of or endorsed by Taylor & Francis. The accuracy of the
Content should not be relied upon and should be independently verified with
primary sources of information. Taylor and Francis shall not be liable for any
losses, actions, claims, proceedings, demands, costs, expenses, damages,
and other liabilities whatsoever or howsoever caused arising directly or
indirectly in connection with, in relation to or arising out of the use of the
Content.
 This article may be used for research, teaching, and private study purposes.
Any substantial or systematic reproduction, redistribution, reselling, loan,
sub-licensing, systematic supply, or distribution in any form to anyone is
expressly forbidden. Terms & Conditions of access and use can be found at
http://www.tandfonline.com/page/terms-and-conditions
Downloaded by [University of Glasgow] at 10:05 21 December 2014
 THE INTERNATIONALJOURNAL OF AVIATION PSYCHOLOGY, 4(1), 1-28
Copyright O 1994, Lawrence Erlbaum Associates, Inc.

FORMAL PAPERS
Downloaded by [University of Glasgow] at 10:05 21 December 2014

Pilot Interaction With Cockpit

Automation 11: An Experimental
Study of Pilots' Model and Awareness
of the
Flight Management System

Nadine B. Sarter and David D. Woods

Department of Industrial and Systems Engineering
The Ohio State University

Technological developments have made it possible to automate more and more

functions on the commercial aviation flight deck and in other dynamic high-
consequence domains. This increase in the degrees of freedom in design has
shifted questions away from narrow technological feasibility. Many concerned
groups, from designers and operators to regulators and researchers, have begun
to ask questions about how we should use the possibilities afforded by technol-
ogy skillfully to support and expand human performance. In this article, we
report on an experimental study that addressed these questions by examining
pilot interaction with the current generation of flight deck automation. Previous
results on pilot-automation interaction derived from pilot surveys, incident
reports, and training observations have produced a corpus of features and
contexts in which human-machine coordination is likely to break down (e.g.,
automation surprises). We used these data to design a simulated flight scenario
that contained a variety of probes designed to reveal pilots' mental model of one
major component of flight deck automation: the Flight Management System

Requests for reprints should be sent to Nadine B. Sarter, Department of Industrial and
Systems Engineering, The Ohio State University, 1971 Neil Avenue, Columbus, OH 43210.
 2 SARTER AND WOODS

(FMS). The events within the scenario were also designed to probe pilots'
ability to apply their knowledge and understanding in specific flight contexts
and to examine their ability to track the status and behavior of the automated
system (mode awareness). Although pilots were able to "make the system
work" in standard situations, the results reveal a variety of latent problems in
pilot-FMS interaction that can affect pilot performance in nonnormal time
critical situations.

The introduction of advanced technology to modern flight decks has

Downloaded by [University of Glasgow] at 10:05 21 December 2014

succeeded in increasing the precision and efficiency of flight operations.

However, recent accidents and incidents involving glass-cockpit aircraft
have suggested that the current generation of cockpit automation may
have created new operational burdens and new kinds of failure modes in
the overall human-machine system (Billings, 1991). Only a limited em-
pirical database is available concerning the nature and circumstances of
existing problems in pilot-automation interaction (Eldredge, Dodd, &
Mangold, 1991; James, McClumpha, Green, Wilson, & Belyavin,l991;
Wiener, 1989). These data consist primarily of either subjective data
obtained from questionnaires and interviews or of in-flight observations
of pilot interaction with one of the core systems of cockpit automation,
the Flight Management System (FMS). The resulting data about pilots'
attitude towards the system and the anecdotal reports of problems indi-
cate that there is a need for further research that will systematically
analyze the nature of and the reasons for FMS-related problems. This
knowledge will be critical for developing countermeasures and improving
pilot-automation interaction.
With this goal in mind, we studied pilot-FMS interaction through three
different methodological approaches that allowed us systematically to col-
lect converging data to describe existing problems and to understand why
they exist. In the first report on our work (Sarter & Woods, 1992b), two
exploratory research activities were described. A survey of pilots7 self-re-
ports of their operational experiences with the FMS and observations of
transition training from a conventional to a glass-cockpit aircraft were used
to gather a corpus of problems with FMS operation. This corpus consisted of
detailed incident descriptions, from which major underlying problem catego-
ries were extracted.
These categories provided the basis for the design of a scenario for an
experimental study of pilots' mental models and their awareness of the
FMS. In this study, we confronted 20 experienced pilots with situations
and tasks that are instances of the previously identified FMS-related
problem categories. The pilots flew the scenario on a part-task training
simulator that had been developed to teach FMS operations. As a result,
it was possible to test the completeness and accuracy of their FMS-related
knowledge as well as their ability to apply this knowledge in specific
situations.
 AWARENESS OF THE FMS 3

INTRODUCTION TO THE FMS

The FMS supports pilots in a variety of tasks, such as flight planning,

navigation (guidance), performance management, and monitoring of
flight progress. One of its major functions-and the function of primary
interest in the context of the reported studies-is automatic flight-path
control.
The major FMS controls in the cockpit are the mode control panel (MCP)
and the multifunction keyboards of two control display units (CDUs; one for
Downloaded by [University of Glasgow] at 10:05 21 December 2014

each pilot). FMS-related cockpit displays are the CDU multifunction dis-
play, two attitude director indicators (ADIs), and two horizontal-situation
indicators (HSIs). Figure 1 illustrates the typical location of these different
FMS components in a generalized glass cockpit.
The CDUs consist of a multifunction control unit (a keyboard) and a data
display. The keyboard is used by pilots to enter data that define a flight path
and to access flight-related data available on various pages within the CDU's
page architecture. The pilot-entered flight path, continuously updated to
reflect current flight status, is presented on the map display of the HSI. This
allows pilots to monitor progress along the path. In the HSI Plan mode, the
pilot can visually check modifications to the active flight plan.
The MCP is used to activate different automatic flight modes: Vertical

FIGURE 1 Flight-deck controls and displays related to pilot-FMS interaction in a

generalized glass cockpit (FMAs = flight mode annunciations).
 4 SARTER AND WOODS

Navigation (VNAV), Lateral Navigation (LNAV), Heading Select (HDG

SEL), and Level Change (LVL CHG). The pilot can also use knobs on the
MCP to dial in targets for individual flight parameters (airspeed, heading,
altitude, and vertical speed), which are tracked by the system if a corre-
sponding automatic flight mode is activated. To find out which FMS modes
are currently active, the pilot can monitor the flight mode annunciations on
the ADI. These provide data on the active (or armed) pitch and roll modes
and on the status of the autopilot(s). They also indicate the status and mode
of the autothrottles, which can be set to manual or automatic mode for speed
Downloaded by [University of Glasgow] at 10:05 21 December 2014

and altitude control. The various FMS interfaces and autoflight functions
provide the pilot with a high degree of flexibility in selecting and combining
levels of automation to respond to different situations and requirements.
It is important to remember that there are various modes of automatic
flight control that range between the extremes of automatic and manual. The
highest level of automatic control occurs in the VNAV and LNAV modes. In
these modes of control, the pilots enter-or, in their words, "program7'-a
sequence of targets into the CDU that defines an intended flight path, and
then activate the automatics by selecting VNAV and/or LNAV through con-
trols on the MCP. The Flight Management Computer automatically controls
the aircraft to follow the desired flight path. At this strategic level of auto-
mation, the FMS pursues a sequence of target values without the need for
further intervention by the pilot. This is particularly helpful in situations that
allow for long-term planning with a low likelihood of deviations from the
plan (e.g., in the cruise phase of flight).
When pilots need to intervene quickly and change flight parameters (e.g.,
in terminal areas), other lower levels of automation are available. Pilots can
enter target values for different flight-path parameters (i.e., airspeed, head-
ing, altitude, vertical speed) on the MCP. They can then activate one of the
corresponding modes (e.g., Heading Select or Level Change), and the target
will be captured and maintained automatically until the target or mode of
control is actively changed by the pilot.
An important characteristic of automatic flight-path control is the high
degree of dynamism. Transitions between modes of control occur in re-
sponse to pilot input and to changes in flight status. Automatic mode changes
can occur automatically when a target value is reached (e.g., when leveling
off at a target altitude) or based on protection limits (e.g., to prevent or
correct pilot input that puts the aircraft into an unsafe configuration).
Both the flexibility of the FMS and the dynamism of flight-path control
impose cognitive demands on pilots. They have to decide which level and
mode of automatic control to use in a given set of circumstances, and they
also have to track the status and behavior of the automation. The latter task
requires that they attend to and integrate data from a variety of indications in
the cockpit such as flight mode annunciations on the ADI, visualization of
the programmed route of flight on the HSI, or the display of target values on
the MCP.
 AWARENESS OF THE FMS 5

METHOD

General Approach

The study was designed based on a phenomenon-driven ethnographic ap-

proach to studying cognitive systems in high-tempo event-driven worlds
(Woods, 1993a). First, we had to identify an experimental environment for
studying pilot-FMS interaction. It seemed important to account for the
numerous concurrent tasks that have to be carried out by the pilot in the real
Downloaded by [University of Glasgow] at 10:05 21 December 2014

operational environment and that could affect his or her FMS-related perfor-
mance. Also, the impact of the high-tempo nature of flight had to be captured
to arrive at valid results. Therefore, a strict laboratory study with a restricted
set of tools and environmental fidelity was rejected. The other extreme on
the scale of possible approaches-a high-fidelity full-mission simulation
study-was rejected because some of its inherent capabilities (e.g., aircraft
motion, outside view) were not essential to the purpose of this study and
because there were high costs associated with obtaining access to such
facilities. As a result, we chose an environment that allowed for realistic
tools and tasks as well as for a fairly high level of fidelity: a part-task
training simulator for FMS operations.
The next important step in conceptualizing the study was designing the
scenario based on predefined phenomena of interest (Woods & Sarter, in
press). This is much more than making the scenario as realistic as possible.
A realistic setting only provides the background against which the scenario
needs to be staged. In this study, the problem categories identified by our
survey and by the training observations represented the phenomena of inter-
est. The scenario-design process involved identification of specific tasks and
events to be linked together in a coherent scenario that would probe these
phenomena. This approach enables the experimenter to trigger behavior of
interest rather than hoping for it to happen accidentally. Although this ap-
proach may underlie many simulation studies, it is often not explicitly laid
out for the reader of a research article. In contrast, this article will provide a
detailed description of the match between phenomena of interest and events
within the simulated scenario.
Based on the scenario, a canonical model of pilot behavior in response to
scenario probes and events was built (for the general case, see Woods,
1993a; for a different, specific example, see Roth, Bennett, & Woods, 1987).
In contrast to normative models, which prescribe one acceptable response to
each task or event, this canonical model describes a set of plausible trajecto-
ries-that is, it describes various possible ways in which pilots may behave
in response to the scenario probes. This model was used to develop a
data-recording instrument to encode observed pilot behavior directly during
each test run.
An observer knowledgeable in FMS operations and the test scenario kept
track of pilots' interaction with the FMS on line by placing checkmarks
 or-in the case of unanticipated behavior or events-comments on the data-
collection sheet. In addition, pilots were asked to describe their reactions to
hypothetical events that could not actually be simulated due to time restric-
tions and about FMS-related knowledge in general. These questions were
asked in low-workload phases of flight without interrupting the simulation.
This allowed us to probe pilots' knowledge within the actual operational
environment instead of questioning them out of context, when their task
would be more related to the retrieval of information than to its application.
A few questions were asked before or after completion of the flight as they
Downloaded by [University of Glasgow] at 10:05 21 December 2014

related to more general topics or to preflight activities.

After each test run, the instructor and the observer walked through the
pilot's behavior and verbal responses during the simulated flight. This de-
briefing helped elicit the instructor's interpretation of any ambiguous or
unanticipated observations and helped prompt the instructor to add other
observations that might have been missed.
The data were deliberately collected during the experiment rather than
extracted after the fact from video and audio recordings of the simulation
runs. Such recordings can be helpful for exploratory studies or in cases
where a knowledgeable observer is not available. But even though retrospec-
tive analysis of videotapes may sometimes reveal unexpected or previously
unattended but interesting behavior, there are disadvantages as well (e.g.,
investigators can be overwhelmed by the amount of data and thus be unsure
of how to abstract broader results from all the details). It is more difficult to
get line pilots and their representatives to agree to participate in a study
when videotaping is involved. In addition, videotape is no substitute for
careful and detailed identification of what one is looking for, based either on
the mapping between phenomena of interest and the specific scenario or on
what one might expect as canonical behavior for knowledge of that field of
practice (Woods, 1993a; Woods & Sarter, in press).

Experimental Scenario

The experimental scenario for this study was designed to address predefined
phenomena of interest. These phenomena had been identified by the corpus
gathering activities (pilot survey and training observations) preceding the
study (see Sarter & Woods, 1992b). The issues were related to (a) pilots'
proficiency in standard tasks, @) pilots' mental model of the functional
structure of the FMS and (c) their awareness of system state and behavior
(mode awareness). In cooperation with a flight instructor, we identified tasks
and events that would best serve to probe these phenomena. The basic flight
context consisted of a flight from Los Angeles to San Francisco that took
approximately 60 min to complete.'

h he actual flight time is longer, but temporary increases in the simulated aircraft speed were
used during quiescent phases of flight to reduce time on the simulator.
 AWARENESS OF THE FMS 7

The following sections provide an overview of the mapping between

phenomena of interest and specific tasks and events within the scenario.
Figure 2 illustrates the flight route and the timing of the tasks and events
throughout the scenario. To better understand the following description of
the scenario, it might be helpful for the reader who is not familiar with
glass-cockpit technology to review the Introduction to the FMS section of
this article.
Downloaded by [University of Glasgow] at 10:05 21 December 2014

AMENDED CLEARANCE
.-.- -..

FIGURE 2 The timing of scenario tasks and events along the flight route.
 8 SARTER AND WOODS

Proficiency in Standard Tasks

Table 1 lists the standard tasks that pilots carried out in the course of the
scenario. Our previous study (Sarter and Woods, 1992b) showed that pilots'
proficiency at standard tasks did not seem to be a major source of difficul-
ties. It was included as part of this scenario to provide additional converging
evidence based on a scenario evolving in real time and involving ilots with
line experience in glass cockpits to confirm the previous results. 2'
Downloaded by [University of Glasgow] at 10:05 21 December 2014

Pilots' Knowledge of the Functional

Structure of the FMS

By functional structure of the FMS we refer to pilots' knowledge about

how the FMS behaves in different flight situations rather than their ability to
simply recite facts about the FMS. For example, do they understand the
sequence of mode changes, their associated indications, and the correspond-
ing aircraft behavior throughout the takeoff roll?
To probe this phenomenon of interest, we built into the scenario a
variety of tasks and situations that permitted inferences about pilots'
knowledge of the system and their ability to apply this knowledge in
actual task contexts. Knowledge of overall FMS functionality was subdi-
vided into six subtopics (discussed hereafter), and specific probes were
developed for each subtopic.

Knowledge of the CDU page architecture. The page architecture of

the FMS CDU contains a huge amount of data that may be relevant at some
point during the flight. Because only a very limited set of data can be
presented on the CDU screen at any given time, pilots need to be able to
navigate through the "hidden" data space. To find out about problems related
to this task, pilots were asked to locate information on CDU pages on the
following topics: (a) single-engine capabilities, (b) wind data for fixes of
flight, (c) available fuel, and (d) localizer frequency and front course for a
runway.
We also asked pilots about their expectations concerning data propagation
throughout the CDU page architecture. After pilots had entered speed and
altitude target values on the cruise page to comply with an amended clear-
ance by ATC, we asked whether they expected these data to propagate to the
descent page to become the targets for their descent.

he group of standard tasks presented in this experiment did not include the Flight Manage-
ment Computer System Performance Initialization, as we had already seen during the training
observations that these tasks did not challenge the pilots. Also, we wanted to focus on tasks that
have to be performed in the dynamic airborne portion of flight rather than on ground tasks that
are not as affected by time pressure or concurrent tasks.
 AWARENESS OF THE FMS 9

TABLE 1
Scenario Probes of Pilots' Proficiency at
Standard FMS-Related Tasks
Route changes
Intercepting a radial
Going directly to a waypoint
Building and executing a hold
Installing an instrument-landing-system approach
Entering crossing restrictions
Unplanned level-off
Downloaded by [University of Glasgow] at 10:05 21 December 2014

Extending the final-approach fix

These probes were intended to test pilots' knowledge of the page architec-
ture of the CDU as well as their ability to use the CDU interface to call up
information and/or pages.

Mode availability and disengagement. After being vectored off-

course by ATC, pilots were asked to recapture the preprogrammed route.
This task was introduced to find out whether pilots were aware of the criteria
that have to be met in order for the LNAV mode to capture the original flight
path.
When being cleared by ATC for the instrument-landing-system approach,
pilots were asked to set up the FMS properly to be able to use the automatic
APPROACH mode. They had to remember that a lower MCP altitude had to
be selected before engaging the APPROACH mode. Without this first step,
the APPROACH mode engagement would not result in the desired start of
descent; rather, the FMS would control the aircraft to maintain the MCP
target altitude.
After localizer and glideslope capture on final descent, pilots were asked
to describe how they would disengage the APPROACH mode if ATC told
them to change heading and altitude for traffic.
These probes allowed us to determine whether pilots were familiar with
the general prerequisites and procedures for engaging or disengaging a mode
and whether they could apply this knowledge to a specific flight context.

FMC logic. After takeoff from Los Angeles, pilots were asked to inter-
cept the LAX 249"radial outbound. To perform this task successfully using
LNAV, the pilot had to understand that the FMS logic is to always fly
towards-not away from-a waypoint. As his original flight plan did not
include any waypoint on the radial, he first had to create a fictitious fix
somewhere on the radial to which the FMS could fly.
After completion of the flight, we asked pilots about functional character-
istics of the VNAV path descent in comparison to the VNAV speed descent.
The questions referred to the way in which either one of these types of
 10 SARTER AND WOODS

descent is initialized, what control mode the system uses to maintain target
speed in either mode, and what is the lowest altitude to which the system
automatically descends.

Effects of partial system failures. During a descent, pilots were asked

about the expected consequences of losing the autothrottles: Would the
aircraft still level off at target altitude, and what would be the consequences
in terms of airspeed? How would they intervene in that case?
After glideslope capture, we disabled the glideslope to simulate a signal
Downloaded by [University of Glasgow] at 10:05 21 December 2014

loss at about 3,000 ft. This allowed us to test whether pilots would realize
what happened, whether they would understand the implications of losing
the glideslope, and how they would react to the failure. In addition, they
were asked about the differences between a glideslope failure above versus
below 1,500 ft above ground level (AGL).
If the glideslope signal is lost above 1,500 ft, the glideslope indicator
(GIs) and the flight director bars disappear from the ADI, and the aircraft
continues its descent at the current rate of descent. A flag indicating unreli-
able glideslope input appears only on the standby attitude indicator.
Glideslope loss below 1,500 ft (where automatic system tests are conducted)
results in both autopilots disengaging and in changes in the mode indications
("FLARE armed" is not annunciated).

Protections. While climbing to 5,000 ft with VNAV engaged, pilots

were asked what other modes they could use for the climb. With respect to
one of the possibilities-the Vertical Speed mode-they also were asked
what happens when an excessive rate of climb is used (the FMS automati-
cally reverts to the LVL CHG mode to maintain a safe airspeed).

Various options for carrying out a task. Pilots were asked to comply
with ATC clearances by using the FMS the same way as in real line operations.
Once they had decided to use a certain mode for a given task, they were asked
about other possible ways of achieving the same goal. This provided us with
information on their knowledge about options provided by the FMS as well as
about their criteria for selecting modes under different circumstances.
Table 2 summarizes the probes built into the scenario to elicit pilots'
understanding of the functional structure of the FMS.

Mode Awareness.

This section describes the probes that were built into the scenario for
testing pilots' mode awareness. They help to determine whether pilots know
what person or system is in charge of controlling the aircraft, what the
 AWARENESS OF THE FMS 11

active target values are, and whether they can anticipate the status and
behavior of the FMS.

Who is in charge? Immediately before takeoff, pilots were asked how

they would abort the takeoff if necessary at approximately 40 kts with the
autothrottles turned on. To cope adequately with the situation, pilots had to
understand what regime the autothrottles follow during takeoff. As shown in
Figure 3, the autothrottles will automatically go to N1 (a critical engine
parameter) until indicated airspeed reaches 64 kts. At and above 64 kts,
Downloaded by [University of Glasgow] at 10:05 21 December 2014

pilots can manually override the autothrottles. Thus, if takeoff is aborted

before 64 kts, the autothrottles must be manually disengaged to prevent them
from advancing again to reach N1.
Pilots' awareness of active mode settings was also probed by checking
whether they (re)activated a corresponding mode after modifying target data
in order to make the system work on reaching a new target state.
TABLE 2
Scenario Probes of Pilots' Knowledge of the Functional Structure of the FMS
Locating data in the CDU page architecture
Tracking data propagation in the CDU
Applying knowledge about mode-capture criteria
Disengaging the automatic APPROACH mode after capturing localizer and glideslope
Intercepting a radial outbound
Questions concerning VNAV Speed mode versus VNAV Path descent mode
Loss of autothrottles during a descent
Loss of glideslope signal
Predicting effects of excessive rate of climb in V/S mode
Describing the different possible ways of doing a task

..<..
...
: :. j i 60
..
...............:..:.......................... ...................................................
i j 64
[AS (krs) j? ... ... .. .. ..
... .. >
.. .:: ::.
AITmode: 1 ARM j. !. N1 i: :i i. i, THR HOLD j

.
::
AIT status .
: :n
and behavior: Pilot A/T I ] Takeoff i A/T
advances advances ; ; thrust j maintains
throttles thrust i. i. is set j thrust lever
manually, levers :: i position; but
pushes to takeoff ! ; throttles can
TOGA thrust; no ii ] now be
buttons manual jj i positioned
intervention i i i manually
::

FIGURE 3 Autothrottle status, behavior, and indications throughout the takeoff roll.
 12 SARTER AND WOODS

What are the active target values? Several probes were used to find
out about pilots' awareness of the current FMS target values. Shortly before
takeoff, they were given an amended takeoff clearance involving a tailwind
component. This required them to remember to change their N1 setting from
reduced to full takeoff thrust.
During the cockpit setup, a pointer to the pilot-calculated N1 target value
can be manually positioned on the forward engine display for reference
purposes. However, if the autothrottles are active during takeoff, as in this
scenario, they use the EMS-calculated N1 target, which is shown on the CDU
Downloaded by [University of Glasgow] at 10:05 21 December 2014

takeoff reference page. To probe pilots' awareness of the relevant N1 value,

the instructor manually positioned the N1 pointer on the engine display to a
different value than the one indicated on the CDU. Pilots were asked which
of the two values would be the target for the autothrottles during takeoff.
During an intermediate climb, the pilot-not-flying (PNF) activated the
Control Wheel Steering (CWS) pitch mode by pulling on the yoke, thus
overriding the active LVL CHG mode. The CWS pitch mode maintains the
vertical rate that corresponds to the pilot-induced yoke position. The pilot-
flying (PF) had to determine whether the aircraft would still level off at the
target altitude that had been preselected on the MCP for the LVL CHG mode.

Anticipation of system status and behavior. Whenever transitions

in aircraft behavior were imminent (e.g., level-off at a target altitude), the
participants were asked what flight mode annunciations they expected to see
on the AD1 throughout the transition.
Table 3 summarizes the probes built into the scenario to test pilots' mode
awareness.

Study Participants

The participants in this study were 20 airline pilots who responded to post-
ings or who were approached by the airline's training department. Participa-
tion was voluntary, and pilots were paid a nominal compensation for their
cooperation. The participating pilots either had a considerable amount of
line experience on the B-737-300 (n = 14), or were about to finish their
fixed-base transition training to the B-737-300 (n = 6). Table 4 describes
their biographical data and flight background.

Procedure

Pilots were asked to fly individually a 60-min scenario on a B-737-300

part-task trainer. This simulator is equipped with all relevant cockpit instru-
ments (including a fully functional FMS with the Electronic Flight Instru-
ment System, communication and navigation radios, and engine displays). It
 AWARENESS OF THE FMS 13

TABLE 3
Scenario Probes of Pilots' FMS-Mode Awareness
Aborted takeoff under 64 kts
Frequent changes in clearances involving mode transitions
Tailwind in takeoff clearance
Incorrect manual N l setting
Activation of Control Wheel Steering during climb
Ask for predictions of ADI-mode indications
Downloaded by [University of Glasgow] at 10:05 21 December 2014

TABLE 4
Age and Flight Backgrounds of Pilots
Pilots
Experienced a Transitioning
Variable M SD M SD

Age (years) 41.1 10.1 41.0 4.0

Total flying time (hr) 8,471 4,539 5,183 1,273
Glass-cockpit experience (hr) 1,011 582 0 0

is based on an actual aircraft database, and it allows for any line-oriented

operation except hand-flying the aircraft below 1,000 ft AGL. The major
differences as compared to a full-mission simulator are that it does not
generate a simulated out-the-window view or provide motion cues. How-
ever, these capabilities were not relevant to the topics investigated in this
study. The simulator does not include any Traffic Alert and Collision Avoid-
ance System equipment.
On arriving at the simulator, pilots were provided with the necessary
paperwork (e.g., charts, approach plates, weather, weight manifest) as well
as the LAX-ATIS and their clearance. The participants were asked to take
their seat in the cockpit, and to act as PF during the flight. They were given
as much time as they needed to familiarize themselves with the cockpit setup
and the intended flight. The instructor told them that weather was not a
consideration, no NOTAMs existed for the flight, and all appropriate check-
lists would be completed during the flight.
The instructor took care of the cockpit setup for the participant. He
occupied the empty seat and acted as PNF and ATC throughout the flight. An
observer was seated behind both pilots to collect behavioral and verbal data
throughout the test run and to introduce scenario events through manipula-
tion of the simulator (e.g., introduction of failures).
With respect to FMS-related tasks, each pilot was given these instructions:

1. All FMS-related work has to be done by the PF (the participant) after

activation of the autopilot at 1,000 ft AGL.
 14 SARTER AND WOODS

2. Altitude changes on the MCP will be taken care of by the PNF [the
instructor] as in actual line operations and the PF can command the
PNF to carry out specified MCP manipulations for him.
3. All tasks should be carried out by the participant the same way as in
real line operations.
4. Don't be in a hurry on the CDU or MCP! We want to keep track of
what you are doing. Speed is not important for our purposes.

At various points during the scenario, pilots were asked to perform or

describe FMS-related tasks, or were asked questions concerning their FMS-
Downloaded by [University of Glasgow] at 10:05 21 December 2014

related knowledge. After completion of the flight, additional questions were

asked concerning FMS logic and operations, and the pilots were given the
chance to ask the instructor about tasks and events that occurred during the
test run.

RESULTS

The data were first analyzed across all of the participants to identify tasks
and events that posed problems to the majority of pilots. Subsequently,
pilots' behavior and misconceptions with respect to these probes were
looked at in greater detail. A dedicated section deals with any significant
differences between the performance of pilots with glass-cockpit line expe-
rience versus those without such experience. No other differences were
apparent for any other pilot factors such as pilot's seat or age. For some tasks
that allowed pilots to choose among several different approaches, the pre-
ferred strategies for the two pilot groups are presented. Finally, problems
related to mode activation that occurred across different tasks are examined
more closely.

Problematic Tasks and Events

Fewer than 6 pilots (30%) had any difficulties carrying out the routine tasks
of changing a route (i.e., creatinglentering new waypoints/airways), inter-
cepting a radial, building or executing a holding pattern, installing an instru-
ment-landing-system approach, and entering crossing restrictions for
waypoints along the route.
On the contrary, more than 14 pilots (70%) showed deficiencies in (a)
aborting a takeoff at 40 kts with autothrottles on, (b) anticipating AD1 mode
indications throughout takeoff roll, (c) anticipating when go-around mode
becomes armed throughout landing, (d) disengaging Approach (APPR) mode
after LOC and GIs capture, (e) explaining speed management, (f) defining
end-of-descent point for VNAV path versus VNAV speed descent, and (g)
describing the consequences of G/S loss above and below 1,500 ft.
 AWARENESS OF THE FMS 15

The first three of these tasks are related to mode awareness either in the
context of dealing with an FMS-related failure or in the sense of anticipating
system status and behavior. The last four tasks point out deficiencies in
pilots' knowledge of the functional structure of the system. The results
revealed in detail the kinds of problems that can arise in pilot-automation
interaction and the misconceptions that pilots can have about the FMS.

Aborted takeoff. Immediately before receiving their takeoff clearance,

pilots were asked what procedure they would use to abort the takeoff at 40
Downloaded by [University of Glasgow] at 10:05 21 December 2014

kts. Although it was emphasized that the takeoff had to be aborted at 40

kts-before Throttle Hold (THR HOLD) is reached at 64 kts, when the pilot
can manually position the throttles-16 pilots (80%) described the proce-
dure as "Throttles back, reversers, and manual brakes." They did not men-
tion that the autothrottles would have to be disconnected to prevent the
throttles from coming back up again after manual intervention. When explic-
itly asked whether they would also disconnect the autothrottles, three partic-
ipants (15%) realized that they had missed that item. 14vo pilots (10%) were
not sure about this question and suggested that they would hold the A/Ts
back manually, "just in case."3
Only four pilots (20%) responded by immediately disconnecting the au-
tothrottles to abort the takeoff. They were asked why this action is necessary,
and all but one pilot properly described the reason. This pilot explained that
he would disconnect the autothrottles because he thought that this was
standard procedure, but he indicated that he was not aware of the conse-
quences of failing to carry out this step.

Anticipation of AD/ indications during takeoff. Pilots were asked for

their expectations concerning AD1 mode indications throughout the takeoff
roll, as these indications are supposed to help monitor whether the system is
working properly and as expected.
The relevant indications that appear in the lower left corner of the AD1 are
N1-that the autothrottles are in charge and will go to takeoff thrust--and
THR HOLD-that the aircraft has reached 64 kn and the autothrottles will
go to takeoff thrust but that they can now be overridden manually by the
pilot. Five of the pilots (25%) expected to see both these indications. Twelve
subjects (60%) only mentioned either THR HOLD (15% of the pilots) or N1
(45% of the pilots) as an indication during takeoff. Another 3 pilots (15%)
could not predict any of the mode indications.

3 ~ the
n debriefing, these pilots argued that they could still hold the throttles back manually to
prevent them from advancing without disengaging them. But it is not clear that they would do
so in the actual situation because, without understanding FMS behavior, it seems unlikely that
they would anticipate the need for manual intervention.
 16 SAKTER AND WOODS

Availability of GA mode. The GA mode becomes available when de-

scending below 2,000 ft radio altitude with the autothrottles armed. Out of
20 pilots, only 5 recalled the altitude at which this occurs. Eight pilots
(40%) knew that the availability of the mode depends upon reaching a
certain altitude, but they did not remember the actual height. Another 4
pilots (20%) replied that they had no idea when the mode becomes avail-
able, and the remaining 3 pilots (15%) assumed that the GA mode is
available upon glideslope capture.
Downloaded by [University of Glasgow] at 10:05 21 December 2014

Disengaging the APPR mode after LOC and G/S capture. When
asked to disengage the APPR mode after localizer and glideslope had been
captured, only 3 pilots (15%)could recall the three ways of accomplishing
this (pushing the TakeofflGo-Around buttons on the throttles, turning both
flight displays and the autopilot off, or retuning the VHF radio). Seven pilots
(35%) did not know of any procedure for disengaging the APPR mode. Three
pilots (15%) were familiar with two of the three different options.
The solutions suggested by the remaining 7 pilots (35%) included at least
one possible approach, but also at least one approach that would not result in
the disengagement of the APPR mode:

1. Six pilots (30%) thought that they could disengage the APPR mode by
pushing the APPR key again.
2. Five pilots (25%) expected that engaging another pitch mode, such as
VIS or ALT HOLD, would get them out of the APPR mode.
3. Five pilots (25%)thought that they would have to disengage either the
A/P or the FDs, but not both.
4. Four pilots (20%) assumed that choosing another roll mode would
solve the problem (e.g., Heading Select or VORLOC).

One interesting characteristic of these different unsuccessful attempts to

disengage the APPR mode is that they seem to be intuitive approaches that
are not supported by the system design.

Speed management and end-of-descent point in VNAV Path versus

VNAVSpeed mode. Knowledge of the control modes (pitch and
power) used to maintain a target airspeed during a descent is important
for pilots to be able to monitor and anticipate aircraft behavior. It allows
them to recognize unexpected activities or the lack of timely aircraft
response. Nine out of 20 pilots knew how the FMS maintains target speed
during a VNAV Path descent. Eight pilots (40%) were aware of the speed
control mode during a VNAV Speed descent. With respect to the end-of-
descent point of a path descent versus a speed descent, the results were
similar: Twelve pilots (60%) were aware of the end of descent during a
 AWARENESS OF THE FMS 17

VNAV Path descent, and 9 pilots (45%) knew at what point the VNAV
Speed descent would end.

Consequences of G/S failure above and below 1,500 ff. After

G/S capture, a G/S signal loss was simulated at approximately 3,000 ft
(before automatic system tests are carried out at 1,500 ft that can detect the
absence of a valid G/S signal and automatically disconnect the autopilot). On
realizing the problem, pilots were asked about the consequences of this
event, and 54% of the pilots provided the correct answer. When asked
Downloaded by [University of Glasgow] at 10:05 21 December 2014

whether a G/S failure at a lower altitude ( ~ 1 , 5 0 0ft) would have different

effects, only 15% of the pilots knew the answer. Twenty-three percent of the
participants did not know the answer to either question.
Although detection time was not measured for this failure, it took some
pilots a rather long time (several minutes in some cases) even to realize the
problem even though they were looking directly at the AD1 (with the G/S
indications and FD bars disappearing) during this phase of flight.

Differences Between Line-Experienced and

Transitioning Pilots

Major differences in performance between line-experienced and transition-

ing pilots were seen only with respect to three of the tasks within the
scenario. First, when asked to intercept the LAX 248" radial, all 6 of the
transitioning pilots had difficulties carrying out the task using LNAV, as
compared to only 7 of the 1 4 experienced pilots. None of the inexperienced
pilots realized the need for building a fictitious waypoint on the radial.
Second, when asked about the consequences of using an excessive vertical
rate of climb in the V/S mode, none of the transitioning pilots could provide
the correct answer, as compared to only 5 (36%) of the experienced partic-
ipants. Last, 5 of the 6 pilots without line experience could not describe
how to program an intermediate descent on the VNAV Cruise page for
avoiding traffic, whereas none of the 14 experienced pilots had any problem
with this task.

Preferred Strategies of FMS Usage

In addition to probes that allowed for only one correct answer or reac-
tion, some situations were built into the scenario that required pilots to
choose among different options to carry out the task. We asked pilots to use
the automation as they would in real line operations. This provided us with
behavioral data on their primary choice of modes for a given task under
specified circumstances. Subsequently, we asked them about other possible
strategies for achieving the same objective.
 18 SARTER AND WOODS

Intercepting a radial outbound without a waypoint at a low altitude.

There are two possible methods for accomplishing this task. First, pilots
can use the VORILocalizer mode (VORLOC), which involves MCP ma-
nipulations, or they can use LNAV, which requires working with the
CDU. As Figure 4 illustrates, most of the pilots with glass-cockpit expe-
rience preferred to use VORLOC (93%), whereas the pilots in transition
to glass cockpits preferred to use LNAV (83%).4 Although it is possible
to use LNAV for this task after one creates a fictitious fix outbound, the
MCP VORLOC mode is the faster and easier method at low altitudes; it
requires less pilot input and no heads-down time as compared to creating
Downloaded by [University of Glasgow] at 10:05 21 December 2014

a fix using the CDU.

Speed-restricted climb to 5,000 ft. Again, there are two options

available to pilots: using the LVL CHG mode via MCP manipulations or
modifying data on the CDU climb page and activating the VNAV mode. In
this case, all of the pilots in transition and 79% of the experienced glass-
cockpit pilots preferred the LVL CHG mode of the MCP (see Figure 5).
Again, using the MCP minimizes heads-down time, which is important as the
aircraft is still at a very low altitude during this task.

Unplanned descent for traffic at FL 290. In this situation, the pi-

lots could either choose the LVL CHG mode on the MCP or they could
program the descent on the CRZ page of the CDU and then activate
VNAV. As Figure 6 shows, the majority of line-experienced pilots chose
to descend using VNAV (79%), whereas most (83%) of the less experi-
enced pilots preferred to use the LVL CHG mode. When asked why they
preferred VNAV, the experienced pilots explained that, because they were
at FL 290, they felt they had enough time to program the CDU. They also
said that they would prefer to modify the VNAV data right away rather
than switch between VNAV and another descent mode at a lower level of
automation, which makes it more difficult for them to keep track of active
modes and targets.

Problems of Mode Activation

Another interesting result refers to failures to engage or reengage a mode

after entering new target values into the MCP or the CDU. This omission
occurred at least once during the scenario for 5 of the 6 transitioning pilots
(total number of omissions = 9). Only 2 of the 14 experienced pilots forgot
to engage an appropriate mode, and this occurred only once for each of

e the pilots in transition (16%) could not think of any second method at all.
4 ~ o m of
 AWARENESS OF THE FMS 19

First Choice
%
100-
90 -

80 -

70 -

60 -

50 -
Downloaded by [University of Glasgow] at 10:05 21 December 2014

40 -

30 - I Transitioning Pilots
(n = 6)
20 -
10-
0 E r F i d Pilots

0-

LNAV I VORLOC
(CDU) (MCP)
.... ....................................
:.: .... . ... . . . .... ...... ... . . ... . ..........

FIGURE 4 Preferred mode and level of automation for intercepting a radial outbound
for experienced versus transitioning pilots.

First Choice
'70

100-

90 -

80 -
70 -
60 -

50 -

40 -

30 - I Transitioning Pilots
(n = 6)
20 -
10 -
0 t y F i d Pilots

0-
VNAV LVL CHG
(CDU) (MCP)

FIGURE 5 Preferred mode and level of automation for a speed-restricted climb at low
altitude for experienced and transitioning pilots.
 20 SARTER AND WOODS

First Choice
%
. . . . . . . . . . . . .... . . ..
100,

I
Downloaded by [University of Glasgow] at 10:05 21 December 2014

Transitioning Pilots
(n = 6)
0 g r : i d Pilots

VNAV LVL CHG

(CDU) (MCP)

FIGURE 6 Preferred mode and level of automation for an unplanned descent for traffic
at high altitude for experienced and transitioning pilots.

them. The problem occurred 4 times with LNAV mode, 6 times with VNAV
mode, and 1 time with LVL CHG mode.
In 7 of the failures to engage a mode, all required entries into the CDU
or the MCP were made, but no mode was activated. In the remaining 4
instances, the pilot would first use an MCP mode (e.g., HDG SEL) to get the
system started towards the target, and then he would enter the new target
data into the CDU, but ultimately he would forget to switch from the MCP
mode to VNAV or LNAV, which use the entered CDU values as targets. The
fact that pilots forgot to engage VNAV or LNAV (rather than an MCP mode)
after entering new target data in the majority of cases may be related to the
spatial separation between the data-entry unit (CDU) and the VNAV and
LNAV buttons on the MCP. It can also be interpreted as an indication of a
flawed mental model of the FMS, in which the close relation between the
CDU (target-entry unit) and the MCP (mode-activation unit) is not well
represented.
Another problem related to mode engagement was the attempt to acti-
vate a mode without the prerequisites for this activation being met. Fifty
percent of the transitioning pilots and 1 of the 14 experienced pilots tried
to engage VORLOC without being in the manual radio mode as required.
Fifty percent of the transitioning pilots and 5 of the 14 experienced pilots
engaged the APPROACH mode without lowering the MCP altitude first,
and they were surprised to find that the aircraft did not start the descent.
 AWARENESS OF THE FMS 21

DISCUSSION

This study supports and expands on the results obtained from the previous
corpus gathering studies of pilot-automation interaction (Sarter & Woods,
1992b). It confirms that most of the difficulties in pilot-automation interac-
tion are related to a lack of mode awareness and to gaps in pilots' mental
models of the functional structure of the automation. These problems seem
to occur primarily in the context of nonnormal time-critical situations such
as an aborted takeoff. Problems related to such situations may be underre-
Downloaded by [University of Glasgow] at 10:05 21 December 2014

ported in surveys because they rarely occur in line operations. In this study,
however, every participant was forced to cope with nonnormal events in the
scenario. In this way, latent problems in pilot-FMS interaction could be
revealed.
For the majority of pilots, it was difficult or impossible to manage the
cockpit automation in three nonnormal time-critical situations in the sce-
nario: (a) an aborted takeoff, (b) the need to disengage an automatic ap-
proach mode for collision avoidance, and (c) loss of the G/S during final
descent. In the case of the aborted takeoff, 65% of all participants did not
understand how the autothrottle controls the aircraft throughout the takeoff.
Fifteen percent of the pilots knew about the ongoing mode activities and
transitions, but they were not capable of applying this knowledge to the
situation at hand. In terms of behavior, this resulted in only four pilots
responding correctly, and one of them did not seem to understand completely
the basis for this action.With respect to the request to disengage the AP-
PROACH mode after localizer and glideslope capture, most of the pilots
knew at least one way of complying with this request. However, 1 4 pilots
also suggested at least one ineffective approach. If, in a real world case, ATC
told the pilot to change heading and/or altitude immediately to avoid a
collision, there would be no time for failed attempts to disengage the mode;
the pilot would have to respond immediately. This problem is related to the
need for an interface design that can indicate available options to help the
pilot intervene quickly and directly when necessary. In the case of the loss of
the GIs during final descent, we observed that it took many pilots fairly long
even to realize that an anomaly had occurred. Although they were looking
directly at the AD1 display at this stage of the simulated flight, it took some
pilots several minutes to realize that the GIs indication and the FD bars had
disappeared, both of which are the only AD1 indications of the problem until
the aircraft descends through 1,500 ft (when automatic system tests are
conducted that can detect an invalid G/S signal, disconnect the autopilot, and
issue alerts to the pilot). This problem illustrates that cueing by absence may
not be a good technique for indicating the presence of an anomaly. Not only
was anomaly detection relatively slow, but about half of the participants
were not aware of the consequences of a loss of the GIs.
The scenario contained a variety of other probes of the pilots' ability to be
 22 SARTER AND WOODS

"ahead of the FMS"-that is, to show the ability to anticipate future system
behavior that can change not only in response to current pilot input, but also
as a result of changes in the environment, previous pilot input, or for
protection purposes (Reason, 1990). For example, only 1 out of 20 partici-
pants could predict the entire sequence of expected mode indications for the
takeoff roll. Similarly, only 5 of the participants knew when to expect the
indication that the go-around mode was available.
The underlying reason for the observed problems seems to be a lack of
mode awareness. In the context of simpler devices and environments, mode
awareness usually refers to the adequate assessment of the currently active
Downloaded by [University of Glasgow] at 10:05 21 December 2014

mode status. But our results show that in the context of the highly dynamic
and complex cockpit environment, other aspects of mode awareness are
more critical. In these systems, the pilots' role has changed from active
manipulator of the aircraft to supervisor of the automated systems. To fulfill
this role, pilots need to have a thorough understanding of what a mode means
in terms of system behavior and have to be "ahead of the FMS."

Operational Costs of Technology-Centered Automation

New automation is developed because of some payback (e.g., precision,

more data, reduced staffing, etc.) or for some beneficiary (e.g, the indi-
vidual practitioner, the organization, the industry, or society). But often
overlooked is the fact that new automated devices also create new de-
mands for the individual and groups of practitioners responsible for oper-
ating and managing these systems. New demands can include new or
changed tasks (e-g., setup, operating sequences) as well as new cognitive
demands. There are new knowledge requirements (e.g., how the automa-
tion functions), communication tasks (e.g., instructing the automation in
a particular case), data-management tasks (e.g., finding the relevant page
within the CDU page architecture), attention demands (e.g., tracking the
state of the automation), and forms of error or failure (e.g., mode error).
This study reveals some of the kinds of costs that can occur in the context
of the current generation of cockpit automation-costs that can be mini-
mized or eliminated through skillful design of human-centered automa-
tion (Billings, 1991).

Mode error and mode awareness. Two of the cost centers associ-
ated with changes in automation are the possibility of new forms of error or
failure and the possibility of creating new cognitive demands for practition-
ers. Interlinked examples of these effects of automation for the glass cockpit
case are mode error and mode awareness.
Devices that allow something to be done one way in one mode and
another way in another mode create the possibility of mode errors, in which
 one executes an intention in a way appropriate to one mode when the device
is actually in another mode (Norman, 1988). Automated systems like those
in the glass cockpit cannot be characterized by a single mode setting. There
are a number of subsystems, each involving a number of possible mode
settings. This increase in the power and flexibility of automated resources
creates a form of operational complexity that increases the potential for
mode errors.
But advanced automation like the FMS extends the kinds of mode-related
problems that can occur because system status and behavior can change
Downloaded by [University of Glasgow] at 10:05 21 December 2014

independent of immediate and direct pilot commands due to situation factors

or protection limits (Sarter & Woods, 1992a). This means that a new cogni-
tive demand is created: the need to maintain awareness of externally induced
mode transitions. As the pilot's role has changed from active manipulator of
the aircraft to supervisor of automated systems, effective situation awareness
requires pilots to stay ahead of the FMS-that is, to be able to anticipate
future system behavior or to detect system failures (Sarter & Woods, 1991).
However, in this study, only 5 out of 20 participants could predict the
operationally most significant mode indications (N1 and THR HOLD) for
the takeoff roll, and only 5 of the participants knew when to expect the
indication that the go-around mode is available.
One way to interpret the results of this study and the complementary
results of Sarter and Woods (1992b) is that many of the observed problems
resulted from a lack of mode awareness-the pilots lost track of system
targets and missed mode changes that occurred independently of immediate
pilot commands. Maintaining mode awareness requires pilots to attend to
and integrate data from a variety of indications in the cockpit such as the
flight mode annunciations on the ADI, the visualization of the programmed
route of flight on the HSI, and the display of target values on the MCP.
Breakdowns in mode awareness may be due to characteristics of these
indications, given the nature of the cognitive demands of high-tempo phases
of flight or nonnormal flight situations. Another contributor to these atten-
tional breakdowns may be limits and gaps in the pilots mental models of the
automated resources.

New knowledge requirements. Transition to glass-cockpit aircraft

requires pilots to learn a great deal about the FMS and other flight-deck
automated subsystems. As the results of this study show, and given the
results of the previous corpus building studies, there are a number of areas
where pilots have gaps in their understanding of the functional structure of
the FMS. By forcing pilots to deal with various nonnormal situations, gaps
or errors in their understanding of how the automation works in various
situations were revealed. Again, the results indicated that pilots do not have
an accurate model of how VNAV descent modes work and that the displays
do not help them in tracking either the targets or the control modes used by
 24 SARTER AND WOODS

VNAV Path and VNAV Speed descents. Overall, this study confirms
previous results (Sarter & Woods, 1992b) and shows that these problems
can occur even with pilots who have relatively extensive glass-cockpit
experience.
Note the interaction between two factors. First, breakdowns in mode
awareness can be due in part to a lack of effective feedback on the state of
the automation and in part due to buggy mental models of the automation.
Second, the lack of feedback on the state of the automation can limit pilots'
ability to learn from experience, to correct or elaborate their mental models
Downloaded by [University of Glasgow] at 10:05 21 December 2014

of system function over time, and to learn to perceive the state of the
automation from the available indications. A third factor further complicates
the difficulty: Many of the flight situations that stress these problems occur
relatively rarely in line operations.
This combination has broad repercussions for training pilots to manage
highly automated aircraft. First, training must go beyond simply providing
pilots with facts about the FMS. The results show that sometimes pilots
possessed knowledge in the sense of being able to recite facts, but that they
were unable to apply the knowledge successfully in an actual flight context.
This is called the problem of inert knowledge. Training must conditionalize
knowledge to the contexts where it is utilized. Second, pilots need to learn
not simply how the automated system works, but also how to work the
system. This requires scenarios and instruction designed around managing
the transitions between different modes of automation. Third, because pilots
do Iearn a subset of methods to be able to make the system work under
routine conditions, situations that challenge their current understanding may
arise relatively infrequently (or go unnoticed as such due in part to lack of
feedback about the state and behavior of the FMS). This means that ongoing
learning programs will need to be devised that help even experienced glass-
cockpit pilots discover and correct subtle bugs in their mental models of the
FMS or to elaborate their understanding of how the automation works in
particular situations in a risk-free environment.

Knowledge miscalibration. The results indicate that pilots have gaps

in their understanding of the functional structure of the FMS. Further, there
are some indications in the data that pilots are miscalibrated with respect to
their understanding of the FMS-that is, pilots may not be aware of the gaps
in their mental models. An expert is well-calibrated if he or she is aware of
the areas and circumstances in which they have correct knowledge and in
which their knowledge is incomplete or limited. If the expert is overconfi-
dent and believes that he or she understands areas in which their knowledge
is in fact incomplete or limited, then that person is said to be miscalibrated
(Wagenaar and Keren, 1986). Note that degree of calibration is not necessar-
ily correlated with expertise.
When we compare pilot responses to questions like "How much do you
 AWARENESS OF THE FMS 25

agree or disagree with the statement: 'There are still modes and features of
the FMS that I don't understand"' (Sarter &Woods, 1992b; Wiener, 1989) to
the behavioral data in this study, there is some indication that glass-cockpit
pilots are overconfident and miscalibrated about how well they understand
the FMS. When forced to cope with flight situations that challenge their
ability to monitor and manage cockpit automation, the number and severity
of pilots7 problems was higher than would be expected from previous survey
data, in particular for pilots with line experience in glass cockpits. Some of
the participants in this study made comments in the post-scenario debrief-
Downloaded by [University of Glasgow] at 10:05 21 December 2014

ings such as: "I never knew that I did not know this. I just never thought
about this situation." Similar results have been obtained in studies of physi-
cian interaction with computer-based automated devices in the surgical op-
erating room (Cook, Potter, Woods, & McDonald, 1991; Moll van Charante,
Cook, Woods, Yue, & Howie, 1992)
There are several factors that could have contributed to the observed
miscalibration. First, areas of incomplete or buggy knowledge could have
remained hidden from pilots because they have the capability to work around
these areas by sticking with a few well-practiced and well-understood meth-
ods. In addition, flight situations that force pilots into areas where their
knowledge is limited and miscalibrated may arise infrequently. Second,
studies of calibration have indicated that the availability of feedback, the
form of feedback and the attentional demands of processing feedback can
effect knowledge calibration (Wagenaar & Keren, 1986). Problems with
ineffective feedback on the state and behavior of the FMS observed in this
study and reported on in previous studies of pilot interaction with cockpit
automation (e.g., Norman, 1990) could have been a factor that contributed to
poor calibration of pilots-that is, a lack of awareness of the gaps in their
mental models of the FMS. The relation between poor feedback and
miscalibrated practitioners was also found in studies of physician-automa-
tion interaction (e.g., Cook et al., 1991). Knowledge miscalibration in pilots,
if it is widespread, is one factor that could lead to underreporting of prob-
lems with cockpit automation in survey studies.

How to manage automated resources. Cockpit automation pro-

vides a large number of functions and options for carrying out a given flight
task under different circumstances. For example, the FMS provides at least
five different mechanisms at different levels of automation for changing
altitude. This flexibility is normally construed as a benefit that allows the
pilot to select the mode or option best suited to a particular flight situation
(e.g., time and speed constraints). However, this flexibility creates new
demands as well. Pilots must learn and know about the functions of the
different modes, how to coordinate which mode to use when, and how to
switch smoothly from one mode to another. In other words, the pilots must
know how the automated system works and must develop skill at how to
 26 SARTER AND WOODS

work the system. To meet the latter criterion, pilots must: (a) learn about all
of the available options; (b) learn and remember how to deploy them across
a variety of operational circumstances, especially rarely occurring but more
difficult or critical ones; (c) learn and remember the interface manipulations
required to invoke the different modes or features; and (d) learn and remem-
ber how to interpret or where to find the various indications about which
option is active or armed and what its associated target values are.
The results of this study indicate that pilots become proficient and
maintain their proficiency on only a subset of the modes and options
Downloaded by [University of Glasgow] at 10:05 21 December 2014

provided by the FMS. Further evidence for this phenomenon was pro-
vided by previous FMS-related studies (Sarter & Woods, 1992b) and by
studies of human-machine interaction in other domains (e.g., Cook,
Woods, & Howie, 1990; Rosson, 1983), where users hardly ever use more
than a small subset of the options provided. This is, in part, a consequence
of the increased costs involved in learning extra functions, but it also
allows practitioners to protect themselves from having to make difficult
decisions due to an increased number of alternatives. In the case of the
FMS, pilots try to manage the system within a set of stereotypical re-
sponses or techniques. In this study, we were able to compare the tactics
selected by pilots with line experience in glass cockpits with those se-
lected by pilots without previous glass-cockpit experience. The results
indicate that, over time, pilots learn to select among the various options
depending on situation factors (e.g., altitude, time constraints) and on
expectations (e.g., the likelihood of deviation from plan). But pilots who
had just finished their transition training were much less sensitive to
these contextual factors. They tended always to use the highest level of
automation independent of context.
Note that, in higher tempo phases of flight, more experienced pilots in our
study chose to use intermediate levels of automation that use the MCP as the
interface rather than higher levels of automation that require CDU interaction.
. The MCP-based modes generally require less interaction, less head-down time,
and less diversion of attention to the interface itself (e.g., remembering the
necessary interface manipulations). In addition, the modes of automation ac-
cessed through the MCP as an interface tend to respond only to direct pilot input
(e.g., the pilot enters a target value and activates a mode of control, and the
automation then responds by capturing and maintaining that target value until
another pilot command is received) and do not initiate a sequence of automated
system activities. This may explain previous results in which pilots saw the
MCP and the CDU as separate systems (Sarter & Woods, 1992b) despite the fact
that, from an engineering point of view, both are part of an integrated FMS.
Operationally, interacting with the MCP modes has a different character than
programming the CDU. This means that general questions about pilots' attitudes
towards cockpit automation in general are ambiguous, and that pilots may differ
from each other and from the investigator in their interpretation of what aspects
of cockpit automation the question refers to.
 SUMMARY

The results of this and previous studies of pilot interaction with cockpit
automation in commercial aviation yield consistent results across diverse
methods. Although pilots seem to be able to make the system work in
standard situations, one of the most important results of this study is the
discovery of latent problems with pilot-FMS interaction that can affect
even experienced pilots' performance in nonnormal time-critical situations.
The severity and importance of these problems is underestimated due to
Downloaded by [University of Glasgow] at 10:05 21 December 2014

several interacting factors:

1. There are gaps in pilots' understanding of the functional structure of

the automation.
2. The opaque interface between pilots and automation makes it difficult
for pilots to track the state and activity of the automation.
3. Pilots may not be aware of the gaps in their knowledge about FMS
function.
4. Pilots can escape from the CDU to the MCP whenever a situation gets
too complicated or time pressure is too high.
5. The flight situations in which these problems produce unmistakable
erformance difficulties may occur infrequently in line observations.

The data in this study, in conjunction with the data from previous studies
(e.g., Norman, 1990; Sarter & Woods, 1992b; Wiener, 1989), point out some
of the costs of the "clumsy" use of technological possibilities from an
operational point of view. These costs should provide input to designers
trying to develop human-centered automation and to trainers trying to de-
velop new instructional programs for developing, maintaining and testing
pilot proficiency in managing automated resources. However, it is important
to remember that the problems in pilot interaction with cockpit automation
are not inherent in the technology itself; rather these problems result from
limitations in how the automation and the human pilots are integrated to-
gether as a joint, distributed cognitive system through both training and
design (Hutchins, 1991; Woods, 1993b).

ACKNOWLEDGMENTS

This work was supported under Cooperative Agreement NCC 2-592 with the
Aerospace Human Factors Research Division of the NASA-Ames Research
Center. Dr. Everett Palmer was the technical monitor.
The authors thank all of the pilots who participated in the study and
shared their experience with us. We are also very grateful for the support and
patience of a large number of people at the collaborating airline who made it
possible to carry out this line of research.
 28 SARTER AND WOODS

REFERENCES

Billings, C. E. (1991). Human-centered aircraft automation philosophy (Tech. Memo. No.

103885). Moffett Field, CA: NASA Ames Research Center.
Cook, R. I., Woods, D. D., & Howie, M. B. (1990). The natural history of introducing new
information technology into a dynamic high-risk environment. In Proceedings of the 34th
annual meeting of the Human Factors Society (pp. 429-433). Orlando, FL: Human Factors
Society.
Cook, R. I., Potter, S. S., Woods, D. D., & McDonald, J. M. (1991). Evaluating the human
engineering of microprocessor-controlled operating room devices. Journal of Clinical Moni-
toring, 7, 217-226.
Downloaded by [University of Glasgow] at 10:05 21 December 2014

Eldredge, D., Dodd, R. S., & Mangold, S. J. (1991).A review and dkcussion ofFlight Management
System incidents reported to the Aviation Safety Reporting System. (Battelle Report, prepared for
the Department of Transportation). Columbus, OH: Volpe National Transportation Systems Center.
Hutchins, E. (1991). How a cockpit remembers its speed Unpublished technical report, San
Diego: University of California, Distributed Cognition Laboratory.
James, M., McClumpha, A, Green, R., W~lson,P., & Belyavin, A. (1991). Pilot attitudes to flight
deck automation. Paper presented at the meeting of the Royal Aeronautical Society, London.
Moll van Charante, E., Cook, R. I., Woods, D. D., Yue, L., & Howie, M. B. (1992). Human-
computer interaction in context: Physician interaction with automated intravenous control-
lers in the heart room. In H. G. Stasser (Ed.)., Analysis, design and evaluation of
man-machine systems 1992. New York: Pergamon.
Norman, D. A. (1988). Thepsychology of everyday things. ~ e York: w Basic.
Norman, D. A. (1990). The 'problem'with automation: Inappropriate feedback and interaction,
not 'over-automation' @p. 585-593). Philosophical Transactions of the Royal Society of
London, B 327.
Reason, J. T. (1990). Human error. Cambridge, England: Cambridge University Press.
Rosson, M. B. (1983). Patterns of experience in text editing. Proceedings of CHI83: Human
Factors in Computing Systems (pp. 171-175). New York: ACM.
Roth, E. M., Bennett, K. B., & Woods, D. D. (1987). Human interaction with an "intelligent"
machine. International Journal of Man-Machine Studies, 27, 479-525.
Sarter, N. B. & Woods, D. D. (1991). Situation awareness: A critical but ill-defined phenome-
non. The International Journal ofAviation Psychology, 1, 45-57.
Sarter, N. B. & Woods, D. D. (1992a). Mode error in supervisory control of automated systems.
In Proceedings of the 36th annual meeting of the Human Factors Society.
Sarter, N. B. & Woods, D. D. (1992b). Pilot interaction with cockpit automation: Opera-
tional experiences with the flight management system. The International Journal of
Aviation Psychology, 2, 303-321.
Wagenaar, W. A. & Keren, G. B. (1986). Does the expert know? The reliability of predictions and
confidence ratings of experts. In E. Hollnagel, G. Mancini, & D. D. Woods (Eds.) Intelligent
decision support in process environments @p. 87-103). New York: Springer-Verlag.
Wiener, E. L. (1989). Human factors of advanced technology ("glass cockpit") transport aircrafi
(NASA Contractor Rep. No. 177528). Moffett Field, CA. NASA-Ames Research Center.
Woods, D. D. (1993a) Process tracing methods for the study of cognition outside of the
experimental psychology laboratory. In G. A. Klein, J. Orasanu & R. Calderwood (Eds.),
Decision making in action: models and methods @p. 228-251). Nowood, NJ: Ablex.
Woods, D. D. (1993b) The price of flexibility in intelligent interfaces. Knowledge-Based
Systems, 6, 1-8.
Woods, D. D., & Sarter, N. B. (in press) Evaluating the impact of new technology on
human-machine cooperation. In J. Wise, V. D. Hopkin, & P. Stager (Eds.) Verification and
validation of complex and integrated human-machine systems. New York: Springer-Verlag.