In Proceedings of the 43rd Annual Meeting of the Human Factors and Ergonomics Society (1999).

Houston, TX: Human Factors Society.

FORMAL ASPECTS OF PROCEDURES:

THE PROBLEM OF SEQUENTIAL CORRECTNESS

Asaf Degani Michael Heymann

San Jose State University, CA; and Technion, Haifa, Israel; and
NASA Ames Research Center NASA Ames Research Center
adegani@mail.arc.nasa.gov heymann@cs.technion.ac.il

Michael Shafto
NASA Ames Research Center, Moffett Field, CA.
Mshafto@mail.arc.nasa.gov

A formal, model-based approach is proposed for the development and evaluation of

the sequences of actions specified in procedures. The approach employs
methodologies developed within the discipline of discrete-event and hybrid systems
control. We demonstrate the proposed approach through an evaluation of a
procedure for handling an irregular engine-start on board a modern commercial
aircraft.

In complex human-machine systems, successful airline’s abnormal procedure for coping with asymmetrical-
operations depend on an elaborate set of procedures provided flap-extension (which can have a significant effect on lateral
to the human operator. These procedures specify a detailed control of the aircraft) had to be rewritten when it was found to
step-by-step process for configuring the machine during be inaccurate. The problem? The power supply for activating
normal, abnormal, and emergency situations. The adequacy of the flaps following asymmetrical flap extension, was different
these procedures is vitally important for the safe and efficient from the standard configuration for this model aircraft. The
operation of any complex system. In high-risk endeavors such airline that originally specified the non-standard power supply
as aircraft operations, maritime, space flight, nuclear power configuration failed to modify the procedure accordingly.
production, and military operations, it is essential that these (The inaccurate procedure was in effect for some five years
procedures be flawless, as the price of error may be before it was detected).
unacceptable. When operating procedures are inadequate for Based on our survey of several U.S. airlines, we have
the task, not only will the system’s overall efficiency be noted that the process of designing a procedure is
thwarted, but there may also be tragic human and material accomplished informally. That is, a Flight Manager and/or
consequences (Degani and Wiener, 1993). several experienced pilots discuss and then (re)-design the
In commercial aviation, for example, crew interaction procedure based on their knowledge, experience, and intuition.
with the aircraft is specified through a set of Standard Once the procedure is reviewed by the regulating agency’s
Operating Procedures (SOPs) (Federal Aviation (e.g., FAA’s) inspector, the procedure is approved, accepted,
Administration, 1995). In the event of a normal task (e.g., and provided to all flight crews. Other industries that we
configuration of the aircraft before takeoff), an abnormal surveyed, such as nuclear power, maritime, and space, use
condition (e.g., high engine temperature on start-up), or an similar procedural design processes.
emergency situation (e.g., engine fire), procedures are set in We believe that current procedural design processes
place to support the crew in managing the situation. should be augmented with an in-depth evaluation of the
Procedures assist the crew along a path of pre-defined procedure in terms of its [1] sequential correctness, [2] ability
sequences of actions; the objective is to quickly “drive” the to deal with out-of-norm configurations, [3] compatibility with
system to some safe, yet still efficient, configuration. It must the user interface, [4] vulnerability to human error, [5]
be recognized, however, that an unpredictable constellation of capability of meeting the demands from the operational
circumstances including machine (e.g., component failure), environment, and [6] consistency with other procedures and
human (e.g., making a mistake), and environmental factors policies. In this paper we suggest an approach for describing
(e.g., low ambient temperature) can interfere with operations and analyzing procedures in terms of sequential correctness.
and lead to a sub-optimal configuration (see Mosier, Palmer,
and Degani, 1992, for one example). APPROACH AND LANGUAGE
From the organization’s point of view, a procedure Procedures constitute sequential execution trees (i.e.,
represents a collective agreement on the “best” way to achieve conditional instruction sequences) of user interaction with the
both safe and efficient operations (Wieringa, Moore, and machine. Their aim is to guide the user in operating the
Barnes, 1992). Nevertheless, there are many documented machine correctly and reliably, so as to achieve well-defined
cases in which the procedures provided to the crews are not task goals and specifications. It is quite clear that in order to
the “best” (Degani and Wiener, 1997). For example, one U.S. formulate a correct and efficient operational procedure, the
procedure designer must have a clear and unambiguous so-called AND state, separated by a dotted line (Figure 1b).
understanding of the machine’s behavior under all (relevant) The resulting super-state S is an abstraction of the two
operating conditions. concurrent processes X and E. Process X is made up of two
The approach proposed in the present paper is aimed at sub-states Y and Z, and process E is identical to the process in
enhancing current practice by augmenting it with a formal Figure 1a. The question as to which sub-state is initially
mathematical methodology that provides a systematic method occupied when entering super-state S is resolved by the small
for procedure “synthesis.” Two elements must be in place to default arrows (), which point to states Y and A.
perform such synthesis: [1] a formal model of the machine’s
behavior and [2] a formal representation of the procedure's S
task goals. Such a model can be based on any one of several X E
existing or emerging modeling formalisms for (untimed) D
discrete-event systems or (timed) hybrid-systems (Ramadge h
and Wonham, 1987; Heymann, Lin and Meyer, 1997). g
Our objective is to develop formal approaches for Y
designing and evaluating procedures (see Degani and FALSE
f [P] C1 A
Heymann, 1999, for a similar approach for evaluating u
interfaces). The focus of this paper is on the sequential v/g
TRUE
correctness problem. From a theoretical standpoint, we strive
for an approach that describes the human-machine- Z e
environment system and its many embedded interactions in a B
clear (e.g., mathematical) language that allows for a detailed
description, synthesis, and analysis. From a practical
standpoint, we seek an approach that provides a reliable design Figure 1b. Concurrency and broadcast.
process, e.g., such that fixing one procedural deficiency will
not generate another deficiency somewhere else—a well- The real subtlety with which Statecharts models
known and common problem in procedure development. concurrency is in its treatment of output events, or actions.
Language Here the machine can generate actions to change its own
configuration. Consider process X in Figure 1b: When event v
The foundation of our approach is a formal description of occurs and the transition labeled v/g is taken, the action g (an
the human-machine system in terms of its behavior. We use output event, denoted with a hat) is immediately activated.
the Finite-State-Machine theory to model system behavior. This event is broadcast to the entire network, and perhaps
The following is a brief description of two graphical causes further transitions in other processes. And indeed, in
representations of this theory: the State Transition Diagrams process E, action g will cause a transition out of state D (into
and the more modern Statecharts formalism (Harel, 1987). A or B depending on how condition C1 is evaluated).
In Figure 1a we have three states A, B, and D (depicted as The ability to arrange processes in a concurrent manner
rounded squares) and several transitions (depicted as arcs). and to broadcast information among processes sums up two
important features of the Statecharts language. These features
of Statecharts allows us to describe the behavior of a system in
D a clear and concise way. Below, we will use the Statecharts
h language to describe one human-machine system.
g
EVALUATION
FALSE A To illustrate our approach we evaluate an abnormal
f [P] C1
procedure used in commercial aviation. In evaluating this
TRUE human-machine-environment system, we [1] describe the
machine and procedure, [2] model the system, [3] define the
e
task goals and specifications, and [4] analyze the necessary
B sequence of events to execute this procedure.
Figure 1a. State-transition-diagram. Machine and Procedure
Normal engine start in the Boeing B-757 aircraft follows
The symbols e, f, g, and h stand for events that trigger this sequence of actions: engagement of the engine starter,
transitions among the machine’s states. The bracketed [P] is a opening of the fuel control switch once the engine is at the
condition, such that the transition from state B to D takes place appropriate speed, and automatic cut-out of the engine starter
when event f occurs and condition P is TRUE (at the same once the engine is running on its own. In the case of abnormal
time). C1 is also a condition such that when g occurs and C1 is start events–such as when the engine is not starting after starter
evaluated FALSE, the machine transitions to A; if C1 is engagement and application of fuel, a high engine temperature
evaluated TRUE, the machine transitions to B. on start-up, or pneumatic or electrical supply interruption–the
The first Statecharts enrichment is concurrency of pilots are instructed to follow a prescribed procedure. The
processes. Two related processes can be placed together in a procedure, the IRREGULAR ENGINE START for aircraft,
specifies the sequence of immediate actions that must be Fuel Control Sw itch
performed by the crew to avoid further damage to the engine Engine
HIG H clo se
T EM P ROT ATI NG
and to shut it down properly. Figure 2 is a copy of the
OFF
procedure as it appears in the pilots’ manual.
run /op en
cu t-o ff/cl ose
gro un d-starte r
ON

IMMEDIATE ACTION
Engin e Start Se lecto r {(t >= 30 sec.).and.
(temp < 1 80) } .a nd. OFF
gn d/g rou nd -sta rter
FUEL CONTROL SWITCH . . . . . . . . . . . . . . . CUTOFF o pe n
ENGINE START SELECTOR. . . . . . . . . . . . . . . . GND AUTO OFF MO TORING I DLE

Motor for 30 seconds or until EGT is below 180, whichever is GND C ONT o pe n gro un d-sta rte r
longer (unless no oil pressure).
FLT
NO RM AL
NOTE cl ose

If starter cutout has occurred, reselect GND when N2 is s wi tch

below 20%
Figure 3. Model.
If problem was other rapid EGT rise:

ENGINE START SELECTOR. . . . . . . . . . . . . . . . .OFF

Model of the System
Figure 3 is a model of engine behavior, given pilot
interactions, during an irregular engine start. Three concurrent
Figure 2. Irregular Engine Start. processes are depicted: Engine, Fuel Control Switch, and
Engine Start Selector. The initial state of the “Engine” process
The sequence of actions for the irregular engine start is to is IDLE (note the small arrow). The pilot starts the engine by
first close the fuel valve (fuel control switch – cutoff), and then first moving the engine start selector switch (depicted in the
engage the ground starter (engine start selector – GND). lower-left process of Figure 3) to GND. This event (gnd), in
These two actions should be done immediately–that is, from turn, broadcasts the event ground starter to the engine. Now
the pilot’s memory—and not by opening and reading a the engine is motoring. Once the engine fan (N2) reaches a
procedures book. Once these two steps are executed, the pilot speed of 25 percent, the pilot places the “Fuel Control Switch”
is then instructed to engage the starter and crank (“motor”) the to ON (depicted in the upper-left process in Figure 3). This
engine for 30 seconds or until the engine’s Exhaust Gas transition from OFF to ON, triggers the output event open (fuel
Temperature (EGT) falls below 180 degrees. This means that valve), which is now broadcast to the “Engine” process.
if 30 seconds have elapsed and the engine temperature is still Once fuel is injected into the engine, the engine speed and
higher than 180 degrees, the pilot should continue to motor the temperature begin to increase. The engine can either stabilize
engine (with the starter engaged), until the temperature at the throttle setting (which is the normal case), or the engine
subsides. The pilot is then cautioned that engine motoring can accelerate and reach undesired speeds and temperatures
should not be continued if there is a no oil pressure, because (which is the abnormal case). From the point of view of the
“dry” motoring will severely damage the moving parts in the pilot, the transition out of motoring into either normal or high
engine. temperature states is non-deterministic. That is, the pilot
The procedure further cautions the pilot that if the ground- cannot foresee when either state will happen (yet the pilot does
starter has disengaged automatically (starter cutout) as part of know–based on historical data, training, and the mere
the normal start, the pilot should re-select ground (starter) existence of the abnormal procedure–that a faulty start may
when the speed of the second stage fan (N2) is below 20 indeed occur). The uncertainty associated with reaching an
percent. In all cases, the pilot should wait for the fan speed to undesired state, yet knowing that it may happen someday, is
drop below 20 percent before engaging the ground starter, the foundation of Standard Operating Procedures and is
because engaging the ground-starter when the engine fan is exactly why procedures are in place.
rotating at a high speed will damage the starter. (This is In the event of a high engine temperature (HIGH TEMP),
somewhat similar to engaging the starter in an automobile the procedure is to close the fuel valve, and as the engine fan
when the car engine is running). The careful reader rotates down, to re-engage the ground-starter (by setting the
immediately notes that there are some problems in the wording starter switch to GND). The procedure tells the pilot to motor
and arrangement of actions, conditions, and notes in this the engine for at least 30 seconds, or continue beyond 30
procedure. And indeed, identifying these deficiencies in a seconds if the engine temperature is above 180 degrees. Once
systematic way is the objective of this paper. We begin by the temperature is below 180 degrees, the pilot should
modeling the system involved in the irregular engine start. disengage the ground-starter.
 ground-starter

Tasks and Specifications [t >=T]

Fail_D
The main objective of the pilot, once a high temperature
occurs, is to remove the fuel source, quickly cool the engine,
HIGH close [t >=T’]
and bring it back to idle. In fact, if the pilot succeeds in ROTATING Fail_C
TEMP
cooling the engine following a hot start, the manual states that
“maintenance personnel may be able to ‘clear the item’ and ground-starter
dispatch the airplane, depending on maximum EGT reached { (t <30 sec.) .or.
and its duration.” (temp >180) } .and. OFF
C1
FALSE
Fail_A
Therefore, the pilot’s task goal is to “drive” the system
back to the idle state. The specification is to perform this task
TRUE
as safely and efficiently as possible, minimizing damage to the
engine and aircraft. It must be recognized, however, that the FALSE
C2 Fail_B
system (the engine and its surrounding environment) can
{ (t >=30 sec.) .and.
interfere with this process and lead to sub-optimal
TRUE (temp <180) } .and. OFF
configurations. Likewise, a pilot can, inadvertently, drive the open
system into an unwanted or sub-optimal configuration. (For
MOTORING IDLE
example, see FAA Airworthiness Directive 88-07-02 that was
issued in response to three documented cases in which Boeing open
ground-starter
B-767 pilots mistakenly shut down an engine during climb,
while intending only to switch off a related sub-system). The close
model, therefore, should be expanded to account for pilot- NORMAL
initiated events that result in sub-optimal configurations. C1:= [oil pressure o.k.]
Figure 4 is an expanded description of the model C2:= [N2<= 20%]
(omitting the “Fuel Control Switch” and “Engine Start
Figure 4. Expanded model.
Selector” process for brevity). Several states and transitions
were added to account for pilot-initiated events and are Analysis of Action Sequences
discussed below: Following a rapid rise in engine temperature Now that we have superimposed the task goals and
(HIGH TEMP), if the pilot fails to take any action [t >= T], or specifications on a model of the human-machine system, we
the pilot mistakenly selects ground-starter (GND), the engine can evaluate the procedure. Specifically, given the current
can be severely damaged. We denote this sub-optimal state as system, we want to identify the safest and most efficient
FAIL_D. If the pilot closes the fuel to the engine, but fails to sequence of actions to get to the IDLE state. One way to trace
motor the engine, this results in another sub-optimal state this path is to “open up” the model in Figure 4 as a sequential
(FAIL_C). When the pilot selects ground-starter, but there is no tree of all possible actions. Figure 5 is a depiction of such a
oil pressure, the engine can be again severely damaged tree (focusing only on the relevant actions).
(FAIL_A). If there is enough oil pressure, but the fan speed is
greater than 20 percent, engaging the starter may damage the HIGH
Fail 0
ground-starter .or. [t >= T]
TEMP Fail_D
starter (FAIL_B). Finally, when the engine is being motored
and the pilot, by slip or mistake, disengages the starter before close
the mandatory 30 seconds or before the engine temperature is
below 180 degrees, he or she may have to re-engage the ROTATING
[t >= T’]
Fail_C
ground-starter. This may not directly drive the system to a
sub-optimal state, but it certainly takes more time and (ground-starter) .and.
increases the potential for damage. (oil pressure o.k) .and. (ground-starter) .and.
(N2> 20%) (NO oil pressure)
Now we can superimpose the task goal on the model. Our
task is to drive the system to IDLE state, but we are willing to (ground-starter) .and.
(oil pressure o.k) .and.
accept FAIL_A, if due to secondary effects (e.g., ruptured oil (N2 <= 20%)
line) beyond our immediate control, unfortunate things occur.
Fail_B
These two “acceptable” end-states are circled with a broken
Fail_A
line in Figure 4. In contrast, FAIL_D, FAIL_C, and FAIL_B are MOTORING
not part of our task goals, but may happen if the pilot, for
whatever reason, performs the wrong action. { (t < 30 sec.) .or.
(temp > 180) } .and. OFF
Our specification called for efficiency and minimization of { (t >= 30 sec.) .and.
(temp < 180) } .and. OFF
engine damage in the process of driving the engine to the IDLE
state. If the pilot disengages the starter, before the mandatory ROTATING
30 seconds or before the engine temperature is below 180
degrees, the task (drive the system to idle) is indeed achieved - IDLE
- but our specification violated. The pilot may have to re-
engage the ground-starter which will certainly takes more time Fi gu r e 5 . Tr ee of ac ti o n s eq ue n ce s .
and, as mentioned earlier, increase the potential for damage.
 The nominal path from high temperature to idle traverses pressure). The use of parentheses implies that a logically
through rotating and motoring. Naturally, this sequence of remote relationship exists between the phrase within the
actions must appear in the procedure. But we must also parentheses and the rest of the sentence. In fact, the oil
recognize that along the way there are numerous pitfalls to pressure is a pre-condition for the previous procedure step
avoid, namely fail states A, B, C, and D. The procedure is the (engaging the ground-starter). The confusing wording of the
only aid to support the pilot along this “treacherous” path. conditional sentence may lead to situations in which the pilot
Based on the model, the sequence of pilot actions following an may stop motoring prematurely (i.e., before the engine
irregular start (e.g., high temp state) is: temperature drops below 180 degrees) or may simply not
1. Close the fuel valve to the engine (the system know what to do!
transitions into rotating state). CONCLUSION
2. Evaluate the situation: if oil pressure is O.K. and fan
speed is at or below 20 percent, engage the ground- As mentioned earlier, it is a basic assumption in all high-
starter (and then transition to motoring), but if fan speed risk industries that for known failures, the procedures supplied
is above 20 percent, wait until it goes down; if there is to the operators provide the “best” way to perform a given
no oil pressure, do not engage the starter (and transition task. This is not the case with the procedure in Figure 2. We
to fail_A). argue that the current process of designing and evaluating
3. Motor for at least 30 seconds and monitor engine procedures can be much improved. New methods for
temperature. Continue motoring until the temperature describing such human-machine-environment interactions
goes below 180 degrees. should replace the more intuitive and ad-hoc processes that are
currently used by most high-risk industries. To assist in such a
However, when we compare this sequence of actions and
change, Human Factors researchers must develop design and
conditions with the procedure in Figure 2, we note an
evaluation methods to deal with far more complex systems and
important discrepancy: The procedure, which must be
situations involving dynamics and timing constraints.
executed immediately as a step-by-step sequence (with no time
Objective methods that will evaluate the vulnerability of a
to think ahead), tells the pilot to close the fuel valve (fuel
given procedure to human error are in great need. Can we
control switch–cutoff) and next immediately engage the
predict where someone may fail in executing a procedure?
ground-starter (engine start selector–gnd). Only then (at the
Are there procedures that are more prone to human error? If
very end of the condition sentence and in small letters), does
so, why?
the procedure stipulate the oil pressure condition–“(unless no
oil pressure).” Furthermore, only after three lines the To conclude, while "deviation from operating procedures"
procedure cautions the pilot not to engage the ground selector is by far the highest-ranking crew-caused factor in aircraft
if N2 is above 20 percent. The order of statements in the accidents (NTSB, 1994), it is also true that many procedures
procedure is not congruent with the order of actual events are inherently incorrect. Such procedures “support” human
needed to drive the engine to idle. This sequential deficiency error. Methods for more systematic, objective, and accurate
in this procedure may lead either to damaging the starter procedure development in high-risk systems are desperately
(FAIL_B) or to the even worse situation of damaging the needed. We cannot continue to develop procedures for
engine because of no oil pressure while motoring (FAIL_A)! increasingly complex and automated systems while still using
informal and ad-hoc techniques that are prone to error. This is
There are also wording problems in this procedure: The
a very important challenge in all high-risk industries.
multi-conditional sentence “Motor for 30 seconds or until EGT
is below 180, whichever is longer (unless no oil pressure)” is REFERENCES
difficult to comprehend. First, the two elements–“30 seconds” Bailey, R. W. (1989). Human performance engineering: Using human
and “until EGT is below 180 degrees”–are difficult to equate; factors/ergonomics to achieve computer system usability (2nd ed.).
in the former time is explicit (30 seconds) and in the latter Englewood Cliffs, NJ: Prentice-Hall.
(until EGT is below 180 degrees) time is implicit (see Degani, A., and Heymann, M. (1999). Pilot Autopilot Interaction: A
formal perspective. In R. Jensen (Ed.), Proceeding of the 10th Aviation
Wickens, 1992 chap. 5; Bailey, 1989 pp. 363-367). Second, it Psychology Symposium. Columbus, OH: Ohio State University.
is not clear whether “until EGT is below 180 degrees” includes Degani, A., and Wiener, E. L. (1993). Cockpit checklists: Concepts,
or excludes the 30 seconds. Third, the logical operator .or. is design, and use. Human Factors, 35(2), 345-359.
followed by a selection criteria–“whichever is longer”-- which Degani, A., and Wiener, E. L. (1997). Procedures in complex systems:
makes the sentence restrictive and therefore confusing. The airline cockpit. IEEE Transactions on Systems, Man, and Cybernetics,
Furthermore, the ordering of the words in the sentence leads to 27(3), 302-312.
reader to believe, initially, that he or she can either do A De Soto, C. B., London, M., and Handel, S. (1965). Social reasoning
and spatial paralogic. Journal of Personality and Social Psychology, 2(4),
(“motor for 30 seconds”) .or. B (“until EGT is below 180”)-- 513-521.
only then to be informed of the selection criteria (“whichever Harel, D. (1987). On visual formalisms. Communications of the ACM,
is longer”) (see De Soto, London, and Handel, 1965 for a 31(5), 514-530.
discussion on logical ordering). Fourth, the use of the term Laughery, K. R., and Wogalter, M. S. (1997). Warning and risk
“longer” is confusing because the pilot is under the impression perception. In G. Salvendy (Ed.), Handbook of human factors and
that he or she should try to “shorten” the time to motor the ergonomics (pp. 1174-1198). New York: John Wiley.
engine. The contextual conflict between these two opposing Mosier, K. L., Palmer, E. A., and Degani, A. (1992). Electronic
checklists: Implications for decision making. Proceeding of the Human
directions appears to mislead readers (see Laughery and Factors Society 36th Annual Meeting (pp. 7-11). Atlanta, GA: Human
Wogalter, 1997). Finally, from a grammatical point of view, Factors Society.
there is a misuse of a parenthetical expression (unless no oil
 National Transportation Safety Board (NTSB). (1994). A review of
flightcrew-involved major accidents of U.S. air carriers, 1978 through 1990
(Safety study, NTSB/SS-94/01). Washington, DC: Author.
R. J. Ramadge and W. M. Wonham, 1987, Supervisory control of a
class of discrete event processes. SIAM J. Control and Optimization, 25(1),
pp. 206-230.
Wickens, C. D. (1992). Engineering psychology and human
performance (2nd ed.). New York: HarperCollins.
Wieringa, D., Moore, C., and Barnes, V. E. (1992). Procedure Writing.
Piscataway, NJ: IEEE Press
.