U.S.

Army Research Institute

for the Behavioral and Social Sciences

Research Report 1780

Assessing the Effectiveness of a Low-

Cost Simulator for Instrument Training for the
TH-67 Helicopter

John E. Stewart II, William C. Barker and Dale S. Weiler

U.S. Army Research Institute
Jerry W. Bonham
Aviation Training Brigade, U.S. Army Aviation Center

David M. Johnson
U.S. Army Research Institute

December 2001
20020103 167
Approved for public release; distribution is unlimited.
U.S. Army Research Institute
for the Behavioral and Social Sciences
A Directorate of the U.S. Total Army Personnel Command

EDGAR M. JOHNSON
Director

Technical review by

Thomas K. Flohr, U.S. Army Aviation Center, DOTDS

Kara A. LatoreUa, NASA Langley Research Center

NOTICES

DISTRIBUTION: Primary distribution of this Research Report has been made by ARI.
Please address correspondence concerning distribution of reports to: U.S. Army
Research Institute for the Behavioral and Social Sciences, Attn: TAPC-ARI-PO, 5001
Eisenhower Ave., Alexandria, VA 22333-5600.

FINAL DISPOSITION: This Research Report may be destroyed when it is no longer

needed. Please do not return it to the U.S. Army Research Institute for the Behavioral
and Social Sciences.

NOTE: The findings in this Research Report are not to be construed as an official
Department of the Army position, unless so designated by other authorized documents.
 REPORT DOCUMENTATION PAGE

2. REPORT TYPE 3. DATES COVERED (from. . . to)

1. REPORT DATE (dd-mm-yy)
July 2000-September 2001
December 2001 Final
5a CONTRACT OR GRANT NUMBER
4. TITLE AND SUBTITLE
Assessing the Effectiveness of a Low-Cost Simulator for
Instrument Training for the TH-67 Helicopter

5b. PROGRAM ELEMENT NUMBER 62785

5c. PROJECT NUMBER A790

6. AUTHOR(S)
John E Stewart II, William C. Barker and Dale S. Weiler (U.S. Army
Research Institute); Jerry W. Bonham (US Army Aviation Center); and
David M. Johnson (U.S. Army Research Institute)

5d. TASK NUMBER 206

5e. WORK UNIT NUMBER

8. PERFORMING ORGANIZATION REPORT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)
US Army Research Institute Aviation Training Brigade
Rotary Wing Aviation Research Unit ATTN: ATZQ BDE OH, Bldg. 4509
ATTN: TAPC ARIIR, Bldg. 5100 Fort Rucker, AL 36362-5000
Fort Rucker, AL 36362-5354

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. MONITOR ACRONYM

U.S. Army Research Institute for the Behavioral and Social Sciences
ARI
5001 Eisenhower Avenue 11. MONITOR REPORT NUMBER
Alexandria, VA 22333-5600
Research Report 1780

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES

14 ABSTRACT (Maximum 200 words): The U.S. Army uses the 2B24 Synthetic Flight Training System (SFTS) for the Instrument Phase
of Initial Entry Rotary Wing (IERW) training. The SFTS is an instrument simulator, mounted on a hydraulic motion platform with no
visuasystem |ts tXhnology dates from the late 1960s. Its cockprt represents the UH-1, which has been replaced by the TH-67
SninXEpS The AmTy is concerned with the age, complexity and costs of the SFTS, at a time when PC-based simulators, ike
he Frasca ^2 P iman/ Skills Trainer (PST) are available. The PST's cockpit represents the TH-67 helicopter. It has a visual display,
but no nota system Thirty-eight IERW students were assigned to experimental (PST) or control (SFTS) groups After 30 hr of
simulator XSboth groups completed 20 hr training in the TH-67. No students were eliminated or set back to later classes. Few
ZSS!SSL ^performance were noted, though SFTS trainees were more likely to indicate that training in the simuator had
hindered performance in the aircraft. The PST seemed inferior to the SFTS in trim control. The research demonstrated that IERW
students could learn instrument skills in a simpler, more economical simulator without hydraulic controls or a motion system.

15. SUBJECT TERMS Instrument Flight Training, Helicopter Flight Training, Simulation Technology, Low-Cost Simulators, Training Effectiveness
Evaluation Methodology.

I|i|p|illllllliillill| 19. LIMITATION OF 20. NUMBER 21. RESPONSIBLE PERSON

(Name and Telephone Number)
ABSTRACT OF PAGES
18. THIS PAGE Dennis C. Wightman
16. REPORT 17. ABSTRACT
Unclassified Unclassified 41 (334) 255-2834/9091
Unclassified Unclassified
Research Report 1780

Assessing the Effectiveness of a Low-

Cost Simulator for Instrument Training for the
TH-67 Helicopter

John E. Stewart II, William C. Barker and Dale S. Weiler

U.S. Army Research Institute

Jerry W. Bonham
Aviation Training Brigade, U.S. Army Aviation Center

David M. Johnson
U.S. Army Research Institute

Rotary-Wing Aviation Research Unit

Dennis C. Wightman, Chief

U.S. Army Research Institute for the Behavioral and Social Sciences
5001 Eisenhower Avenue, Alexandria, Virginia 22333-5600

December 2001

Army Project Number Personnel Performance and

2O262785A790 Training Technology

Approved for public release; distribution is unlimited.

in
FOREWORD

The Rotary-Wing Aviation Research Unit of the U.S. Army Research Institute for the
Behavioral and Social Sciences (ARI) is located at Fort Rucker, Alabama. The ARI Aircrew
Performance Team is committed to enhancing aviation training in the Army. One means by
which this can be achieved is through the optimization of simulation-augmented Initial Entry
Rotary-Wing (IERW) training programs. The advent of PC-based simulators and other training
devices which are more reliable, simpler, and much less expensive than their predecessors, could
potentially revolutionize the expansion of synthetic flight for all levels of training. A research
program in this area was deemed necessary because few reliable benchmarks exist as to how this
new technology can best be employed in an IERW environment.

Currently, the use of simulation in IERW training is limited to the Instrument Phase,
which begins after the student pilot has mastered contact flight (i.e., visual flight rules). The
simulators presently in use are complex, expensive, and based upon late 1960s technology. The
research project comprising this report concerned itself with the comparison of two simulator
technologies: a low-cost, PC-based simulator, represented by the Frasca 342 Primary Skills
Trainer, and the currently operational 2B24 Synthetic Flight Training System. Student pilots
were assigned to one simulator or the other on a random basis, and completed the Instrument
Phase of training. Results indicated that training outcomes were equally successful, regardless
of the simulator. The major difference was the much lower operational cost of the Frasca 342.
The results of the research were briefed to the Commander, Aviation Training Brigade, on 22
August 2001. A briefing of the preliminary findings was presented at the U.S. Army Aviation
and Missile Command Aviation Science and Technology Review, on 27 June 2001.

%^Jh*J&"
TA M. SIMUTIS
technical Director
ACKNOWLEDGEMENT

Field research on the scale of the present study requires the participation of many key
personnel. This project was no exception. Lear Siegler Services, Inc. (LSSI) is the training
support contractor at Fort Rucker. The instructor pilots (IPs) responsible for training the student
pilots in the Frasca simulator, and also the 2B24 comparison simulator, were employees of LSSF
These persons were: Larry Grimm, James Kale, Blaine Pendleton, David Sheppard, and Irvin
Starrak. They also provided ARI with invaluable usability feedback on the Frasca 342 simulator.
The IPs belonged to two training Flights, and the Flight Commanders, Lisa Bailey and Henry
Witmer, provided essential support in managing the simulation activities as well as coordinating
the collection of performance data on the flightline, where a large portion of the Advanced
Instruments training took place. LSSI administrative and management staff assisted with the
selection of students from class rosters, the initial certification of the Frasca 342, and the
recruitment of IPs. Hats off to: Samuel Denton, Joseph Holmes, Robert Mead, Samuel Mowery,
and Robert Price, for their dedication and support. Flight evaluations were a crucial part of the
study, and required the services of three Standardization IPs, one of these military, the other two,
Department of the Army Civilians. These were: CW3 Michael Alberich, Mr. Robert Boutwell,
and Mr. David Hatcher, all of the 1st Battalion, 223rd Aviation Regiment. CW2 Keith Miller,
who is currently on casual assignment to ARI, assisted with the day-to-day operation of the
Frasca 342. Mr. James Hughes, Chief of the Student Support Branch, Directorate of Plans,
Training, Mobilization and Security, was immensely helpful in making available data on the
class standing and overall grades of students who participated in this research. Finally, the high
quality shroud and frame assembly, which controlled light levels in the Frasca simulator, was the
handiwork of Marco Verardo of CAE International, the main on site support contractor at ARI.
The high level of professionalism and dedication of these and others involved in the conduct of
the research is gratefully appreciated by ARI.

VI
ASSESSING THE EFFECTIVENESS OF A LOW-COST SIMULATOR FOR INSTRUMENT
TRAINING FOR THE TH-67 HELICOPTER

EXECUTIVE SUMMARY

Research Requirement:

The U.S. Army uses a simulator, the 2B24 Synthetic Flight Training System (SFTS) for
the Instrument Phase of Initial Entry Rotary Wing (IERW) training. The SFTS is a dedicated
instrument trainer with no visual display system. It is mounted on a five degree of freedom
hydraulic motion base. Its technology dates from the late 1960s; furthermore, it represents the
UFI-1, which has been replaced by the TH-67 primary training helicopter. The U.S. Army
Aviation Center (USAAVNC) is concerned with the age, complexity, and operating costs of the
SFTS, at a time when simpler, more affordable and versatile, PC-based simulators are available.
An example of such a simulator is the Frasca 342 Primary Skills Trainer (PST) proof-of-concept
simulator. The PST has a cockpit modeled after the TH-67, and is equipped with a low-cost
visual display system. It was loaned to ARI by the manufacturer for purposes of conducting
usability and transfer of training research. USAAVNC's quest for a new-technology simulator
provided an excellent opportunity to conduct a training effectiveness assessment.

Procedure:

The training effectiveness research compared two simulator technologies in a routine

training environment. Thirty-eight IERW flight students participated. They were assigned
randomly to either the PST (experimental group) or the SFTS (control group). A simple two-
group design was used. The only difference was the simulator in which the participants received
the 30 hr of instruction. After completion of the simulator phase, both groups reported to the
flightline to complete the last 20 hr of Instrument Phase in the TH-67. Five instructor pilots (IPs)
familiar with the PST trained both experimental and control group students. Three
Standardization IPs administered the evaluation checkrides in simulator and aircraft.

Findings:

Few significant differences in measures of performance (checkride scores; hours to

proficiency; usability ratings of the simulators) were evident. No student pilots in either
condition were set back to later classes or eliminated from training. On the postexperimental
questionnaire, control group participants were more likely than their experimental group
counterparts to indicate that some things learned in the simulator had hindered their performance
in the aircraft. The PST was rated as significantly inferior in trim control to the SFTS, but
significantly better for instrument take offs and for instrument approaches. Most other
differences were nonsignificant, though overall it appeared that the PST had more appeal to
students than did the SFTS. The most frequent spontaneous complaint from PST students
concerned poor trim control; for the SFTS, it was dissimilarity of its cockpit from that of the
TH-67.

vn
Utilization of Findings:

The research demonstrated that EERW students could learn Instrument Phase flight skills
in a simpler, more economical simulator. The complexity of the hydraulically-actuated flight
controls and the motion cueing system of the SFTS appear unnecessary for successful instrument
training. Measures of performance and evaluative input from student pilots and IPs pinpointed
PST technical issues, mostly related to software, that would require resolution before this type of
simulator could be acquired by the Army. In brief, the software aerodynamic model and
instrumentation would have to represent the TH-67, both physically and functionally. Finally,
the presence of a visual display system on the PST suggests that an improved version of this type
of simulator could support training beyond the Instrument Phase. A future research project,
employing such a simulator, could determine the range of flight skills, both Primary (visual flight
rules) and Instrument that could be trained.

vm
CONTENTS

Page
INTRODUCTION. 1

Training Effectiveness Research 1

2
Instrument Training in Army Aviation
The Frasca 342 Primary Skills Trainer (PST) Low-Cost Flight Simulator 3

4
METHOD
4
Overview
Participants and Design ->
Procedure ^
Measures of Performance °
Hypotheses '
7
RESULTS

Participants '
8
Technical Problems
Measures of Performance 8
20
DISCUSSION

Training Effectiveness of the PST 20

Assessment by Target Audience 20
21
CONCLUSIONS
23
REFERENCES

APPENDIX A: Student Proficiency Records A-l

B: TH-67 Frasca Primary Skills Trainer (PST) Questionnaire B-l
C: Student Background Questionnaire C-l

IX
CONTENTS (Continued)

LIST OF TABLES
Page

Table 1. Frasca Primary Skills Trainer VS. Synthetic Flight Training System 4

2. Mean Instrument Phase Put-up and Checkride Scores (SDs in Parentheses) 8

3. Mean Hours to Proficiency for Basic Instruments (BI) Flight Maneuvers 10

4. Mean Hours to Proficiency for Advanced Instruments (AI) Flight

Maneuvers H

5. Mean Student Questionnaire Ratings (6-pt. Scale) by Item Content 12

6. Content Categories of Spontaneous Comments of Participants on

Postexperimental Questionnaire 13

7. Comparison of Instructor Pilot (IP) vs. Student Pilot (SP) Attitudes Toward
Simulation. (6-pt. Scale) 14

8. Mean Instructor Pilot Ratings by Item Content (6-pt. Scale) 15

9. Instructor Pilot (IP) vs. Student Pilot (SP) Ratings of Simulator Effectiveness
by Item Content, for All Maneuvers (6-pt. Scale) 16

10. Content Categories of Spontaneous Comments of Instructor Pilots on

Postexperimental Questionnaire 17

11. Pearson Correlations of Times to Proficiency on Basic Instruments (BI)

Maneuver Tasks with BI and Advanced Instruments (AI) Checkride
Scores (N = 38) 18

12. Pearson Correlations of Times to Proficiency on Advanced Instruments

(AI) Maneuver Tasks with Basic Instruments (BI) and AI Checkride
Scores (N = 38) '. 19
 ASSESSING THE EFFECTIVENESS OF A LOW-COST SIMULATOR FOR
INSTRUMENT TRAFNING FOR THE TH-67 HELICOPTER

John E. Stewart II, William C. Barker, and Dale S. Weiler

Army Research Institute Rotary Wing Aviation Research Unit

Jerry W. Bonham
Aviation Training Brigade. US Army Aviation Center

David M. Johnson
Army Research Institute Rotary Wing Aviation Research Unit

Introduction

Training Effectiveness Research

Cost and training effectiveness have been, and still are, important issues for transfer of
training (TOT) research. The Army has long been concerned with the training effectiveness of
simulators that are simpler and cheaper, when compared to those currently in operation (Caro,
Jolley, Isley, & Wright, 1972; Prophet & Boyd, 1970). Paul Caro (1988) discusses research,
conducted in the early 1970s, which compared the training effectiveness of a high fidelity,
expensive cockpit procedures trainer, for the OV-1 aircraft (Device 2-C-9), with a simpler
plywood mock-up of the same aircraft. The mock-up had been built by carpenters at a cost of
$30.00. Although in many ways dissimilar to the aircraft, the training device had functional
fidelity, in that it could provide the same critical cues for practicing cockpit procedural tasks, as
did its high-realism counterpart. Caro defines both of these devices as simulators, as opposed to
generic trainers. His criterion for the definition is that they were designed to present precisely
the cues and response opportunities necessary for performing procedural tasks in the aircraft. In
other words, the spatial and functional relationships of the simulated levers, dials, and switches
were the same in the low-cost simulator as in the OV-1. Transfer of skills to the aircraft should
be the same, regardless of the cost of the simulator. Prophet and Boyd (1970) compared three
groups of Army aviators, all of whom were transitioning to the OV-1 tactical reconnaissance
aircraft. One group trained in the high-fidelity procedures trainer, the second in the plywood
mock-up, and the third trained exclusively in the aircraft. Each group received five trials in the
device, and subsequently five in the aircraft. On the first trial in the aircraft, both training device
groups performed equally well, and only slightly worse than the aircraft-only baseline group.
Prophet and Boyd concluded that the transfer characteristics of both trainers were essentially the
same, despite cost differences, and that pretraining in the mock-up was about as effective as
training in the aircraft alone. Both this study, and Caro, et al.'s evaluation of a plywood mockup
of the U-21 fixed wing aircraft cockpit, were practical applications of TOT methodology. Then-
main objective was to determine the functionality of very low-cost simulators that were cheaper
than contemporary training devices developed for the Army, and far cheaper than the hourly cost
of training in the aircraft they represented. The research confirmed that greater training
efficiency is possible, to the extent that low-cost simulators can be developed, so long as the
critical functions being trained are preserved.
 The issue addressed by Prophet and Boyd is even more important today. The advent of
powerful, affordable, PC-based simulator technology promises to replace the older simulators
with a new generation requiring less support and maintenance. The main difference between 1970
and the present is that the less expensive alternative no longer has to be a relatively crude mock-
up. Instrument training is a case in point.

Instrument Training in Army Aviation

U.S. Army initial entry rotary wing (IERW) flight training. The Instrument Phase of
IERW lasts a total of eight weeks and consists of 50 flight hours. Of these, 30 take place in the
Synthetic Flight Training System (SFTS); the remainder in the aircraft. The SFTS is a dedicated
instrument trainer based on the UH-1 helicopter. It is mounted on a five degree of freedom
motion platform, has a high fidelity UH-1 cockpit, hydraulically loaded controls, and a complete
Instructor-Operator Station (IOS). Its technology dates from the late 1960s, and it has been
operationally employed at the U.S. Army Aviation Center (USAAVNC) at Fort Rucker, AL,
since the early 1970s. It is a dedicated instrument flight training simulator; hence, it has no visual
display system. SFTS training, in the EERW program, consists of 20 training days, each lasting
1.5 hr. Training in the TH-67 primary training helicopter spans 20 training days, with each
session lasting 1 hr. Student pilots who begin the Instrument Phase of ERW have successfully
completed the Primary Phase, which consists of a total of 60 flight hr of contact (visual) flight
training. This includes successful completion of unsupervised solo flight, and an end of Primary
Phase checkride. Primary flight training takes place on the flightline using the TH-67; no
simulation is employed.

Training effectiveness of the SFTS. Caro (1972) performed an evaluation of the Army's "
then-new SFTS. He reported that introduction of the simulator reduced instrument training time
in the aircraft by approximately 90%. Prior to the introduction of the SFTS and the revamped
instrument training program, 60 hr aircraft time and 26 hr in a modified Link 1-CA-l Trainer were
required to obtain the Army Standard Instrument Card. Introduction of the new simulator and
revised training program reduced aircraft training time to 6.5 hr, supplemented by approximately
43 hr simulator time. The length of the instrument training course was reduced from 12 to 8
weeks. The SFTS remains operational today, which is a tribute to its training effectiveness. Cost
effectiveness is another issue altogether.

Research has shown that, with modifications, the SFTS could be used as a primary, visual
flight trainer. The U.S. Army Research Institute for the Behavioral and Social Sciences (ARI) has
supported the Aviation Training Brigade (ATB) in the execution of simulation transfer of training
(TOT) research at Fort Rucker, AL. Research involving a modified SFTS was performed by Jack
Dohme in the late 1980s and early 1990s (Dohme, 1991; 1995). Dohme converted the SFTS
into a visual simulator by the addition of image generators and three 69 cm monitors. The
software flight model was also modified to allow for hovering and low speed flight. Dohme was
able to demonstrate that pretraining in this visual simulator saved training time and task iterations
in the training aircraft, at that time the UH-1H. Implications of these findings were that the SFTS,
with similar modifications, could be used for contact as well as for instrument training. Although
the experiments were successful, USAAVNC did not introduce simulation into the primary flight
training syllabus.

Complexity, cost and obsolescence of the SFTS. ATB is concerned with the complexity,
as well as the operational and maintenance costs of the aging SFTS. Thirty-two SFTS cockpits
are located at Fort Rucker. These are used primarily for DERW Instrument Phase training. Each
training simulator requires complex hydraulic equipment to operate the motion cueing system.
Computer equipment, now outdated, is also required, as well as a climate control system to cool
the electronics and hydraulics. If parts break, they must sometimes be custom machined,
replacements being unavailable from the original source of supply. These drawbacks
acknowledged, its lack of versatility remains. The aerodynamic software model does not allow
for the practice of low speed or hovering flight; some flight maneuvers (e.g., termination to a
hover) cannot be performed. Furthermore, the need for the expensive and complex motion
cueing system has never been demonstrated. Finally, the SFTS represents an interim training
aircraft that has been withdrawn from service. The UH-1 configuration of flight instruments is
quite different from that of the TH-67. This requires student pilots to re-adapt their instrument
scan patterns to the aircraft when they go to the flightline. All of these concerns pose a cogent
argument for replacing the SFTS at a time when cheaper, more versatile, and more supportable
simulators are available.

The Frasca 342 Primary Skills Trainer (VST) Low-Cost Flight Simulator

The PC-based Frasca 342 PST, on loan to ARI by the manufacturer, (Frasca International,
Inc., Urbana, IL) is a proof-of-concept testbed for a simulator that is both more cost effective and
more versatile than the SFTS. The PST is configured to simulate the characteristics of the TH-67
primary training aircraft. The cockpit is modeled after that of the aircraft with dual flight controls,
as well as actual aircraft flight instruments and system indicators. The avionics can be used for
training instrument flight rules (EFR) operations. The instrument panel does depart from that of
the IFR-configured TH-67 on one important aspect. The PST has the full complement of flight
instruments on the right (pilot) side only. Any operational instrument trainer would require a
Military Standard, two-pilot IFR cockpit. For this study, wallpaper instruments were applied to
the left (copilot) side of the panel, to give students a perspective on where the flight instruments
are located on that side for the aircraft.

The flight controls incorporate a passive spring control loading system designed to
produce a control feel similar to that of the aircraft. A force trim system for cyclic pitch and roll
control is included. Engine, electrical, hydraulic, and mechanical failures can be simulated. A
variety of environmental conditions including wind speed, wind direction, turbulence, visibility,
cloud ceiling, and day/dusk/night illumination can also be simulated.

The simulator provides helicopter sound cues, including engine, transmission, main rotor,
and wind noise, plus warning tones. The out-the-window view is projected onto three screens at
the front of the cockpit. The visual field of view subtended by the front screen is 99 degrees
horizontal by 25 degrees vertical, with 640 x 480 pixels. The computer-generated imagery in the
database includes: an airfield and runway, with approach and runway lights, taxi lanes, helipad,
hangars, and control tower. A stage field with a tower, helipad, two parking pads, and a
functioning windsock are also included, as are navigational beacons and ground stations. All
locations in the visual database are accurately modeled and internally consistent in terms of
latitude, longitude, and magnetic compass orientation. Four Pentium 75 computers control the
host computer and visual display system.

Integrated into the simulator system is an IOS, consisting of a generic desktop PC (486
DX 66) running an interactive, Windows 3. l™-based program. From this station the instructor,
or a console operator, can control environmental conditions, system failures, and aircraft location.
The instructor, or operator, can selectively freeze any or all flight controls to allow training of
specific perceptual-motor cues and skills. He or she can track the aircraft's location in three
dimensions, monitor in real time a wide range of flight data (including aircraft location in six
dimensions: x, y, z, pitch, roll, yaw), select and record key snapshots of these data, and print the
recorded file of these snapshots. Table 1, below, compares the major components of the PST to
those of the SFTS.

Table 1

Frasca Primary Skills Trainer VS. Synthetic Flight Training System

Primary Skills Trainer (PST) Synthetic Flight Training System (SFTS)

1990s Technology 1970s Technology.
No Motion System. Motion System (Five degrees of freedom).
Full Aerodynamic Flight Model. Flight Model at Speeds > 40 KT.
Spring-loaded Controls. Hydraulically-loaded Controls.
PC-based (Pentium 75; 80486). Non-PC-based.
Visual Display System No Visual Display System.
No Console Operator Required. Console Operator Required.
TH-67 Cockpit. UH-1H Cockpit.

Method
Overview

This training effectiveness assessment compared the performance of two training devices,
in a routine, operational training environment. Before such a study could be undertaken, the
Frasca PST had to be evaluated by U.S. Army military, civilian, and contractor instructor pilots
(IPs) to determine if basic instrument (BI) and advanced instrument (AI) flight maneuver tasks
could be adequately trained in the simulator. Certification by ATB is required for any such use of
a simulator or training device. This was important because, unlike other research in which student
pilots are pretrained in a simulator, in this study the PST would be employed as an operational
instrument trainer. The IPs who would be conducting training in the simulator would also need to
be familiarized with the device. They were asked, in the course of their training, to note any
discrepancies between the PST's instruments and handling, and those of the TH-67. Student
pilots were selected from successive classes and assigned either to the PST or to a comparison
SFTS simulator. The same U.S. Army Aviation Program of Instruction (POI) that is used for all
EERW instrument training was employed.

Participants and Design

Participants were 38 EERW flight students from even-numbered classes (2000-14 through
2001 -08), who were starting the Instrument Phase of EERW. Data collection began 27 July 2000
and ended 25 June 2001. Mean age was 26.11 years, (SD = 3.14); mean score on the Flight
Aptitude Selection Test (FAST) was 128.52 (SD = 12.56). Of the 38 participants, 30 were male,
34 described themselves as Caucasian, four as Hispanic. The sample consisted of 1 First
Lieutenant, 18 Second Lieutenants, and 19 Warrant Officer Is. Only those student pilots with
midrange grades (84-89) in Primary Phase training were selected. Those who had been set back
from earlier classes, or awarded additional hours due to performance deficiencies were not eligible
for selection, nor were those with primary flight grades 90 and above. Students with flight
experience, prior to ffiRW, were also excluded. A two-group TOT paradigm was used. The only
difference between the two groups was the simulator in which they received the first 30 hr of
instruction. The experimental group trained in the PST, the control group in the SFTS. Both
groups then went to the flightline for 20 hr of training in the TH-67 helicopter. Two participants
("stick buddies") were assigned to each group. They were matched, based on their performance
in primary training. The sample was balanced by gender, so that when a female student was
assigned to the experimental group, another was assigned to the control group. One important
consideration in selection of participants was weight. This was dictated by performance
limitations of the TH-67, especially during the hot, humid summer months. For this reason, no
student pilots weighing in excess of 86 kg were selected. The aircraft normally carries two
student pilots and one IP.

Procedure

Five civilian contract IPs from two training flights at Fort Rucker, who volunteered to
participate in the study, were trained to use the Frasca PST. All were Caucasian males. One
withdrew from the project after training two students in the SFTS. For all five, mean age was 51
years (SD - 8.49). The youngest was 39, the oldest, 59. Total flight hours, in a variety of
aircraft types, ranged from 4,550 hr to 15,000 hr (M = 10,487.50; SD = 4,347.87). All had had
extensive experience training students in the SFTS. For the four who completed the
postexperimental questionnaire, experience as IPs ranged from 14 to 31 yr (M = 25.0; SD =
8.04). Experience as Instrument Phase IPs ranged from 1.5 to 24 yr (M = 11-13, SD = 9.80).
They also had provided evaluative input on the suitability of the PST for training the standard
flight maneuvers for the Instrument Phase of IERW. Each IP was responsible for training a pair
of stick buddies. They would alternate between PST and SFTS for each successive class trained,
in order to maintain balance in the research design. Each pair of stick buddies would have the
same training IP throughout the Instrument Phase. Three U.S. Army standards and evaluation IPs
(SBPs) administered the Basic Instrument (BI) checkride in the simulator after 12 hr of training
had been completed, and the final advanced instrument (AI) checkride in the aircraft, after
completion of the 50 hr Instrument Phase, for both treatment conditions. Efforts were undertaken
for the same SIP to administer both checkrides (BI in the simulator, AI in the aircraft) for the pair
of participants in the experimental or control condition, though this was not always possible. For
these three SIPs, all Caucasian males, mean age was 48.67 years (SD = 7.77). The youngest was
40, the oldest, 55. Total flight hours, in a variety of aircraft types, ranged from 3,400 hr to
H'OOO hr (M = 10,433.33; SD = 6,091.25). SIPs, like the IPs, alternated between experimental
and control participants, for consecutive classes, whenever possible.

During the BI phase of instrument training, the student learns the fundamentals of flying
the aircraft, without visual reference cues, in controlled airspace. For example, the student must
learn to make standard rate (3°/sec) turns, timed turns, climbs, descents, and to accelerate and
decelerate, all within the standards of IFR. The AI phase teaches the fundamentals of radio
navigation, which includes holding and approach procedures, using a variety of radio navigation
aids, including non-directional beacons (NDBs), very high frequency omni directional radio range
(VOR) transmitters, runway approach localizers (LOC), and instrument landing system
(ILS) aids.

Two retired Army aviators, one a civilian Department of the Army computer specialist, the
other, a contractor, alternated as IOS operators. One had previously been rated in OH-58 A/C
helicopters; the other, in various models of the AH-1 helicopter. They were supplemented by an
active duty Army aviator, a Chief Warrant Officer 2, rated in the AH-64A helicopter, who was on
casual assignment.

Measures of Performance

Flight grades. During the Instrument Phase of IERW, student progress is evaluated on
a daily basis, from Training Day 61 until the final checkride on Training Day 100. The
cumulative hours are noted, and a letter grade is given for each maneuver task performed.
A grade of B indicates performance of the task to standard. Any grade less than a B indicates that
performance is substandard for a particular maneuver. Performance to standard indicates
successful performance of the task, with reference to formal criteria as published in the Instrument
Phase Flight Training Guide (USAAVNC, 2001). All flight maneuver tasks are defined in terms
of relevant parameters (e.g. airspeed, altitude, heading) and the deviation allowed (e.g., plus or
minus 10°). A student not performing with the specified parameters for a given task has not met
the standard. Performance to standard is not expected of a student the first time that he or she
attempts it; in this instance, a grade of NM (standard not met) is appropriate. The grade sheets
(USAAVNC Form 463) used for instrument training clearly delineate the hourly point at which
performance to standard (P3 level) is expected, for each maneuver task. The culmination of BI
and AI is a checkride, administered by an evaluation SIP who did not provide the daily training.
Checkride performance is indicated by numerical scores, not grades (although these are
traditionally referred to as grades). The training IP attempts to estimate the student's
performance on the checkride by providing a "put-up" score independent of the final checkride
score. The SIP does not know the put-up score while administering the checkride. Put-up and
checkride scores are then averaged into an evaluation grade. Both put-up and checkride grades
have their advantages; the training IP is familiar with the student's performance over a long period
of trials, whereas the SIP, who has not observed the student over this period, observes a
"snapshot" of his or her performance.
 Student proficiency records. Each IP was asked to complete a proficiency record (see
Appendix A) for each student participant, noting the hourly point at which he or she demonstrated
performance to standard (proficiency) in the simulator for BI tasks, and later in the aircraft, for AI
tasks. Fourteen BI maneuvers and twelve AI maneuvers were included on the form. This form
was supplemental to the much more comprehensive Form 463, and was employed in the study
because the data could be tabulated much more easily and rapidly.

Postexperimental questionnaire. For purposes of assessing training usability, supplemental

questionnaires were administered to PST and SFTS participants (Appendix B). A modified form
of this same questionnaire was administered to the four IPs who participated in the daily training
of students in the PST and SFTS. These questionnaires asked the student to evaluate the training
effectiveness of the simulator, for specific maneuver tasks that had been performed. Students
were also encouraged to make spontaneous comments about the simulator and training
experience.

Use of desktop flight simulators. Another supplementary questionnaire (Appendix C),

given to each student at the outset of instrument training, was designed to elicit information as to
whether or not he or she had used a desktop PC-based flight simulator (e.g., Microsoft Flight
Simulator 2000), and if yes, for how many hours in the past month. The intent of this one page
questionnaire was to determine if prior use of popular aviation-oriented software and gameware
conferred any advantage in learning the IERW instrument maneuvers.

Hypotheses

The PST is closer in configuration to the TH-67 than is the SFTS. Consequently, it would
be reasonable to expect that student pilots would reach proficiency in the aircraft sooner if trained
in the PST rather than the SFTS. This is because instrument scan patterns learned in the
PST should transfer to the TH-67 more readily; whereas those learned in the SFTS would have to
be unlearned, once the student began training on the flightline. However, it is acknowledged that
this advantage may be short-lived, in that 20, 1 hr training sessions are spent in the aircraft,
providing sufficient time for re-adaptation.

Results
Participants

Two participants, both in the SFTS (control) group, were eliminated from the study for
medical reasons unrelated to the research. Eliminations in both cases occurred too late for an
alternate to serve as a replacement. In both instances, the student pilots had completed the
simulator phase of training, and their classes had gone to the flightline for training in the TH-67.
The final sample size of 38 participants comprised 20 in the PST group, and 18 in the SFTS
group. Mean age of the PST participants was 26.85 years (SD = 3.23) vs. 25.28 years for the
SFTS (SD = 2.87); this difference was nonsignificant. Mean FAST score for the PST group was
121.20 (SD = 15.40), vs. 130.04 (SD = 14.62) for the SFTS group. This difference approached
significance (t = -1.81, df = 36, p < .08). Since actual FAST scores were not easily obtainable
from records, the investigators had to rely on self-reports.

Technical Problems

At the start of the evaluation of the PST, IPs noticed some technical issues.
Those that were considered serious enough to prevent the research from being conducted (erratic
altimeter readings, inoperative attitude indicator, inoperative and occasionally erratic turn
indicator) were resolved. After consulting with the IPs, and ATB Flight Standards, the research
team decided to proceed with the project, in spite of several problems and anomalies that were as
yet unresolved. These were: A tendency to gain, rather than lose altitude in a turn, calibration of
the turn needle and attitude indicator that were non-standard, an inoperative trim ball, inconsistent
readings from the magnetic compass, and anomalous readings on the torque indicator. Initially, it
was feared that these unresolved problems would preclude the effective instruction of BI skills;
however, IPs learned to work around these deficiencies, so that both BI and AI skills could be
taught. These issues were all due to problems with the software flight model.

Measures of Performance

Checkride scores. Due to heterogeneity of variance considerations for a few of these

measures in the present study, and for the sake of consistency, the Mann-Whitney U-test was
employed for all measures of performance. The Mann-Whitney U test has been described as one
of the most robust of the nonparametrics (Hays, 1973). In cases where the larger of the two
samples contains 20 or more observations, the statistic approximates the normal distribution
and can be interpreted as the normal deviate z. BI and AI put-up and checkride scores were
compared for the two groups of students. Only AI put-up scores showed any significant
difference between conditions (z = 1.96, p_ < .05), indicating that IPs anticipated better
performance from PST than from SFTS participants. Put-up and checkride grades appear in
Table 2. No student pilots in either the PST or SFTS groups were set back to later classes or
eliminated from flight training; all completed the Instrument Phase successfully and graduated
with their classes.

Table 2

Mean Instrument Phase Put-up and Checkride Scores (SDs in Parentheses)

Condition N Basic Basic Advanced Advanced

Instruments Instruments Instruments (AI) Instruments
(BI) Put-up (BI) Checkride Put-up (Al)'Checkride
Experimental 20 89.05 (3.33) 87.50 (7.42) 89.00(4.19) 85.35 (7.28)
(PST)
Control 18 89.33 (2.91) 88.83 (4.09) 86.11(4.90) 85.94 (6.80)
(SFTS)
 Hours to proficiency. The supplementary proficiency record was designed to capture, in
the IP's judgment, the hourly point at which the student pilot performed 14 BI and 12 AI
maneuver tasks to standard (a grade of B). As in the case of traditional flight grades, the
comparisons of proficiency times showed few significant differences between groups. Means,
SDs, and significance levels for BI tasks appear in Table 3; those for AI tasks appear in Table 4.
For the BI tasks listed in Table 3, standard rate turns, in emergency panel mode, showed a
significant difference favoring the SFTS. Climbs and descents, in emergency panel mode, also
showed a difference favoring the SFTS, but this did not reach conventional levels of significance.
Partial or emergency panel signifies a situation in which certain key instruments (e.g., attitude
indicator) fail, and the pilot must use the remaining instruments to control the aircraft. U-tests
performed on the AI maneuver tasks revealed no significant between-group differences in hours
to proficiency, though for all 12 maneuvers, PST students performed to standard in .2 to 3.0 hr
less time than did their SFTS counterparts, with an overall mean difference of 1.54 hr. Although
these differences were not significant, the time savings may be meaningful, since much of AI time
is aircraft time, which is costlier than time in the simulator. Still, it must be concluded that
students trained in the PST had no advantage over those trained in the SFTS, not supporting the
hypothesis that the former would attain proficiency in fewer hours. A caveat is in order: The two
significant differences in Table 3 could be so simply by chance, judging from their small number.
Table 3

Mean Hours to Proficiency for Basic Instruments (BI) Flight Maneuvers

Maneuver Condition Mean SD H<

Accelerate/Decelerate PST 9.43 1.57 .82
SFTS 9.29 1.98
Climbs/ Descents PST 5.90 2.06 .19
SFTS 7.04 2.43
Climbs/ Descents, Emergency PST 10.04 1.28 .06
Panel SFTS 9.19 1.61
Climbing/Descending Turn PST 9.45 1.61 .73
SFTS 9.62 2.19
Climbing/Descending Turn, PST 10.99 1.25 .10
Emergency Panel SFTS 9.93 2.27
Simulated Engine Failure PST 8.23 1.59 .13
SFTS 9.09 1.82
Standard Rate Turn PST 5.78 2.17 .33
SFTS 5.09 1.69
Standard Rate Turn, PST 9.72 1.50 .005
Emergency Panel SFTS 8.24 1.52
Steep Turn PST 7.30 2.20 .36
SFTS 7.93 2.30
Straight and Level Flight PST 4.38 1.13 .09
SFTS 3.95 1.11
Straight and Level Flight, PST 8.43 1.99 .19
Emergency Panel SFTS 7.71 1.13
Timed Turn PST 6.55 2.03 .93
SFTS 6.68 2.17
Unusual Attitude Recovery PST 7.18 1.39 .17
SFTS 8.07 1.81
Unusual Attitude Recovery, PST 9.68 1.14 .15
Emergency Panel SFTS 9.10 1.49

10
Table 4

Mean Hours to Proficiency for Advanced Instruments (AI) Flight Maneuvers

Maneuver Condition Mean SD E<

En Route Navigation PST 33.57 3.75 .18
SFTS 34.98 4.89
Instrument Landing System PST 38.38 4.43 .94
(ILS) Approach SFTS 38.57 6.27
Instrument Takeoff PST 35.42 3.64 .28
SFTS 36.53 3.79
Localizer Approach PST 38.60 4.69 .14
SFTS 40.78 4.63
Localizer Holding PST 39.30 5.55 .57
SFTS 40.26 6.12
Lost Communication PST 36.54 5.97 .29
SFTS 38.77 4.28
Non Directional Beacon (NDB) PST 37.09 5.69 .31
Approach SFTS 38.94 6.34
Non Directional Beacon (NDB) PST 37.08 3.44 .70
Holding SFTS 38.44 5.83
Precision Approach PST 36.49 5.37 .35
SFTS 38.06 4.89
Radio Communication PST 35.67 3.89 .13
SFTS 38.72 6.04
VHF Omni Directional Radio PST 34.92 3.34 .33
(VOR) Missed Approach SFTS 36.62 4.50
VOR Holding PST 37.85 4.20 .73
SFTS 38.74 5.56

Student postexperimental questionnaire responses. Recall that student pilots were

administered a questionnaire in order to assess their perceptions of the training effectiveness of
their respective simulators. They were also given the opportunity to provide additional evaluative
input via an open-ended question at the end of the questionnaire. The reader should note that the
postexperimental questionnaire does not include all of the same tasks that were sampled as
measures of proficiency. AI tasks in particular, are subsumed under broader categories. This was
done primarily to keep the number of questionnaire items at a manageable level, to insure a high
rate of compliance. Means and standard deviations for each item rated appear in Table 5. All
items are presented in the order in which they appeared on the questionnaire. One item,
pertaining to the adequacy of the visual display system on the PST, was omitted because its
ambiguity led to widespread misinterpretation. An examination of these responses shows that, in
general, students rated both the PST and SFTS as training-effective, though the PST tended to
enjoy more favorable ratings (though not significantly different in most instances). The most

11
noteworthy finding, for purposes of this research, is the significant tendency of SFTS participants
to indicate that training in the device hindered their performance in the TH-67 (item 7; see
Appendix B), as opposed to their PST counterparts.

Table 5

Mean Student Questionnaire Ratings (6-pt. Scale)1 by Item Content

Description PST SFTS j£l.

Mean SD Mean SD
Attitudes Toward Simulation
Simulation is a useful tool. 5.45 .69 5.33 .67 .56
Simulation saves time in the aircraft. 5.30 .87 5.17 .92 .65
Simulator can't be built that handles like aircraft. 4.05 1.00 3.78 1.00 .50
Time saved in aircraft from simulation is negligible. 2.95 1.01 2.89 1.02 .84
Tasks learned in a simple simulator will transfer to 4.95 .51 5.11 .76 .42
aircraft.
Some things learned in the simulator hindered my 2.55 1.43 3.56 1.38 .03
performance in the aircraft
Simulator Effectiveness Ratings: Basic Instruments
Straight and level flight. 5.40 .68 4.67 1.19 .03
Timed turns. 5.20 .83 4.72 1.18 .15
Steep turns. 5.00 1.08 4.83 .79 .30
Climbs and descents. 5.05 .76 4.67 .97 .24
Trim control. 2.45 1.79 3.61 1.46 .02
Standard rate turns. 5.00 1.12 4.89 .90 .51
Climbing turns. 4.80 1.06 4.89 .90 .24
Acceleration/ Deceleration. 5.00 1.03 4.61 .85 .13
Descending turns. 4.85 .99 4.83 .86 .89
Simulated engine failure at altitude. 4.80 1.00 4.65 1.06 .68
Unusual attitude recovery 5.30 .87 5.17 .79 .54
Simulator Effectiveness Ratings: Advanced Instruments
Instrument takeoff. 4.45 1.47 3.38 1.54 .03
Missed approaches. 5.15 1.09 4.72 1.02 .14
Holds. 5.45 .76 4.78 1.31 .07
Instrument approaches. 5.55 .76 4.72 1.27 .02

1. A high rating indicates strong agreement or that the simulator was very effective.
2. Comparisons were via Mann-Whitney U test.

12
 In addition to the Likert-based scaled question items, space was provided for comments
at the end of the questionnaire. A total of nine control group and 13 experimental group
participants responded with open-ended comments. Table 6 presents a summary of the comments
by category. The reader should be aware that most participants who made spontaneous
comments made several. A glance at this table shows that, among participants who trained in the
PST, problems with trim control was the foremost concern. The second most frequently
mentioned deficiency was the difficulty managing power, especially in turns. Collective pitch
control adjustments were seen as challenging. Three participants remarked that, at the very least,
a functioning horizontal situation indicator (HSI) was needed on the copilot's (left) side of the
cockpit. Other comments concerned what seemed like a fixed-wing aerodynamic model, and the
lack of a motion cueing system. In spite of the perceived deficiencies, there were six spontaneous
laudatory comments about the Frasca PST. These took the form of having enjoyed the training
experience, and how the Frasca PST had the potential of becoming an outstanding training device,
if the indicated problems were fixed. There were also positive comments on the visual display
system, and its potential as a confidence-builder for Instrument Meteorological Conditions (IMC)
final approaches and break-outs. By contrast, most spontaneous comments regarding the SFTS
were concerned with those differences, primarily instrumentation and aerodynamic model, which
hindered transfer of instrument skills to the aircraft. Most of these concerned the necessity of
relearning navigation, radio, and instrument cross-check procedures in the TH-67 on the
flightline. One participant believed that adding a visual display would greatly enhance the SFTS's
effectiveness. There were three positive statements about the training effectiveness of the SFTS.

Table 6

Content Categories of Spontaneous Comments of Participants on Postexperimental Questionnaire

Simulator Content Category Number of

Mentions
PST Problems with trim control. 11
General positive comments about simulator effectiveness 6
and training experience.
Visuals were helpful. 5
Avionics, instruments transfer to TH-67. 5
Power adjustments difficult, especially in turns. 5
Need HSI on copilot side also. 3
Simulator flies like fixed-wing aircraft. 2
Simulator should have motion cueing system. 1
SFTS Simulator too different in control touch, instruments, from 8
TH-67.
Simulator good for teaching basic principles; had very little 3
trouble adjusting to TH-67.
Visual display would be helpful. 1

13
 Instructor pilot questionnaire responses. The four IPs, who trained student pilots for the
duration of the study, were administered essentially the same postexperimental questionnaire.
Minor semantic changes were made to it, so that the perspective was that of instructor rather than
student. The small number of responses, plus the fact that these were within- and not between-
subject ratings, renders statistical tests of differences impractical. Nevertheless, a side-by-side
comparison of IPs' and students' evaluations of the two training devices may be useful. Table 7
compares IP and student pilot attitudes toward simulation as a training tool, based upon responses
to these five questions in the post questionnaire. Table 8 presents the means and standard
deviations for IPs' ratings of training effectiveness, comparing the two simulators. The first five
questions, which concerned the respondents' attitudes toward simulation in general, and were
irrelevant to comparison between the simulators, were only asked once. Table 9 is a supplement
to Table 8, comparing directly the ratings given by IPs to those given by student pilots (SPs). An
examination of Table 9 shows that student pilots tended to be more favorable toward the Frasca
PST and less favorable toward the SFTS, than did instructors. It is also quite evident that the
main perceived deficiency of the PST involved trim control.

Table 7

Comparison of Instructor Pilot (IP) vs Student Pilot (SP) Attitudes toward Simulation
(6-pt. Scale)1

Description IP SP
Mean SD Mean SD
Simulation is a useful tool. 6.00 .00 5.39 .68

Simulation saves time in the aircraft. 5.50 1.00 5.24 .88

Simulator can't be built that handles like 5.25 .96 3.92 1.00
the aircraft.
Time saved in aircraft from simulation is 2.00 .82 2.92 1.05
negligible.
Tasks learned in a simple simulator will 5.00 .00 5.03 .64
transfer to the aircraft

1. A high rating indicates strong agreement with the item.

14
Table 8

Mean Instructor Pilot Ratings by Item Content (6-pt. Scale)

Description PST SFTS

Mean SD Mean SD
General Attitude Toward the Simu ator
Some things students learned in simulator
hindered performance in aircraft 5.50 .58 4.75 .50
Simulator Effectiveness Ratings: Basic Instruments
Straight and level flight. 5.50 .58 5.25 .50
Timed turns. 4.25 2.22 5.25 .50

Steep turns. 4.25 2.22 5.25 .50

Climbs and descents. 4.25 .96 5.25 .50
Trim control. 1.00 .00 4.75 .50
Standard rate turns. 3.75 2.22 5.25 .50
Climbing turns. 3.00 1.63 5.25 .50
Acceleration/ Deceleration. 4.25 .96 5.25 .50
Descending turns. 3.00 1.63 5.25 .50
Simulated engine failure at altitude. 5.25 .50 5.00 .82
Unusual attitude recovery. 5.25 .50 5.25 .50
Simulator Effectiveness Ratings: Advanced Instruments
Instrument takeoff. 5.50 .58 3.75 1.89
Missed approaches. 6.00 .00 5.00 .82
Holds. 5.75 .50 5.00 .82
Instrument approaches. 6.00 .00 5.50 .50

1. A high rating indicates strong agreement or that the simulator was very effective.

15
Table 9

Instructor Pilot (IP) vs. Student Pilot (SP) Ratings of Simulator Effectiveness
by Item Content, for All Maneuvers (6-pt. Scale)1

Description PST SFTS

Mean Mean
IPs SPs IPs SPs

Some things learned in simulator 5.50 2.55 4.75 3.56

hindered performance in aircraft.
Straight and level flight. 5.50 5.40 5.25 4.67
Timed turns. 4.25 5.20 5.25 4.72

Steep turns. 4.25 5.00 5.25 4.83

Climbs and descents. 4.25 5.05 5.25 4.67
Trim control. 1.00 2.45 4.75 3.61
Standard rate turns. 3.75 5.00 5.25 4.89
Climbing turns. 3.00 4.80 5.25 4.89
Acceleration/ Deceleration. 4.25 5.00 5.25 4.61
Descending turns. 3.00 4.85 5.25 4.83
Simulated engine failure at altitude. 5.25 4.80 5.00 4.65
Unusual attitude recovery. 5.25 5.30 5.25 5.17
Instrument takeoff. 5.50 4.45 3.75 3.38
Missed approaches. 6.00 5.15 5.00 4.72
Holds. 5.75 5.45 5.00 4.78
Instrument approaches. 6.00 5.55 5.50 4.72

1. A high rating indicates strong agreement or that the simulator was very effective.

Instructor pilot spontaneous comments. IPs did provide some comments to the open-
ended question at the end of the questionnaire. Of the four IPs who completed the questionnaire,
two provided this input. Table 10 presents these responses by category. In spite of the small
number of responses, we can see some similarity to students' perceived strengths and weaknesses
of the two simulators. The Frasca PST was seen as a good proof of the concept of a low-cost
simulator, but one in need of improvement, especially with regard to the software flight model and
trim control. These deficiencies were seen as having diminished its effectiveness in the study. On
the other hand, IPs perceived the 2B24 SFTS as a good BI trainer whose AI usefulness was
hindered by a lack of additional navigation radios and instruments, and the general dissimilarity in
configuration of its cockpit to that of the TH-67. One P mentioned that this was initially
confusing to students, who eventually were able to readjust to the TH-67's instrument array.

16
Table 10

Content Categories of Spontaneous Comments of Instructor Pilots on Postexperimental

Questionnaire

Simulator Content Category Number of

Mentions
PST Flight model is of poor quality.
Not having an operating trim indicator hindered experiment.
RMI and HSI should be duplicated on left side of cockpit.
Visual display system benefits training.
Visuals, terrain database need to be more accurate.
Having cockpit similar to TH-67 benefits training.
Instructor Operator Station needs to be more accessible
SFTS Does not have VHF navigation radios.
A great BI trainer, but not for AI.
Cannot teach instrument takeoff.
Students initially confused by differences in instruments, but
catch up to PST students after about seven flights.

Use of desktop simulators. Four of the 38 participants reported having used a desktop
flight simulator within the past year. This reported rate of usage is lower than expected by the
investigators, when compared with the results of a study by Dunlap and Tarr (1999), who found
that 47% of U.S. Navy primary flight training students reported having used PC-based simulation
software. The low rate of self-reported usage in the present study precludes any meaningful
comparisons between users and nonusers. This question was asked because USAAVNC is
interested in the possible benefits of pretraining on commercially available flight simulation
software.

Supplemental correlational analyses. In addition to the between-group comparisons,

correlational analyses were run on the time to proficiency data, with BI and AI put-up and
checkride grades as the criteria. This was essentially an exploratory data analysis, because
no prior hypotheses were entertained as to which, if any, BI and AI maneuver tasks would
correlate the most highly (or at all) with the criteria. The reader should also note that the
measurement of these two variables was quite different; for the maneuvers, it was the training
IP's estimate of how many hours were required for the student pilot to reach proficiency on a
given task; for the checkrides, it was the overall numerical score that the SIP gave the student.
One possible benefit of these analyses would be insight into which maneuver tasks showed the
strongest relationship to graded checkride performance. It should be recalled that checkrides are
administered by U.S. Army IPs, whereas the daily performance evaluations are performed by
contractor IPs; hence, the two are independent measures of student performance. Table 11
presents correlations for BI maneuver tasks on BI and AI checkride grades; Table 12 presents
the same for AI maneuver tasks.

17
Table 11

Pearson Correlations of Times to Proficiency on Basic Instruments (BI) Maneuver Tasks

with BI and Advanced Instruments (AI) Checkride Scores (N = 38)

Grades BI Checkride AI Checkride

Score Score
AI checkride score. .04 1.00
BI put up score. .50b .43b
AI put up score. .20 .30
BI average time to proficiency (grand mean). -.62b -.40b
AI average time to proficiency (grand mean). -.18 -.38a
Maneuver Tasks
Acceleration/deceleration. -.23 -.30
Climbing/descending turn. -.30 -.38a
Climbing/descending turn, emergency panel. -.27 -.24
Climbs and descents. -.43b -.36a
Climbs/descents, emergency panel. -.32a -.34a
Simulated engine failure at altitude. -.38a -.31
Standard rate turn. -.44b -.31
Standard rate turn, emergency panel. -.09 -.36a
Steep turns. -.01 -.47b
Straight and level flight. -.52b -.23
Straight and level flight, emergency panel. .36a -.27
Timed turns. -.51b .12
Unusual attitude recovery. -.39a -.10
Unusual attitude recovery, emergency panel. -.26 -.43b

Note, a = p < .05; b = p<.01

An examination of Table 11 reveals that BI times to proficiency correlated significantly

with BI checkride scores for eight of the 14 tasks. All correlations were in the consistent
direction with one exception; the significant positive correlation between straight and level flight
(emergency panel) and BI checkride grade appears to be somewhat counter-intuitive, in the sense
that the longer it took a student to master the task, the higher was his or her BI grade. This
finding could be due to the fact that this is the first of the emergency panel maneuvers, and that it
simply might take longer to master the first time. Other correlations were in the expected
direction, indicating that the sooner a student pilot demonstrated proficiency on a task, the higher
he or she was likely to score on the BI checkride. Note that six of the BI times to proficiency for
certain maneuvers also correlated significantly with scores on the end of phase AI checkride. It is
also noteworthy that the average times to proficiency (the grand mean) of all these maneuver
tasks were highly correlated with both checkride scores. However, the same (grand mean)
average times for AI maneuvers only correlated with scores on the AI checkride.

18
 Also of interest is the lack of a correlation between BI and AI checkride scores; in fact,
the correlation is effectively zero. The reason for this lack of correlation is not simple to explain.
One possibility could be the early timing of the BI checkride, which evaluates skills that have only
recently been mastered, and which are the building blocks to the more complex AI skills to be
acquired later. Thus, the BI skills may be assessed before they have had time to become fully
integrated. Another, more mundane, explanation could be the weather, which obviously affects
performance in the aircraft and not the simulator. Students in this study from one class in
particular, after scoring high on their BI evaluations, had to complete their AI checkrides under
marginal weather conditions. As a result, students with BI scores in the 90s received AI scores in
the 80s, which were adequate to pass, but nonetheless disappointing. SIPs indicated that winds
and poor weather conditions that day, adversely affected performance of all students who took a
checkride.

Table 12

Pearson Correlations of Times to Proficiency on Advanced Instruments (AI) Maneuver Tasks

with Basic Instruments (BI) and AI Checkride Scores (N = 38)

Maneuver Tasks BI Checkride AI Checkride

Score Score
En route navigation. .10 .23
ILS approach. .13 .18
Instrument takeoff. .16 .44b
Localizer approach. .05 -.10
Localizer holding. -.23 -.10
Lost communication procedures. .17 .17
NDB approach. .10 -.25
NDB holding. -.11 .45b
Precision approach. .25 .10
Radio communication procedures. .06 -.44b
VOR holding. .08 .43b
VOR missed approach. -.03 .11

Note. b = p_<.01

Table 12 shows quite a different picture for the AI tasks. None of the 12 AI maneuvers
correlated significantly with the BI checkride scores, whereas four maneuvers correlated
significantly with the AI checkride scores. Even so, most of the correlations, for both BI and AI
times to proficiency, seem internally consistent and do show some potential of utility as
performance measures. The grand mean times to proficiency for both BI and AI maneuvers
tended to correlate strongly and consistently with their respective checkride scores. Although
these measures may have utility for predicting checkride performance, a much larger sample size
than the current one would be required to explore this possibility. Hence, any further post hoc
correlational or regression analyses would be beyond the scope of this report.

19
 Discussion

Training Effectiveness of the PST

The hypothesis that instrument skills learned in the PST would show stronger transfer to
the aircraft than those trained in the SFTS, was not confirmed. Instead, students were able to
complete Instrument Phase successfully, regardless of the simulator type. Data collected as part
of the training effectiveness assessment imply that the PST, with modifications, could be
successfully employed as an instrument trainer for U.S. Army EERW student pilots. No student
pilots who trained in the PST were set back to later classes, nor were any eliminated from flight
training. The results of the study do not show that the PST has a clear advantage over the SFTS
as to training outcomes. This could be attributed to at least three factors: First, the measures of
performance, especially the traditional flight grades and checkride scores, may have been
insensitive to any differences that existed between the two simulators (see Dohme, 1995, for a
discussion of the limitations of flight grades). Secondly, the previously mentioned technical
problems with the Frasca PST may have countered any benefits inherent in using a device that
was more similar to the aircraft. Third, the POI used for this research project was designed for
the SFTS, a non-visual simulator; no maneuvers were attempted in the PST that could not be
performed in the SFTS. Obviously, the Army would not find these problems acceptable in a
production device, and would require remediation before such a device could be fielded. All of
these things considered, it would still seem that such a PC-based training device, by virtue of its
greater economy of operation and maintenance, would be a viable candidate for replacing the
SFTS. Furthermore, since the PST is a visual simulator, it would be reasonable to suppose that,
were the Army to acquire this type of device, it would find training applications beyond the
training of instrument flight skills. Effective exploitation of contact DBRW training would require
upgrades to the software flight model and perhaps to the terrain database as well.

Assessment by Target Audience

An important part of the evaluation of the PST was the collection of input from the
students who trained in it, and the IPs who would be conducting the training in this type of
simulator. These respondents comprise a sample of those who will be using the device in the
future. Students were generally more positive in their evaluations of the PST than the SFTS
(though not significantly so, for most items) in spite of the perceived handling problems of PST.
By contrast, IPs were more ambivalent, favoring the SFTS for BI, but acknowledging that the
PST had advantages for AI training. There was consensus among IPs, that the primary deficiency
of the PST was poor trim control. This high degree of agreement was reflected in a mean rating
of 1 on a 6-point scale, with a standard deviation of 0. Likewise, student pilots assigned to the
PST gave it lower ratings on trim control than those assigned to the SFTS; in fact, this was the
one questionnaire item on which they gave significantly higher ratings to the SFTS. This leaves
little doubt that the Frasca PST had a serious trim problem that, in the experience of both IPs and
students, diminished its training effectiveness.

20
 One perceived advantage of the PST was the similarity in the location of the basic flight
instruments and radios to those in the TH-67, obviating a relearning of instrument scan patterns
after returning to the aircraft. The dissimilarity in scan patterns, and the necessity of relearning
the correct ones, was mentioned frequently by students who had trained in the SFTS. Another
perceived advantage for the PST was the presence of a visual display system. During an IMC
approach, the pilot must be able to acquire the runway visually in order to terminate to a landing.
This can only be done in a simulator with a visual display system. Both IPs and students believed
that breaking out of the overcast, to see the runway in the correct location, was reinforcing and
tended to build self-confidence. This finding is important in light of the fact that historically,
instrument trainers and simulators have not had visual display systems, because of a lack of a
perceived need for them. The inclusion of a visual display system also imparts versatility beyond
instrument training.

Recall that the most common complaints concerning the PST were its poor trim control
and its anomalous responses to pitch and power changes. The ARI research team was unable to
resolve all the software issues related to these problems, which forced the IPs to work around
them during training. Likewise, the calibration of the attitude indicator and turn indicator in the
PST were determined to be nonstandard, adding to the challenge. These factors made it difficult
to perform the standard rate turns, climbs, and descents that must be learned during the first nine
hours of Instrument Phase. The IPs understood that the PST was a prototype proof-of-concept
simulator and that these problems would have to be resolved in a production simulator. In brief,
it seems that the Frasca 342 PST was a potentially effective training device marred by a poor
software aerodynamic model.

Conclusions

The reader should not lose sight of the fact that this was a comparison of two simulation
technologies, not just two simulators. In this sense, it was shown that a PC-based simulation
environment can train Instrument Phase tasks successfully. One important implication is the
functionality of a future generation of training simulators, in terms of what is required to train
IERW tasks. This research project demonstrated that student pilots could receive the full
simulator portion of Instrument Phase training, in a PC-based, non-motion simulator with a simple
visual display system. Such a simulator would also be more economical to operate and maintain
than the one currently employed. Secondly, it provided evaluative input from the student pilots
themselves, as to its perceived effectiveness for training various IERW flight maneuvers. The
study uncovered technical problems with the prototype training device, which, though they likely
detracted from its effectiveness as an instrument simulator, pinpointed issues that must be
resolved before any device of this type is acquired by the U.S. Army. Should the Army decide to
acquire a PST-like simulator in the future, it appears that such a device would be usable for
training BI and AI skills, with the proviso that the software aerodynamic model, and
instrumentation, represent the TH-67, both physically and functionally. In the current state of the
art for PC technology, this challenge can easily be met.

One encouraging finding was that, regardless of the simulator in which students trained,
the time required to reach proficiency in the simulator on BI tasks, in many cases correlated

21
significantly with the scores received on the AI checkride. This imparts some degree of validity to
the employment of proficiency-based criteria instead of traditional flight grades as indices of
mastery. It also lends support to Dohme's (1995) position on the superiority of training to
standard to lock-step, hours-based training. Future investigation of the sensitivity and validity of
proficiency-based measures of performance would seem a worthwhile undertaking, in light of
these findings.

The investigators found one aspect of the research to be very difficult. This was finding
reliable and comparable cost figures for direct comparison of the PST and the SFTS. It is obvious
from its complexity, age, and maintenance burden that the SFTS is a more expensive device to
operate. Its operation requires a large, climate-controlled building, and hydraulics that generate
heat, consume power, and require frequent maintenance. The PST, on the other hand, consists of
four Pentium 75 computers, plus one 486 DX 66. A more current version would have more up to
date PC equipment, which could easily be expanded and upgraded. This could include a more
current visual display system. Thirty-two PST-like training simulators could be located in any
institutional building with sufficient floor space, and would not require additional climate control
equipment, outside ofthat needed for the comfort of the occupants. Civilian computer specialists
could maintain the PSTs, with little additional training. This stands in contrast to the
specialization required to support the SFTS. It should go without saying, then, that cost
advantages could be realized by acquiring PC-based simulators, with no detriment in training
effectiveness. Each SFTS also requires a specialized console operator, who can either be a
contractor, an Army civilian, or an enlisted soldier. The IP, who is in the simulator cab, cannot
access the controls on the IOS, due to his or her physical location. By contrast, the PST has a
very simple IOS, located on a table behind the cab, consisting of one PC, with an interactive
program that can be mastered by the IP and the student.

One final difference in the two simulators should be noted. The SFTS is a dedicated
instrument simulator, and, without an upgrade, cannot support Primary Phase flight skills training.
By contrast, the PST used in this study was equipped with a simple, forward-projection visual
display, which could potentially support some visual flight maneuvers. The PST display
technology is dated. Recently, ARI has integrated a current generation of PC-based displays
using rear-projection technology that are far superior. At this writing, an effective EERW
instrument training simulator could consist of the following components: A high-end PC-based
three-to-five-channel imaging system (1024 x 768 pixels), with 60 Hz display update rate, three to
five 2.5 m rear-projection visual displays, a PC or minicomputer-based host computer, a fully-
populated IFR cockpit shell, and control-loading system. No motion system would be needed;
however, a simple seat-shaker may be desirable. A rough unit cost estimate for such a simulator
would be approximately $1,000,000.

An added benefit of such a system would be its potential for training a much broader range
of flight maneuver tasks than the SFTS. Exactly what tasks remains an empirical question. A
future research project, employing such an updated simulator, could determine the range of flight
skills, both Primary and Instrument Phase, which could be trained.

22
 References

Caro, P. W. (1972). Transfer of instalment flight training and the Synthetic Flight Training
System. (Prof. Pap. 7-72). Alexandria, VA: Human Resources Research Organization.

Caro, P. W., Jolley, O. B., Isley, R ,N., & Wright, R. H. (1972). Determining training device
requirements in fixed-wing aviator training (Tech. Rep. 72-11). Alexandria, VA: Human
Resources Research Organization.

Caro, P. W. (1988). Flight training and simulation. In Weiner, E. L., & Nagel, D. C. (Eds.)
Human factors in aviation (pp. 229-260). New York: Academic Press.

Dohme, J. A. (1991). Transfer of training from a low-cost helicopter simulator. Paper

presented at the Sixth International Symposium on Aviation Psychology, Columbus,
OH, 29 April-4 May.

Dohme, J. A. (1995). The military quest for flight training effectiveness. In Larsen, W.E.,
Rändle, R.J., & Popish, L.N. (Eds.) Vertical flight training, (pp. 87-130). NASA Reference
Publication 1373 (DOT/FAA/CT-94/83). Moffett Field, CA: National Aeronautics
and Space Administration.

Dunlap, S., & Tarr, R. (1999). Micro-simulator systems for immersive learning
environments. Proceedings of the 21st Interservice/Industry Training Simulation and
Education Conference (pp. 506-511). Orlando, FL: I/ITSEC

Hays, W.L. (1973). Statistics for the social sciences (pp. 778-780). New York: Holt, Rinehart
& Winston.

Prophet, W. W., & Boyd, H. A. (1970). Device-task fidelity and transfer of training: Aircraft
cockpit procedures training (Tec. Rep. 70-10). Alexandria, VA: Human Resources
Research Organization.

United States Army Aviation Center (2001, February). Flight training guide. TH-67 Initial Entry
Rotary Wing Aviator Course: Instrument Phase. Fort Rucker, AL

23
Appendix A

Student Proficiency Records

AR] IERW Research Project: Simulator Proficiency Record: Basic Instruments

SP Name Rank SSN

Instructor Date

Training Device (Check One) Frasca PST 2B24 SFTS

Hours to Proficiency
Please indicate for each task the Hours at which the SP first demonstrated proficiency. If SP did
not show proficiency on the task during the simulator training, please indicate "Not demonstrated"
Basic Instruments Tasks Full Panel Emer°encv Panel

Straight/Level Flight
Standard Rate Turns
Compass Turns
Timed Turns
Climbing/Descending Turns
Steep Turns
Accelerate/ Decelerate
SEF at Altitude
Climbs/Descents
Unusual Attitude Recovery
Trim Control
Comments:

A-l
Appendix A (Continued)

AR1IERW Research Project: Aircraft Proficiency Record: Advanced Instruments

SP Name Rank SSN

Instructor Date

Training Device (Check One) Frasca PST 2B24 SFTS

Hours to Proficiency (i.e., Standard) in the TH-67 Aircraft

Please indicate for each task the Hours at which the SP first demonstrated proficiency in the
aircraft, during the IA Phase of BERW.
Advanced Instrument Tasks Hours to Proficiency Comments

Instrument Takeoff
Radio Communication
Missed Approach

Radio Navigation
Lost Communication
Holding:
NDB
VOR
LOC
Instrument Approach:
ADF
PAR
ILS
LOC
Emergency Procedures
Comments:

A-2
Appendix B

TH-67 Frasca Primary Skills Trainer (PST) Questionnaire (SFTS questionnaire was identical
except for simulator named)

Administer after completion of Training

Name "Last Four"

We would like to ask you some questions about your perceptions regarding simulation in general,
and your experience in the Frasca TH-67 Primary Skills Trainer in particular. We are asking for
your name and the last four digits of your SSN for data analysis purposes only. Your answers
to the questions will reported as aggregated data along with the averages of other people who
completed the questionnaire. Your particular responses will not be identifiable.

On the questions that follow, Please indicate your impressions by placing an X in the appropriate
box on the rating scale. We appreciate your cooperation in completing the questionnaire.

PART I: GENERAL. The following questions pertain to simulation in general, and your
perceptions of its role in rotary wing training.

1. All in all, I believe that simulation is an effective tool for initial flight training.

Strongly Agree Agree Disagree Disagree Strongly

Agree Somewhat Somewhat Disagree

2. Simulation is a good investment in that it saves training time in the aircraft.

Strongly Agree Agree Disagree Disagree Strongly

Agree Somewhat Somewhat Disagree

3. In spite of all the technology, a simulator can't be built that handles like a real aircraft.

Strongly Agree Agree Disagree Disagree Strongly

Agree Somewhat Somewhat Disagree

1-1
Appendix B (Continued)

4. Use of simulators may save some flight training hours, but this is generally a
negligible amount.

Strongly Agree Agree Disagree Disagree Strongly

Agree Somewhat Somewhat Disagree

5. Skills acquired, even in a simple simulator, should transfer to the aircraft.

Strongly Agree Agree Disagree Disagree Strongly

Agree Somewhat Somewhat Disagree

PART II: EVALUATION OF THE SIMULATION EXERCISE

We are interested in your perceptions of the effectiveness of the TH-67 Primary Skills
Trainer (PST) in the context of the training exercise, which you just completed. Your responses
to the following questions would be of great value to us.

6. I believe that that the out-the-window view of the PST was adequate for Basic Instrument
training. (IF THE VISUAL DISPLAY WAS NOT USED, INDICATE "NOT APPLICABLE")

Strongly Agree Agree Disagree Disagree Strongly Not

Agree Somewhat Somewhat Disagree Applicable

7. I believe that some of the techniques that I had to learn to fly the PST hindered my
performance in the TH-67 aircraft.

Strongly Agree Agree Disagree Disagree Strongly

Agree Somewhat Somewhat Disagree

PART III: Basic Instruments

The following questions concern the extent to which you believe that Basic Instruments training in
the Frasca PST affected your performance in the TH-67 aircraft. For those question items below,
please indicate the degree to which you think the PST was/was not helpful in doing this, for those
maneuvers listed below.

B-2
Appendix B (Continued)

8. Straight/Level Flight

Very Helpful Helpful Somewhat Somewhat a A Hindrance Very Much a

Helpful Hindrance Hindrance

9. Timed Turns

Very Helpful Helpful Somewhat Somewhat a A Hindrance Very Much a

Helpful Hindrance Hindrance

10. Steep Turns

Very Helpful Helpful Somewhat Somewhat a A Hindrance Very Much a

Helpful Hindrance Hindrance

11. Climbs/Descents

Very Helpful Helpful Somewhat Somewhat a A Hindrance Very Much a

Helpful Hindrance Hindrance

12. Trim Control

Very Helpful Helpful Somewhat Somewhat a A Hindrance Very Much a

Helpful Hindrance Hindrance

13. Standard Rate Turns

Very Helpful Helpful Somewhat Somewhat a A Hindrance Very Much a

Helpful Hindrance Hindrance

14. Climbing Turns

Very Helpful Helpful Somewhat Somewhat a A Hindrance Very Much a

Helpful Hindrance Hindrance

B-3
Appendix B (Continued)

15. Acceleration/Deceleration

Very Helpful Helpful Somewhat Somewhat a A Hindrance Very Much a

Helpful Hindrance Hindrance

16. Compass Turns

Very Helpful Helpful Somewhat Somewhat a A Hindrance Very Much a

Helpful Hindrance Hindrance

17. Descending Turns

Very Helpful Helpful Somewhat Somewhat a A Hindrance Very Much a

Helpful Hindrance Hindrance

18. Simulated Engine Failure/Altitude

Very Helpful Helpful Somewhat Somewhat a A Hindrance Very Much a

Helpful Hindrance Hindrance

19. Unusual Attitude Recovery

Very Helpful Helpful Somewhat Somewhat a A Hindrance Very Much a

Helpful Hindrance Hindrance

PART IV. Advanced Instruments

20. Instrument Takeoff

Very Helpful Helpful Somewhat Somewhat a A Hindrance Very Much a

Helpful Hindrance Hindrance

B-4
Appendix B (Continued)

21. Missed Approach

Very Helpful Helpful Somewhat Somewhat a A Hindrance Very Much a

Helpful Hindrance Hindrance

22. Holding: NDB, VOR, LOC

Very Helpful Helpful Somewhat Somewhat a A Hindrance Very Much a

Helpful Hindrance Hindrance

23. Instrument Approach

Very Helpful Helpful Somewhat Somewhat a A Hindrance Very Much a

Helpful Hindrance Hindrance

COMMENTS (OPTIONAL):

B-5
I .. '*

Appendix C

Student Background Questionnaire

U.S. Army Research Institute for the Behavioral and Social Sciences, (ARI) Fort Rucker Field
Unit

Use of Commercial Desktop Flight Simulator and Aviation-Related Games

We are interested in determining the extent to which IERW student pilots have had experience in the use
of PC-based flight simulation and aviation video games. While traditionally thought of as recreational, these
programs may be potentially valuable teaching tools. This information will be used strictly for research purposes
by ARI.

PART I: General Background Questions

Name IERW Class Today's Date_

Last Four Numbers from SSN Race DOB

Gender M F Rank. AFAST SCORE (estimate, if unknown)

Prior to IERW, how many flight hours have you had?

None

Hr fixed wing

Hr rotary wing

PART II: Use of Commercially-available desktop flight simulations (such as Microsoft Flight Simulator, Strike
Eagle, Falcon and RAH-66 Comanche).

In the past year, I have run a desktop flight simulator or aviation-related game on a PC.

Yes No

IF YES. Please indicate the name(s) of the program(s) and the approximate number of hours used.

Name of Program Approximate hours per month

C-l