Understanding Aviation Risk
Andrew Rose
Operational Analysis
NATS
Southampton, UK
andrew.rose@nats.co.uk
Abstract
The paper seeks to highlight the challenges facing the
aviation industry in the need to better understand and
predict operational risk. It looks at the types of data
available to improve the understanding of risk and
discusses the need to bring it together using a common
risk currency. The relationships between the different
types of data are briefly addressed in regard to both
understanding the current level of risk and predicting
what it will be in the future. The conclusions are that a
more comprehensive view of risk is required and that
the fusion of incident based data together with risk
exposure data provides a method for achieving this. The
paper highlights the potential for future aviation
regulation to be risk reduction based. The paper has
been generated as a result of effort being undertaken at
NATS to understand risk, but reflects it from a whole
aviation viewpoint.
Keywords: aviation safety, operational risk, data
association, risk assessment.
1 Introduction
Despite unparalleled growth in aviation, the industry has
maintained a steady decline in the number of accidents
each year [1]. The reasons for this success are
widespread, but undoubtedly the collection and intelligent
use of safety data has been a key factor.
The collection and intelligent use of safety data has been a
cornerstone in the management of safety in aviation. Over
the last decade aviation has developed extensive processes
for the collection of safety data and varying levels of
capability for its analysis. Analysis of this data has led to
a number of organizations adopting measures for the
monitoring of safe operations.
NATS, like others in the aviation community, are always
looking to make the business safer and in doing so have
recognised the need to have a more cohesive view of
operational risk. With the ever increasing demands of
capacity and environment, there is a need for operational
risk to become an equitable partner in the debate.
Through the development of its Strategic Plan for Safety
NATS has endeavored to generate predictions on future
risk within specific topic areas. This work has focused on
trying to predict the effect of system improvements and
has highlighted the need to better understand the
interactions between an array of operational risk
information.
This purpose of this paper is to describe the challenges
that the aviation industry faces in the better understanding
and prediction of risk. The paper attempts to outline the
activities that NATS is working on in this field and to
encourage the industry in joining together to meet the
challenges it highlights.
2 The challenges
As safety within the industry has improved, the
availability of simple lessons from accidents and incidents
has reduced. To sustain the successful safety trend the
industry has had to increasingly focus on the wider and
lower level sources of safety information.
There are a plethora of safety related data sources that are
available which could, and often do, guide the
management of aviation risk. Where they are used to
make risk decisions it is generally in an individual and
isolated way and without a clear understanding of their
relative importance. The problem to those with a high
level responsibility for ensuring aviation safety is to
identify the highest priorities to ensure that they are using
resources effectively.
The challenge this sets the industry is to bring this
complete range of safety related information together in a
cohesive way to better understand and manage its risks.
To achieve this requires the data to be fused together
through a common understanding of the represented risks
and their relative importance.
The value of such an understanding of risk within an
organisation is clear in that it will enable better informed
decisions in the management of risk. There is however a
much wider benefit to the aviation community if a
common understanding of risk can be developed.
1381
The ever increasing integration of aviation systems
demands a more effective understanding of risk across the
aviation environment. Decisions taken in one
organisation or one part of the domain have an
increasingly significant impact on risk within other parts
of the industry. It is only through a common
understanding of this overall risk that those decisions can
be made in an informed way.
Ultimately this leads to opportunities to manage the
regulation of aviation in a different way. A common
understanding and view of risk provides a measure to
ensure that the aviation system is controlling its risk and is
making changes that contribute to an overall risk
reduction.
The greatest value from collecting historical safety
performance data has to be in using it to look forward, to
predict what the risk will be in the future. Risk prediction
today is often at the level of looking at trends of safety
information and assuming those trends will continue.
This however does not recognise the many variables that
affect safety performance and does not enable effective
prediction of the affects of changes to the system. Efforts
to make better predictions of risk have highlighted the
need to understand the complex relationships between a
wide range of data.
3 Types of Data
To better understand the challenge of generating a
cohesive picture and prediction of risk requires
consideration of the range of data available.
3.1 Incident data
Incident data has always been the core to the aviation
industrys understanding of risk. Incident, or event based
data, is data that is generated on the basis of something
having occurred that is, or could be, an indication of the
unsafe operation of the system. Incident data can be
considered in three main categories.
3.1.1 System generated data
System generated data can be characterised by
information that is captured automatically as the operation
progresses. It does not require human intervention to
ensure that an event is captured and can therefore be
considered as highly reliable. Aviation has pioneered the
automatic collection and analysis of a wide range of data,
most significantly that of the aircraft flight data [2]. In the
air traffic service field there is a wide range of data
gathered automatically from the radar and radio systems.
Examples include: aircraft separation monitoring, aircraft
conflict alerting, unauthorised airspace incursion
monitoring and radio frequency congestion monitoring.
In addition the increasing use of automated and complex
systems brings with it a significant amount of data on
system availability or performance.
Assuming the integrity of the systems used to generate
these data sets are robust, these data sources can be
considered to be reliable and unbiased in their view of the
performance that they are measuring.
3.1.2 Human reporting data
Aviation has been at the forefront of open reporting
systems, encouraging employees to report incidents and
occurrences [3]. Processes have been put in place to
allow and encourage employees to report safety related
events. The data this generates is obviously dependent
upon the specific requirements for reporting, but is also
significantly impacted by a number of other factors.
The factors affecting human incident reporting data have
been well analysed but need to be carefully considered
when generating an overall view of risk [4]. A view of
risk based on purely incident reporting data is only the
risk that is made visible by the employees reporting it.
The factors effecting the reporting of incidents can be
significant and therefore obscure the real picture. In
essence you are only measuring the risk that the
employees choose, or are able, to report. Ultimately you
dont know what you dont know.
Clearly a group of employees with a mature safety culture
should provide a good level of reporting, however that
will still be influenced by personalities, organisational
politics and the physical difficulties in producing the
reports. Key factors to consider are the barriers to people
reporting as requested, particularly the difficulty of
reporting and the disinclination to report due to a culture
of blame, or perceived blame, within the organisation [5].
The data provided by employees reporting errors or
system deficiencies is extremely valuable and provides
probably some of the best risk information available
across aviation as a whole. It is therefore important to
consider ways to overcome the barriers to reporting and to
maximise the value from this data set. Possibilities for the
simplification of reporting include simple event capture
methods, where an employee makes a simple data entry to
flag a particular event. Although such a method does
not provide the richness of data available in a written
report, the improved coverage could significantly
outweigh that loss.
3.1.3 Sample data
Aviation makes extensive use of auditing to ensure the
safe adherence to procedures designed to control risks.
These audits provide a valuable indication of conformance
within the organisation and are used to control risks,
however they are rarely used to provide a better
1382
understanding of risk. Among other things an audit is
likely to identify events and occurrences that could, but
may not, form part of the reported understanding of risk.
An audit however is only a sample of the performance of
a particular part of the system and therefore to use it to
understand overall risk requires extrapolation across the
system. Obviously there are a number of factors that
influence how the findings in one area/time period relate
to another, but if understood then they can be used to
enhance the view of risk.
On a lower level there are other forms of sample data that
again provide opportunities to inform the view of risk
within the industry. The implementation of LOSA (Line
Operations Safety Audit) and other human performance
observations provide a rich source of safety information
that need to become part of an organisations
understanding of its risk.
3.2 Risk exposure data
Although incident or event based data has historically
been the mainstay of understanding aviation risk, there are
other data sets that are significant in determining the
overall risk to aviation. In its simplest form this may just
be data that measures how often you perform an activity
that is exposed to risk. It will however also include data
that has a much more complex relationship to the resultant
risk.
Traditionally some forms of exposure data have been
applied in a simplistic way to analyse their effect. For
example if a certain type of event tends to happen every X
flight hours then an assumption may be made that it
continues to happen on the same relative frequency as the
number of hours changes. In many cases this simple
relationship may be valid, or at least a useful
approximation to use. In other cases however the
relationship may be much more complex, where for
example an increase in flight hours creates a
disproportionate increase in human workload or decrease
in human performance.
There is potentially a vast array of potential exposure
information that could be of use in informing the
understanding of risk. The following table provides an
idea of some of the data that might be available to
consider in an air traffic management environment.
Traffic levels
Staffing levels
Employee experience and training
Types of traffic & complexity
Visibility conditions
System availability
System accuracy
RF frequency usage levels
Safety culture of the employees
It is only through understanding the relationships between
the data and risk that it can reliably be used to inform the
understanding of risk. Without that understanding there
can be no clear understanding of how much impact one
factor has compared to another. With such a wide range
of data the relationships involved are going to be complex
and difficult to accurately model. It is however likely that
simplified relationships could be defined to ensure a wider
and more effective understanding of risk.
Historical comparison will likely provide some
explanation of the relationships but a fuller understanding
may come from the perceptions of those directly involved
in the operation. Their perceptions of how the data
relates, although not necessarily complete, provide an
excellent basis for defining a high value relationships.
Another option may be to consider a perfect steady state
condition, one where the level of none of the factors
identified gave cause for concern. Any deviation from the
perfect state can then be considered as an event and
treated as described in section 3.1. The difficulty here is
that incident data generally consists of discrete events that
can have a risk associated with them. A deviation from
the norm can however be a prolonged situation and so
does not easily lend itself to the same type of analysis.
Despite the potential drawbacks of such an approach,
there is merit in examining such approaches in more
detail.
3.3 Risk mitigation effectiveness data
In a complex and high risk system such as aviation there
are many risk mitigation measures (barriers) put in place
to control risk. These measures are designed to ensure
that events that do occur do not develop into the serious
incidents and accidents that the system is aiming to avoid.
It is debatable whether the measures of effectiveness of
these barriers are any different from the measures of
exposure factors described above. In essence this
probably depends upon how the barriers are considered in
the system context. Not withstanding this, performance
data for barriers will be a valuable source of data in the
overall cohesive understanding of risk.
4 Common understanding of risk
To allow the wide sources of aviation safety data to be
used in an inclusive way requires a common
understanding of how they relate to each other.
The FAA (Federal Aviation Administration) has been
working on ways to combine multiple sources of aviation
safety data [6]. The approach here has been to use expert
judgment to provide a relative weighting to the different
data sources to enable them to be plotted on a common
1383
graph. This approach makes a significant step in bringing
data together to provide a common view and enables those
responsible for managing risk to see the trends of a range
of data in a comparable way. The drawback to such an
approach is that the simplification of the relationship
between the data sets on the basis of their source may hide
the relative risk that individual data points represent.
An alternative approach is to identify each piece of data
with a common, and measurable, safety currency. Once
the data has a common currency the relative importance
of each piece is predefined when they are combined.
Safety has been described as a construct and a concept [7]
and is therefore not a finitely measurable indicator. Risk,
defined as the measure of probability and impact of an
incident, is however a generally accepted quantifiable
measure. Although not necessarily comprehensively
defined across the industry, it is used to assess and
monitor safety performance [4]. The definition of risk as
adopted in this work is based upon the latest ICAO
thinking and is; the assessment, expressed in terms of
predicted probability and severity, of the consequence(s)
of a hazard taking as reference the worst foreseeable
situation.
The agreement and assignment of that risk value to safety
data is a major subject that would merit a paper in its own
right. It is however worthwhile covering it briefly in this
paper so that the difficulties in applying it to a wide range
of safety data can be seen.
4.1 Risk assignment concept
There is a range of risk weighting concepts and risk
matrices in use across the aviation industry [8]. The
outcome of work to review these different approaches has
been to formalise one approach that attempts to overcome
the many drawbacks identified in the process [8]. In
summary the proposal is to ask three discrete questions
about the event you wish to risk rate:
a) What could have been the likely worst case
outcome of this event?
b) What barriers were effective in ensuring that the
event did not reach the likely worst case
outcome?
c) How often would this event be likely occur?
The response to these three questions places the event at a
particular place in a risk matrix.
In the context of the analysis of the resulting risk data, as
described in this paper, the question regarding frequency
is not required. Individual events can be rated for their
own individual risk contribution; the contribution due to
their relative frequency is addressed by the summation of
their individual risks. The use of a frequency in the
assignment of individual risk leads to a distortion of its
overall contribution to risk when the individual events are
combined.
To overcome this problem, the proposal is to use only a
two dimensional matrix with questions about the potential
outcome and the barriers to prevent that outcome. An
example of such a matrix is shown in figure 1.
Major Accident with significant loss
of life
D C B A
Limited Accident scenario with low
potential for loss of life
D C C B
Minor Accident with some injuries
and damage
E E D C
Degradation of safety margins but
with little direct consequences
E E E D
Normal
Intervention
Non-normal
safety nets
Abnormal human
intervention
Providence (or
consequences
occurred)
Defences that avoided
consequences
Reasonable worst casepotential
consequences
Figure 1.
This matrix provides the framework on which to assess
the risk of any incident or event on a similar scale. The
selection of a row and column based upon the answers to
the questions raised provides for a risk category (A to E)
to enable the incident to be prioritised. To enable it to be
used in a number of different domains within aviation, for
example for aircraft operators, air traffic service providers
and airport operators, guidance can be given for each
category that matches the types of events and risks that
manifest themselves in that domain.
The risk category enables prioritisation and qualitative
analysis to be performed. To enable quantative analysis
each of the risk categories can be numerically weighted to
allow it to be combined to give an overall risk value [4].
An example set of risk weightings is shown in figure 2. In
this example matrix an A risk event is worth 100 and an E
risk worth between 1 and 10 depending upon its location
in the matrix. These values are examples and would need
to be adjusted through statistical and expert analysis.
D (15) C (50) B (85) A (100)
D (12) C (40) C (60) B (90)
E (8) E (10) D (30) C (60)
E (1) E (5) E (10) D (15)
Figure 2.
1384
5 Data relationships
5.1 Current risk
The application of a common risk currency to all your
available sources of incident data provides a means to
generate one view of the current, and historical, level of
risk. The incident data that forms this view can either be
taken as it stands and accepted as a partial view, but one
based on all the incident data available, or could be
extrapolated to provide a fuller view of the risk across the
organisation.
As highlighted in section 3.1.2 human reporting data is
subject to a number of pressures and issues, not least of
which is the reporting culture of the employees who are
expected to raise the reports. Employee safety culture
assessments provide one indication of that culture [9] and
could therefore be used as a factor to overcome low or
unreliable incident reporting rates within particular parts
of an organisation.
Data on events that is obtained as part of a sampling
activity also requires extrapolation to give a better view of
overall risk within the organisation. To do such
extrapolation requires an understanding of the relationship
that the event has to the types of exposure data available.
In its simplest form that relationship could just be a direct
relationship between the number of reported events and
the overall quantity of the activity being undertaken. For
example the number of misheard radio transmissions
could be considered to be directly proportional to the
number of flights, number of flight hours, or perhaps most
appropriately the number of radio transmissions made. It
is however likely that the true relationship is far more
complex, involving such things as the experience of the
controller and pilot, levels of interference on the system,
workload and the native language of the parties involved.
There is however a compromise needed to allow optimum
use of the information that is actually available. It may be
that using the rate per flight hour is the best extrapolation
available and to do this is better than not extrapolating the
data at all. If further data could be made available to
improve the correlation then that may be worthwhile but
there is clearly a limit to the value of the complexity of the
process used to provide a view of risk that can never be
entirely accurate.
5.2 Predicted risk
There is clearly much to be gained from the fusion of
historical safety data in the understanding of risk. The
main benefit however comes from using that fused data to
predict risk and, more importantly, to predict risk as a
result of changes to the system. An historical review of
risk data often reveals trends that do not purely match
those to be expected relative to the simple exposure
factors that are traditionally considered. Clearly there will
be a degree of random exposure to events that no tool will
be able to predict, but with a wide base of risk information
the random effects should be minor compared to the
underlying risk trends.
The challenge this creates is in identifying how the risk
indicators are influenced by the more complex sources of
exposure data. It will be through the understanding of the
complex relationships that exist between historical safety
data and risk exposure data that predictions will be
possible.
In the every day operation the people involved are
constantly making risk judgments based on their
understanding of the factors that affect it. They are in a
small way making safety predictions based on their expert
judgment of the situation. If that understanding could be
captured, along with the data on which they base those
decisions, then the process could be repeated to allow a
longer term quantative prediction of risk to be made.
The complexity of the relationship and the availability of
comprehensive exposure data is again likely to be a
limiting factor in any such approach. The data that may
be available real-time to those managing risks on a day to
day basis may not be available in an historical and reliable
format. There will however be a significant pool of
exposure data that will be available, as per the example in
section 3.2, and the fusion of it to allow its optimum use
needs to occur. If the decision making process in the
operational environment relies heavily on other sources of
information, there may well be merit in attempting to
capture it to improve the understanding of future risk.
5.3 Describing future system changes
Once the relationships between the data sources and
resultant risk are understood, to be able to predict the
change in risk from a future change, requires that change
to be described in ways that reflect those relationships.
For example a system change to an air traffic management
system would need to be described in ways that are part of
the risk understanding, for example through the decrease
in workload, or the increase in a particular type of
interaction that carries risk. Clearly to be able to describe
changes in this way will involve detailed expert
understanding of the change and how it effects the
operation. However if the risk is measured in a way that
reflects the users understanding of their risks, then the
step of describing changes in that way is a logical one.
A further advantage of this approach is that by describing
the benefits of the change in the same terms which are
used to measure risk, the resulting effect can be measured.
This then facilitates and effective feedback mechanism for
the monitoring of risk improvements.
1385
6 Regulating risk
The aviation environment is one of tight safety regulation.
Changes to the aviation system require careful review and
demonstration that they meet agreed regulatory
requirements. These requirements are however generally
domain orientated and described as to meet certain risk
probability goals. Such an approach does not necessarily
enable the most effective management of risk within the
aviation environment.
The regulation of safety within European aviation is
controlled by various different organisations responsible
for different parts of the domain. For example aircraft
design and operation is regulated by the European
Aviation Safety Agency (EASA) but air traffic service
provision is regulated by individual state regulators. Such
an approach makes the coordinated management of
overall aviation risk difficult; however there are proposals
to bring all the regulation together under EASA.
As the aviation regulation process becomes more
coordinated the ability to regulate the overall industry on
the basis of risk reduction becomes an exciting possibility.
CANSO (Civil Air Navigation Services Organisation) has
identified Prescriptive, excessively complex safety
Regulation as one of the issues that the aviation industry
faces and has set a goal for Simplified, performance
based safety regulation covering the whole aviation
chain. [10].
The present certification processes require changes to be
assessed on how they, once implemented, achieve certain
probability goals. For example in very simplistic terms
they might require that the probability of a catastrophic
accident is less that one in 10 million flight hours. The
problem with this approach is that assessing probabilities
as low as this is very difficult, but more importantly that it
doesnt necessarily take into account the state of the
system as it is now.
If the risk of aviation is widely understood and measured
then an alternative approach to regulation can be
considered. Changes to the system could be described in
ways that reflect the data set used to measure the risk at
the present time and therefore their effect on future risk
can be predicted. This would enable the certification of
changes on the basis of the positive reduction in risk to
aviation that they provide.
The major benefit of such an approach is that it would
allow the aviation system to develop and continually
reduce its risk rather than trying to achieve goals that may
not reflect the actual present performance of the system.
The other significant benefit that it brings is that it enables
the effect of changes to be monitored in a way that can
determine their effectiveness and the accuracy of their
predicted performance.
7 Conclusion
It is well recognised that aviation has been successful in
using safety data to manage its risks. Not withstanding
this, the challenges that face the industry through growth
and increased integration mean that improved ways to
understand and predict risk are required.
A more comprehensive and unified view of risk
throughout the industry would benefit the effective and
efficient management of risk. To provide this requires
both a common understanding of risk and also an
understanding of the complex relationships that exist
between the many types of safety and risk exposure data
that are available.
The common understanding of risk is an area that is under
much consideration in the industry and there is
widespread recognition in the value of a simple common
process to allow comparable risk data to be developed.
That common risk understanding not only needs to work
across varied types of incident data but also varying
sources of data across the industry.
The data relationships that will allow the wide range of
safety data to be fused together will be complex and
diverse. They will need to be determined from both data
analysis and expert opinion to allow models to be
developed for the data to be combined. Inevitably
compromises will be required between complexity and
achievability, but the overall goal will need to be one of
making the best use of data available.
With a cohesive view of risk based on the incident and
exposure data available, the greatest value will be to use
that data to predict future risk. To do this will require
changes to be described in ways that can be applied to that
fused risk data set so that their effect can be predicted.
Ultimately such an approach opens up new possibilities
for the future regulation of aviation safety. The present
methods of goal based regulation could be replaced with
regulation based on ensuring aviation risk reduction.
NATS is pursuing efforts in all the areas outlined in this
paper and welcomes input and coordination with other
interested industry partners.
The author would like to thank those at NATS and British
Airways who provided the knowledge, understanding and
inspiration to enable this paper to be written.
The opinions expressed in this paper are the authors and
do not necessarily represent those of NATS.
1386
References
[1] IATA, Annual Safety Report, April 2007.
[2] Gerard van Es, Advanced Flight Data Analysis, 14
th
European Safety Seminar, March 2002.
[3] Chris Sullivan, Who cares about CAIR?, Annual
Conference of the Australia and New Zealand Society of
Air Safety Investigators, June 2001.
[4] Andrew Rose, Measuring operational safety in
aviation, Aircraft Engineering and Aerospace
Technology, 2006.
[5] Andrew Rose, Free Lessons in aviation safety,
Aircraft Engineering and Aerospace Technology, 2004.
[6] Thomas Lintner & Steven Smith, Future Concepts
for Measuring Aviation Safety, Performance, &
Effectiveness, System Safety Society Conference 2008.
[7] Steven Vick, Degrees of Belief: Subjective
Probability and Engineering Judgment, ASE Press, 2002
[8] Jari Nisula, Practical solutions for Risk Assessment
in Flight Operations, FAA Conference on Risk Analysis
and Safety Performance in Aviation, 2006
[9] Martin Neil et al., Measuring & Managing
Culturally Inspired Risk, 2002
[10] Civil Air Navigation Services Organisation
(CANSO), Global vision on the future of air navigation
services, 2007
1387
