The Use of Safety Cases in Certification and Regulation1 Prof.

Nancy Leveson Aeronautics and Astronautics/Engineering Systems MIT Introduction Certification of safety-critical systems is usually ased on evaluation of !"et"er a system or #roduct reduces ris$ of s#ecific losses to an acce#ta le level. T"ere are ma%or differences& "o!ever& in "o! t"at decision is made and on !"at evidence is re'uired. T"e term Safety Case "as ecome #o#ular recently as a solution to t"e #ro lem of regulating safety-critical systems. T"e term arises from t"e (SE )(ealt" and Safety E*ecutive+ in t"e ,.-.& ut different definitions seem to e rife. To avoid confusion& t"is #a#er uses t"e term .assurance cases/ for t"e general term and limits t"e use of t"e term .safety case/ to a very s#ecific definition as an argument for !"y t"e system is safe. T"is #a#er e*amines t"e use of safety cases and some dangers associated !it" t"eir use. T"e first im#ortant distinction is et!een ty#es of regulation. Types of Regulation Safety assurance and certification met"ods differ greatly among industries and countries. Safety assurance met"ods commonly used can e ro$en into t!o general ty#es& !"ic" determine t"e ty#e of argument used in t"e assurance or certification #rocess0 1. Prescriptive0 Standards or guidelines for #roduct features or develo#ment #rocesses are #rovided t"at are used to determine !"et"er a system s"ould e certified. a. Product0 S#ecific design features are re'uired& !"ic" may e )a+ s#ecific designs as in electrical codes or ) + more general features suc" as fail-safe design or t"e use of #rotection systems. Assurance is usually #rovided y ins#ection t"at t"e design features #rovided are effective and im#lemented #ro#erly. In some industries& #ractitioners are licensed ased on t"eir $no!ledge of t"e standards or codes of #ractice. Assurance t"en ecomes t"e res#onsi ility of t"e licensed #ractitioner& !"o can lose t"eir license if t"ey fail to follo! t"e standards. 2rgani3ations may also e esta lis"ed t"at #roduce standards and #rovide certification& suc" as t"e ,L rating. . Process0 (ere t"e standards s#ecify t"e #rocess to e used in #roducing t"e #roduct or system or in o#erating it )e.g.& maintenance or c"ange #rocedures+ rat"er t"an s#ecific design features of t"e #roduct or system itself. Assurance is ased on !"et"er t"e #rocess !as follo!ed and& sometimes& on t"e 'uality of t"e #rocess or its artifacts. T"e #rocess re'uirements may s#ecify i. 4eneral #roduct or system develo#ment #rocesses and t"eir artifacts& suc" as re'uirements s#ecifications& test #lans& revie!s& analyses to e #erformed& and documentation #roduced. ii. T"e #rocess to e used in t"e safety engineering of t"e system and not t"e general develo#ment #rocess used for t"e #roduct. 5. Performance-based or goal-setting approaches focus on desired& measura le outcomes& rat"er t"an re'uired #roduct features or #rescri#tive #rocesses& tec"ni'ues& or #rocedures. T"e certification aut"ority s#ecifies a t"res"old of acce#ta le #erformance and a means for assuring t"at t"e t"res"old "as een met. 6asically& t"e standards set a goal& !"ic" may e a ris$ target& and usually it is u# to t"e assurer to decide "o! to accom#lis" t"at goal. Performance- ased regulation s#ecifies defined results !it"out
1

T"is #a#er !ill a##ear in t"e Nov/7ec 5811 issue of t"e 9ournal of System Safety.

s#ecific direction regarding "o! t"ose results are to e o tained. An e*am#le is a re'uirement t"at an aircraft navigation system must e a le to estimate its #osition to !it"in a circle !it" a radius of 18 nautical miles !it" some s#ecified #ro a ility. :"ile in t"e #ast most assurance !as #rescri#tive )eit"er #roduct or #rocess+& t"ere "as een interest in #erformance- ased regulation and assurance y government agencies& starting in t"e ,.S. during t"e ;eagan administration& often s#ear"eaded y #ressure from t"ose eing certified. A similar movement& ut muc" more successful& !as started in 4reat 6ritain around t"e same time& some of it stemming from t"e Cullen re#ort on t"e Pi#er Al#"a accident <5=. Certification in t"e ,.S. #rimarily uses #rescri#tive met"ods& ut mi*es t"e t!o ty#es )#roduct and #rocess+. Commercial aircraft& for e*am#le& are certified ased on air!ort"iness standards re'uiring s#ecific features )e.g.& o*ygen systems and life #reservers+& and more general features suc" as fail-safe design. Certification also re'uires t"e use of various ty#es of safety analysis tec"ni'ues& suc" as >ault (a3ard Analysis& and general engineering develo#ment standards. NASA also uses ot" #roduct and #rocess standards. :"ile t"e Nuclear ;egulatory Commission re'uires #rescri#tive assurance for nuclear #o!er #lants& t"e American Nuclear Society in 588? called for t"e use of ris$-informed and #erformance- ased regulations for t"e nuclear industry& arguing t"at .;is$-informed regulations use results and insig"ts from #ro a ilistic ris$ assessments to focus safety resources on t"e most ris$-significant issues& t"ere y ac"ieving an increase in safety !"ile simultaneously reducing unnecessary regulatory urden #roduced y deterministic regulations/ <1= Similar arguments "ave een made a out >AA regulations and #rocedural "and oo$s eing infle*i le and inefficient and rule-ma$ing ta$ing too long. ;ecommendations "ave een made to redesign t"e rulema$ing #rocess y moving to #erformance- ased regulations !"ere a##ro#riate& ut t"is ty#e of certification is controversial& #articularly !it" res#ect to "o! t"e #erformance goals are set and assured. Assurance Cases 2ften& certification is a one-time activity t"at follo!s t"e develo#ment #rocess and occurs efore t"e #roduct or system is allo!ed to e mar$eted or used. >or com#le* systems& suc" as aircraft and nuclear #o!er #lants& certification may involve ot" initial a##roval and oversig"t of t"e o#erational use of t"e system. C"anges to t"e original system design and certification asis may re'uire recertification activities. All certification is ased on .arguments/ t"at t"e certification a##roac" "as een follo!ed. Ins#ection and test may e used if t"e certification is ased on follo!ing a #roduct standard. If t"e certification is ased on t"e #rocess used& engineering artifacts or analyses may e re'uired and revie!ed. Performance- ased regulation may re'uire a #articular ty#e of analysis )suc" as t"e use of s#ecific ty#es of #ro a ilistic ris$ assessment+ or may allo! any ty#e of reasoning t"at su##orts "aving ac"ieved a #articular #erformance goal. As an e*am#le& t"e ,.S. 7e#artment of 7efense in Mil-Std-@@5 <1@= uses a #rescri#tive #rocess t"at details t"e ste#s t"at must e ta$en in t"e develo#ment of safety-critical systems to ensure t"ey are safe. T"e #ur#ose of t"e SA; )safety assessment re#ort+& !"ic" is used as t"e asis for certification& is to descri e t"e results of t"e #rescri ed ste#s in t"e standard. T"e SA; contains t"e artifacts of t"e #rescri ed #rocess& suc" as a Safety Plan )!"ic" must e a##roved y t"e 7o7 at t"e eginning of t"e develo#ment of t"e system+& a Preliminary (a3ard Analysis& a System (a3ard Analysis& a Su system (a3ard Analysis& an 2#erating System (a3ard Analysis& etc. T"e 7o7 evaluates t"e 'uality of t"e #rocess artifacts #rovided in t"e SA; as t"e asis for a##roving use of t"e system. :"ile NASA "as recently een influenced y t"e nuclear #o!er community em#"asis on #ro a ilistic ris$ analysis& traditionally it "as ta$en )and continues to em#"asi3e+ an a##roac"

similar to t"e ,.S. 7o7. T"e ,.S. >AA )>ederal Aviation Aut"ority+ a##roac" for civil aviation "as also een over!"elmingly #rescri#tive and t"e initial certification ased on t"e 'uality of t"e #rescri ed #rocess used to develo# t"e aircraft and t"e im#lementation of various air!ort"iness standards in t"e aircraftAs design. 2#erational oversig"t is ased on ins#ection as !ell as feed ac$ a out t"e safety of t"e o#erations #rocess. ;ecently& t"e >AA "as moved to create a re'uirement for a safety management system y t"ose develo#ing or o#erating aviation systems in order to s"ift more of t"e res#onsi ility for safety to t"e airframe manufacturers and airlines. T"e ty#e of evidence re'uired and assurance arguments used are straig"tfor!ard !it" #rescri#tive regulation& ut #erformance- ased regulation re'uires a more com#le* argument and evaluation strategy. :"ile t"e term .safety case/ may e used in #rescri#tive regulation& it is more commonly used in a #erformance or goal- ased regulatory regime. Performance- ased Regulation and Safety Cases 4overnment oversig"t of safety in England started after t"e >li* oroug" e*#losion in 1BC?& ut t"e term safety case seems to "ave emerged from a re#ort y Lord Cullen on t"e Pi#er Al#"a disaster in t"e offs"ore oil and gas industry in 1B@@ !"ere 1DC #eo#le died. T"e Cullen re#ort on t"e Pi#er Al#"a loss& #u lis"ed in 1BB8& !as scat"ing in its assessment of t"e state of safety in t"e industry <5=. T"e Cullen re#ort concluded t"at safety assurance activities in t"e offs"ore oil industry !ere0 Too su#erficialE Too restrictive or #oorly sco#edE Too genericE 2verly mec"anisticE 7emonstrated insufficient a##reciation of "uman factorsE :ere carried out y managers !"o lac$ $ey com#etencesE :ere a##lied y managers !"o lac$ understandingE >ailed to consider interactions et!een #eo#le& com#onents and systems. T"e re#ort suggested t"at regulation s"ould e ased around .goal setting/ !"ic" !ould re'uire t"at stated o %ectives e met& rat"er t"an #rescri ing t"e detailed measures to e ta$en <51=& i.e.& #erformance- ased rat"er t"an #rescri#tive. In suc" a regime& res#onsi ility for controlling ris$s s"ifted from government to t"ose !"o create and manage "a3ardous systems in t"e form of self-regulation. T"is a##roac" "as een ado#ted y t"e 6ritis" (ealt" and Safety E*ecutive and a##lied !idely to industries in t"at country. T"e 6ritis" safety case #"iloso#"y is ased on t"ree #rinci#les <B& 1C=0 T"ose !"o create t"e ris$s are res#onsi le for controlling t"ose ris$s Safe o#erations are ac"ieved y setting and ac"ieving goals rat"er t"an y follo!ing #rescri#tive rules. :"ile t"e government sets goals& t"e o#erators develo# !"at t"ey consider to e a##ro#riate met"ods to ac"ieve t"ose goals. It is u# to t"e managers& tec"nical e*#erts& and t"e o#erations/maintenance #ersonnel to determine "o! accidents s"ould e avoided. All ris$s must e reduced suc" t"at t"ey are elo! a s#ecified t"res"old of acce#ta ility. :"en #erformance- ased or goal- ased certification is used& t"ere are differences in "o! t"e #erformance or goals are s#ecified and "o! t"e evaluation !ill e #erformed. In 1BC?& t"e creation of t"e (ealt" and Safety E*ecutive )(SE+ !as ased on t"e #rinci#le t"at safety management is a matter of alancing t"e enefits from underta$ing an activity and #rotecting t"ose t"at mig"t e affected y it& essentially cost- enefit analysis )C6A+. T"e (SE also instituted t"e related conce#t of ALA;P or .as lo! as reasona ly #ractical/ and !idely used #ro a ilistic ris$ analysis as t"e asis for t"e goals. Eac" of t"ese is controversial.

T"e nuclear #o!er industry !as #ro a ly t"e first to use #ro a ilistic ris$ analysis as a asis for certification. In t"e ,nited -ingdom& t"e Nuclear Installations Act of 1BDF re'uired covered facilities to create and maintain a safety case in order to o tain a license to o#erate. T"e nuclear industry "as #laced #articular em#"asis on t"e use of Pro a ilistic ;is$ Assessment )P;A+ !it" t"e use of tec"ni'ues suc" as >ault Tree and Event Tree Analysis. 6ecause of t"e use of standard designs in t"e nuclear #o!er community and very slo! introduction of ne! tec"nology and innovation in designs& "istorical failure rates are often determina le. 2t"er #otentially "ig"-ris$ industries& suc" as t"e ,.S. nuclear su marine community& ta$e t"e o##osite a##roac". >or e*am#le& S,6SA>E does not allo! t"e use of P;A <15=. Instead& t"ey re'uire 2GE )2 %ective Guality Evidence+& !"ic" may e 'ualitative or 'uantitative& ut must e ased on o servations& measurements& or tests t"at can e verified. Pro a ilistic ris$ assessments& for most systems& #articularly com#le* systems& cannot e verified. A second uni'ue as#ect of t"e 6ritis" a##roac" to safety assurance and re'uired y t"e (SE is argumentation and a##roval ased on !"et"er ris$s "ave een reduced as lo! as is reasona ly #ractica le )ALA;P+. Evaluating ALA;P involves an assessment of t"e ris$ to e avoided& an assessment of t"e sacrifice )in money& time and trou le+ involved in ta$ing measures to avoid t"at ris$& and a com#arison of t"e t!o. T"e assumed level of ris$ in any activity or system determines "o! rigorous& e*"austive and trans#arent t"e ris$ analysis effort "as een. .T"e greater t"e initial level of ris$ under consideration& t"e greater t"e degree of rigor re'uired to demonstrate t"at ris$s "ave een reduced so far as is reasona ly #ractica le./ <C=. T"e a##lication of ALA;P to ne! systems& !"ere .reasona ly #ractical/ "as not yet een defined& is 'uestiona le. Not increasing t"e accident rate in civil aviation a ove !"at it is today does seem li$e a reasona le goal given t"e current lo! rate& for e*am#le& ut it is not clear "o! suc" an evaluation could e #erformed for t"e ne! tec"nologies )suc" as satellite navigation and intensive use of com#uters+ and t"e ne! and very different #rocedures t"at are #lanned. T"ere are also et"ical and moral 'uestions a out t"e acce#tance of t"e cost- enefit analysis underlying t"e ALA;P #rinci#le. :"ile none of t"ese more controversial as#ects of assurance and certification need to e #resent !"en using a .safety case/ a##roac"& t"ey are #art and #arcel of t"e "istory and foundation of safety cases and #erformance- ased regulation. Potential !imitations of Safety Cases A .safety case/ may e and "as een defined in many !ays. In t"is #a#er& t"e term is used to denote an argument t"at t"e system !ill e acce#ta ly safe in a given o#erating conte*t. T"e #ro lem is t"at it is al!ays #ossi le to find or #roduce evidence t"at somet"ing is safe. ,nli$e #roving a t"eorem using mat"ematics )!"ere t"e system is essentially .com#lete/ and .closed&/ i.e.& it is ased on definitions& t"eorems and a*ioms and not"ing else+& a safety analysis is #erformed on an engineered and often social system !"ere t"ere is no com#lete mat"ematical t"eory to ase arguments and guarantee com#leteness .5 T"e main #ro lem lies in #syc"ology and t"e notion of a mindset or frame of reference. .In decision t"eory and general systems t"eory& a mindset is a set of assum#tions& met"ods or notations "eld y one or more #eo#le or grou#s of #eo#le !"ic" is so esta lis"ed t"at it creates a #o!erful incentive !it"in t"ese #eo#le or grou#s to continue to ado#t or acce#t #rior e"aviors& c"oices& or tools. T"is #"enomenon of cognitive bias is also sometimes descri ed as mental inertia& groupthink& or a paradigm& and it is often difficult to counteract its effects u#on analysis and decision-ma$ing #rocesses./ <55= An im#ortant com#onent of mindset is t"e conce#t of confirmation ias. Confirmation bias is a tendency for #eo#le to favor information t"at confirms t"eir #reconce#tions or "y#ot"eses
5

Even !it" suc" a mat"ematical asis& #u lis"ed and !idely acce#ted mat"ematical #roofs are fre'uently found later to e incorrect.

regardless of !"et"er t"e information is true. Peo#le !ill focus on and inter#ret evidence in a !ay t"at confirms t"e goal t"ey "ave set for t"emselves. If t"e goal is to #rove t"e system is safe& t"ey !ill focus on t"e evidence t"at s"o!s it is safe and create an argument for safety. If t"e goal is to s"o! t"e system is unsafe& t"e evidence used and t"e inter#retation of availa le evidence !ill e 'uite different. Peo#le also tend to inter#ret am iguous evidence as su##orting t"eir e*isting #osition <H=. E*#eriments "ave re#eatedly found t"at #eo#le tend to test "y#ot"eses in a one-sided !ay& y searc"ing for evidence consistent !it" t"e "y#ot"esis t"ey "old at a given time <18& 1H=. ;at"er t"an searc"ing t"roug" all t"e relevant evidence& t"ey as$ 'uestions t"at are #"rased so t"at an affirmative ans!er su##orts t"eir "y#ot"esis. A related as#ect is t"e tendency for #eo#le to focus on one #ossi ility and ignore alternatives. In com ination !it" ot"er effects& t"is one-sided strategy can o viously ias t"e conclusions t"at are reac"ed. Confirmation iases are not limited to t"e collection of evidence. T"e s#ecification of t"e information is also critical. >isc"off& Slavin& and Lic"tenstein conducted an e*#eriment in !"ic" information !as left out of fault trees. 6ot" novices and e*#erts failed to use t"e omitted information in t"eir arguments& even t"oug" t"e e*#erts could e e*#ected to e a!are of t"is information. >isc"off et al attri uted t"e results to an .out of sig"t& out of mind/ #"enomenon <?=. In related e*#eriments& an incom#lete #ro lem re#resentation actually im#aired #erformance ecause t"e su %ects tended to rely on it as a com#re"ensive and trut"ful re#resentationIt"ey failed to consider im#ortant factors omitted from t"e s#ecification. T"us& eing #rovided !it" an incom#lete #ro lem re#resentation )argument+ can actually lead to !orse #erformance t"an "aving no re#resentation at all <58=. T"ese #ro lems are not easy to eliminate. 6ut t"ey can e reduced y c"anging t"e goal. T"e aut"orAs com#any !as recently "ired to conduct a non-advocate safety assessment of t"e ne! ,.S. Missile 7efense system for t"e "a3ard .inadvertent launc"&/ !"ic" !as t"e ma%or concern at t"e time <1F=. T"e system safety engineers conducting t"e inde#endent safety assessment did not try to demonstrate t"at t"e system !as safe& everyone !as already convinced of t"at and t"ey !ere going to de#loy t"e system on t"at elief. T"e develo#ers t"oug"t t"ey "ad done everyt"ing t"ey could to ma$e it safe. T"ey "ad asically already constructed a .safety case/ argument during develo#ment t"at !ould %ustify t"eir elief in its safety. 6y la!& "o!ever& t"e government !as re'uired to #erform an inde#endent ris$ analysis efore de#loyment and field testing !ould e allo!ed. T"e goal of our inde#endent assessment !as to s"o! t"at t"ere !ere scenarios !"ere inadvertent launc" could occur& not to s"o! t"e system !as safe. T"e analysis found numerous suc" scenarios t"at "ad to e fi*ed efore t"e system could e de#loyed& resulting in a si* mont" delay for t"e Missile 7efense Agency and e*#enditure of a large amount of money to fi* t"e design fla!s. T"e difference in results !as #artly due to a ne!& more #o!erful analysis met"od !e used ut also involved t"e different mindset and t"e different goal& !"ic" !as to identify unrecogni3ed "a3ards rat"er t"an to argue t"at t"e system !as safe )t"at inadvertent launc" could not occur+. Engineers al!ays try to uild safe systems and to verify to t"emselves t"at t"e system !ill e safe. T"e value t"at is added y system safety engineering is t"at it ta$es t"e o##osite goal0 to s"o! t"at t"e system is unsafe. 2t"er!ise& safety assurance ecomes sim#ly a #a#er e*ercise t"at re#eats !"at t"e engineers are most li$ely to "ave already considered. It is for e*actly t"is reason t"at (addon-Cave recommended in t"e Nimrod accident re#ort t"at safety cases s"ould e rela eled .ris$ cases/ and t"e goal s"ould e .to demonstrate t"at t"e ma%or "a3ards of t"e installation and t"e ris$s to #ersonnel t"erein "ave een identified and a##ro#riate controls #rovided/ <F=& not to argue t"e system is safe. A final #otential #ro lem !it" safety cases& !"ic" "as een critici3ed in t"e off-s"ore oil industry a##roac" to safety cases and !it" res#ect to t"e 7ee#!ater (ori3on accident )and !as also involved in t"e >u$us"ima 7aic"i nuclear #o!er #lant events+& is not using !orst-case analysis <@=. T"e analysis is often limited to !"at is li$ely or e*#ected& not !"at could e

catastro#"ic. Sim#ly arguing t"at t"e most li$ely case !ill e safe is not ade'uate0 Most accidents involve unli$ely events& often ecause of !rong assum#tions a out !"at is li$ely to "a##en and a out "o! t"e system !ill o#erate or e o#erated in #ractice. Effective safety analysis re'uires considering !orst cases. 6ut !"ile t"eoretical arguments against safety cases are interesting& t"e #roof is really .in t"e #udding./ (o! !ell "ave t"ey !or$ed in #racticeJ "#perience $ith Safety Cases T"e use of #erformance- ased regulation "as not necessarily #roven to e etter t"an t"e ot"er a##roac"es in use. 2ne of t"e most effective safety #rograms ever esta lis"ed& S,6SA>E <15=& !"ic" "as "ad no losses in t"e #ast ?@ years des#ite o#erating under very dangerous conditions& is t"e almost total o##osite of t"e goal- ased orientation of t"e 6ritis" form of t"e safety case. T"e s#ectacular S,6SA>E record is in contrast to t"e ,.S. e*#erience #rior to t"e initiation of S,6SA>E& !"en a su marine loss occurred on average every t!o to t"ree years. S,6SA>E uses a very #rescri#tive a##roac" as does t"e civil aviation community& !"ic" "as also een a le to reduce accident rates do!n to e*tremely lo! levels and $ee# t"em t"ere des#ite t"e tendency to ecome com#lacent after years of "aving very fe! accidents. ,nfortunately& careful evaluation and com#arison et!een a##roac"es "as not een done. Most #a#ers a out safety cases e*#ress #ersonal o#inions or deal !it" "o! to #re#are a safety case& ut not !"et"er it is effective. As a result& t"ere is no real evidence t"at one ty#e of regulation is etter t"an anot"er. T"e use or at least #oor use of safety cases "as een im#licated in accident re#orts. T"e est $no!n of t"ese is t"e Nimrod aircraft cras" in Afg"anistan in 588D. A safety case "ad een #re#ared for t"e Nimrod& ut t"e accident re#ort concluded t"at t"e 'uality of t"at safety case !as gravely inade'uate <F=0 .. . . t"e Nimrod safety case !as a lamenta le %o from start to finis". It !as riddled !it" errors. . . Its #roduction is a story of incom#etence& com#lacency& and cynicism K T"e Nimrod Safety Case #rocess !as fatally undermined y a general malaise0 a !ides#read assum#tion y t"ose involved t"at t"e Nimrod !as Lsafe any!ayA ) ecause it "ad successfully flo!n for H8 years+ and t"e tas$ of dra!ing u# t"e Safety Case ecame essentially a #a#er!or$ and Ltic$ o*A e*ercise./ T"e criticisms of safety cases contained in t"e Nimrod re#ort include0 T"e Safety Case ;egime "as lost its !ay. It "as led to a culture of L#a#er safetyA at t"e e*#ense of real safety. It currently does not re#resent value for money. T"e current s"ortcomings of safety cases in t"e military environment include0 ureaucratic lengt"E t"eir o scure languageE a failure to see t"e !ood for t"e treesE arc"aeological documentary e*ercisesE routine outsourcing to industryE lac$ of vital o#erator in#utE dis#ro#ortionalityE ignoring of age issuesE com#liance-only e*ercisesE audits of #rocess onlyE and #rior assum#tions of safety and Ls"elf-!areA. Safety cases !ere intended to e an aid to t"in$ing a out ris$ ut t"ey "ave ecome an end in t"emselves. Safety cases for LlegacyA aircraft are dra!n u# on an Las designedA asis& ignoring t"e real safety& deterioration& maintenance and ot"er issues in"erent in t"eir age. Safety cases are com#liance-driven& i.e.& !ritten in a manner driven y t"e need to com#ly !it" t"e re'uirements of t"e regulations& rat"er t"an eing !or$ing documents to im#rove safety controls. Com#liance ecomes t"e overriding o %ective and t"e argumentation tends to follo! t"e same& re#etitive& mec"anical format !"ic" amounts to no more t"an a secretarial e*ercise )and& in some cases& "ave actually een #re#ared y secretaries in outside consultant firms+. Suc" safety cases tend also to give t"e ans!er t"at t"e customer or designer !ants& i.e. t"at t"e #latform is safe.

 Large amount of money are s#ent on t"ings t"at do not im#rove t"e safety of t"e system (addon-Cave& t"e aut"or of t"e Nimrod accident re#ort& concluded t"at safety cases s"ould e renamed .ris$ cases/ and made t"e follo!ing recommendations )among ot"ers+0 Care s"ould e ta$en !"en utili3ing tec"ni'ues suc" as 4oal Structured Notation or LClaims-Arguments-EvidenceA to avoid falling into t"e tra# of assuming t"e conclusion )Lt"e #latform is safeA+& or loo$ing for su##orting evidence for t"e conclusion instead of carrying out a #ro#er analysis of ris$. )Note t"e similarity to t"e concerns e*#ressed in earlier a out mindset and confirmation ias.+ Care s"ould e ta$en !"en using 'uantitative #ro a ilities& i.e. numerical #ro a ilities suc" as 1 * 18-D e'uating to .;emote/. Suc" figures and t"eir associated nomenclature give t"e illusion and comfort of accuracy and a !ell-"oned scientific a##roac". 2utside t"e !orld of structures& num ers are far from e*act. Care s"ould e ta$en !"en using "istorical or #ast statistical data. T"e fact t"at somet"ing "as not "a##ened in t"e #ast is no guarantee t"at it !ill not "a##en in t"e future. Pi#er Al#"a !as ostensi ly .safe/ on t"e day efore t"e e*#losion on t"is asis. T"e etter a##roac" is to analy3e t"e #articular details of a "a3ard and ma$e a decision on !"et"er it re#resents a ris$ t"at needs to e addressed. Care needs to e ta$en to define t"e #rocess !"ere y ne! "a3ards can e added to t"e ;is$ Case& incor#orated in t"e (a3ard Log& and dealt !it" in due course& and "o! original assum#tions a out "a3ards or 3ones are to e re-e*amined in lig"t of ne! events. 2nce !ritten& t"e safety case s"ould e used as an on-going o#erational and training tool. T"ere are all too many situations !"ere a com#re"ensive safety case is !ritten& and t"en it sits on a s"elf& gat"ering dust& !it" no one #aying attention to it. In suc" situations t"ere is a danger t"at o#erations #ersonnel may ta$e t"e attitude& .:e $no! !e are safe ecause !e "ave a safety case/. Conclusions To avoid confirmation ias and com#liance-only e*ercises& assurance cases s"ould focus not on s"o!ing t"at t"e system is safe ut in attem#ting to s"o! t"at it is unsafe. It is t"e em#"asis and focus on identifying "a3ards and fla!s in t"e system t"at #rovides t"e .value-added/ of system safety engineering. T"e system engineers "ave already created arguments for !"y t"eir design is safe. T"e effectiveness in finding safety fla!s y system safety engineers "as usually resulted from t"e a##lication of an o##osite mindset from t"at of t"e develo#ers. :"atever is included in t"e assurance case& t"e follo!ing c"aracteristics seem im#ortant0 T"e #rocess s"ould e started early. T"e assurance case is only useful if it can influence design decisions. T"at means it s"ould not e done after a design is com#leted or #re#ared in isolation from t"e system engineering effort. If safety cases are created only to argue t"at !"at already e*ists is safe& t"en t"e effort !ill not im#rove safety and ecomes& as a##arently "as "a##ened in t"e #ast& sim#ly #a#er e*ercises to get a system certified. 2ne result mig"t e un%ustified com#lacency y t"ose o#erating and using t"e systems. T"e assum#tions underlying t"e assurance case s"ould e continually monitored during o#erations and #rocedures esta lis"ed to accom#lis" t"is goal. T"e system may e !or$ing& ut not t"e !ay it !as designed or t"e assum#tions may turn out to e !rong& #er"a#s ecause of #oor #rediction or ecause t"e environment "as c"anged. C"anges to t"e system and its environment may "ave een made for all t"e rig"t reasons& ut t"e drift et!een t"e system as designed and t"e system as enacted is rarely if ever analy3ed or understood as a !"ole& rat"er t"an eac" #articular deviation a##earing sensi le or even "el#ful to t"e individuals involved.

To ma$e maintaining t"e assurance case #ractical& t"e analysis needs to e integrated into system engineering and system documentation so it can e maintained and u#dated. Safety assurance is not %ust a one-time activity ut must continue t"roug" t"e lifetime of t"e system& including c"ec$ing during o#erations t"at t"e assum#tions made in t"e assurance argument remain true for t"e system com#onents and t"e system environment. In t"e aut"orAs e*#erience& t"e #ro lems in u#dating and maintaining safety assurance do not arise from t"e form of t"e assurance documentation or in u#dating t"e argument once t"e need for it is esta lis"ed& ut in relating t"e assurance case to t"e detailed design decisions so t"at !"en a design is c"anged& it is #ossi le to determine !"at assum#tions in t"e safety analysis are involved. T"e analysis s"ould consider !orst cases& not %ust t"e li$ely or e*#ected case )called a design basis accident in nuclear #o!er #lant regulation+. T"e analysis needs to include all factors& t"at is& it must e com#re"ensive. It s"ould include not %ust "ard!are failures and o#erator errors ut also management structure and decision-ma$ing. It must also consider o#erations and t"e u#dating #rocess must not e limited to develo#ment and certification ut must continue t"roug" t"e o#erational #art of t"e system life cycle. To e most useful& 'ualitative and verifia le 'uantitative information must e used& not %ust #ro a ilistic models of t"e system. T"e integrated system must e considered and not %ust eac" "a3ard or com#onent in isolation.

;eference 1. American Nuclear Society& M;is$-Informed and Performance-6ased ;egulations for Nuclear Po!er Plants&M Position Statement ?D& 9une 588?. 5. T"e (on. Lord Cullen& T"e Pu lic In'uiry into t"e Pi#er Al#"a 7isaster& Nols. 1 and 5 );e#ort to Parliament y t"e Secretary of State for Energy y Command of (er Ma%esty& Novem er 1BB8+. H. Sidney 7e$$er& T"e >ield 4uide to ,nderstanding (uman Error& As"gate Pu lis"ers& 588D. ?. 6. >isc"off& P. Slovic& and S. Lic"tenstein& M>ault Trees0 Sensitivity of Estimated >ailure Pro a ilities to #ro lem ;e#resentation&M 9. E*#erimental Psyc"ology0 (uman Perce#tion and Performance& vol. ?& 1BC@. F. C"arles (addon-Cave& T"e Nimrod ;evie!& (C 185F& London0 T"e Stationery 2ffice Limited& 2ct. 5@& 588B. D. (ealt" and Safety E*ecutive& MSafety Case ;egulations for 2ffs"ore 2il 7rilling&M 588F. C. -at"ryn (eiler& MIs t"e Australian Mining Industry ;eady for a Safety Case ;egime&M H1st International Conference of Safety in Mines ;esearc" Institute& 6ris ane& Australia& 2ct. 588F. @. 2liver A. (ouc$& M:orst Case and t"e 7ee#!ater (ori3on 6lo!out0 T"ere 2ug"t to e a La!&M Evironmental La! ;e#orter& ?8 EL; 118HD& Nov.& 5818. B. 9.;. Inge& MT"e Safety Case0 Its 7evelo#ment and ,se in t"e ,nited -ingdom&M E'ui#ment Safety Assurance Sym#osium 588C& 6ristol ,.-. 18. -unda& Oiva )1BBB+& Social Cognition0 Ma$ing Sense of Peo#le& MIT Press& IS6N BC@85D5D11?HF& 2CLC ?8D1@BC?. 11. N.4. Leveson& Safe!are0 System Safety and Com#uters& Addison :esley Pu lis"ers& 1BBF 15. N.4. Leveson& Engineering a Safer :orld& MIT Press& in #roduction )to a##ear 5811+& "tt#0//sunnyday.mit.edu/safer-!orld.

1H. Nic$erson& ;aymond S. )1BB@+& MConfirmation 6iasE A , i'uitous P"enomenon in Many

4uisesM& ;evie! of 4eneral Psyc"ology )Educational Pu lis"ing >oundation+ 5 )5+0 1CFP558& 1?. N2PSA& "tt#0//no#sa.gov.au/safety.as#& 588F. 1F. Steven 9. Pereira& 4rady Lee& and 9effrey (o!ard. MA System-T"eoretic (a3ard Analysis Met"odology for a Non-advocate Safety Assessment of t"e 6allistic Missile 7efense System&M AIAA Missile Sciences Conference& Monterey& CA& Nov. 588D. 1D. ;asc"e& T )5881+ M7evelo#ment of a safety case met"odology for t"e Minerals Industry P a discussion #a#er&M MIS(C& ,niversity of Gueensland. 1C. Ian Sutton& MPre#aring and Managing a Safety Case in t"e Process Industries&M "tt#0//$nol.google.com/$/ian-sutton/safety-cases/5vuF88dgll ?m/HHQ. 1@. ,.S. 7e#artment of 7efense& MStandard Practice for System Safety&M MIL-ST7-@@57& >e ruary 18& 5888. 1B. Nectra 4rou#& MLiterature ;evie! on t"e Perceived 6enefits and 7isadvantages of t"e ,Safety Case ;egime/& at "tt#0//!!!."se-data ases.co.u$/researc"/misc/sc?858@H.#df. 58. -.9. Nicente and 9. ;asmussen& MEcological Interface 7esign0 T"eoretical >oundations&M IEEE Trans. Systems& Man& and Cy ernetics& vol. 55& no. ?& 9uly/Aug. 1BB5. 51. :"yte& 7. )1BBC+ .Moving t"e goal#osts0 T"e deregulation of safety in t"e #ost #i#er al#"a offs"ore oil industry/ "tt#0//!!!.#sa.ac.u$/c#s/1BBC/!"yt.#df. 55. :il$inson& P )5885+ .Safety case0 success or failureJ/ Seminar #a#er 5 National ;esearc" Centre for 2(S ;egulation& AN, Can erra 5H. :i$i#edia& Mindset& "tt#0//en.!i$i#edia.org/!i$i/Mindset.