U.S.
Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Accidents & Control Room
Influence
Karen Butler
Supervisor, Accident Investigation Team
PHMSA
Central Region
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Objectives of the CRM Rule
• Create an environment to help assure controllers will be
successful in maintaining pipeline safety and integrity
– Assure pipeline operators are addressing fatigue risks
in the control room
• Verify that procedures, systems and equipment are well
thought out, and function as designed.
-2-
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Safety Pyramid Is Why
Serious Failure
Reportable Significant and
Minor Events
Un-reportable
Intervention by
Controllers, other
Personnel or
Safety Systems
Recorded
Near Miss, Identified
Un-recorded
Near Miss, Undetected
Control Room Management Processes can help assure that developing situations
don’t climb the pyramid to become Significant Events or Serious Failures
USDOT/PHMSA, 06-24-2011 -3-
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
General Observations
• Even though Section G of the CRM rule exists
– Operators are not looking at Fatigue or control room
elements on all reportable events
• Excavation Damage is a primary opportunity to
identify controller response and evaluate specifics
associated with ROC alarms
– To prove Fatigue was an issue in an accident,
• If possible, two week work/sleep histories should be
captured quickly
• Don’t forget the drug and alcohol
U.S. Department of Transportation
Hazardous Liquid
Pipeline and Hazardous Materials
Safety Administration
U.S. Department of Transportation
Hazardous Liquid
Pipeline and Hazardous Materials
Safety Administration
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
General Observations
•Procedures and connections not fully developed
•Emergency Procedure Crosslink
•O&M Procedure Crosslink
–Abnormal, Normal, Specific
–Enhanced Level of Detail
–More than Generic Company Wide Procedures
•Leak Detection
•Communications and Power
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
General Observations
•Make sure:
–Specifically identify in the roles and responsibility
documents if the controller could shut down the pipeline
•Remember the supervisor functions
• Emergency?
•Shutdown Only?
•ESD?
•LASD?
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
General Overview
• Controllers were not informed of things happening in the
field that could affect them:
–Pressure restrictions due to integrity management
–PLIDCO sleeve installations that required a reduced
pressure or flow condition
–Pig runs that impacted control room operations
–Line maintenance that impacted thru put.
- 10 -
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
General Overview
• Controllers were unaware of things happening in the field
that could affect them:
– New pipeline facility construction and start up on a
moments notice without adequate training or preparation
– No control room input to control changes or set up for
new facilities
– Mergers or Acquisitions and suddenly the controller
without training
- 11 -
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
General Overview
• Previous accidents/incidents have indicated that
communication with the control room is not always
adequate nor informative
– Training needed with all parties
– Lack of understanding between locations for similar
terminology
- 12 -
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
General Overview
• Include in procedures how deficiencies that are identified
in RCFA investigations relative to the control room will
be communicated and addressed
• Procedures should identify and explicitly address the
contribution of erroneous training
- 13 -
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
General Overview
• Procedures identify how training is impacted by lessons
learned or near miss events
• Contributory factors may provide training and procedure
revision insights
• Document how deficiencies found in accident/incident
investigations have been corrected
• Document how lessons learned are shared with all
controllers
- 14 -
U.S. Department of Transportation
Pipeline and Hazardous Materials
Lessons Learned
Safety Administration
Bellingham, WA 1999
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Little Known Lessons Learned
• Server load is important
• When server reloads occur should be well thought out
• If a command doesn’t go the first time, don’t just keep
trying endlessly
• When an asset is not behaving like normal, don’t assume all
is well and wait to shutdown.
• When anyone in the company sees a concerning element
associated with operations, call the control room
• All employees should be trained to call the control room
and actions employed
• Controller training and understanding can have a significant
impact
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Different Incident Picture Inserted
Natural Gas Compressor Station
Explosion
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Lessons Learned
• If you have a local control room that is used routinely or in
emergencies,
– Make sure any emergency equipment that is necessary
to monitor is in plain site and accessible
– Make sure valve status’s associated with even fuel lines
are understood
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Different Accident Picture Inserted
Multiple Pipelines
Liquid Line MIC in Casing
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Lessons Learned
• Instrumentation alarms are important
• Controllers should not have to rely on verbal
communication between each other if they have to also
answer phones
• Automatic display changes when controllers monitor
multiple facilities is not a good idea unless adequate
screens exist
• Make sure field locations know what to call certain valves
• Make sure the field locations and control room know
locations as the same name
• If you don’t have check valves, better make sure backflow
from tankage is prevented
• If go through an integrity audit and promises to implement
ROC alarms, follow it through to implementation
• Poor maintenance
U.S. Department of Transportation
Pipeline and Hazardous Materials
Lessons Learned
Safety Administration
• If you automatically disable alarms, make sure that when
this automatic provision is lifted, alarm setpoint values
return to where they were
• Make sure you have enough staff to send all that may have
influenced the event from the control room to a drug and
alcohol test quickly
• Make sure instrumentation that the control room needs is
installed properly and has the a high maintenance priority
• If you have more than one controller responsible for a
portion of the system, make sure what they control comes
into their alarm or event environment
• Make sure that training matches procedures
• Verify controller displays are accurate
• Have maximum and minimum alarms controllers cannot
move
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
The Importance of Alarm Management
¾ Console 4 (Controller #1)
– High and Low for May 21 – May 23
• High - 4,630 Per Day, Low – 1015 Per Day
• 192 Per Hour Average, 42 Per Hour Average
¾ Console 2 (Controller #2)
– High and Low for May 21-May 23
• High – 6,202 Per Day, Low – 1,343
• 258 Per Hour Average, 55 Per Hour Average
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Different Accident Picture Inserted
Liquid Pipeline Stopple Fitting Leak
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Lessons Learned
• Controllers need to understand what is being done in the
field
– Stopple installations
• Purpose and how it can fail
• Consistent training
– Slack line conditions
• Understand the support systems
– Leak Detection
• False alarms can mask real problems
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Different Accident Picture Inserted
Liquid Pipeline
Plidco Sleeve Leak and Ignition
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Lessons Learned
¾ Know and Abide by the Manufacturer’s Fitting
Recommendations (Proper Installation, How the
Fitting Works, Pressure Restrictions)
¾ Work to make sure operating procedures are correct
and well understood
¾ Assist in making sure the control room understands
what you are doing in the field
¾ Be alert for AOC’s
¾ Keep an eye out for ignition sources that should not
be present at the job site
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Different Accident Picture Inserted
Liquid Pipeline HVL
2” Bypass Stopple Leak and Stuck Pig
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Lessons Learned
¾Use Stopple Fittings correctly, as indicated by
the manufacturer
¾Make sure the Control Room knows what
maintenance activity you are performing in the
field
¾If you stick a pig, carefully construct your
bypass and make sure you communicate your
actions to all impacted including the control
room
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Just for Information
• If providing Trends on pressure or flow,
make sure that the information makes
some sense with the evidence before
sending them
– Received information as pressures from
transmitters that been left on line but
were considered not used
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Different Accident Picture Inserted
Liquid Pipeline
Plant with
Jurisdictional Piping
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Lessons Learned
• If a particular alarm is part of a required action
and procedure, make sure it is always working
– High Level In the Accumulator
– Personnel were to dump the accumulator
• Disconnected during a control room and
never reconnected
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Different Accident Picture Inserted
Liquid Pipeline – NTSB Investigation
SCC
Lessons Learned
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
• Leak detection needs improved on downed pipelines
• Support staff may also suffer from fatigue
• Forced points can lead to confusion
• Think leak first
• Don’t work around the controller
• Make sure those giving instructions to the controller are
trained
• Follow procedures
• Don’t wait for the public to find your leaks or ruptures
• If controllers tell you they need a change, listen and act
• Don’t dismiss comments from others in the control room
• Instrumentation needs to be in the correct location
• Leak detection should be set up correctly for pipeline and
facility maintenance also
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Different Incident Picture Inserted
Natural Gas Pipeline
NTSB Investigation
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Lessons Learned
• Upgrade your scada system if it is older and you have
limited functions or concerns
• Know and train controllers on how equipment fails when
loosing power
• Review where remote controlled valves are located versus
population
– ***Drills can be helpful if done right
• Isolate the area away from commodity quickly – remote
valves first, manual second
– If you don’t have them, add them
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Lessons Learned
• Think leak control versus town loss
• Anticipate problems when changing controls and normal
power supply arrangements
• Enhanced field to control room communication needed
when performing maintenance activities, especially when
impacting SCADA data.
• Alarms due to maintenance can hide other issues
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Different Incident Picture Inserted
Natural Gas Transmission Pipeline
Cracked Girthweld in a Casing
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Lessons Learned
– Value of Controller Experience
– Instrumentation weakness
– Common Lines feed a fire
– Do not start up when a known fire is present
– Make sure controllers know where line break
valves are located and what they operate on
– Make sure controllers understand where cross-
connects are located on common line
operations
– Unless OQ’d and planned for through training,
supervisors are NOT controllers
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Different Accident Picture Inserted
Liquid Pipeline
Crude Oil
Vibration
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Lessons Learned
• Don’t make it hard for the public to call in a release
• If you have a 3rd party take your 800 calls, make sure they
transfer the calls regarding potential leaks promptly
• If you are in a control room operating a new facility, think
about vibration and pulsation issues
• Make sure control valves and VFDs don’t fight each other
• Instrumentation needs to be in the right place and
supported correctly
• Listen to field and control room personnel
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Different Accident Picture Inserted
Liquid Pipeline
Jet Fuel A
Overpressure
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Lessons Learned
• Electrical power outage occurred and communications was
lost with a pump station
• The operator manned the facilities and took over local
control
• The next day, power was restored
• Local personnel gave instructions to the controller to open
certain valves and start specific pumps and tested multiple
flow paths including an area thru a trap that the controller
could not see values for pressure
• The field instructions were performed as requested by the
controller
• The controller pumped against a closed valve and over
pressured the line
• Clocks in Field Equipment and Scada were not Synch.
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Lessons Learned
• Not only controllers can be fatigued, fatigue was not
confirmed
• The field needs to understand what the controllers can see
and what they cannot if they are providing instructions
• Responsibility needs to be clear
• Training should be implemented for the field and the
controller
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Different Accident Picture Inserted
Liquid Pipeline
Crude Oil
Commissioning Overpressure
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Lesson’s Learned
• When preparing to commission facilities, don’t assume the
controls are correctly set up in a PLC/RTU load
• Delay putting commodity in the system until controls can
sufficiently be tested if at all possible
• Verify PIDs are correct before introducing any pressure or
commodity into the system
• Make sure the local control room display screens are correct
if technicians are trained to use them for calibrations
• Don’t reset something repeatedly in order to clear an
override safety and keep checking something that does not
make sense
• Always be aware of liquid sumps
• Confirm open and closed control paths
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Different Accident Picture Inserted
Liquid Pipeline
River Flooding
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Lessons Learned
• The procedures in the Control Center were not adequate in
that the controller shut in the line but because it was on a
general downslope, it kept flowing into the river.
• The operator had valves they could have shut immediately
to provide greater protection of the river put did not do it.
– Use what you have
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Different Incident Picture Inserted
Natural Gas Transmission
Vent Valve Malfunction
Work Around
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
What Happened and Lesson Learned
• A compressor unit is shutdown and a vent valve
inadvertently opens
• Two alarms are received by the controller after shutdown
pertaining to the open vent valve
• By procedure, controller was supposed to call the field but
does not and simply acknowledges the alarms
• The Vent valve blew for at least 8 hours
• Part of the reason alarms were available is this problem had
happened several times in the past and a work around had
been implemented
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Other Accident Learning
• Relief valve, relief valve, relief valve
• WHY IMPACTS THE CONTROL ROOM
– If they activate, remember it is a sign that the process
moved beyond the planned range
• Vibration or pulsation?
• Trash? Where is it coming from – Line Up?
• Other operator sending you problems?
• Poor maintenance technicians
– In many cases, there are meters that could be compared
routinely to find releases rather than blowing and being
reported
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Other Accident Learning
ENGINE SHUTDOWN VALUES and LINE BREAK VALVES
• If you have a special permit or have a limited pressure on
compressor units, make sure that your engine shutdowns
are set correctly.
• If you have automatic line break valves, make sure that
your engine controls are set up to shutdown rather than
continue pumping against a closed valve.
– Whether doing maintenance or remotely controlled
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Other Accident Learning
• Control room asked to perform unusual activities without
any warning due to a leak at an anomaly but were not told
about it, were just told an integrity dig was happening
– Infrequently used procedures and advance training?
• Control room experienced a leak and did everything right
but in the process of investigating it was determined that
the control room had inadequate instrumentation to know if
any of the product was moving to the end of the pipeline
– One pressure transmitter and one flow meter at the
beginning of the pipeline
– Other operator personnel had to be called to know more
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Objectives of the Rule
• Ask:
– CAN THE CONTROLLER SUCCEED AT THE
ROLES AND RESPONSIBILITIES ASSIGNED IN
CURRENT ENVIRONMENT?
– WILL THIS AFFECT THE CONTROLLERS
OPPORTUNITY TO SUCCEED?
• Don’t Know:
– Ask a controller, we will.
- 53 -
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Key Takeaways
• Make sure your controllers know field terminology
• Stress to the field the need to communicate activities with
the control room
• Use low minimum and high maximum pressure alarms that
cannot be changed by controllers
– If zero’s, use an algorithm with knowledge associated
with line up and past experience on timing
• Take advantage of ROC alarms
• Know your leak detection systems
• Routing commodity through traps or sumps to various
locations seems to cause more than one problem
• Implement the needed instrumentation correctly and
maintain it
Key Takeaways
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
• Use what is available on metered volumes or pressure
changes to pick up flow on down pipelines
– Laterals and existing meter locations should be reviewed
for possible LUAF
– The public should not be the operator’s only leak
detection system
• If procedures require an alarm, make sure the technology
works
• Consistency in training versus procedures
• If commissioning, assign responsibility to monitor pressures
• Training, training and more training for all parties
• Make sure controllers know and understand how equipment
will fail, especially related to power loss
• Think outside the box……… to learn……….as on the next
slide……….
U.S. Department of Transportation
Pipeline and Hazardous Materials
Safety Administration
Lesson’s Learned from SRCR
• Not an accident
– Welker Operator and Monitor Set up Fails
– Determined to have manufacturer issues
– SRCR is filed because more than one location existed
with this equipment
• But in the investigation it is determined:
– An ESD switch did not work
– Overpressure was seen in remote control room but page
acknowledge on another alarm did not allow it to be
seen
– Displays did not alarm correctly
• Alarm Logger was incorrectly programmed – gave
temperature value instead of pressure
