Overview of IEEE P3006.
7 Draft
Recommended Practice for Determining g the
Reliability of “7 x 24” Continuous Power
Systems in Industrial and Commercial
Facilities
Presented By
Robert Schuerger,
Schuerger PP.E.
E
HP Critical Facility Services delivered by EYP MCF
Origins of P3006.7
• All of the IEEE Color Books are
being
g revised and repackaged
p g as
“3000 Series” Standards
• IEEE Gold Book, Std. 493-2007
has Chapter 8 “7 x 24”
Continuous Power Facilities
• Chapter 8 has become P3006.7
• P3006.7
P3006 7 includes
i l d both
b th electrical
l ti l
distribution and mechanical
cooling systems used in critical
facilities
1
P3006.7 Developed by:
Power System Reliability P3006.7 Working Group
Working Group: Chair: Robert Schuerger
Ch i R
Chair: Robert
b t Arno
A Members:
• Robert Arno
Members:
• Jose Cay II
• William Braun
• Raymond Chiu
• Timothy J. Coyle
• Edwin Cothran
• Neal Dowling • Ann’claude Coutu
• Peyton Hale • Neal Dowling
• Masoud Pourali • Addam Friedl
• Robert Schuerger • Joaquin Fuster
• Michael Simon • Gardson Githu
• Peter Gross
• Christopher C Thompson, Jr
• Ian Levine
• Joseph Weber
• Michael Simon
• Sonny K. Siu
Reliability
• Today the term “reliability” is used as an umbrella definition
covering a variety of subjects including availability,
durability, quality and sometimes the function of the product
• Currently for critical facilities, when the subject is reliability, it
is very common to see statements about “five 9’s”
2
Five 9’s refers to Availability
Total Time - Downtime
Availability =
Total Time
• Availability (A): Availability is the long-term average
fraction of time that a component or system is in service
and satisfactorily performing its intended function
• Five 9’s means an availability of 99.999%.
Availability
Availability = Total Time - Downtime
Total Time
In terms used for statistics, for a constant failure rate:
•Downtime = Time it takes to repair the failure
•Average downtime = Mean time to Repair (MTTR)
•Average uptime = Mean Time Between Failures (MTBF)
•Total time = (MTBF) + (MTTR)
MTBF
Availability =
(MTBF + MTTR)
3
Availability of 99.999%
This could be:
• 5.3 minutes of downtime each year or
• 1.75 hours of downtime every 20 years
Availability does not specify how often an outage
occurs
Number of
outages Length of Failure rate MTBF MTBF
Availability per year outage (failure/hour) (hours) (years)
0.99999 315 1 second 3.60E-02 27.81 0.0032
5.3
0.99999 1 minutes 6.05E-04 1,652.80 0.189
1.75
0.99999 0.05 hours 5.71E-06 175,200 20
Reliability has a specific definition
• Reliability (R) is the probability
that a product or service will
operate properlyl for
f a specified
f d
period of time under design
operating conditions without
failure.
• Reliability = Probability of
Successful Operation
(for a specific time period)
• Probability of Failure =
(1 – Reliability)
4
Reliability and Availability – both important metrics
• Availability gives the average percentage of “uptime”
• Reliability gives an indication of long it will operate before it
f il
fails
Number of
Failure rate - MTBF MTBF Reliability
Availability outages per
failure/hour (hours) (years) (1 year)
year
0.99999 315 3.60E-02 27.81 0.0032 0%
0.99999 5.3 6.05E-04 1,652.8 0.189 0%
0.99999 0.05 5.71E-06 175,200 20 95.12%
MTBF – Mean Time Between Failures
Performing Reliability Analysis
• There are several ways to perform calculate reliability and
availability
• P3006.7 p
presents several methods
– Reliability Block Diagrams (RBD)
– Fault Tree Analysis (FTA)
– Failure Mode Effects and Criticality Analysis (FMECA)
• Each method has a different approach to the analysis
– RBD is a model of the system flow; the one-line diagram or
piping
p p g diagram
g
– FTA starts at the top with the failure to be analyzed and works
down the “tree” with all of the potential causes of the top event
– FMECA is a deep dive into how all of the components or
systems can fail (failure modes), what the effect of that failure
would be and how critical it would be to the overall mission
10
5
Performing Reliability Analysis – RBD
• Many of the comparisons between electrical and mechanical
systems in P3006.7 have been done using Reliability Block
g
Diagrams
• The individual components are represented by blocks.
Figure 8 — RBD of Utility power to two fused disconnects, two
transformers, and two circuit breakers, either one of which can
power the Main Switchgear
11
Performing Reliability Analysis – FTA
• Some of the comparisons in P3006.7 have been done using
Fault Tree Analysis
• Boolean Algebra with “OR,”
OR, “AND,”
AND, etc. are used to analyze the
fault tree
= OR
= AND
= Basic Event
= Repeated
Event
= Undeveloped
Event
Figure 10 – Example Fault Tree
12
6
Performing Reliability Analysis – FTA
• Fault Tree for a top event of “loss of power to the Main
Switchgear” fed by Utility power from two fused
disconnects,, two transformers,, and two circuit breakers
Figure 11 – FTA for power to Main Switchgear
13
Availability for multiple components
Series System – Each block is a Single Point of Failure (SPOF)
This is also
System Block 1 Block 1
Availabilityy = A = 0.9 A = 0.9
9
“AND” for
FTA
Each 0.81 = 0.9 x 0.9
component
has availability
of 90% or 0.9 Parallel System (having redundancy)
(one “9”)
Block 1
A = 0.9 This is also
System
Availability = “OR” for
Block 1 FTA
A = 0.9
0.99 = 1 – [ (1- 0.9) x (1- 0.9)]
14
7
Reliability vs. time
• For a single block that has
a constant failure rate ,
reliability as a function of
time is:
R(t) = e-t
15
Reliability for multiple components
• For two blocks in series with failure rates of 1 and 2, the
reliability as a function of time R(t) is:
R(t) = R(1) X R(2) = e- (1 + 2) t This is also “AND” for FTA
Assy: Block
A Bl k 1 Assy: Block
A Bl k 2
FR: 6.53E-4 FR: 4.15E-4
Start End
1::1
16
8
Reliability for multiple components
• For two parts in parallel with redundancy, where 1 out of 2
is necessary for successful operation, the reliability as a
function of time R(t) is:
R(t) = R(1) + R(2) - [R(1) X R(2)] This is also
R(t) = e-1t + e-2t - [e- (1 + 2) t] “OR” for FTA
Assy: Block 1
FR: 6.53E-4
Start 1::1 1::2 End
1::1
Assy: Block 2
FR: 4.15E-4
17
Component failure data is required to
perform Reliability analysis
• The primary source of failure and repair rates used for the
modelingg of critical electrical ((and mechanical)) distribution
systems is IEEE Gold Book, Standard 493-2007
Recommended Practice of the Design of Reliable Industrial
and Commercial Power Systems
• A large part of the of the data in the IEEE Gold Book was
provided by the Army Corp of Engineers which was collected
as p
part of the Power Reliabilityy Enhancement Programg ((PREP))
• Another source is the Reliability Analysis Center (RAC) Non-
electronic Parts Reliability Data
18
9
Data center nomenclature – N
• A single piece of equipment by itself is “N”
• If there are two, but both are needed to carry the load, that is
still “N”
N
• “N” means the “number needed”
19
Data center nomenclature – N+1
• “N + 1” means that there is a
spare unit; one is needed to
carry the load and the second
one is redundant
• This is “component redundancy,”
since only the UPS (or generator)
is redundant
20
10
Data center nomenclature – 2N
• “2N” means that there are two
systems; one system is needed
to carry the load and the
second one is redundant
• This is “system redundancy,”
since there two separate
systems
21
When in doubt – be specific
The design below is “2N” for the UPS system, but “N+1”
for the standby generators
22
11
Special equipment - STS
• Static Transfer Switches
(STS) were developed
d l d to
improve the reliability of
power to the IT equipment
• STS is designed to transfer
between sources in ¼ of a
cycle and therefore the IT
equipment is unaffected
ff d
23
Single and Dual cord IT equipment
Another very important
consideration is the
configuration of the power
supplies
l in the
h IT equipment
itself
• “Single cord” IT equipment
means there is only one
power cord supplying it
• “Dual
Dual cord
cord” IT equipment
has two power cords, each
one individually capable of
carrying the load
24
12
Dual cord IT equipment
• We will see later in this
p
presentation that it is veryy
important to properly connect
the Dual cord IT equipment so
each cord is powered by a
different source
25
Special Mechanical equipment - CRAC
• Computer Room Air Conditioning (CRAC) units have been
used to provide cooling since the early main frame
computers were developed
26
13
Figure 15 – N + 1 generators and UPS
The design below is
“N+1” for the standbyy
generators and UPS system
to single cord loads
27
Figure 16 – 2N electrical distribution
The design below is “2N” for the UPS system and standby
generators
28
14
Figure 17 – N+1 gens, 2(N+1) UPS system
The design below is N+1 standby generators and
2(N+1) UPS system
29
Figure 18 – Distributed Redundant UPS
The design below is N+1 standby generators and
Distributed Redundant 2 of 3 UPS system
30
15
Reliability and availability – single cord
Name Description of Critical MTBF MTTR Inherent Probability of
Distribution System (years) (Hours) Availability Failure
Figure 15:
Gen (2-3), UPS (4-5)
N + 1 (GEN 7.4 11.29 0.9998253 49.11%
12 single cord loads
+ UPS)
Figure 16: 2X [Gen (2-2), UPS (4-4)]
2N (GEN + 12 STS/PDU 8.9 10.96 0.9998592 39.82%
UPS) single cord loads
N+1 GEN Gen (2-3), 2X [UPS (4-5)]
2(N + 1) 12 STS/PDU 8.9 11.06 0.9998576 39.47%
UPS single cord loads
N+1 Gen: Gen (2-3), DR (2-3) X [UPS
DR (2-3) (2-3)], 12 STS/PDU 8.7 11.08 0.9998549 40.71%
UPS single cord loads
MTBF – Mean Time Between Failures
31
Reliability and availability – dual cord
Name Description of Critical MTBF MTTR Inherent Probability of
Distribution System (years) (Hours) Availability Failure
2N Gen 2X [Gen (2-2), UPS (4-4)]
68.9 2.57 0.9999958 6.96%
2N UPS 12 dual cord loads
N + 1 Gen (2-3), 2N
N+1 Gen
UPS(4-4) 12 dual cord 66.6 2.50 0.9999957 7.57%
2N UPS
loads
Figure 17:
N+1 Gen Gen (2-3), 2X [UPS (4-5)]
67.5 2.50 0.9999958 7.18%
2(N + 1) 12 dduall cord
d lloads
d
UPS
Figure 18:
Gen (2-3), DR (2-3) X
N+1 Gen
[UPS 2-3], 12 dual cord 65.9 2.52 0.9999956 7.69%
DR (2-3)
loads
UPS
MTBF – Mean Time Between Failures
32
16
Figure 27 – Chilled Water Central Plant
The design below is N+1for the cooling towers, chillers
and pumps and N+4 for the CRAH units
33
Figure 29 – Air Cooled Chilled Water Plant
The design below is N+1for the chillers and pumps and
N+4 for the CRAH units
34
17
Figure 30 – Air & Water Cooled CWP
The design below is N+N for the air cooled and water
cooled chillers, pumps, etc. and N+4 for the CRAH units
35
Reliability and availability – Mechanical
Name Description of Critical MTBF MTTR Inherent Probability of
Distribution System (years) (Hours) Availability Failure
2 of 3 [CT, Dedicated
(CWP, WC CH, PWP),
Figure 27 77.9 6.65 0.9999903 1.72%
SWP] + (14 of 18) CRAH
units
2 of 3 [CT, CWP, WC CH,
Figure 28 PWP, SWP] + (14 of 18) 78.1 6.60 0.9999904 1.62%
CRAH units
2 of 3 [ AC CH, PWP,
Figure 29 SWP] + (14 of 18) CRAH 495 13.46 0.9999969 0.95%
units
1 [CT, CWP, WC CH,
PWP] + 2 [AC CH, 2
Figure 30 226 8.96 0.9999955 2.17%
PWP] + (2 of 3 SWP) +
(14 of 18) CRAH units
MTBF – Mean Time Between Failures
36
18
Figure 32 – 2N Electrical System feeding
N+1 Water Cooled CWP
The design below powers the 2 out of 3 water cooled
chillers, pumps, etc. from a 2N electrical system
37
Figure 33 – 2N Electrical System feeding
N+1 Water Cooled CWP
The design below uses 3 switchboards with circuit breaker
transfer pairs from a 2N electrical system
38
19
Figure 34 – 2 of 3 Electrical &
Mechanical Systems
The 2 of 3 Distributed Redundant electrical system with a
2 of 3 Water Cooled Chiller plant and N+4 CRAH units
39
Reliability and availability – Mechanical
Description of Critical MTBF MTTR Inherent Probability of
Name
Distribution System (years) (Hours) Availability Failure
Figure 34
2 of 3 Mech Swbds
without 5.2 2.78 0.9999389 61.62%
feeding (14 of 18) CRAH
ATSs
2 of 3 Mech Swbds
Figure 34 feeding (14 of 18) CRAH 70.2 2.00 0.9999968 6.86%
with 6 ATS for 6 CRAH
2 of 3 Mech Swbds
Figure 34
feeding (14 of 18) CRAH
with 147.4 1.20 0.9999991 3.52%
with two contactors in
contactors
each
2N feeding 2 Mech Swbds
with ATS feeding one set
Figure
g 32 of chillers,, etc. + (14
( of 18)) 90.7 1.61 0.9999980 5.69%
CRAH with two contactors
in each
2N feeding 2 of 3 Mech
Swbds with breaker
Figure 33 transfer pairs + (14 of 18) 192.3 1.15 0.9999993 2.79%
CRAH with two contactors
in each
MTBF – Mean Time Between Failures
40
20
Cost Optimization is greatly assisted with
Reliability and Availability analysis
Availability
99.9999
99.999
99.99
99.9
99.0
9.0
Cost $
41
Questions?
Robert Schuerger, PE
Principal Reliability Analysis
bschuerger@hp.com
HP Critical Facilities Services, Inc.
delivered by EYP MCF
42
21
