ECE566

Enterprise Storage Architecture

Spring 2024

Storage Area Network (SAN)

Tyler Bletsch
Duke University

Adapted from the course “Information Storage and Management v2”

(module 5-6), published by EMC corporation.

Includes additional content cited inline.

Classical Fibre Channel SAN

2
 What is a SAN?
SAN

It is a high-speed, dedicated network of servers and shared storage devices.

• Centralizes storage and management

• Enables sharing of storage resources across multiple servers at
block level
• Meets increasing storage demands efficiently with better
economies of scale
• Common SAN deployments are:
 Fibre Channel (FC) SAN: uses FC protocol for communication
 IP SAN: uses IP-based protocols for communication

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 5: Fibre Channel Storage Area Network 3
 SAN block diagram
Initiator (client)
Direct block request User program
(e.g. read of /dev/sda) open(), read(), mkdir(), etc.

Kernel
VFS
(Virtual File System) Target
(server)
Choose based on directory mountpoint

ext4 ext4 nfs

FS driver FS driver FS driver
Kernel
Forward file IO call
Read-block/Write-block
(these are not NAS protocols)
(open, read, mkdir, etc.) SAN server
(this is a NAS protocol) disk routing logic

SAN HBA Local disk or NIC

RAID array

Physical disks
Ethernet

SAN SAN HBA

NAS server

4
 SAN block diagram
Initiator (client)
Direct block request User program
(e.g. read of /dev/sda) open(), read(), mkdir(), etc.

Kernel
VFS
(Virtual File System) Target
What kind of network is this? (server)
Choose based on directory mountpoint

ext4 ext4 nfs

FS driver FS driver FS driver
Kernel
Forward file IO call
Read-block/Write-block
(these are not NAS protocols)
(open, read, mkdir, etc.) SAN server
(this is a NAS protocol) disk routing logic

SAN HBA Local disk or NIC

RAID array

Physical disks
Ethernet

SAN SAN HBA

NAS server

5
 History and protocols of SAN: FCP

• Original SAN: Fibre Channel Protocol (FCP)

• A network standard for sending SCSI frames,
totally separate from Ethernet/IP
• In fact, VERY different network design from Ethernet/IP
• First on market in 1993, this protocol was the only SAN for many years
• Successive technology versions labeled by data rate
• 1Gb/s in 1997, 2Gb/s in 2001, 4Gb/s in 2004, … 256Gb/s in 2020
• Expensive, single purpose network (only for SAN, nothing else)

• Good news! It’s finally dying (kinda)!

• Despite 128Gb/s and 256Gb/s standards, little hardware for that
• Some markets still love it (e.g. mainframe, some VM hosts, others)
• Will probably never “fully” die, but will get smaller and more niche

• So our coverage will be light

6
 History and protocols of SAN: FCoE

• Weird off-shoot: Fibre Channel over Ethernet (FCoE)

• I used to teach this a bunch – it was very popular for a while!
• Now it’s largely dead except for a few special areas, e.g. Cisco Unified
Computing System (UCS)
• I’m going to skip teaching much of it, except for this:
• It promised Fibre Channel on the same network as Ethernet ☺
• Because Fibre Channel was so different, you couldn’t run it on
‘normal’ Ethernet – you needed special “Data Center Ethernet”
(DCE), so all new hardware, thus ruining the potential benefit 
• Meanwhile, if you wanted SAN on regular Ethernet, iSCSI came
along (see next slide)…
• Conclusion: why pay huge money when you could pay no money?

Source 7
 History and protocols of SAN: iSCSI

• Cheap option: iSCSI

• It’s SCSI protocol over IP
• The SCSI storage protocol that everything uses!
+
• The Ethernet/IP protocols that everything uses!
• Result: Cheap! ☺
• Developed in 1998, standardized in 2000
• This is probably what you’d invent: “Just send SCSI over IP”

8
 History and protocols of SAN: FCIP (lol)

• Comedy option: FCIP

• It’s FCP over IP? Invented after iSCSI existed? Why???
• This thing died almost immediately upon launch
• I’ve never taught it

• Why bring it up?

• These slides used to say “FCoE is the future, FCIP is dead”
• Now they say “FCoE is dead, like FCIP”
• Predicting the future is hard
• The industry moves fast
• Sometimes stuff dies

9
 How we’ll proceed

• I’m going to show some info about traditional FCP

• It’s still around
• All SAN protocols borrow terminology from it
• We’ll skip some of the nasty details
• Then we’ll contrast with iSCSI

10
 Understanding Fibre Channel
• High-speed network technology
 Latest FC implementation supports speed up to 256 Gb/s
• Highly scalable
 Theoretically, accommodate approximately 15 million devices

IP
Network FC SAN

Clients Application Servers Storage Arrays

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 5: Fibre Channel Storage Area Network 11
 Cables
• SAN implementation uses
 Copper cables for short distance
 Optical fiber cables for long distance
• Two types of optical cables: single-mode and multimode
Cladding Core Cladding Core

Single-mode Multimode
Light In Light In

Carries single Can carry multiple beams of

beam of light light(a)simultaneously
Multi-mode Fiber (b) Single-mode Fiber
Single-mode Fiber

Distance up to Used for short distance

Cladding Core
10km (Modal dispersion weakens
signal strength after certain Light In
distance)
(a) Multi-modeFiber
Multimode Fiber

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 5: Fibre Channel Storage Area Network 12
 Connectors
• Attached at the end of a cable
• Enable swift connection and disconnection
of the cable to and from a port Standard Connector
• Commonly used connectors for fiber optic
cables are:
 Standard Connector (SC)
 Duplex connectors
 Lucent Connector (LC) Lucent Connector

 Duplex connectors
 Straight Tip (ST)
 Patch panel connectors
 Simplex connectors
Straight Tip Connector

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 5: Fibre Channel Storage Area Network 13
 Interconnecting Devices
• Commonly used interconnecting devices in FC SAN are:
 Hubs, switches, and directors
• Hubs provide limited connectivity and scalability
• Switches and directors are intelligent devices
 Switches are available with fixed port count or modular design
 Directors are always modular, and its port count can be increased
by inserting additional 'line cards' or 'blades'
 High-end switches and directors contain redundant components

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 5: Fibre Channel Storage Area Network 14
 Fibre Channel Switch (FC-SW) Connectivity
• Creates a logical space (called fabric) in which all nodes
communicate with one another using switches
 Interswitch links (ISLs) enable switches to be connected together
• Provides dedicated path between nodes
• Addition/removal of node does not affect traffic of other nodes
Servers
Storage Array

FC Switch FC Switch

Server Interswitch Links

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 5: Fibre Channel Storage Area Network 15
 Terminology

• Initiator: Client using • Target: Storage

block devices controller with the
data

Servers
Storage Array

FC Switch FC Switch

Server Interswitch Links

16
 Fibre Channel Protocol Stack
Upper Layer Protocol
Example: SCSI, HIPPI, ESCON, ATM, IP

FC-4 Upper Layer Protocol Mapping

FC-2 Framing/Flow Control

FC-1 Encode/Decode

FC-0 1 Gb/s 2 Gb/s 4 Gb/s 8 Gb/s 16 Gb/s

FC Layer Function Features Specified by FC Layer

Mapping upper layer protocol (e.g. SCSI) to
FC-4 Mapping interface
lower FC layers
FC-3 Common services Not implemented
FC-2 Routing, flow control Frame structure, FC addressing, flow control
8b/10b or 64b/66b encoding, bit and frame
FC-1 Encode/decode
synchronization
FC-0 Physical layer Media, cables, connector

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 5: Fibre Channel Storage Area Network 17
 World Wide Name (WWN)
• Unique 64 bit identifier
• Static to node ports on an FC network
 Similar to MAC address of NIC
 World Wide Node Name (WWNN) and World Wide Port Name
(WWPN) are used to uniquely identify nodes and ports
respectively
World Wide Node Name (for the array)
5 0 0 6 0 1 6 0 0 0 6 0 0 1 B 2
0101 0000 0000 0110 0000 0001 0110 0000 0000 0000 0110 0000 0000 0001 1011 0010

Format Company ID Port Model Seed

Type 24 bits 32 bits

World Wide Port Name (for the HBA port)

1 0 0 0 0 0 0 0 c 9 2 0 d c 4 0
Format Reserved Company ID Company Specific
Type 12 bits 24 bits 24 bits

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 5: Fibre Channel Storage Area Network 18
 Zoning
Zoning
It is an FC switch function that enables node ports within the fabric to be
logically segmented into groups, and communicate with each other within
the group.

• Zone set comprises zones Zone 1

• Each zone comprises zone

members (HBA and array Server
ports)
FC SAN
• Benefits
 Restricts RSCN traffic Storage Array
 Provides access control
Array port

Zone 2
Servers HBA Port

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 5: Fibre Channel Storage Area Network 19
 Types of Zoning

Servers Switch Domain ID = 15

Port 5

Zone 2

Port 1
WWN 10:00:00:00:C9:20:DC:40 FC Switch Storage Array

Port 12
Port 9
Zone 3

WWN 10:00:00:00:C9:20:DC:56

WWN 50:06:04:82:E8:91:2B:9E

Zone 1

WWN 10:00:00:00:C9:20:DC:82 Zone 1 (WWN Zone) = 10:00:00:00:C9:20:DC:82 ; 50:06:04:82:E8:91:2B:9E

Zone 2 (Port Zone) = 15,5 ; 15,12
Zone 3 (Mixed Zone) = 10:00:00:00:C9:20:DC:56 ; 15,12

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 5: Fibre Channel Storage Area Network 20
 Ethernet SAN:
iSCSI and FCoE

21
 Module 6: IP SAN and FCoE

Lesson 1: IP SAN
During this lesson the following topics are covered:
• Drivers for IP SAN
• IP SAN Protocols: iSCSI and FCIP
• Components, topologies, and protocol stack for iSCSI and FCIP

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 22
 Drivers for IP SAN
• IP SAN transports block-level data over IP network
• IP is being positioned as a storage networking option because:
 Existing network infrastructure can be leveraged
 Reduced cost compared to investing in new FC SAN hardware and
software
 Many long-distance disaster recovery solutions already leverage IP-
based network
 Many robust and mature security options are available for IP
network

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 23
 IP SAN Protocol: iSCSI
• IP based protocol that is used to connect host and storage
• Encapsulates SCSI commands and data into an IP packet and
transports them using TCP/IP

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 24
 Components of iSCSI
• iSCSI initiator
 Example: iSCSI HBA (hardware, rare nowadays)
 Example: iSCSI software initiator (pure software, common)
• iSCSI target
 Storage array with iSCSI support
 iSCSI gateway – enables communication with FC storage array
• IP network iSCSI Gateway
Storage Array
Server

IP
FC Port
iSCSI HBA

iSCSI Port

Storage Array

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 25
 iSCSI Host Connectivity Options
• Standard NIC with software iSCSI initiator
 NIC provides network interface
 Software initiator provides iSCSI functionality
 Requires host CPU cycles for iSCSI and TCP/IP processing
• TCP Offload Engine (TOE) NIC with software iSCSI initiator
 Moves TCP processing load off the host CPU onto the NIC card
 Software initiator provides iSCSI functionality
 Requires host CPU cycles for iSCSI processing
• iSCSI HBA
 Offloads both iSCSI and TCP/IP processing from host CPU
 Simplest option for boot from SAN

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 26
 iSCSI Topologies: Native iSCSI
• iSCSI initiators are either directly attached to storage array or
connected through IP network
 No FC component
• Storage array has iSCSI port
• Each iSCSI port is configured with an IP address

IP
Server Storage Array
iSCSI HBA iSCSI Port

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 27
 iSCSI Topologies: Bridged iSCSI
• iSCSI gateway is used to enable communication between iSCSI
host and FC storage
• iSCSI gateway works as bridge between FC and IP network
 Converts IP packets to FC frames and vice versa
• iSCSI initiator is configured with gateway’s IP address as its target
• iSCSI gateway is configured as FC initiator to storage array
iSCSI Gateway

IP
Server
iSCSI HBA

FC SAN
Storage Array
Server FC HBA FC Port

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 28
 Combining FC and Native iSCSI Connectivity
• Array provides both FC and iSCSI ports
 Enable iSCSI and FC connectivity in the same environment
 No bridge devices needed

iSCSI Port

IP
Server
iSCSI HBA

FC SAN
Storage Array
FC Port
Server FC HBA

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 29
 iSCSI Protocol Stack
OSI Model iSCSI Initiator iSCSI Target

Layer 7 Application SCSI Commands and Data SCSI

Layer 5 Session iSCSI Login and Discovery iSCSI

Layer 4 Transport TCP Windows and Segments TCP

Layer 3 Network IP Packets IP

Layer 2 Data Link Ethernet Frames Ethernet

Interconnect

Ethernet IP TCP iSCSI SCSI Data

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 30
 iSCSI Discovery
• For iSCSI communication, initiator must discover location and
name of target on a network
• iSCSI discovery takes place in two ways:
 SendTargets discovery
 Initiator is manually configured with the target’s network portal
 Initiator issues SendTargets command; target responds with required
parameters
 Internet Storage Name Service (iSNS)
 Initiators and targets register themselves with iSNS server
 Initiator can query iSNS server for a list of available targets

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 31
 iSCSI Name
• iSCSI name is a unique iSCSI identifier that is used to identify
initiators and targets within an iSCSI network
• Two common types of iSCSI names are:
 iqn: iSCSI Qualified Name
 iqn.2008-02.com.example:optional_string
 eui: Extended Unique Identifier
 eui.0300732A32598D26

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 32
 Module 6: IP SAN and FCoE

Dancing on the grave of

Lesson 2: Fibre Channel over Ethernet (FCoE)
During this lesson the following topics are covered:
• Drivers for FCoE
• Components of FCoE network
• FCoE frame mapping
• Converged Enhanced Ethernet (CEE)

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 33
 This slide was
FCoE Motivation, translated written in 2016

(My personal take on the evolution of FCoE)

Pro-FCoE: “Oh my god, can we stop running two networks with

separate hardware, terminology, cables, and teams? It’s just
data!”

Anti-FCoE: “But-but-but– my job security!! I mean, uh, your

packets will be haunted if they don’t pass through this special
network that costs 10 times a commodity Ethernet network!”

Compromise: We’ll just make a variant of Ethernet so the

hardware still costs 10 times as much, then everyone’s happy!

34
 Updated for
FCoE Motivation, translated 2024…

(My updated personal take on the evolution of FCoE)

Pro-FCoE: *doesn’t exist*

Anti-FCoE: “lmao it’s dead”

35
 Safe to ignore! Just

Drivers for FCoE included for those

curious about history.

• FCoE is a protocol that transports FC data over Ethernet network

(Converged Enhanced Ethernet)
• FCoE is being positioned as a storage networking option
because:
 Enables consolidation of FC SAN traffic and Ethernet traffic onto a
common Ethernet infrastructure
 Reduces the number of adapters, switch ports, and cables
 Reduces cost and eases data center management
 Reduces power and cooling cost, and floor space

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 36
 Data Center Infrastructure – Before Using Safe to ignore! Just
included for those
curious about history.
FCoE Servers Servers

Server Server

FC IP
Switches Switches

FC LAN
Switches

Storage Array Storage Array

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 37
 Data Center Infrastructure – After Using Safe to ignore! Just
included for those
curious about history.
FCoE Servers Servers

Server Server

FCoE
Switches

FC LAN
Switches

Storage Array Storage Array

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 38
 Data Center Infrastructure – with iSCSI

Servers Servers

Server Server

Ethernet
Switches

LAN

Storage Array Storage Array

39
 Safe to ignore! Just

Components of an FCoE Network included for those

curious about history.

• Converged Network Adapter (CNA)

• Cable
• FCoE switch

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 40
 Safe to ignore! Just

Converged Network Adapter (CNA) included for those

curious about history.

• Provides functionality of both – a

standard NIC and an FC HBA
 Eliminates the need to deploy separate
adapters and cables for FC and Ethernet
communications
• Contains separate modules for 10 Gigabit
Ethernet, FC, and FCoE ASICs
 FCoE ASIC encapsulates FC frames into
Ethernet frames

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 41
 Safe to ignore! Just

Cable included for those

curious about history.

• Two options are available for FCoE cabling

 Copper based Twinax cable
 Standard fiber optical cable

Twinax Cable Fiber Optical Cable

Suitable for shorter distances (up to Can run over longer distances
10 meters)
Requires less power and are less Relatively more expensive than
expensive than fiber optical cable Twinax cables

Uses Small Form Factor Pluggable Uses Small Form Factor Pluggable
Plus (SFP+) connector Plus (SFP+) connector

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 42
 Safe to ignore! Just

FCoE Switch included for those

curious about history.

• Provides both Ethernet and

FC switch functionalities FC Port FC Port FC Port FC Port

• Consists of FCF, Ethernet

bridge, and set of CEE ports Fibre Channel Forwarder (FCF)
and FC ports (optional)
 FCF encapsulates and de-
Ethernet Bridge
encapsulates FC frames
• Forwards frames based on
Ethertype Ethernet Ethernet Ethernet Ethernet
Port Port Port Port

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 43
 Safe to ignore! Just

FCoE Frame Mapping included for those

curious about history.

OSI Stack

7 - Application
FCoE Protocol Stack FC Protocol Stack

6 - Presentation FC - 4 FC - 4 Protocol map

5 - Session FC Layers FC - 3 FC - 3 Services

4 - Transport FC - 2 FC - 2 Framing

3 - Network FCoE Mapping FC - 1 Data enc/dec

2 - Data Link 2 - MAC FC - 0 Physical

IEEE 802.1q
Layers
1 - Physical 1 - Physical
Ethernet

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 44
 Safe to ignore! Just

Converged Enhanced Ethernet included for those

curious about history.

• Provides lossless Ethernet

• Lossless Ethernet requires following functionalities:
 Priority-based flow control (PFC)
 Enhanced transmission selection (ETS)
 Congestion notification (CN)
 Data center bridging exchange protocol (DCBX)

EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 45
 Summary

• Classic Fibre Channel Protocol (FCP):

• Special-purpose network that wraps SCSI
• Lossless circuit-switched network
• Hosts talk via Host-Bus Adapters (HBAs)
• Hosts labeled with World-wide Node Name (WWNN);
each port has its own World-wide Port Name (WWPN)
• iSCSI
• Send SCSI packets over plain TCP/IP Ethernet networks
• Can be bridged to classic FCP
• Endpoints talk via plain network card (in software) or HBA (hardware)
• Endpoints labeled with iSCSI Qualified Names (IQNs)

46
 SAN protocol tradeoffs

• Classic Fibre Channel (FC):

+ Separate network no contention between storage and regular traffic
– Separate network means having to buy more hardware
~ Available in physical links up to 64Gb/s*
+ Compatible with all the classic FC stuff you may already have
• iSCSI:
+ Uses commodity cheap Ethernet
– Storage traffic is in contention with regular traffic
~ Ethernet available in 1Gb (very cheap), 10Gb (common in datacenter), 40Gb
(somewhat common), 100Gb (expensive)

47
* Faster standards exist, but not much hardware available for those