FlexPod Technical Bootcamp

Nexus Switching Functions and Configuration

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 1
Nexus Switching Functions and Configuration
Agenda:
• Nexus Switching for FlexPod
 Nexus 5000 Functions in FlexPod
 Logical Topology Diagrams for FlexPod Models
 5500 Data and Storage Features in FlexPod
• Nexus 5500 Data Networking
 Base setup
 vPC Setup
 Optimize and Verify
• Nexus 5500 Storage Networking
 UCS FC Ports and VSANs
 NetApp FCoE Configuration
 Fibre Channel Aliasing and Zoning

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
FCoE and 10 GbE Based Architecture

Cisco UCS
• Redundant Fabric Interconnects
• FEX-connected blade chassis
• FC, FCoE, iSCSI, NAS Storage
Access
• 10 Gig Ethernet port channel data

Nexus 5500
• Redundant Data and Storage
Switching Fabric
• FC, FCoE, iSCSI, NAS
Connectivity

NetApp FAS
• Redundant Controllers
• FC, FCoE, iSCSI, NAS

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3
Fibre Channel and 10 GbE Based Architecture

Cisco UCS
• Redundant Fabric Interconnects
• FEX-connected blade chassis
• FC, FCoE, iSCSI, NAS Storage
Access
• 10 Gig Ethernet port channel data

Nexus 5500
• Redundant Data and
Storage Switching
Fabric
• FC, FCoE, iSCSI, NAS
Connectivity

NetApp FAS
• Redundant Controllers
• FC, FCoE, iSCSI, NAS

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4
High Availability Switching of Data and Storage Traffic

To Data Center
Aggregation or
Network Core

Nexus 5500

Nexus 5500 Functions:

• Multi-protocol central communications hub of
the FlexPod system
• UCS system storage connectivity to NetApp
system, FC, FCoE, or IP-based
UCS Fabric
Interconnect • UCS Server communication to other systems
in the Data Center access layer
• “Northbound” communications from the
FlexPod to the Data Center Aggregation layer
NetApp Array and the Enterprise Core.

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
FCoE and 10 GbE Based Architecture
NetApp Array:
• Ethernet attached to Nexus 5500 with 10 Gig
port channels
• Storage access via FCoE encapsulation, or
IP-based (iSCSI, NAS)
• Both controllers linked to both SAN Fabrics

Nexus 5500:
• Switches FC encapsulated traffic on the UCS
links to FCoE encapsulation on the NetApp
side
• Ethernet connectivity supports data traffic, IP-
based storage traffic (iSCSI, NAS)

UCS Fabric Interconnects:

• FI to Chassis I/O Module links support FCoE,
data, and server management
• Storage traffic to Nexus over Fibre Channel,
or via IP-based storage protocols
FC SAN A
• Each FI linked to only one SAN Fabric
FC SAN B
Ethernet
FCoE
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6
Fibre Channel and 10 GbE Based Architecture
NetApp Array:
• Ethernet and Fibre channel attached to
Nexus 5500
• Storage access via FC or IP-based (iSCSI,
NAS)
• Both controllers linked to both SAN Fabrics

Nexus 5500:
• Switches FC traffic on the UCS links to FC
encapsulated links on the NetApp side
• Ethernet connectivity supports data traffic and
any IP-based storage traffic (iSCSI, NAS)

UCS Fabric Interconnects:

• FI to Chassis I/O Module links support FCoE,
data, and server management
• Storage traffic to Nexus over Fibre Channel,
or via IP-based storage protocols
FC SAN A
• Each FI linked to only one SAN Fabric
FC SAN B
Ethernet
FCoE
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
 Nexus 5500 Hardware
Nexus 5548 (5548P & 5548UP)

32 x Fixed Unified Ports 1/10 GE or 1/2/4/8 FC

Expansion Module

Fabric Interconnect
Not Active on Nexus Out of Band Mgmt
10/100/1000 USB Flash

Console Fan Module Fan Module Power Entry Power Entry

N + N Redundant FANs N + N Power Supplies

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8
 Nexus 5500 Hardware
Nexus 5596UP 3 Expansion Modules

48 x Fixed Unified Ports 1/10 GE or 1/2/4/8 FC

Fabric Interconnect Out of Band Mgmt

Not Active on Nexus 10/100/1000 Console USB Flash

Power Supply Fan Module Fan Module Fan Module Fan Module

N + N Power Supplies N + N Redundant FANs

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
 Nexus 5500 Hardware Overview
Data and Control Plane Elements
Expansion Module

DRAM
10 Gig CPU Intel DDR3
Jasper
Forest South
Gen 2 UPC Gen 2 UPC Gen 2 UPC Bridge

Flash
12 Gig
Memory

PCIe x8 NVRAM

Unified Crossbar Fabric PEX 8525 Serial

Gen 2 ç 4 port PCIE
Switch
Console
PCIe x4

PCIE PCIE PCIE

Dual Gig Dual Gig Dual Gig

Gen 2 UPC ... Gen 2 UPC

0 1 0 1 0 1
L2
L1

Mgmt 0

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10
 Nexus 5500 Hardware
Nexus 5500 Expansion Modules

 Nexus 5500 expansion slots

 Expansion Modules are hot swappable (Future support for
L3 OIR)
 Contain forwarding ASIC (UPC-2)

8 x 1/10GE + 16 unified ports L3 module for

16 x 1/10GE individually 160G of L3 I/O
8 x 1/2/4/8G FC
configurable as 1/10GE bandwidth
or 1/2/4/8G FC

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11
 Unified Ports – Flexible Ethernet, FCoE and Fibre Channel
 Unified Port supports multiple
transceiver types Any Unified Port can be configured as
 1G Ethernet Copper/Fibre Ethernet Fibre Channel

Fibre Fibre
 10G Ethernet Copper/Fibre Channel or Channel
Traffic Traffic
 10G DCB/FCoE Copper/Fibre
 1/2/4/8G Fibre Channel

 Unified Port supports a Unified

Access Layer across the entire
Fabric
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
 Where and How is storage connected?
iSCSI iSCSI NAS NAS
FCoE SAN Appliance Gateway Appliance Gateway
Computer System Computer System Computer System Computer System Computer System

Application Application Application Application Application

File System File System File System File System File System
Volume
VolumeManager
Manager Volume Manager Volume Manager I/O Redirector I/O Redirector
SCSI Device Driver SCSI Device Driver SCSI Device Driver NFS/CIFS
NFS/CIFS
The Flexibility of FCoE Driver
iSCSI Driver
TCP/IP Stack
iSCSI Driver
TCP/IP Stack
TCP/IP Stack
NIC
TCP/IP Stack
NIC
a Unified Fabric
NIC NIC NIC

Block Block
I/O I/O File I/O
Transport
‘Any RU to Any SAN IP IP IP IP
Spindle’ NIC NIC NIC NIC
TCP/IP Stack TCP/IP Stack TCP/IP Stack TCP/IP Stack
iSCSI Layer iSCSI Layer File System File System
FCoE Bus Adapter FC HBA Device Driver FC HBA

FC Block I/O FC

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14
 N-Port Virtualization Forwarding

SAN A SAN B • Fabric Interconnect operates in N_Port

Proxy mode (not FC Switch mode)
Simplifies multi-vendor interoperation
FLOGI
NPIV NPIV Simplifies management
FDISC
F_Port F_Port • Nexus 5500 switch sees the 6200 as a
FC End Host with many N_Ports and
VSAN 10 VSAN 20 many FC IDs assigned
N_Proxy N_Proxy • Server facing ports function as F-proxy
6200-A
ports
6200-B
• Server vHBA pinned to an FC uplink in the
vFC 1 vFC 2 vFC 1 vFC 2 same VSAN. Round Robin selection.
F_Proxy F_Proxy • Provides multiple FC end nodes to one
F_Port off an FC Switch
• Eliminates need for FC domain on UCS
N_Port N_Port Fabric Interconnect
• All zoning on the Nexus 5500 NPIV
vHBA 0 vHBA 1 vHBA 0 vHBA 1
switch, not UCS.

Server 1 Server 2
VSAN 1 VSAN 1

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15
• vPC is a Port-channeling feature extending link
aggregation to two separate physical switches

• Allows the creation of resilient Layer-2 topologies

based on Link Aggregation.
Spanning Tree Protocol (STP) no longer the primary
means of loop prevention
Physical Topology Logical Topology
• Provides more efficient bandwidth utilization
All links are actively forwarding Virtual Port Channel

• vPC maintains independent control planes L2

• vPC switches are joined together to form a “domain” Si Si

• If using double-sided vPC, ensure unique domain id’s

are used with only two switches in each domain.

vPC domain Non-vPC vPC

Increased BW with vPC

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16
Applications in FlexPod

• vPC allows the Nexus 5500 switches

to appear as a single device at Layer-
2 vPC Peers

• UCS Fabric Interconnects build port

channels as if they were connected to
a single upstream switch Virtual
Port
Channels
• Each NetApp controller also builds
port channels as if connected to a
single upstream switch
• Eliminates Spanning Tree Protocol
(STP) blocked ports
• Provide fast convergence upon
link/device failure

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17
 • vPC peer – a vPC switch, one of a pair

vPC peer vPC peer link • vPC Primary, Secondary – roles assigned by
keepalive link election or using role priority
• Operational Primary or Secondary – current
functional role of a peer after recovering from
a failure (device or connection)
Primary Secondary
• vPC member port – one of a set of ports (port
vPC peer channels) that form a vPC
• vPC – the combined port channel between the
vPC peers and the downstream device
vPC
• vPC peer link – Link used to synchronize state
between vPC peer devices, must be 10GbE.
vPC Also carry multicast/broadcast/flooding traffic,
member and data traffic in case of vPC member port
port failure
• vPC peer keepalive link – the peer keepalive
link between vPC peer switches. It is used to
carry heartbeat packets
• CFS – Cisco Fabric Services protocol, used
for state synchronization and configuration
validation between vPC peer devices
Orphan
Port • Orphan port - Non-vPC member port

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18
• Peer Link carries both vPC data and control
traffic between peer switches
• Carries any flooded and/or orphan port traffic

• Carries STP BPDUs, IGMP updates, etc. vPC Peer

Link
• Carries Cisco Fabric Services messages (vPC
control traffic) 5k01 5k02

• Carries IP multicast traffic

• Minimum 2 x 10GbE ports

• ALL VLANS used on vPC PORTS MUST BE

PRESENT ON THE PEER-LINK

5020 (config)# interface port-channel 10

5020 (config-if)# switchport mode trunk
5020 (config-if)# switchport trunk allowed <BETTER TO ALLOW ALL VLANS>
5020 (config-if)# vpc peer-link
5020 (config-if)# spanning-tree port type network

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19
• Peer Keep-alive provides and out of band heartbeat Peer Keepalive
between vPC peers can be carried
over the OOB
• Purpose is to detect and resolve roles if a Split Brain
(Dual Active) occurs management
network
• Messages sent on 1 second interval with 5 second
timeout int mgmt 0
• 3 second hold timeout on peer-link loss before triggering
recovery
• Should not be carried over the Peer-Link
• Use the mgmt0 interface in the management VRF
• Can optionally be a dedicated link, 1Gb is adequate (first
16 ports on 5020 are 1/10GE ports)
• 3rd option, use a routed inband connection over L3
infrastructure (using SVI’s in the default VRF)

dc11-5020-1(config)# vpc domain 20

dc11-5020-1(config-vpc-domain)# peer-keepalive destination 172.26.161.201 source
172.26.161.200 vrf management
Note:
--------:: Management VRF will be used as the default VRF ::--------

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20
 Why vPC and not VSS or Stackwise?
WAN Core FC

Core

Core
L3 Aggregation
L2

Access Edge

 LAN and SAN utilize different High Availability Models

 SAN is dual fabric, LAN is fully meshed fabric

 vPC enables ‘both’ architectures at the edge (single device models not acceptable to SAN
engineers)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21
 Nexus 5500 Series Layer-3 Support
(Outside the scope of FlexPod itself)

160Gbps (240Mpps)
Layer 3 processing

1) Remove Fans 1) Install L3 Expansion Module(s)

2) Replace Daughtercard with L3 enabled daughtercard 2) Install License and enabled NX-OS features
3) Install License and enabled NX-OS features

Nexus 5548P/UP Nexus 5596UP

 Ordered with L3 daughtercard installed  At FCS one Layer 3 Expansion Module
or order a FRU for an L2 5548
 Support for OIR of Layer 3 Expansion
 Daughtercard can be replaced while in Module and/or up to three Layer 3
the rack Expansion Modules (Future, not EC’d)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22
Nexus Switching Functions and Configuration
Agenda:
• Nexus Switching for FlexPod
 Nexus 5000 Functions in FlexPod
 Logical Topology Diagrams for FlexPod Models
 5500 Data and Storage Features in FlexPod
• Nexus 5500 Data Networking
 Base setup
 vPC Setup
 Optimize and Verify
• Nexus 5500 Storage Networking
 UCS FC Ports and VSANs
 NetApp FCoE Configuration
 Fibre Channel Aliasing and Zoning

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27
“Part I” from Deployment Guide; Base Setup items
The following actions:
• Execute the Cisco Nexus 5548 setup script.
 Or set up manually, admin credentials, management interface IP, access
methods (telnet, ssh, snmp)
• Enable the appropriate Cisco Nexus features and licensing.
- feature fcoe
- feature npiv
- feature fport-channel-trunk
- feature lacp
- feature vpc

• Set global configurations.

 Spanning Tree port type defaults, global QoS settings, aligned with example
configuration in Deployment Guide
• Create necessary VLANs including NFS and management.
 Add VLANs with names for identification, identify VLANs to be included on
vPC peer link
• Add individual port descriptions for troubleshooting.
 Best practice for configuration management, identify each ports function
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28
“Part I” from Deployment Guide; vPC Setup
Complete the following actions:
< see following slides for detail steps on vPC setup >
• Create necessary port channels including the vPC peer-link.
• Add Port Channel configurations.
• Configure virtual Port Channels (vPCs) to UCS fabric
interconnects and NetApp controllers.
• Configure uplinks into existing network infrastructure, preferably
by using vPC.
• Save the configuration.

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29
 vPC
Domain 20
• Define the vPC Domain
 Two switches belong to a vPC Domain
 Role priority can control vPC primary function vPC
 vPC Domain must be unique, distinct from any vPC Peers Domain 10
adjacent domain
vpc domain 23
role priority 10

Virtual
• Establish physical Peer-Keepalive
Port
 Mgmt0 interface, or dedicated SVI Channels
 Point to peer’s address, ensure traffic will NOT be
routed over the vPC Peer-Link
peer-keepalive destination 10.61.185.70 source
10.61.185.69

• Create vPC Peer-Link

 Minimum 2 x 10 GigE Interfaces in a
port channel
 vPC peer-link command on port
channel (the peer-link itself is not defined as a
vPC)

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 30
 vPC
Domain 20
• Create port-channel interfaces
 Links to UCS and NetApp systems each assigned to
separate port-channel interfaces
vPC
 Use LACP for port-channel definition (mode active) vPC Peers Domain 10

• Assign vPC numbers to port-channels

 Use consistent numbering across the peer switches
 Each UCS Fabric Interconnects and NetApp controller Virtual
has a separate port channel Port
Channels
• Ensure configurations are consistent
 vPC requires configuration consistency
between peer switches
 Type-1 inconsistencies can cause
interfaces not to be brought up

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31
 vPC
Domain 20
• Use “show vPC” command to ensure
switches are peering
vPC
• Verify individual vPCs are in an “up” vPC Peers Domain 10
state
• To troubleshoot look for either
connectivity issues or configuration Virtual
mismatch Port
Channels
pod1-5548a(config-if)# sh vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 101

Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status: success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 0
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled

vPC Peer-link status

----------------------------------------------------------
id Port Status Active vlans
-- ---- ------ ---------------------------------------
1 Po1 up 1

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 32
Optimize Layer-2 Switching on Nexus 5500
vPC
Domain 20
• vPC Auto-Recovery
 Ensures vPC recovery after outages
 Allows single switches to bring vPC’s online vPC
 “auto-recovery” command under vPC domain vPC Peers Domain 10

• Identify Orphan Ports

 If non-portchannel devices are attached to Nexus 5500
pair (outside of FlexPod equipment itself) Virtual
 “vpc orphan-ports suspend” command Port
Channels

• Enable UDLD globally

 Best Practice for uni-directional link detection on fibre
links, not vPC-specific

• Use vPC peer-switch command

 Improves failover times by keeping a consistent STP
id between vPC peers
 Only needed if Nexus 5500 is acting as Spanning-
Tree root for Layer-2 domain

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 33
Layer-3 Considerations “peer-gateway” command
tells the vPC to respond
to the physical MAC
< This step required if running Layer-3 on Nexus address of its peer
5500 pair, or on upstream vPC switches >

• For Non-RFC compliant end hosts

 Device required to send packets to the MAC address
returned in ARP response (HSRP virtual MAC) L3
 Some non-compliant devices use the MAC address of the L2
sender device (Switch physical MAC)

 NAS devices (i.e. NETAPP Fast-Path) have been found to

do this

• vPC Peer Gateway - NX-OS 4.2(1)

 Allows a vPC peer to respond both the the HSRP virtual VLAN 200 VLAN 100
and the real MAC address of both itself and it’s peer

• Also enable “ip arp synchronize”

 Allows arp table synchronization over vPC peer link to
improve failover convergence for layer 3 flows

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 34
Nexus Switching Functions and Configuration
Agenda:
• Nexus Switching for FlexPod
 Nexus 5000 Functions in FlexPod
 Logical Topology Diagrams for FlexPod Models
 5500 Data and Storage Features in FlexPod
• Nexus 5500 Data Networking
 Base setup
 vPC Setup
 Optimize and Verify
• Nexus 5500 Storage Networking
 UCS FC Ports and VSANs
 NetApp FCoE Configuration
 Fibre Channel Aliasing and Zoning

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 35
UCS-facing Storage Ports, and VSAN assignments
• Create VSANs for fabric "A" or "B" on respective Nexus platform.
 Each Nexus 5500 only participates in one of the two VSANs
vsan database
vsan 101 name Fabric_A
• Create necessary SAN port channels to be connected to UCS
Fabric Interconnects. If the NetApp is also FC connected, use
the same procedure for the interfaces connecting the NetApp.
Interface san-port-channel 1
Channel mode active
Interface fc1/29-30
Channel-group 1 force
No shutdown
• Assign to VSAN appropriate FC interfaces.
vsan database
Vsan 101 interface fc1/29-30
Vsan 101 interface san-port-channel 1

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 36
NetApp-facing FCoE VLAN and vPC Port configuration
• Create the FCoE VLAN and map it to the VSAN.
vlan 101
fcoe vsan 101

• Configure physical port channel members to allow the VLAN associated

to the VSAN in use on the switch. < Note, the VSAN will be different on each 5500
switch so the vlan allowed statements will not match on both sides of the vPC. Ensure that the
data VLANs do match across switches. >
interface Ethernet1/1
description ice3270-1a:e2a
switchport mode trunk
switchport trunk native vlan 2
switchport trunk allowed vlan 101,3170
channel-group 11 mode active
interface Ethernet1/2
description ice3270-1b:e2a
switchport mode trunk
switchport trunk native vlan 2
switchport trunk allowed vlan 101,3170
channel-group 12 mode active

• Make the port channels vPC’s

interface port-channel11
description ice3270-1a
spanning-tree port type edge trunk
vpc 11
interface port-channel12
description ice3270-1b
spanning-tree port type edge trunk
vpc 12
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 37
NetApp FCoE Configuration, Virtual Fibre Channel ports
• Create the Virtual Fibre Channel (vfc) ports and bind them to the port-
channel interfaces that link to the NetApp controllers
interface vfc11
bind interface port-channel11
switchport description ice3270-1a:2b
no shutdown
interface vfc12
bind interface port-channel12
switchport description ice3270-1b:2b
no shutdown
• Add the new vfc interfaces to the VSAN database associated to the
correct VSAN active on the switch
vsan database
vsan 101 interface vfc11
vsan 101 interface vfc12

• Verify presence of either Fibre Channel or FCoE attached initiators and

targets with show flogi database, show fcns database
commands. The PWWN identifiers will be needed for the zoning to be
completed in the next step, and LUN mapping on the NetApp system

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 38
Storage Configuration, Aliases, Zoning and Zoneset
< Note, DCNM can also be used to perform these functions through a graphical interface >
• Create device aliases on each Cisco Nexus 5548 for the NetApp
controllers and each service profile using corresponding fabric
PWWN.
device-alias database
device-alias name ice3270-1a_2a pwwn 50:0a:09:81:8d:7d:92:bc
device-alias name ice3270-1b_2a pwwn 50:0a:09:81:9d:7d:92:bc
device-alias name VM-Host-Infra-01_A pwwn 20:00:00:25:b5:01:0a:1f
device-alias name VM-Host-Infra-02_A pwwn 20:00:00:25:b5:01:0a:1e

• Create zones for each service profile and assign devices as

members by using Fibre Channel aliases.
zone name VM-Host-Infra-01_A vsan 101
member device-alias VM-Host-Infra-01_A
member device-alias ice3270-1a_2a
member device-alias ice3270-1b_2a
zone name VM-Host-Infra-02_A vsan 101
member device-alias VM-Host-Infra-02_A
member device-alias ice3270-1a_2a
member device-alias ice3270-1b_2a

• Activate the zoneset.

zoneset name flexpod vsan 101
member VM-Host-Infra-01_A
member VM-Host-Infra-02_A
zoneset activate name flexpod vsan 101

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 39
Thank you.