TECDCN-2002

Next Generation Data

Center Infrastructure

Brenden Buresh – Principal Systems Engineer

Nicolas Delecroix – Technical Marketing Engineer
Thomas Scheibe – Product Management
Azeem Suleman – Principal Technical Marketing Engineer
Matthias Wessendorf – Technical Marketing Engineer
Who We Are?
Thomas Scheibe Brenden Buresh Matthias Wessendorf
Product Management Principal Systems Technical Marketing
@thomas0002 Engineer Engineer
@BrendenBuresh @matteq4er

Nicolas Delecroix Azeem Suleman

Technical Principal Engineer TME
Marketing Engineer @azeem_suleman
@ndelecroix

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda

• Introduction
• VXLAN / EVPN Technology
• NX-OS Standalone Programmability
• Data Center Network Manager
• Application Centric Infrastructure (ACI)
• ACI Programmability
• Conclusion

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
 • 08:30 – 10:30 (2 hours)
• 10:30 – 10:45 (Break)
Coffee and Lunch • 10:45 – 12:45 (2 hours)
Breaks • 12:45 – 14:30 (Lunch)
• 14:30 – 16:30 (2 hours)
• 16:30 – 16:45 (Break)
• 16:45 – 18:45 (2 hours)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
 Thomas Scheibe
Product Management
Introduction
How to Build the Network for the Cloud?

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Designing Networks is Easy …

… When Architected Correctly

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Mobility

Identity is Not Tied to Location

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Network Perspective Application Perspective
Connectivity, Infrastructure Endpoints, Compute & Storage Tiers

Automation

Security Perspective
Segmentation
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
 Workload Context is Key
ACTIVATION ANALYTICS

On-Premise
IT Intent
Lifecycle

Application Application Application

Performance Policy Cycle Portability ASSURANCE
Private
Management Cloud

Business Insights, Policy Inference, Infra Independence

User Satisfaction Enforcement, Forensics IaaS or On-premise
Public
Cloud

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Segmentation
Macro, Micro, Nano, Context Aware

1. End-end segmentation Compliance monitoring

needed Cloud 1

2. Flexibility to enforce in Cloud 2

Cloud 3
underlay and overlay

3. Mapping of domains/
tenants

Public & Private cloud Bare metal Virtual Container Fabrics Traditional network

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Abstraction to Capture Intent
Network, Application and Security
Profile
EPG EPG EPG

Consumer/ Consumer/
Provider Contracts Provider Contracts

EPG (End Point Group) = Security Zone, App Tier, Physical Location, ..
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Insights & Assurance
From Re-active to Pro-active

Visibility and Change

forensics Analysis

Process Compliance
inventory Checks

Application Connectivity
insight Analysis

Network Network
insights Verification

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
 Nexus Cloud Scale Foundation

Industry Leading ASIC Automation

Technology • Fabric Automation with
• Multi-speed 10/25G & 40/100G the same set of Nexus
• Flexible deployment profiles products (DCNM & ACI)
based on TCAM tile
configurations

Visibility Security
• FlowTable and Event • L2 (MACSec) and L3
triggered export (CloudSEC) Encryption
• Streaming data plane at line rate
statistics • Secure flow export

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Performance & Cost
Leverage Latest ASIC Technology

Server Silicon Cloud Scale Technology

X86 Processors 14nm 16nm ASICs ->7nm

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
 Road to 400G+
Optics

50/400G 100/400G
10/40G 100G QSFP
QSFP DD QSFP DD
Switch

10/40G 25/100G 50/400G 100/400G

ASIC Technology

28nm 14/16nm 7nm

2017 2018 2019 2020

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Pluggable Multispeed Interfaces

SFP QSFP

Pluggable Options Pluggable Options

• 100M SFP • 100M SFP (via QSFP)
• 1G SFP • 1G SFP (via QSA)
• 10G SFP+, Twinax, AOC • 10G SFP+, Twinax, AOC (via QSA)
• 25G SFP+, Twinax, AOC • 25G SFP+, Twinax, AOC (via SLIC)
• 40G QSFP, Twinax, AOC
• 50G Twinax, AOC (via SLIC)
• 100G QSFP, Twinax, AOC
Host: 1/10/25G Network: 10/40/50/100G

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
 Re-Use Cabling for 10/40/100G 40/100G BiDi
Shipping
With “2 Fiber” Optics

Examples
2 fiber MMF: BiDi
SMF: LR4, SM-SR
LC connector

SFP Note: Trade-off between

QSFP
fiber cost and optics cost
2 fiber 8 fiber MMF: SR4
LC connector MTP connector SMF: PSMF4
Breakout possible

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
QSFP DD – Avoiding the Sins of the Past
Lessons from the Past
1G  10/25G or 40G  100G transitions resulted in same high volume form factor being adopted. Why?

SFP SFP28 QSFP QSFP28 QSFP DD

1G to 10/25G 40G to 100G to

XEN CFP4
XFP 400G CFP
Journey PAK
Journey

X2 CPAK
CFP2

o System & network requirements do not change. Same port density per RU to maintain proven fabric designs
o Limited impact on system ecosystem – strong leverage
o Multi-speed switch port options – slower optics in higher speed ports
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Evolving Network Designs Routed Fabric
Traditional 3 Tier
VXLAN Bridging / Routing
DC Network Design
• VXLAN Flood & Learn
• VXLAN EVPN
• Separate Management Tools
(e.g. Nexus Fabric Manager)
DC Core

DC
PODs ACI
APIC • VXLAN Routing
• Policy Controller (APIC)
• Consistent policy across physical
VPC in Access
and virtual network
Routed Aggregation & Core
• Multi-hypervisor (VMware,
MSFT, OVS)
• Endpoint agnostic (bare metal,
VM, container)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
 API
Network Stack
Orchestration/ Policy
Multi Site
Federation

Insights Assurance Telemetry Data

Fabric Level API’s

Network Semantics
DCNM APIC (DCNM, APIC)
Application Semantics
(APIC)

Fabric (NX-OS) Fabric (ACI)

Device Level API’s

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
 Brenden Buresh
VXLAN / EVPN Principal SE

Technology
VXLAN Introduction
 Data Center “Fabric” Journey (Standalone)
Layer-3 HSRP HSRP

Layer-2

Spanning-Tree

Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2

Baremet al Hypervisor Hypervisor Hypervisor Baremet al Hypervisor Baremet al Baremet al Hypervisor Hypervisor

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
 Overlay Based Data Center: Edge Devices
Network Overlays Host Overlays

Overlay Overlay
VTEP VTEP VTEP VTEP - - - -

Hybrid Overlays
VTEP VTEP VTEP VTEP
Baremetal Baremetal Baremetal Baremetal Hypervisor Hypervisor Hypervisor Hypervisor

• Router/Switch End-Points
Overlay • Virtual End-Points only
• Protocols for Resiliency/Loops
• Single Admin Domain
• Traditional VPNs
• VXLAN, NVGRE, STT
•
- - VTEP VTEP
VXLAN, OTV, VPLS, LISP, FP

VTEP VTEP
Hypervisor Hypervisor Baremetal Baremetal

• Physical and Virtual

• Resiliency and Scale
• Cross-Organizations/Federation
• Open Standards
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Understanding Overlay Technologies

Overlay Services
• Layer-2 Underlay Transport
Tunnel Encapsulation
• Layer-3 Network
• Layer-2 and Layer-3

Data-Plane
Control-Plane
• Overlay Layer-2/Layer-3 Unicast Traffic
• Peer-Discovery
• Overlay Broadcast, Unknown Unicast,
• Route Learning and Distribution
• Local Learning
Multicast traffic (BUM traffic) forwarding
• Ingress Replication (Unicast)
• Remote Learning
• Multicast

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Overlay Taxonomy - Underlay

Layer-3
Interface Spine Spine Spine Spine

Peering

Underlay
Edge Device Leaf Leaf Leaf Leaf Leaf Leaf Leaf

LAN
Segment

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

Virtual
Server Physical
Server

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
Overlay Taxonomy - Overlay

Tunnel Encapsulation
Spine
(VNI Namespace)
Spine Spine Spine

Overlay
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

LAN
Segment

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

Virtual
Server Physical
Server VTEP: VXLAN Tunnel End-Point
VNI/VNID: VXLAN Network Identifier
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
 Introducing VXLAN

MAC 802.1q IP Payload CRC

Src and Dst
Src, Dst VTEP VTEP IP
and Hop-by- UDP Dst VXLAN
Address Port 4789 VNI
Hop MAC Original Layer-2 Frame

Data-Plane
Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC
(VXLAN)
14-byte + 20-byte +8-byte + 8-byte* = 50 Bytes
of total overhead UDP Src Port
Hash of L2/L3/L4
headers of
original Frame

*plus 4-byte if IEEE 802.1q exists as part of Inner MAC Header

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
VXLAN Frame Format – MAC in IP Encapsulation
Field Value Bites Total Field Value Bites Total
Dest. MAC Address Next-Hop MAC Address 48 Source Port L2/L3/L4 Hash 16

(4 Bytes Optional)
Src. MAC Address Next-Hop MAC Address 48

8 Bytes
Destination Port 4789 (UDP) 16

14 Bytes
VLAN Type 0x8100 16 UDP Length 16
VLAN ID Tag 16 Checksum 0x0000 16
Ether Type 0x0800 16

Outer UD VXLA CR
Outer IP Inner MAC Payload
MAC P N C

Field Value Bites Total

Field Value Bites Total
IP Header Misc. Data 72
VXLAN Flags RRRRIRRR 8
Protocol 0x11 (UDP) 8
20 Bytes

8 Bytes
Reserved 24
Header Checksum Various 16
VNI 16M Possible Segments 24
Source IP Src, VTEP IP 32
Reserved 8
Destination IP Dest. VTEP IP 32

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
No Path Diversity

Spine
• Equal Cost Multi-Pathing (ECMP)
uses Header information to form
Path Diversity

Leaf
• Some Tunnel Protocol provide no
Leaf
101010110101010
10101010
Baremetal
AS#65500 diversity in IP or Protocol Header Baremetal

• As a Result, all Packets travel

the same Path
Spine

• No Path Diversity or Entropy

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Introducing VXLAN – Entropy

Spine
• VXLAN provides variable UDP
Source Port in Outer Header
• Hash of the inner Layer-2/Layer-
VTEP VTEP
101010110101010
10101010
Baremetal
AS#65500 3/Layer-4 Headers of the original Baremetal

Ethernet Frame.
• Enables entropy for ECMP Load
Spine
balancing in the Network

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Introducing VXLAN – Entropy
Spine

VTEP VTEP

AS#65500

Entropy Spine

happens here

Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC
Data-Plane (VXLAN)
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
 A Scale Out Architecture

More Spine – More Bandwidth – More Resiliency

• Leaf Spine Spine Spine Spine

• Smallest Operational Entity

• Spines
• Wide vs. Big
Leaf Leaf Leaf Leaf Leaf Leaf Leaf

• Uplinks
• Symmetric to all Spines or Pods
More Leaf – More Ports – More Capacity
• SAYG: Scale as You Grow

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
 Folded Clos Topology – Device Roles
• Spine
• Interconnecting Leafs and Border
Leafs Spine Spine Spine Spine

• Leaf (VTEP)
• Virtual Machines
• Physical Machines
•
Leaf Leaf Leaf Leaf Leaf Leaf Leaf
FEXes
• 3rd Party Switches
• UCS FIs
• Blade Switches WAN

• Border Leaf (VTEP)

• External Connectivity

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
 Folded Clos Topology – Device Roles
• Border Spine WAN
• Interconnecting Leafs and Border
Leafs Spine Spine Spine Spine

• External Connectivity

• Leaf (VTEP)
• Virtual Machines
• Physical Machines Leaf Leaf Leaf Leaf Leaf Leaf Leaf

• FEXes
• 3rd Party Switches
• UCS FIs
• Blade Switches

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
 The Super-Spine

SuperSpine

• Scale Out
SuperSpine SuperSpine

• Not Limited to Port Density

• Simpler Capacity Planning

• Beyond a Single Server Room

• Allows Interconnecting Pods
Spine Spine Spine Spine Spine Spine Spine Spine

• Retains Intra-Pod Topology with

Flexible Inter-Pod Connectivity
Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf

POD 1 POD 2

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
VXLAN Gateway Types
Egress packet is IEEE 802.1q
Ingress VXLAN packet tagged interface. packet is
on RED segment VTEP BRIDGED to new VLAN

• VXLAN to VLAN Bridging

• (Layer-2 Gateway) VXLAN Layer-2
Gateway

Egress packet is IEEE 802.1q

Ingress VXLAN packet tagged interface. packet is
on RED segment VTEP ROUTED to new VLAN

• VXLAN-to-VLAN Routing
• (Layer-3 Gateway)
VXLAN Router

Ingress VXLAN packet Egress VXLAN packet is

on RED segment VTEP ROUTED to new segment

• VXLAN-to-VXLAN Routing
• (Layer-3 Gateway)
VXLAN Router

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
EVPN Deep Dive
 What is VXLAN and EVPN ?
• EVPN
• VXLAN
• Standards based Control-Plane
• Standards based Encapsulation
• RFC 7432
• RFC 7348
• Uses Multiprotocol BGP
• Uses UDP-Encapsulation
• Uses Various Data-Planes
• Transport Independent
• VXLAN (EVPN-Overlay), MPLS,
• Layer-3 Transport (Underlay)
Provider Backbone (PBB)
• Flexible Namespace
• Many Use-Cases Covered
• 24-bit field (VNID) provides ~16M
• Bridging, MAC Mobility, First-Hop
unique identifier
& Prefix Routing, Multi-Tenancy
• Allows Segmentation
(VPN)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Introducing Ethernet VPN (EVPN)

EVPN MP-BGP – RFC 7432

MPLS Provider Backbone Bridges Overlay (NVO3)

(draft-ietf-l2vpn-evpn) (draft-ietf-l2vpn-pbb-evpn) (draft-ietf-bess-evpn-overlay)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
 VXLAN and EVPN Related RFCs & Drafts (IETF)
ID Title Category
RFC 7348 Virtual Extensible Local Area Network Data Plane

RFC 7432 BGP MPLS based Ethernet VPNs Control Plane

draft-ietf-bess-evpn-overlay A Network Virtualization Overlay Solution using EVPN Control Plane

draft-ietf-bess-evpn-inter-subnet-forwarding Integrated Routing and Bridging in EVPN Control Plane

draft-ietf-bess-l2vpn-evpn-prefix-
IP Prefix Advertisement in E-VPN Control Plane
advertisement

draft-tissa-nvo3-oam-fm NVO3 Fault Management / OAM Management Plane

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
EVPN Layer-2 Services (1)
Single Subnet per EVI Multiple Subnets per EVI
• VLAN-based • VLAN-aware

• Per EVI BGP Route Distinguisher / Router Target per EVI / VNI
• BGP Route-Target constrain mechanism to limit propagation (import/export)

• 1:1:1 mapping • 1:1:N mapping

• VNI to EVI to Single Broadcast • VNI to EVI to Multiple Broadcast
Domain (Bridge Domain) Domains (Bridge Domains)
• Ethernet Tag ID must be 0 • Ethernet Tag ID is to differentiate
Bridge Domains

VID = VLAN ID
VNI = VXLAN Network Identifier
EVI = EVPN Virtual Instance TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
EVPN Layer-2 Services (1a)
(draft-ietf-bess-evpn-overlay – Section 5.1.2)

• VLAN-based

VID
10 VNI EVI

Route Target: 65000:30000

VID = VLAN ID
VNI = VXLAN Network Identifier
EVI = EVPN Virtual Instance TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN Layer-2 Services (1a)
(draft-ietf-bess-evpn-overlay – Section 5.1.2)

• VLAN-based

VID
10 VNI EVI

Route Target: 65000:30000

VID = VLAN ID
VNI = VXLAN Network Identifier
EVI = EVPN Virtual Instance TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN Layer-2 Services (1b)
(draft-ietf-bess-evpn-overlay – Section 6.1)

• VLAN-based
[2]:[0]:[0]:[48]:[0050.569f.d495]:[32]:[192.168.22.33]

VID
10 VNI EVI

Route Target: 65000:30000

VID = VLAN ID
VNI = VXLAN Network Identifier
EVI = EVPN Virtual Instance © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN Layer-2 Services (1b)
(draft-ietf-bess-evpn-overlay – Section 6.1)

• VLAN-based
[2]:[0]:[0]:[48]:[0050.569f.d495]:[32]:[192.168.22.33]

VID
10 VNI EVI

Route Target: 65000:30000

VID = VLAN ID
VNI = VXLAN Network Identifier
EVI = EVPN Virtual Instance © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
 EVPN Layer-2 Services (1c)
(RFC 7432 – Section 6.3)

• VLAN-based • VLAN-aware
[2]:[0]:[0]:[48]:[0050.569f.d495]:[32]:[192.168.22.33]

VID
10
VID
VNI EVI VNI EVI
VID
10 20
VID
30

[2]:[0]:[20]:[48]:[0050.569f.d495]:[32]:[192.168.22.33]

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
EVPN IP-VRF Services (2)
Interface-Less Model Interface-Full Model (2 Modes)
• Route-Type 5 only • Core-facing IRB
• Next-Hop is remote VTEP • Unnumbered Core-facing IRB
(Optional)
• Two extended communities
• Encapsulation Extended Community • Route-Type 5
• Router’s MAC Address (remote VTEP) • Next-Hop is remote IRB
• One or two extended communities
• Encapsulation Extended Community
• Router’s MAC Address (remote VTEP)

• Route-Type 2
• Containing Router MAC or MAC/IP

Route Type 2 = MAC/IP Route

Route Type 5 = IP Prefix Route
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
EVPN IP-VRF Services (2a)
(draft-ietf-bess-evpn-prefix-advertisement – Section 5.4.1)

• Interface-Less

NVE IP: 10.22.22.34

EVPN Router MAC: 0200.0ADE.DE22

VTEP VTEP

[5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop)
Update Encap:8 (VXLAN)
Router MAC:0200.0ade.de22

TECDCN-2002
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
EVPN IP-VRF Services (2a)
(draft-ietf-bess-evpn-prefix-advertisement – Section 5.4.1)

• Interface-Less

NVE IP: 10.22.22.34

EVPN Router MAC: 0200.0ADE.DE22

VTEP VTEP

[5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop)
Update Encap:8 (VXLAN)
Router MAC:0200.0ade.de22

TECDCN-2002
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
EVPN IP-VRF Services (2b)
(draft-ietf-bess-evpn-prefix-advertisement – Section 5.4.2)

• Interface-Less • Interface-Full (Core-facing IRB)

NVE IP: 10.22.22.34 NVE IP: 10.22.22.34

EVPN Router MAC: 0200.0ADE.DE22

EVPN Router MAC: 0200.0ADE.DE22

VTEP VTEP VTEP VTEP

[5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0] [5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop) BGP 10.22.22.34 (Next-Hop)
Update Encap:8 (VXLAN) Update Encap:8 (VXLAN)
Router MAC:0200.0ade.de22 Router MAC:0200.0ade.de22

[2]:[0]:[0]:[48]:[0200.0ade.de22]:[32]:[10.22.22.34]
BGP 10.22.22.34 (Next-Hop)
Update Encap:8 (VXLAN)
Router MAC:0200.0ade.de22

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN IP-VRF Services (2b)
(draft-ietf-bess-evpn-prefix-advertisement – Section 5.4.2)

• Interface-Less • Interface-Full (Core-facing IRB)

NVE IP: 10.22.22.34 NVE IP: 10.22.22.34

EVPN Router MAC: 0200.0ADE.DE22

EVPN Router MAC: 0200.0ADE.DE22

VTEP VTEP VTEP VTEP

[5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0] [5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop) BGP 10.22.22.34 (Next-Hop)
Update Encap:8 (VXLAN) Update Encap:8 (VXLAN)
Router MAC:0200.0ade.de22 Router MAC:0200.0ade.de22

[2]:[0]:[0]:[48]:[0200.0ade.de22]:[32]:[10.22.22.34]
BGP 10.22.22.34 (Next-Hop)
Update Encap:8 (VXLAN)
Router MAC:0200.0ade.de22

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN IP-VRF Services (2c)
(draft-ietf-bess-evpn-prefix-advertisement – Section 5.4.3)

• Interface-Less • Interface-Full
(Unnumbered Core-facing IRB)

NVE IP: 10.22.22.34 NVE IP: 10.22.22.34

EVPN Router MAC: 0200.0ADE.DE22

EVPN Router MAC: 0200.0ADE.DE22

VTEP VTEP VTEP VTEP

[5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0] [5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop) BGP 10.22.22.34 (Next-Hop)
Update Encap:8 (VXLAN) Update Encap:8 (VXLAN)
Router MAC:0200.0ade.de22 Router MAC:0200.0ade.de22

[2]:[0]:[0]:[48]:[0200.0ade.de22]:[0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop)
Update Encap:8 (VXLAN)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN IP-VRF Services (2c)
(draft-ietf-bess-evpn-prefix-advertisement – Section 5.4.3)

• Interface-Less • Interface-Full
(Unnumbered Core-facing IRB)

NVE IP: 10.22.22.34 NVE IP: 10.22.22.34

EVPN Router MAC: 0200.0ADE.DE22

EVPN Router MAC: 0200.0ADE.DE22

VTEP VTEP VTEP VTEP

[5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0] [5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop) BGP 10.22.22.34 (Next-Hop)
Update Encap:8 (VXLAN) Update Encap:8 (VXLAN)
Router MAC:0200.0ade.de22 Router MAC:0200.0ade.de22

[2]:[0]:[0]:[48]:[0200.0ade.de22]:[0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop)
Update Encap:8 (VXLAN)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN IRB Services (3)

Symmetric Inter-Subnet Asymmetric Inter-Subnet

Forwarding Forwarding
• Bridge->Route/Route->Bridge • Bridge->Route->Bridge
• Symmetric VNI in both • Different (Asymmetric) VNI
directions depending on directions
• Adjacency contains Remote • Adjacency contains Remote
VTEP,VRF VTEP,VRF and End-Points
• Optimal for Scale • Potential Sub-Optimal for Scale
• Flexible Configuration • Consistent Configuration

VTEP = VXLAN Tunnel End-Point

VRF = Virtual Routing and Forwarding
VNI = VXLAN Network Identifier
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
EVPN IRB Services (3a)
(Traditional Bridging – Depending on EVPN Layer-2 Services)

• Symmetric IRB • Asymmetric IRB

VNI 30000 (L2VNI) VNI 30000 (L2VNI)

V1 V2 V1 V2

MA MA
MAC IP IP MAC MAC IP IP MAC
C C

192.168.22.33 192.168.22.44 192.168.22.33 192.168.33.44

192.168.22.44

Bridge Bridge

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
EVPN IRB Services (3b)
(draft-ietf-bess-evpn-inter-subnet-forwarding – Section 4 and 5)

• Symmetric IRB • Asymmetric IRB

VNI 50000 (L3VNI) VNI 40000 (L2VNI)

VNI 30000 (L2VNI)

V1 V2 V1 V2

MA MA MA MA
IP IP IP IP
C C C C

192.168.22.33 192.168.33.44 192.168.22.33 192.168.33.44

Bridge -> Route -> Route -> Bridge Bridge -> Route -> Bridge

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
EVPN IRB Services (3c)
(draft-ietf-bess-evpn-inter-subnet-forwarding – Section 5.1)
• Symmetric IRB
VNI 50000 (L3VNI)

V1 V2

MA MA
IP IP
C C

192.168.22.33 192.168.33.44

[5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop)
Update Encap:8 (VXLAN)
Router MAC:0200.0ade.de22

TECDCN-2002

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN IRB Services (3c)
(draft-ietf-bess-evpn-inter-subnet-forwarding – Section 5.1)
• Symmetric IRB
VNI 50000 (L3VNI)

V1 V2

MA MA
IP IP
C C

192.168.22.33 192.168.33.44

[5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop)
Update Encap:8 (VXLAN)
Router MAC:0200.0ade.de22

TECDCN-2002

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN IRB Services (3d)
(draft-ietf-bess-evpn-inter-subnet-forwarding – Section 4.1)
• Symmetric IRB • Asymmetric IRB
VNI 50000 (L3VNI) VNI 40000 (L2VNI)

VNI 30000 (L2VNI)

V1 V2 V1 V2

MA MA MA MA
IP IP IP IP
C C C C

192.168.22.33 192.168.33.44 192.168.22.33 192.168.33.44

[5]:[0]:[0]:[24]:[192.168.22.0]:[10.22.22.34]
BGP
10.22.22.34 (Next-Hop)
Update
Encap:8 (VXLAN)

[2]:[0]:[0]:[48]:[0200.0ade.de22]:[32]:[10.22.22.34]
BGP 10.22.22.34 (Next-Hop)
Update Encap:8 (VXLAN)
Router MAC:0200.0ade.de22
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN IRB Services (3e)
• Symmetric IRB and Asymmetric IRB is NOT interoperable per-se
• Routing is implemented different
• Symmetric IRB: Bridge -> Route -> Route -> Bridge
• Asymmetric IRB: Bridge -> Route -> Bridge
• Symmetric IRB uses Route-Type 2 with two VNI
• L3VNI for routing and L2VNI for bridging
• Asymmetric IRB uses Route-Type2 and Route-Type 5
• Type 2 with L2VNI for bridging and inter-subnet forwarding (known VNI/VTEP)
• Type 5 with L3VNI for inter-subnet forwarding (see IP-VRF Services)
• If implemented

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
Conclusions - Cisco’s EVPN Implementation
• VLAN-based • VLAN-aware
• VLAN to VNI to EVI • Multiple VLAN to VNI to EVI
• More granular control (route-target) • No true MAC separation (potential)

• Interface-Less • Interface-Full
• Follows classic routing • Additional overhead (2 routes and additional
lookup)
• No need for MAC routes in routing
• Asymmetric IRB
• Symmetric IRB • Extensive usage of Adjacency Tables
• Adjacency Tables are preserved • More Centralized Gateway-like
• Configuration is flexible • “Consistent” configuration necessary if Distributed
Gateway is required

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
EVPN Operations
 EVPN - Host and Subnet Route Distribution
• Host Route Distribution
RR RR decoupled from the Underlay
Spine Spine Spine Spine
protocol

Overlay • Use MultiProtocol-BGP (MP-

BGP) on the Leaf nodes to
Leaf Leaf Leaf Leaf Leaf Leaf Leaf distribute internal Host/Subnet
Routes and external reachability
information
• Route-Reflectors (RR) deployed
for scaling purposes

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
 EVPN Control Plane - Host and Subnet Routes
• BGP EVPN NLRI*

Spine Spine Spine Spine

• Host MAC (Route Type 2)
• MAC only, Single VNI, Single
Route Target
Overlay
• Host MAC+IP (Route Type 2)
Leaf Leaf Leaf Leaf Leaf Leaf Leaf • MAC and IP, Two VNI, Two Route
Target, Router MAC

• Internal and External Subnet

Prefixes (Route Type 5)
• IP Subnet Prefix, Single VNI,
Single Route Target

*NLRI: Network Layer Reachability Information (BGP Update Format)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
 Host Advertisements
Type
Type MAC // Length
MAC Length L2VNI // RT
L2VNI IP / Length L3VNI / RT Next-Hop Seq.

22 0000.3001.1101 // 48
0000.3001.1101 48 3001, 65500:3001 10.200.200.101
Spine Spine Spine Spine

22 0000.3001.1102 // 48
0000.3001.1102 48 3001, 65500:3001 10.200.200.104

Overlay • Host MAC (Route Type 2)

2 0000.3002.2101 / 48 3002, 65500:3002 10.200.200.107
• MAC
• MPLS Label1 (L2VNI*)
•
Leaf Leaf Leaf Leaf Leaf Leaf Leaf
101010110101
01010101010
101010110101
01010101010
101010110101
01010101010
Route Target for MAC-VRF

• MAC attributes are Mandatory

Baremetal Baremetal Baremetal

Host A Host B Host C

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101

*L2VNI: VNI for all Bridging operation (”VLAN-VNI”)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
 Ethernet Tag
Ethernet Identifier
V2# show bgp l2vpn evpn 0000.3001.1101 (Ethtag)
Segment
Identifier (ESI) MAC Address MAC
Route Type: Length Address
BGP routing table information
MAC/IP for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.10.10.101:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[0000.3001.1101]:[0]:[0.0.0.0]/216,
version 4
Paths: (1 available, best #1)
Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked

Advertised path-id 1
Next-Hop
IP Address
Path type: internal, path
L2VNIis valid, is best path, no labeled nexthop
AS-Path: NONE, path(MPLS
sourced internal to AS
Label1)
10.200.200.101 (metric 3) fromL2VNI
10.10.10.201 (10.10.10.201)
Encap:8
Route Target
Origin IGP, MED not set, localpref 100, weight 0
VXLAN
Received label 3001
Extcommunity: RT:65500:3001 ENCAP:8
Originator: 10.10.10.101 Cluster list: 10.10.10.201

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
 Host Advertisements
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.

2 0000.3001.1101 / 48
Spine Spine
3001, 65500:3001
Spine Spine
192.168.10.101 /32 • Host MAC+IP
5000, 65500:5000 10.200.200.101(Route Type 2)

• MAC and IP
2 0000.3001.1102 / 48 3001, 65500:3001 192.168.10.102 /32 5000, 65500:5000 10.200.200.104
• MPLS Label1 (L2VNI)
2 0000.3002.2101 / 48
Overlay 3002, 65500:3002 192.168.20.101 /32 • Route
5000, 65500:5000 Target for MAC-VRF
10.200.200.107

• MPLS Label2 (L3VNI*)

Leaf
101010110101
01010101010
Leaf Leaf Leaf
101010110101
01010101010
Leaf Leaf Leaf
101010110101
01010101010
• Route Target for IP-VRF
• Router MAC

• IP Attributes are Optional

Baremetal Baremetal Baremetal
• Populated through ARP/ND
Host A Host B Host C
MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101

*L3VNI: VNI for all Routing operation (”VRF-VNI”)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
 Ethernet Tag
Ethernet Identifier
V2# show bgp l2vpn evpn 0000.3001.1101 (Ethtag)
Segment
Identifier (ESI) MAC Address MAC
Route Type: Length Address
BGP routing table information
MAC/IP for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.10.10.101:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[0000.3001.1101]:[32]:[192.168.10.101]/272,
version 4
Paths: (1 available, best #1) IP Address
Length
Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked IP Address

Advertised path-id 1
Next-Hop L3VNI
IP Address
Path type: internal,L2VNI
path is (MPLS
valid, is best path, no labeled nexthop
Label2)
AS-Path: NONE, path
(MPLSsourced
Label1) internal to AS
10.200.200.101 (metric 3) from 10.10.10.201 (10.10.10.201)
Encap:8
Origin IGP, MED not set, localpref 100, weight 0 VXLAN
Received label 3001 5000
Extcommunity: RT:65500:3001 RT:65500:5000 ENCAP:8 Router MAC:0200.0ade.de01
Originator: 10.10.10.101 Cluster list: 10.10.10.201
L2VNI L3VNI
Route Target Router MAC
Route Target

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
 Subnet Route Advertisements
Type IP / Length L3VNI / RT Next-Hop Seq.

5 192.168.10.0 /24 5000, 65500:5000 10.200.200.101

Spine Spine Spine Spine
• Internal and External Subnet
Prefixes (Route Type 5)
Overlay • IP Prefix
• MPLS Label (L3VNI)
Leaf Leaf Leaf Leaf Leaf Leaf Leaf
• Route Target for IP-VRF
•
101010110101
01010101010

Router MAC

• Populated through External

Routing Protocol
Subnet A
192.168.10.0/24

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
 Subnet Route Advertisements
Type IP // Length
IP Length L3VNI // RT
L3VNI RT Next-Hop Seq.

5 192.168.10.0 /24
192.168.10.0
Spine
/24
Spine
5000, 65500:5000
5000, 65500:5000
Spine Spine
10.200.200.101 • IP Prefix Learning
5 192.168.10.0 /24
192.168.10.0 /24 5000, 65500:5000
5000, 65500:5000 10.200.200.104
• via BGP with VRF-Lite
• via LISP on Nexus 7000/7700
5 192.168.20.0 /24
Overlay 5000, 65500:5000 10.200.200.107 • via other routing protocol (static
or dynamic)
Leaf Leaf Leaf Leaf Leaf Leaf Leaf

• Default: Export of IP Host and IP

101010110101 101010110101
01010101010 01010101010

Prefix Routes advertisements

• Filter and Summarize where
appropriate
Subnet A Subnet A Subnet B
192.168.10.0/24 192.168.10.0/24 192.168.20.0/24

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
 Ethernet Tag
Ethernet Identifier
Segment
V2# show bgp l2vpn evpn 192.168.10.0 (Ethtag)
Identifier (ESI) IP Address
Route Type: IP Address
Length family
BGP routing table information
IP Prefix for VRF default, address L2VPN EVPN
Route Distinguisher: 10.10.10.101:3
BGP routing table entry for [5]:[0]:[0]:[24]:[192.168.10.101]/224,
version 4
Paths: (1 available, best #1)
Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked

Advertised path-id 1
Next-Hop
IP Address
Path type: internal, path
L3VNIis valid, is best path, no labeled nexthop
AS-Path: NONE, path (MPLS
sourced internal to AS
Label)
10.200.200.101 (metric 3) from L3VNI
10.10.10.201 (10.10.10.201)
Encap:8
Origin IGP, MED not set, localpref 100, weight 0 Router MAC
Route Target VXLAN
Received label 5000
Extcommunity: RT:65500:5000 ENCAP:8 Router MAC:0200.0ade.de01
Originator: 10.10.10.101 Cluster list: 10.10.10.201

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
 VXLAN and BGP EVPN – Putting it Together
Control-Plane (BGP EVPN)
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.
3001 5000
2 0000.3001.1101/48 192.168.10.101/32 10.200.200.101
65500:3001 65500:5000

Dst VTEP IP L2VNI Dst MAC Dst IP

10.200.200.101 3001 0000.3001.1101 192.168.10.101

Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC
Data-Plane (VXLAN)

Bridging

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
 VXLAN and BGP EVPN – Putting it Together
Control-Plane (BGP EVPN)
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.
3001 5000
2 0000.3001.1101/48 192.168.10.101/32 10.200.200.101
65500:3001 65500:5000

Dst VTEP IP L3VNI Router MAC Dst IP

10.200.200.101 5000 0200.0ade.de01 192.168.10.101

Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC
Data-Plane (VXLAN)

Routing

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
 Routing and the Router MAC – Ethernet
Router MAC

SMAC DMAC SIP DIP

Payload
0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101

SMAC DMAC SIP DIP SMAC DMAC SIP DIP

Payload Payload
0000.3001.1101 0000:3002.2101 192.168.10.101 192.168.20.101 0000.3001.1101 0000.3002.2101 192.168.10.101 192.168.20.101

Switch Switch
SVI10 SVI20
192.168.10.1 192.168.20.1

interface: Eth2/1 interface: Eth2/1

MAC: 0200.0ade.de01 MAC: 0200.0ade.de07
Baremetal IP: 10.200.200.1 IP: 10.200.200.7 Baremetal

Host A Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
 Routing and the Router MAC – VXLAN
Router MAC

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101

SMAC DMAC SIP DIP SMAC DMAC SIP DIP

Payload Payload
0000.3001.1101 0000:3002.2101 192.168.10.101 192.168.20.101 0000.3001.1101 0000.3002.2101 192.168.10.101 192.168.20.101

SVI10 SVI20
192.168.10.1 VTEP
VXLAN VTEP 192.168.20.1

interface: NVE1 interface: NVE1

MAC: 0200.0ade.de01 MAC: 0200.0ade.de07
Baremetal IP: 10.200.200.1 IP: 10.200.200.7 Baremetal

Host A Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
 Packet Walk – ARP Request

VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf

Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
ARP Request for 192.168.10.102 Spine

SMAC: DMAC: SIP DIP VXLAN SMAC DMAC

ARP Request for
0000.3001.1101 FFFF.FFFF.FFFF 192.168.10.102
10.200.200.101 239.0.0.1 30001 0000.3001.1101 FFFF.FFFF.FFFF

Spine

TOR3 TOR4
VLAN 101 (Green)

Leaf Leaf

Host

Host B
MAC: 0000.3001.1102
VXLAN EVPN

VLAN 202 (Blue)

IP: 192.168.10.102
Host

ARP Request for 192.168.10.102

SMAC: DMAC:
Host C
0000.3001.1101 FFFF.FFFF.FFFF MAC: 0000.3002.2101
IP: 192.168.20.101

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
 Packet Walk – ARP Response

VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf

Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
ARP Response for 192.168.10.102 Spine

SMAC: DMAC:
0000.3001.1102 0000.3001.1101
SIP DIP VXLAN SMAC DMAC
ARP Response for
192.168.10.102
10.200.200.103 10.200.200.101 30001 0000.3001.1102 0000.3001.1101
Spine

TOR3 TOR4
VLAN 101 (Green)

Leaf Leaf

Host

Host B
MAC: 0000.3001.1102
VXLAN EVPN

VLAN 202 (Blue)

IP: 192.168.10.102
Host

ARP Response for 192.168.10.102

SMAC: DMAC:
Host C
0000.3001.1102 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.20.101

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
 Packet Walk – Bridging

VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf

Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine
SMAC DMAC SIP DIP

0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102

SIP DIP VXLAN SMAC DMAC SIP DIP

Spine
Payload
10.200.200.101 10.200.200.103 30001 0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102

TOR3 TOR4
VLAN 101 (Green)

Leaf Leaf

Host

Host B
MAC: 0000.3001.1102
VXLAN EVPN

VLAN 202 (Blue)

IP: 192.168.10.102
Host

SMAC DMAC SIP DIP Host C

0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102 MAC: 0000.3002.2101
IP: 192.168.20.101

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
 Packet Walk – Routing

VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf

Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
SMAC DMAC SIP DIP Spine

0000.3001.1101 2020.0000.AAAA 192.168.10.101 192.168.20.101

SIP DIP VXLAN SMAC DMAC SIP DIP

Spine
Payload
10.200.200.101 10.200.200.104 50001 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101

TOR3 TOR4
SMAC DMAC SIP DIP
VLAN 101 (Green)

Leaf Leaf
2020.0000.AAAA 0000.3002.2101 192.168.10.101 192.168.20.101

Host

Host B
MAC: 0000.3001.1102
VXLAN EVPN

VLAN 202 (Blue)

IP: 192.168.10.102
Host

Host C
MAC: 0000.3002.2101
IP: 192.168.20.101

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
 Packet Walk – Routing (Silent Host)

VLAN 101 (Green)

SIP DIP VXLAN SMAC DMAC SIP DIP

10.200.200.101
TOR1
10.200.200.102 50001 0200.0ade.de01 0200.0ade.de07
TOR2
192.168.10.101 192.168.20.101
Payload

Host
Leaf Leaf

Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine

VLAN 202 (Blue)

SMAC DMAC SIP DIP

0000.3001.1101 2020.0000.AAAA 192.168.10.101 192.168.20.101

Spine

TOR3 TOR4
VLAN 101 (Green)

Leaf Leaf

Host

Host B
MAC: 0000.3001.1102
VXLAN EVPN

VLAN 202 (Blue)

IP: 192.168.10.102
Host

Host C
MAC: 0000.3002.2101
IP: 192.168.20.101

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
 Packet Walk – Routing (Silent Host)

VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf

Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine

VLAN 202 (Blue)

SIP DIP VXLAN SMAC DMAC
ARP Request for
192.168.20.101
10.200.200.102 239.0.0.1 30002 AGM FFFF.FFFF.FFFF

Spine

TOR3 TOR4
VLAN 101 (Green)

Leaf Leaf
ARP Request for 192.168.20.101
Host
SMAC: DMAC:
AGM FFFF.FFFF.FFFF
Host B
MAC: 0000.3001.1102
VXLAN EVPN

VLAN 202 (Blue)

IP: 192.168.10.102
Host

Host C
MAC: 0000.3002.2101
IP: 192.168.20.101

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
 Packet Walk – Routing (Silent Host)

VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf

Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine

VLAN 202 (Blue)

Spine

TOR3 TOR4
VLAN 101 (Green)

Leaf Leaf

Host

Host B
MAC: 0000.3001.1102
VXLAN EVPN

VLAN 202 (Blue)

IP: 192.168.10.102
Host

Host C
ARP Response for 192.168.20.101MAC: 0000.3002.2101
IP: 192.168.20.101
SMAC: DMAC:
0000.3002.2102 AGM
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
VXLAN Design
Considerations
Underlay Design Review
MTU and Overlays

•
•

•
•

*Cisco Nexus 5600 only supports a MTU of 9192 Byte for Layer-3 Traffic
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
 Interface Principles

• Spine Spine Spine Spine

• Leaf Leaf Leaf Leaf Leaf Leaf Leaf

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
 IP Addressing Principles
Routing
Identifier Rendezvous
Point
• p2p* Links / IP
Unnumbered
• Spine Spine Spine Spine

•
Leaf Leaf Leaf Leaf Leaf Leaf Leaf

• VTEP
Routing
• Loopback Identifier

• p2p Agg: 10.1.1.0/24

RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
RP Agg: 10.254.254.0/24
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
 Unicast Routing – OSPF and IS-IS

•
•
• •
• •
•
• •

•
• •

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
Unicast Routing – BGP

• eBGP Underlay Routing –

Service Provider style
• Two Different Models
• Two-AS
• Multi-AS

• BGP is a Distance Vector

Protocol
• actually Path Vector Protocol
• AS* are used to calculate the
Path (AS_Path)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
Unicast Routing – eBGP Two-AS Model

All-Spine AS#65500
Spine Spine Spine Spine •
•
•

Leaf Leaf Leaf Leaf Leaf Leaf Leaf •

•

All-Leaf AS#65501 •
•

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
Unicast Routing – eBGP Multi-AS Model

All-Spine AS#65500 •
•
Spine Spine Spine Spine

•
Leaf Leaf Leaf Leaf Leaf Leaf Leaf
•

•
•

•
•

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
Unicast Routing – eBGP Model

Spine Spine Spine Spine

•
•
•
•
Leaf Leaf Leaf Leaf Leaf Leaf Leaf

•
•
•
•

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
 Multicast Enabled Underlay – PIM ASM

• PIM Any-Source-Multicast (ASM) RP RP

Spine Spine Spine Spine

• Platform Support
• Nexus 9000 / Nexus 7000 (F3/M3)
• ASR 1000 / ASR 9000
Underlay
• RP Redundancy Leaf Leaf Leaf Leaf Leaf Leaf Leaf

• PIM Anycast-RP or MSDP

• Source-Trees (Unidirectional)
• 1 Source Tree per VTEP per
Multicast Group
Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
Underlay – PIM ASM with PIM Anycast-RP

RP RP
Spine Spine Spine Spine

S,G S,G
S,G S,G
Underlay S,G

Leaf Leaf Leaf Leaf Leaf Leaf Leaf

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 97
 Why Do I Need Multicast Again?

Destination Group
239.1.1.1
Spine (0100.5E01.0101)
Spine Spine Spine

3
MAC VNI VTEP MAC VNI VTEP
0000.3001.1101 30001 E1/12
Overlay
3 0000.3002.2101 30002 E1/4

2
SMAC:
VTEP
MAC_LEAF1
VTEP VTEP VTEP VTEP VTEP VTEP
ARP Request for 192.168.10.102
DMAC: 0100.5E01.0101

Underlay
Src MAC: 0000.3001.1101
Dst MAC: FFFF.FFFF.FFFF SIP: IP_LEAF1
DIP: 239.1.1.1 ARP Request for 192.168.10.102
1 4 Src MAC: 0000.3001.1101
UDP Dst MAC: FFFF.FFFF.FFFF

VXLAN VNID: 30001

MAC VNI
VNI VTEP
VTEP
Overlay

ARP Request
Baremetal Baremetal 0000.3001.1102
0000.3001.110 30001
30001 E1/8
E1/8 Baremetal

SMAC: 0000.3001.1101 2
Host A DMAC: FFFF.FFFF.FFFF Host B 0000.3001.1101 30001 LEAF1 Host C
MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 98
Things to Remember
Multicast Enabled Underlay

• Multi-Destination Traffic (Broadcast, Unknown Unicast, etc.) needs to be

replicated to ALL VTEPs serving a given VNI
• Each VTEP is Multicast Source & Receiver
• For a given VNI, all VTEPs act as a Sender and a Receiver
• Head-End Replication will depend on hardware scale/capability
• Resilient, efficient, and scalable Multicast Forwarding is highly desirable
• Choose the right Multicast Routing Protocol for your need (type/mode)
• Use redundant Multicast Rendezvous Points (Spine/Aggregation generally preferred)
• 99% percent of Overlay problems are in the Underlay (OTV experience)

Keep in Mind
Overlay Convergence = Underlay Convergence!

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 99
 Underlay – Ingress Replication
• A Packet Multiplication
• EVPN assists no Peer, VNI Topology
Spine Spine Spine Spine

• Various Platform Support

• Nexus 9000 Underlay
• Ingress Replication
•
Leaf Leaf Leaf Leaf Leaf Leaf Leaf
Host sends 1 Packet to Edge-Device
• Edge-Device Encapsulates 1 Packet
and multiplies it
• Ingress VTEP sends 1 Packet per
Neighbor

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 100
Overlay Design Review
Overlay
• EVPN MP-BGP Primer
• Overlay Configuration Steps
1. Define the VTEP Interface

2. Establish the MP-BGP Control Plane

3. Define VLAN-VXLAN Mapping

4. VXLAN Routing Configuration

5. VXLAN HW Gateway Redundancy

6. Connect the Fabric to External Networks

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 102
EVPN MP-BGP Primer (1)
 Virtual Routing and Forwarding (VRF) VRF Info
VRF Info Name: VRF-A
Layer-3 segmentation for tenants’ routing space Name: VRF-A RD: 50000:1.1.1.2 (auto)
RD: ImpRR
Route-Target
50000:1.1.1.1 (auto) RR 65000:50000 (auto)
 Route Distinguisher (RD): Imp Route-Target 65000:50000 Exp Route-Target 65500:50000 (auto)
VRF Info (auto)
Exp Route-Target 65500:50000 (auto)
Name: VRF-A
8-byte field, VRF parameters; unique value to make VPN IP routes unique:
RD: RD + VPN IP prefix (auto)
50000:1.1.1.3
Imp Route-Target 65000:50000 (auto)
 Selective distribute VPN routes: Exp Route-Target 65500:50000 (auto)

1
V to define the import/export rules for VPNv4 routes
Route Target (RT): 8-byte field, VRF parameter, unique value
V2

 VPN Address-Family:
Distribute the MP-BGP VPN routes
RR BGP Route-Reflector

V3 iBGP Adjacency

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 103
EVPN MP-BGP Primer (2)
 BGP Advertisement
 VPN-EVPN: RD:[MAC_A][IP_A]
 BGP Next-Hop: V1
 Virtual Routing and Forwarding (VRF)  Route Target: 65500:50000
 Label: 50000
Layer-3 segmentation for tenants’ routing space
RR RR
 Route Distinguisher (RD):
8-byte field, VRF parameters; unique value to make VPN IP routes unique: RD + VPN IP prefix

 Selective distribute VPN routes:  MAC_A / IP_A >> LOCAL

 Route-Type2
Route Target (RT): 8-byte field, VRF parameter, unique value
1
V to define the import/export rules for VPNv4 routes
 MAC_A / IP_A >> V1
V2
 Route-Type2

 VPN Address-Family:
Distribute the MP-BGP VPN routes
RR  BGP Route-Reflector

V3  iBGP Adjacency

 Host A
 MAC_A / IP_A
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 104
Overlay
• EVPN MP-BGP Primer
• Overlay Configuration Steps
1. Define the VTEP Interface

2. Establish the MP-BGP Control Plane

3. Define VLAN-VXLAN Mapping

4. VXLAN Routing Configuration

5. VXLAN HW Gateway Redundancy

6. Connect the Fabric to External Networks

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 105
 1

Define VTEP Interface (VXLAN Tunnel End-Point)

# Features & Globals Enables VTEP support (only required on Leaf or Border)
feature bgp
feature nv overlay
nv overlay evpn
Enables EVPN Control-Plane in BGP
# Spine (S1)

# Leaf (V1) Configure the VTEP interface

interface nve1 RR RR RR RR
source-interface loopback0 Use a Loopback for Source Interface
host-reachability protocol bgp

iBGP
V2
Enable BGP for Host reachabilityV
1

V3
*Simplified BGP configuration; would have 4 BGP peers (RR)
IGP not shown

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 106
Overlay
• EVPN MP-BGP Primer
• Overlay Configuration Steps
1. Define the VTEP Interface

2. Establish the MP-BGP Control Plane

3. Define VLAN-VXLAN Mapping

4. VXLAN Routing Configuration

5. VXLAN HW Gateway Redundancy

6. Connect the Fabric to External Networks

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 107
 2

Building the Overlay Control-Plane

# Features & Globals
feature bgp
feature nv overlay
nv overlay evpn

# Spine (S1) Dynamic BGP neighbor

router bgp 65500
router-id 10.10.10.S1
address-family ipv4 unicast
RR RR RR RR
neighbor 10.10.10.0/24 remote-as 65500
update-source loopback0 Activate L2VPN EVPN under each BGP neighbor
address-family l2vpn evpn
send-community both iBGP

route-reflector-client V1 V2

# Leaf (V1)
router bgp 65500
router-id 10.10.10.V1 Send Extended BGP Community
address-family ipv4 unicast to distribute EVPN route attributes
neighbor 10.10.10.S1 remote-as 65500 V3
update-source loopback0 *Simplified BGP configuration; would have 4 BGP peers (RR)
address-family l2vpn evpn IGP not shown
send-community both
*

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 108
Overlay
• EVPN MP-BGP Primer
• Overlay Configuration Steps
1. Define the VTEP Interface

2. Establish the MP-BGP Control Plane

3. Define VLAN-VXLAN Mapping

4. VXLAN Routing Configuration

5. VXLAN HW Gateway Redundancy

6. Connect the Fabric to External Networks

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 109
Logical Construct of Multi-Tenant VXLAN EVPN
Tenant A (VRF A)
SVI SVI SVI SVI
A B N X

VLAN A Layer-3 VNI X’

VLAN B VLAN N

Layer-2 VNI A’ Layer-2 VNI B’ Layer-2 VNI N’ VLAN X

• One VLAN maps to one Layer-2 VNI per Layer-2 segment • 1 Layer-3 VNI per Tenant
• A Tenant can have multiple VLANs, therefore multiple (VRF) for routing
Layer-2 VNIs • VNI X’ is used for routed
• Traffic within one Layer-2 VNI is bridged packets
• Traffic between Layer-2 VNI’s is routed

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 110
 3

Define VLAN-VXLAN Mapping

Example VLAN Based CLI
# Features
 CLI offers a simplified method of mapping a feature vn-segment-vlan-based
802.1Q VLAN ID to a VXLAN VNI
# VLAN to VNI mapping
• VLAN to VNI configuration on a per-Switch based vlan 43 VLAN to Layer-2 VNI mapping
vn-segment 30000
• VLAN becomes “Switch Local Identifier”
• VNI becomes “Network Global Identifier” # Activate Layer-2 VNI for EVPN
evpn
• 4k VLAN limitation per-Switch still applies vni 30000 l2 Enables EVPN
rd auto Control-Plane for
• 4k Network limitation across fabric has been removed Layer-2 Services
route-target import auto
• Dependent on VLAN Space! route-target export auto

# Activate Layer-2 VNI on VTEP

interface nve1
source-interface loopback0
host-reachability protocol bgp
Bridge-
Domain
member vni 30000 Enables Layer-2
mcast-group 239.239.239.100 VNI on VTEP and
suppress-arp Suppress ARP
ethernet VLAN VNI vxlan

Multi-Tenancy Full (MT-Full)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 111
 3

Port-Local VXLAN-VXLAN Mapping

vlan 3100
 Available on N9K from 7.0(3)l1(2) Release vn-segment 31000
vlan 3101
 Allows to map the same 802.1Q VLAN tag to different vn-segment 31001
vlan 3102
VNIs on different interfaces of the same leaf node
vn-segment 31002
vlan 3103
 Current limit is 100 PV mappings per interface, and
vn-segment 31003
total 1K L2 VNIs per leaf !
interface Ethernet1/7
switchport mode trunk
VXLAN switchport vlan mapping enable
Underlay
switchport vlan mapping 3000 3100
switchport vlan mapping 3001 3101
switchport trunk allowed vlan 3100,3101
!
interface Ethernet1/8
switchport mode trunk
switchport vlan mapping enable
switchport vlan mapping 3000 3102
switchport vlan mapping 3001 3103
switchport trunk allowed vlan 3102-3103
3001
3000
3001 3000

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 112
 3

Define VLAN-VXLAN Mapping

Example BD Based CLI
# VLAN to VNI mapping
• Configuration is extended to allow Per-Port vni 30000
VLAN to Bridge-Domain mapping. Bridge-
Domain will be mapped to VXLAN VNI bridge-domain 100
VLAN to Bridge Domain mapping
member vni 30000
• VLAN to VNI configuration on a per-Port based
encapsulation profile vni MyProfile
• VLAN becomes “Port Local Identifier”
dot1q 43 vni 30000
• Bridge-Domain becomes “Switch Local Identifier”
# Interface Configuration
• VNI becomes “Network Global Identifier” interface Ethernet 1/12
• 4k VLAN limitation resides only on a per-Dot1Q no switchport
service instance 1 vni
Trunk encapsulation profile MyProfile default
• Independent from VLAN Space!

Bridge-
Domain

ethernet VLAN VNI vxlan

Multi-Tenancy Full (MT-Full)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 113
Overlay
• EVPN MP-BGP Primer
• Overlay Configuration Steps
1. Define the VTEP Interface

2. Establish the MP-BGP Control Plane

3. Define VLAN-VXLAN Mapping

4. VXLAN Routing Configuration

5. VXLAN HW Gateway Redundancy

6. Connect the Fabric to External Networks

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 114
Gateway Functions in VXLAN
VXLAN Routing

Layer-3 Boundary

VY
VX

V2 V2

V3 Layer-3 Boundary V3

V1 V1

Centralized Gateway Distributed Gateway

• Extra Bridging hop before and after Routing • Route or Bridge at Leaf
• Centralized Gateway (Aggregation) for Routing • Distributed Gateway (Anycast) for Routing
• Large amounts of state => convergence issues • Disaggregate state by scale out
• Scale problem for large Layer-2 domains • Optimal Scalability
• Works with VXLAN Flood & Learn or EVPN • Requires VXLAN/EVPN!

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 115
 Centralized Gateway (FHRP) *
VXLAN Routing

• Centralized Routing in a Layer-2 VXLAN

VY
Network
• Routing between VNI ( Different Subnet) VX
• Bridging within VNI (Same Subnet)

• Inter-VXLAN Routing at V2
Core/Aggregation Layer
• vPC provides MAC state synchronization V3
and HSRP peering
• Redundant VTEPs share Anycast VTEP IP
address in the Underlay V1

*Only Flood&Learn Host Y

VNI 30001
Host A
VNI 30000

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 116
 Distributed IP Anycast Gateway*
VXLAN/EVPN

• Distributed Routing with IP Anycast

Gateway (Integrated Route/Bridge IRB)
• Routing between VNI (Different Subnet)
• Bridging within VNI (Same Subnet)
V2
• Inter-VXLAN Routing Leaf/Access
Layer V3
• All Leafs share gateway IP and MAC for a
Subnet (No HSRP)
• A Host will always find its Gateway directly V1
attached anywhere it moves
Host Y
VNI 30001
*Requires EVPN Control-Plane.
Host A
VNI 30000

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 117
Different Integrated Route/Bridge (IRB) Modes
 Overlay Networks do follow two
slightly different integrated
?
Route/Bridge (IRB) semantics
 Asymmetric V2
• Route and Bridge on the ingress VTEP
• Bridge on the egress VTEP
V3
 Symmetric
• Route on both the ingress and egress
V1
VTEPs

 Cisco follows Symmetric IRB Host Y

VLAN 55
Host A
VLAN 43

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 118
Routing in VXLAN
 VNI utilized for providing isolation at
Layer-2 and Layer-3 across VXLAN
• Received frames must be mapped to
specific VNI for VXLAN transport
V2
• The VLAN-to-VNI mapping is performed
on Routing
V3
 All Routed Traffic uses the VNI
assigned to the VRF
VLAN
V1

Host Y
VLAN VNI 30001
Host A
VNI 30000

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 119
Asymmetric IRB
 Asymmetric
• Similar to Inter-VLAN routing
• Source and Destination VNIs have to exist
on leaf nodes where routing happens V2
• Post Routing traffic shares destination VNI
with Bridged traffic
V3
• Not very suitable for distributed Routing
From Host A via VLAN/VNI “blue” routed
at V1 to VNI “red” reaching destination V1
VLAN “red”
From Host Y via VLAN/VNI “red” routed Host Y

at V3 to VNI “blue reaching destination

VLAN 55
Host A
VLAN “blue” VLAN 43

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 120
Symmetric IRB
 Symmetric
• Similar to creating a Transit Segment
• Regardless of where Source or Destination
VNIs exist V2
• Post Routing traffic uses different VNI than
Bridged traffic
V3
• Additional VNI for Routing traffic (per VRF)
From Host A via VLAN “blue” routed at V1
to VNI “purple” reaching destination VLAN V1
“red”
From Host Y via VLAN “red” routed at V3 to Host Y
VNI “purple” reaching destination VLAN VLAN 55

“blue”
Host A
VLAN 43

• Used in Cisco VXLAN/EVPN

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 121
Host Subnet Redistribution
 Host “A” is a silent Host
• Not known via ARP/IP I know Subnet “A”

 How can Host “Y” reach Host “A”

• Host “A” and “Y” are in different V2
VLAN/Subnet

 Route for Host “A”-Subnet will be V3

advertised by V1 and V2
 Host “Y” will reach either V1 or V2 V1
based on ECMP
From V1 or V2, Host “A” can be reached
Host Y
 VLAN 55

via Layer-2 Segment Host A

VLAN 43

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 122
Asymmetric vs. Symmetric IRB
VXLAN Routing Block Diagram

 Asymmetric IRB  Symmetric IRB

• Used in Cisco VXLAN/EVPN

Layer-2 VNI VRF VNI

V1 V3 V1 V3
Layer-2 VNI

Layer-2 VNI

Layer-2 VNI

Layer-2 VNI
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 123
 4

Routing in VXLAN – VRF Routing Instance

# VLAN to L3 VNI Mapping

VNI 50000 vxlan

vlan 2500
vn-segment 50000 VLAN to Layer-3 VNI mapping

# Define SVI for VRF routing instance

interface Vlan2500
VLAN to Layer-3 VNI mapping
no shutdown
mtu 9216 - ip forward required
vrf member VRF-A
ip forward

# VRF configuration for “customer” VRF

VRF-A vrf context VRF-A
vni 50000
VRF context definition
rd auto - VNI
address-family ipv4 unicast
route-target both auto
- Route-Distinguisher
ethernet ethernet
route-target both auto evpn - Route-Targets
# Activate Layer-3 VNI on VTEP
- IPv4 and/or IPv6
Enables Layer-3 VNI on VTEP interface nve1
and associate it to VRF (one entry member vni 50000 associate-vrf
per tenant/VRF)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 124
 4

Enable Distributed IP Anycast Gateway*

Configuration Example for “BLUE” (V1 & V3) Configuration Example for “RED” (V1-3)
# Features # Features
feature interface-vlan feature interface-vlan

# VLAN to L2 VNI mapping (MT-Lite) # VLAN to L2 VNI mapping (MT-Lite)

vlan 43 vlan 55
vn-segment 30000 vn-segment 30001

# Anycast Gateway MAC, inherited by any interface # Anycast Gateway MAC, inherited by any interface
(SVI) using “fabric forwarding” (SVI) using “fabric forwarding”
fabric forwarding anycast-gateway-mac fabric forwarding anycast-gateway-mac
0002.0002.0002 0002.0002.0002

# Distributed IP Anycast Gateway (SVI) # Distributed IP Anycast Gateway (SVI)

interface vlan 43 interface vlan 55
no shutdown no shutdown
vrf member VRF-A vrf member VRF-A
ip address 11.11.11.1/24 tag 12345 ip address 98.98.98.1/24 tag 12345
fabric forwarding mode anycast-gateway fabric forwarding mode anycast-gateway

*Requires EVPN Control-Plane. VRF and BGP configuration not shown

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 125
 4

Routing in VXLAN – Advertise Local IP Subnets

VNI 50000 vxlan

# Route-Map for Redistribute Subnet
route-map REDIST-SUBNET permit 10
match tag 12345

# Control-Plane configuration for VRF (Tenant)

router bgp 65500
…
vrf VRF-A
address-family ipv4 unicast
VRF/Tenant definition advertise l2vpn evpn
within
VRF-AOverlay Control-Plane redistribute direct route-map REDIST-SUBNET
maximum-paths ibgp 2

ethernet ethernet

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 126
Overlay
• EVPN MP-BGP Primer
• Overlay Configuration Steps
1. Define the VTEP Interface

2. Establish the MP-BGP Control Plane

3. Define VLAN-VXLAN Mapping

4. VXLAN Routing Configuration

5. VXLAN HW Gateway Redundancy

6. Connect the Fabric to External Networks

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 127
 5

VXLAN HW Gateway Redundancy (vPC)

Southbound Connectivity

 VXLAN vPC Domain Configuration

Classic Ethernet
 Configure VXLAN specific vPC Peer-
Link Configuration
 Extend the IP Interface (Loopback) V5
configuration for the VTEP
• Secondary IP address (Anycast) is used as V4
the Anycast VTEP address
• Both vPC VTEP switches need to have the
identical secondary IP address configured
under the loopback interface
Host D
VNI 30000

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 128
 5

VXLAN HW Gateway Redundancy (vPC)

Southbound Connectivity
# VLAN to VNI mapping (MT-Lite)
vlan 55
vn-segment 30000 interface loopback0
ip address 10.10.10.5/32
ip address 10.10.10.99/32 secondary
# VTEP IP Interface; Source/Destination for all
VXLAN Encapsulated Traffic.
 Primary IP address is used for Orphan Hosts
 Secondary IP is for vPC Hosts (same IP on both
vPC Peers)
interface loopback0 Add Secondary IP to VTEP Loopback V5
ip address 10.10.10.5/32
ip address 10.10.10.99/32 secondary
V4
# VTEP configuration using Loopback as source.
Destination Group for VNI 30001 is “239.1.1.2”
interface nve1
source-interface loopback0
host-reachability protocol bgp
member vni 30000
mcast-group 239.239.239.100 Host D
suppress-arp VNI 30000
member vni 50000 associate-vrf

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 129
 5

VXLAN HW Gateway Redundancy (vPC)

Do Not Forget!
# VPC Domain Configuration
vpc domain 99 interface loopback0
peer-switch ip address 10.10.10.5/32
ip address 10.10.10.99/32 secondary
peer-keepalive destination V4-mgmt source v5-mgmt
peer-gateway
ip arp synchronize

# VPC Peer-Link
interface port-channelXX V5
switchport mode trunk
vpc peer-link
V4
# VPC Domain Routing Adjacency*
Routed Interface (SVI) for routing
interface Vlan3999 adjacency across VPC Peer-Link
no shutdown
ip address 10.254.254.1/30
interface loopback0
ip router ospf 1 area 0.0.0.0 ip address 10.10.10.4/32
ip ospf network point-to-point ip address 10.10.10.99/32 secondary
ip pim sparse-mode Host D
VNI 30000

*Best practice on Border Leaf nodes is to enable peering also for each defined VRF (on dedicated SVI interfaces)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 130
Overlay
• EVPN MP-BGP Primer
• Overlay Configuration Steps
1. Define the VTEP Interface

2. Establish the MP-BGP Control Plane

3. Define VLAN-VXLAN Mapping

4. VXLAN Routing Configuration

5. VXLAN HW Gateway Redundancy

6. Connect the Fabric to External Networks

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 131
 6

VXLAN/EVPN Fabric External Routing

 The Border Leaf/Spine provides Layer-
2 and Layer-3 connectivity to external
networks
 Flexible routing protocol options for
external routing V2
 Today, VRF-lite allows to extend VRFs
outside of the fabric V3
VBL
 With Nexus 7000/7700 and F3, LISP
becomes available for fabric extension V1

WAN

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 132
 6

VXLAN/EVPN Fabric External Routing

VRFs for External Routing

need to exist on Border Leaf
VBL
VRF VRF VRF V2
A B C

V3
Interface-Type Options:
• Physical Routed Ports V1
• Sub-Interfaces
• VLAN SVIs over Trunk Ports Peering Interface can
be in Global or Tenant VRF

WAN

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 133
 6

VXLAN/EVPN Fabric External Routing (eBGP)

VXLAN Fabric Side Configuration

# Sub-Interface Configuration
interface Ethernet1/1
no switchport
interface Ethernet1/1.10
mtu 9216
VBL
VRF VRF VRF V2
encapsulation dot1q 10 A B C
vrf member VRF-A
ip address 10.254.254.1/30 V3
# eBGP Configuration
router bgp 65500
…
vrf VRF-A
Advertise external learned routes V1
address-family ipv4 unicast into EVPN (Route-Type 5)
advertise l2vpn evpn
aggregate-address 10.0.0.0/8 summary-only
neighbor 10.254.254.2 remote-as 65599
update-source Ethernet1/1.10
address-family ipv4 unicast
WAN
Advertise an aggregate of the internal prefixes
Ensure that non-necessary routes are not advertised
towards the External Network
AS# 65599

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 134
 6

VXLAN/EVPN Fabric External Routing (eBGP)

VBL
VRF VRF VRF V2
WAN Router Side Configuration A B C

# Interface Configuration
interface Ethernet1/1
V3
vrf member VRF-A
ip address 10.254.254.2/30

# eBGP Configuration V1
router bgp 65599
…
vrf VRF-A
address-family ipv4 unicast
neighbor 10.254.254.1 remote-as 65500
update-source Ethernet1/1
address-family ipv4 unicast WAN
AS# 65599

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 135
 6

VXLAN/EVPN Fabric External Routing (OSPF)

VXLAN Fabric Side Configuration

# Sub-Interface Configuration
interface Ethernet1/1
no switchport
interface Ethernet1/1.10 VBL
VRF VRF VRF V2
mtu 9216
encapsulation dot1q 10 A B C
vrf member VRF-A
ip address 10.254.254.1/30 V3
ip router ospf 1 area 0.0.0.0
ip ospf network point-to-point

# BGP Configuration V1
router bgp 65500
…
address-family l2vpn evpn
retain route-target all Advertise external learned routes
vrf VRF-A
address-family ipv4 unicast into EVPN (Route-Type 5)
advertise l2vpn evpn
redistribute bgp 100 route-map OSPF-BGP* WAN
Ensure that non-necessary routes are not advertised
towards the External Network Redistribute internal prefixes with route-map

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 136
Migrating to VXLAN
EVPN
Converting from vPC
VXLAN Design Considerations
VXLAN Mode: Scalability:
• Flood-and-Learn • The number of VXLAN VNIs
• With EVPN control Plane • The number of VTEP peers
• The number of EVPN tenants
BUM Traffic Handling:
• The number of VXLAN Host IP routes
• Multicast replication
• The number of VXLAN Host MAC addresses
• Unicast/ingress replication
• The number of IPv4/IPv6 LPM routes
Deployment Scenarios: • The number of Ingress replication peers
• Brown field vs green field
• Investment protection
• Multi-vendor environment?

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 139
VXLAN EVPN Loop Avoidance Considerations

Layer 2
Domain

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 140
VXLAN EVPN Loop Avoidance – Option 1

Single logical
connection to the
external L2 domain

Layer 2
Domain

Add BPDU-Guard on all the server facing interfaces

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 141
VXLAN EVPN Loop Avoidance – Option 2

Single logical
connection to the
external L2 domain

Layer 2
Domain

Add BPDU-Guard on all the server facing interfaces

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 142
Starting Point – Brownfield Network (vPC Based)
WAN - Core  Starting from a traditional network (named
‘Brownfield’)
 Brownfield network could be built
L3 leveraging vPC, STP, FabricPath or even
L2 3rd party devices
Migration considerations equally apply to all
those cases
 Assumption is also that network services
are connected in traditional fashion (routed
mode at the aggregation layer)
vPC Based Brownfield Network

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 143
And What About FabricPath?
WAN - Core  Starting from a traditional network (named
‘Brownfield’)
 Brownfield network could be built
L3 leveraging vPC, STP, FabricPath or even
L2 3rd party devices
Migration considerations equally apply to all
those cases
 Assumption is also that network services
are connected in traditional fashion (routed
mode at the aggregation layer)
vPC Based Brownfield Network

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 144
Deployment
Building Small Initial VXLAN EVPN POD
The end goal is to migrate endpoints and network services to the ACI fabric

WAN - Core

Greenfield VXLAN EVPN Fabric

Brownfield Network
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 145
Integration
Connecting Brownfield and Greenfield Networks
First step: creating a L2 connectivity path

WAN - Core

Back-to-back vPC for

L3 avoiding L2 loops

L2

L2 Trunk

Greenfield VXLAN EVPN Fabric

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 146
Endpoints Integration
Mapping VLANs to L2 VNIs

WAN - Core

Map VLAN10/L2 VNI1

and VLAN20/L2 VNI2
L3
L2

VLAN 10 VLAN 10 VLAN 20 VLAN 20 VLANs 10, 20

App1 Web App1 Web App2 Web Greenfield VXLAN EVPN Fabric
App2 Web 10.20.20.11
10.10.10.10 10.10.10.11 10.20.20.10

 Endpoints connected to different VLANs in the brownfield network

 Each legacy VLAN is trunked to the VXLAN fabric and mapped to a dedicated L2 VNI

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 147
Endpoints Integration
Use Case 1: VLAN == VNI
WAN - Core Greenfield VXLAN EVPN Fabric

Map VLAN10/L2 VNI1

and VLAN20/L2 VNI2
L3
L2

VLAN 10 VLAN 10 VLAN 20 VLAN 20 VLANs 10, 20

L2
App1 Web App1 Web
Broadcast
10.10.10.10 10.10.10.11 Domain

L2
App1 Web App1 Web Broadcast
10.20.20.10 10.20.20.11 Domain
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 148
Endpoints Integration
Use Case 1: VLAN == VNI
WAN - Core Greenfield VXLAN EVPN Fabric

Map VLAN10/L2 VNI1

and VLAN20/L2 VNI2
L3
L2

VLAN 10
Map VLAN10/L2 VNI1
VLAN 10 VLAN 20 VLAN 20

Web1
App1 Web App1 Web
10.10.10.10 10.10.10.11

Web2
App1 Web App1 Web
10.20.20.10 10.20.20.11

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 149
Endpoints Migration
1 - Single VCenter Server Scenario

WAN - Core
Greenfield VXLAN EVPN Fabric

L3
L2

Compute
Clusters

100.1.1.3
VM
100.1.1.99
VM
100.1.1.7
VM VM
Compute
VM
Cluster BD
VM VM
Existing
VM
VM
App New Compute
Mgmt Cluster Clusters
DVS

vCenter Managed
DVS
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 150
Endpoints Migration
1 - Single VCenter Server Scenario

1.1 Connect the new ESXi hosts to the vCenter managed DVS
WAN - Core

L3
L2

100.1.1.3
VM
100.1.1.99
VM
100.1.1.7
VM VM
VM
BD
VM VM
Existing
VM
VM
App
DVS

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 151
Endpoints Migration
1 - Single VCenter Server Scenario

1.2 Migrate VMs to the new ESXi cluster

WAN - Core

Migrated VMs still

leverage the gateway
L3 in the Brownfield
network
L2

VM VM

100.1.1.3
VM
100.1.1.99 100.1.1.7
VM VM
VM
BD
VM VM
Existing
VM
App
DVS

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 152
Endpoints Migration
2 – Multiple VCenter Servers Scenario

WAN - Core
Greenfield VXLAN EVPN Fabric

L3
L2

Compute
Clusters

100.1.1.3
VM
100.1.1.99
VM
100.1.1.7
VM VM
Compute
VM
Cluster BD Mgmt Cluster
VM VM
Existing vCenter2
VM
VM
App New Compute
Mgmt Cluster Clusters
DVS

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 153
Endpoints Migration
2 – Multiple VCenter Servers Scenario

2.1 Connect new ESXI servers to a second DVS

WAN - Core

L3
L2

100.1.1.3
VM
100.1.1.99
VM
100.1.1.7
VM VM
VM
BD
VM VM
Existing
VM
VM
App

DVS New DVS

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 154
Endpoints Migration
2 – Multiple VCenter Servers Scenario
2.2 Migrate VMs to the new ESXi cluster*
WAN - Core

Migrated VMs still

leverage the gateway
L3 in the Brownfield
network
L2

VM VM

100.1.1.3
VM
100.1.1.99
VM
100.1.1.7
VM VM VM
VM
BD
VM VM
Existing
VM
VM
App

DVS New DVS

*Cross-vCenter vMotion supported with vSphere 6.0

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 155
Default Gateway Considerations
Existing Design VXLAN EVPN Fabric

HSRP
Default GW

Subnet 1 = VLAN 10 L2 Bridging Subnet 1 = VNI

VM VM VM
P P P
VM

 Default Gateway up to this point is still deployed in the Brownfield network

 VXLAN EVPN fabric initially provides only L2 connectivity services

 L2 path between the two networks leveraged by migrated hosts to reach the default gateway

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 156
Migrate Default Gateway to the VXLAN Fabric
WAN - Core
Greenfield VXLAN EVPN Fabric

Anycast Default
Gateway

L3 L3
L2 L2
VLAN 10 VLAN 20

10.10.10.11 10.20.20.11

10.10.10.10 10.20.20.10

Any IP - Anywhere

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 157
Migration
Routing Between Brownfield and Greenfield
 Routing between Brownfield and Greenfield may still be required
• Handling communication to IP subnets that remain only on
Brownfield (default gateway remains on aggregation devices)
• Handling communication with the WAN
Existing Design VXLAN EVPN Fabric

L3 Routing

HSRP
Default GW

IP Subnet 2 = VLAN 30 IP Subnet 1 = L2 VNI 1

V V V
M M M V V V
P P P P M M M

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 158
 Migration
Routing Between Brownfield and Greenfield

WAN - Core
Greenfield VXLAN EVPN Fabric
Default Gateway for L3 Links
VLAN 30

L3

L2

L3

L2
VLAN 30
10.10.10.11
VLAN 30 NOT carried
on the vPC connection
10.30.30.10

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 159
Moving L4-L7 Services
 Migrating Network Services
Example of Firewall Services Migration
Starting point:
Active/Standby FW nodes
(routed mode*) connected to
the Aggregation layer
switches
WAN - Core Greenfield VXLAN EVPN Fabric
Active Standby

*Similar considerations apply for services in transparent mode

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 161
Migrating Network Services
Move the Standby Node to the VXLAN Fabric

WAN - Core Greenfield VXLAN EVPN Fabric

Active

FW Keepalives and
state synchronization
Standby

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 162
Migrating Network Services
Disconnect the Active Node from the Brownfield Network

WAN - Core Greenfield VXLAN EVPN Fabric

Active

Active
FW activated on the
VXLAN fabric

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 163
Migrating Network Services
Both Firewall Nodes Connected to the VXLAN Fabric

WAN - Core
Greenfield VXLAN EVPN Fabric

Standby Active

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 164
Interconnecting Multiple
Sites
VXLAN and DCI
Overlays Evolve and Spread

DC Local Overlay

End-to-End Overlay SS SS SS SS

S S S S S S S S

L L L L .... L L L L L .... L
Single Logical Data Center

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 167
Changing the Paradigm with Overlays

DC Local Overlay

Multi-Site Overlay SS SS SS SS

S S S S S S S S

L L L L .... L L L L L .... L
Multiple Logical Data Center

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 168
 VXLAN Evolves as the Control Plane Evolves!
Early Years
Yet Another Encapsulation
 Flood & Learn (Multicast-based)
 Data-Plane only Yesterday
VXLAN for the Data Center – Intra-DC
 Control-Plane
 Active VTEP Discovery
Today
 Multicast and Unicast
VXLAN for DCI – Inter-DC
 DCI Ready
 ARP/ND caching/suppress
 Multi-Homing
 Failure Domain Isolation
 Loop Protection
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 169
 Inter-X Connectivity
VXLAN Multi-Pod VXLAN Multi-Fabric VXLAN Multi-Site

EVPN Control- BGP EVPN EVPN Control- EVPNFabric

Control-Plane EVPNFabric
Control-Plane EVPNFabric
Control-Plane
#1 BGP EVPN EVPNFabric
Control-Plane
#2
Fabric #1 Fabric #2 #1 #2
Plane Domain 1 Plane Domain 2 Domain 1 Domain 2 Domain 1 Domain 2

Overlay Overlay Overlay Overlay Overlay Overlay

VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE
P P P P P P P P P P P P P P P P P P P P P P P P

Bar Bar Bar Bar

em em em em Bar Bar Bar Bar Bar Bar Bar Bar
eta eta eta eta em em em em em em em em
l l l l etal etal etal etal etal etal etal etal

DCI DCI
Single Data-Plane – End-to-End Data-Plane Domain 1 Data-Plane Domain 2 Data-Plane Domain 1 Data-Plane Domain 2
Data-Plane Data-Plane

 Single Fabric with End-to-  Multiple Fabrics – Normalized  Multiple Fabrics with
End Encapsulation through Ethernet Integrated DCI (DCI2)
 Build Hierarchy in the  Multiple Fabrics Interconnect  Integrated DCI – Scaling
Underlay – Flatten it in the using DCI (Layer 2 and Layer 3) within and between
Overlay Fabrics

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 170
VXLAN EVPN – Multi-Pod

Underlay Extension
VTEP VTEP VTEP VTEP

Spine Spine Spine Spine Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Pod 1 Pod n

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 171
Multi-Pod Characteristics – ”The Single”

 Single Overlay Domain – End-to-End Encapsulation

 Single Overlay Control-Plane Domain – End-to-End EVPN Updates
 Single Underlay Domain End-to-End
 Single Replication Domain for BUM
 Single VNI Administrative Domain

Building Underlay Hierarchies – Non Hierarchical Overlay

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 172
 Multi-Pod End-to-End Encapsulation

Underlay Extension
VTEP VTEP VTEP VTEP

Spine Spine Spine Spine Overlay Spine Spine Spine Spine

VTEP VTEP
10.1.1.1 VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
10.2.2.7
VTEP

Unicast

Pod 1 Pod n
Baremetal Baremetal

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 173
Multi-Pod VXLAN Tunnel Adjacencies

Underlay Extension
VTEP VTEP VTEP VTEP

Spine Spine Spine Spine Overlay Spine Spine Spine Spine

VTEP VTEP VTEP

10.1.1.1 10.1.1.4 10.2.2.7
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Switch# show nve peers

Interface Peer-IP VNI Up Time
---------- ----------- ------ ----------
nve1
Pod 110.1.1.4 30000 03:18:06
Pod n
nve1 10.2.2.7 30000 00:12:23

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 174
 Multi-Pod Underlay Extension
POD1 Underlay Routing Table POD2 Underlay Routing Table

Leaf: Border: Leaf: Border: Border: Leaf: Border: Leaf:

10.1.1.1 10.1.1.101 10.2.2.1 10.2.2.101 10.2.2.101 10.2.2.1 10.1.1.101 10.1.1.1
10.1.1.2 10.1.1.102 10.2.2.2 10.2.2.102 10.2.2.102 10.2.2.2 10.1.1.102 10.1.1.2
10.1.1.3 10.2.2.3 10.2.2.3 10.1.1.3
10.1.1.4 10.2.2.4 10.2.2.4 10.1.1.4
10.1.1.5 10.2.2.5 Underlay Extension 10.2.2.5 10.1.1.5
10.1.1.6 10.2.2.6
VTEP VTEP VTEP
10.2.2.6
VTEP
10.1.1.6
10.1.1.7 10.2.2.7 10.2.2.7 10.1.1.7
Border (PIP) Border (PIP) Border (PIP) Border (PIP)
10.1.1.101 10.1.1.102 10.2.2.101 10.2.2.102
Spine Spine Spine Spine Spine Spine Spine Spine

VTEP VTEP
10.1.1.1
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
10.2.2.7
VTEP

Pod 1 Pod 2

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 175
Multi-Pod BUM Replication

Underlay Extension
VTEP VTEP VTEP VTEP

Spine Spine Spine Spine Overlay Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

BUM

Pod 1 Pod 2
Baremetal

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 176
Multi-Pod Challenges – ”The Single”

 Single Overlay Domain – End-to-End Encapsulation

• Scaling the VXLAN EVPN Network
 Single Overlay Control-Plane Domain – End-to-End EVPN Updates
• Overlay Control-Plane Update Propagation
 Single Underlay Domain End-to-End
• Network must be extended in Underlay (VTEP to VTEP reachability)
 Single Replication Domain for BUM
• One BUM flooding domain through out all connected Pods

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 177
 VXLAN Multi-Site https://tools.ietf.org/html/draft-sharma-multi-site-evpn
Functional Components
Site-External DCI
(IP Routing and Increased
Border Gateways MTU Support)
(Key Functional Components of
VXLAN Multi-Site Architecture)

VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Spine Spine Spine Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Site-Internal Fabric
Site 1 (Common VXLAN and Site n
BGP-EVPN Functions)
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 178
VXLAN Multi-Site Characteristics

 Multiple Overlay Domains – Interconnected & Controlled

 Multiple Overlay Control-Plane Domains – Interconnected & Controlled
 Multiple Underlay Domains - Isolated
 Multiple Replication Domains for BUM – Interconnected & Controlled
 Multiple VNI Administrative Domains – Phase 2

Underlay Isolation – Overlay Hierarchies

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 179
VXLAN Multi-Site
Main Use Cases

Scale-Up Model to Build a

Large Intra-DC Network

Data Center Interconnect (DCI)

Integration with Legacy Networks

(Coexistence and/or Migration)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 180
VXLAN Multi-Site Inter-Site Network
Routing Table
Underlay Isolation Border Site1: Border Site2:
10.1.1.101 10.2.2.101
10.1.1.102 10.2.2.102
10.1.1.111 10.2.2.222

Multi-Site VIP Multi-Site VIP

10.1.1.111 Site-External DCI 10.2.2.222
VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Border (PIP) Border (PIP) Border (PIP) Border (PIP)
10.1.1.101 10.1.1.102 10.2.2.101 10.2.2.102
Spine Spine Spine Spine Spine Spine Spine Spine

Site 1 Underlay Site n Underlay

Routing Table Routing Table
Border: Leaf: Border: Leaf:
VTEP 10.1.1.101 10.1.1.1 VTEP
10.2.2.101 10.2.2.1
10.1.1.1
VTEP VTEP VTEP VTEP VTEP
10.1.1.102 10.1.1.2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP
10.2.2.7
10.2.2.102 10.2.2.2
VTEP VTEP

10.1.1.111 10.1.1.3 10.2.2.222 10.2.2.3

10.1.1.4 10.2.2.4
10.1.1.5 10.2.2.5
10.1.1.6 10.2.2.6
Site 1 Site n
10.1.1.7 10.2.2.7

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 181
 VXLAN Multi-Site
Introducing the Border Gateway

Overlay Multi-Site

Border Gateway (BGW)

- Anycast Cluster -
Multi-Site VIP Multi-Site VIP
10.1.1.111 10.2.2.222
VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine

Any VTEP

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Site 1 Site n

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 182
Multi-Site – VXLAN Tunnel Adjacencies
BG102# show nve peers
Interface Peer-IP VNI Up Time
---------- ----------- ------ ----------
nve1 Overlay
10.1.1.1 Multi-Site
30000 00:12:16
nve1 10.1.1.4 30000 03:18:06
nve1 10.2.2.222 30000 00:12:23

Multi-Site VIP Multi-Site VIP

10.1.1.111 10.2.2.222
VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine

VTEP VTEP VTEP

10.1.1.1
VTEP VTEP VTEP
10.1.1.4VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
10.2.2.7
VTEP

Leaf1-1# show nve peers

Leaf2-7# show nve peers
Interface Peer-IP VNI Up Time
Interface Peer-IP VNI Up Time
Site 1
---------- ----------- ------ ---------- Site n------
---------- ----------- ----------
nve1 10.1.1.4 30000 03:18:06
nve1 10.2.2.222 30000 00:12:25
nve1 10.1.1.111 30000 00:12:23
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 183
 Border Gateway to Cloud

Layer-3
Network

BGW BGW BGW BGW BGW BGW BGW BGW BGW BGW
Site 1 Site 2 Site n
Spine Spine Spine Spine Spine Spine

Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 184
 Border Gateway Between Spine and Super-Spine

Super-Spine Super-Spine

BGW BGW BGW BGW BGW BGW BGW BGW BGW BGW
Site 1 Site 2 Site n
Spine Spine Spine Spine Spine Spine

Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 185
 Border Gateway on Spine

Super-Spine Super-Spine

BGW BGW BGW BGW BGW BGW BGW BGW BGW BGW
Spine Spine Spine Spine Spine Spine Spine Spine Spine Spine
Site 1 Site 2 Site n

Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 186
 Border Gateway Back-to-Back

BGW BGW BGW BGW

Site 1 Site 2
Spine Spine Spine Spine

Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 187
VXLAN Multi-Site
Border Gateways Deployment Considerations

 Border Gateways used for two main functions: Anycast Border Gateways

Interconnecting each site to the Inter-Site network (for

BGW BGW BGW BGW

• VTEP VTEP VTEP VTEP

East-West traffic flows)

• Connecting each site to the external Layer 3 domain (for
North-South traffic flows)
• May also be used to connect endpoints and/or network
service nodes (FWs, ADCs) Site 1

 Possible deployment models:

VPC Border Gateways
• Anycast Border Gateways (currently supported) BGW BGW

VPC Border Gateways (planned for Q3CY18)

VTEP VTEP

 BGW function enablement in the VXLAN EVPN fabric:

• BGWs as leaf nodes
• BGWs as spine nodes (Border-Spines)
Site 1

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 188
VXLAN Multi-Site
Anycast Border Gateway (1)
Anycast Border Gateway
 Up to 4 Border Gateways
Multi-Site VIP
10.1.1.111  Border Gateway Support

BGW BGW BGW BGW

 Leaf 7.0(3)I7(1), Spine 7.0(3)I7(2)
VTEP
PIP-BGW1
VTEP
PIP-BGW2
VTEP
PIP-BGW3
VTEP
PIP-BGW4  Common Multi-Site Virtual IP (Multi-Site VIP)
10.1.1.101 10.1.1.102 10.1.1.103 10.1.1.104
across BGWs
• Multi-Site VIP for communication between the
Multi-Site VIP
10.1.1.111 Border Gateways in different Sites
• Multi-Site VIP for communication between
Border Gateways and Leaf nodes within a Site
 Individual Primary IP (PIP) per BGW
• Used for Broadcast, Unknown Unicast and
Multicast (BUM) replication
• PIP for communication with Single-Homed
Site 1
endpoints (routed only), intra- and inter-Site

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 189
VXLAN Multi-Site
Anycast Border Gateway (2)
Anycast Border Gateway
Type: 00 IP: 10.1.1.101
4 System MAC: 00:00:00:00:00:01
Ethernet Segment: 00:00:07 VNI: 30010  Per-VNI Designated Forwarder (DF) election
• Each BGW can serve as DF for a single or
BGW BGW BGW BGW
a set of Layer-2 VNIs
VTEP VTEP VTEP VTEP
• DF election and assignment is automatic
 Using BGP EVPN Route Type 4 for DF election
DF DF DF DF
30010 30011 30012 30099

• Operator Managed Assignment (Type: 00)

BGP EVPN
• Six Octet Site Identifier (System MAC:
00:00:00:00:00:01)
RR RR
Spine Spine
• Multi-Site Discriminator (Ethernet-
Segment: 00:00:07)
• Originators IP Address (PIP): 10.1.1.101
• Layer-2 VNI: 30010
Site 1

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 190
 VXLAN Multi-Site
Anycast Border Gateway (3)
External
Connectivity Anycast Border Gateway
Point-to-Point L3 Links
(Physical/Sub-Interfaces)  Single-Homed End-Points only connected with
L3 links
• Services Appliance (i.e. Firewall, ADC etc.)
BGW BGW BGW BGW
VTEP
PIP-BGW1
VTEP
PIP-BGW2
VTEP
PIP-BGW3
VTEP
PIP-BGW4
• External routers
10.1.1.101 10.1.1.102 10.1.1.103 10.1.1.104
• No SVI support on BGW nodes
.1 .1
 Advertised and Reachable through Individual
Point-to-Point L3 Links
Primary IP Address (PIP)
Point-to-Point L3 Links
(Physical/Sub-Interfaces)
ADC ADC
• Intra-Site: Leaf nodes use PIP to reach the device
ADC ADC
connected to Border Gateways
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102 • Inter-Site: Remote Border Gateways use PIP to
reach the device connected to Border Gateways
VTEP

Site 1

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 191
Anycast BGW vs. VPC Border Gateway
Anycast Border Gateway VPC Border Gateway
• Up to 4 BGW • 2 BGW with physical VPC Peer-Link
• Shared Nothing • Small Deployments
• Simple Failure Scenarios • End-Point or Network Services
• Any Deployments Connectivity on BGW
• No End-Point or Network Services • Migration Use-Cases (Brownfield)
Connectivity on BGW • Pseudo-BGW to BGW
• Greenfield Deployments • Classic Ethernet/FabricPath to VXLAN
EVPN

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 192
VXLAN Multi-Site
VPC Border Gateway and Transit Traffic

VPC Border Gateway

Multi-Site VIP  Common Multi-Site Virtual IP (Multi-Site VIP)
10.1.1.111
across BGWs
BGW BGW • Multi-Site VIP for Inter-Site transit
communication (transit)
VTEP VTEP
PIP-BGW1 PIP-BGW2
10.1.1.101 VPC VIP 10.1.1.102

Common VPC Virtual IP (VPC VIP) across BGWs

10.1.1.121
•
Multi-Site VIP
10.1.1.111
• Used by default for external communication
• Used for Broadcast, Unknown Unicast and
Multicast (BUM) replication
 Individual Primary IP (PIP) per BGW
• Used for external communication with
“advertised-pip”
Site 1

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 193
VXLAN Multi-Site
VPC Border Gateway and Locally Attached End-Points

VPC Border Gateway

Multi-Site VIP  Single- or Dual-Homed End-Points
10.1.1.111
• Services Appliance (i.e. Firewall, ADC etc.)
BGW BGW
VTEP VTEP • Physical or Virtual Servers
VPC VIP
10.1.1.121  Advertised and Reachable through VPC Virtual
IP Address (VPC VIP)
Multi-Site VIP
10.1.1.111 • Intra-Site: Leaf nodes use VPC VIP to
ADC Baremetal reach End-Points connected to Border
ADC EP
Gateways
0000.3010.1102 0000.3010.1101
192.168.10.102 192.168.10.101 • Inter-Site: Remote Border Gateways use
VPC VIP to reach End-Points connected to
Border Gateways
• Traffic potentially traverses VPC Peer-Link
Site 1

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 194
VXLAN Multi-Site
VPC Border Gateway and Designated BUM Forwarder

VPC Border Gateway

 VPC-based Designated Forwarder Election
BGW BGW
VTEP VTEP
 Per-Site Designated Forwarder (DF) election
DF VPC VIP
10.1.1.121
• Using same approach as in VPC
• Best Path to Rendezvous-Point or VPC
Primary Node
• Same VPC node is elected DF for all the
Layer-2 VNIs

Site 1

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 195
VXLAN Multi-Site
BUM Traffic Forwarding
Overlay Multi-Site

VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

BUM

Site 1 Site n
Baremetal

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 196
VXLAN Multi-Site
BUM Replication Modes (Multicast Intra-Site)

Overlay Multi-Site

Ingress Replication

VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine

Multicast Multicast

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Site 1 Site n

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 197
VXLAN Multi-Site
BUM Replication Modes (Ingress Replication Only)

Overlay Multi-Site

Ingress Replication

VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine

Ingress Replication Ingress Replication

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Site 1 Site n

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 198
VXLAN Multi-Site
BUM Replication Modes (Mixed Mode Intra-Site)

Overlay Multi-Site

Ingress Replication

VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine

Ingress Replication Multicast

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Site 1 Site n

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 199
VXLAN Multi-Site
BUM Traffic Policing
Overlay Multi-Site

Storm Control
VTEP VTEP Broadcast 0-100% VTEP VTEP

BGW BGW
Unknown Unicast 0-100% BGW BGW

Multicast 0-100%
Spine Overlay Site 1
Spine Spine Spine Spine Overlay Site n
Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

BUM

Site 1 Site n
Baremetal

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 200
VXLAN Multi-Site
Connectivity to the External Layer 3 Domain

 The BGW nodes can also be used to provide Layer-3 external

connectivity to each site
 Different connectivity models are supported
• VRF-Lite peering with external WAN Edge routers
• MP-BGP EVPN peering with external WAN Edge routers (Shared
Border deployment model, aka GOLF)
• Dedicated or shared pair of WAN Edge routers across sites

 External Layer-3 network may be different from the DCI

network used for inter-site communication

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 201
 VXLAN Multi-Site
Border Gateways and VRF-Lite to External Routers

Dedicated physical Separate IPv4/IPv6 routing

interfaces / sub- peering for each VRF (IGP
interfaces for each VRF External
VRF-AVRF-B VRF-C or eBGP)  Separate IPv4/IPv6 routing peering for
Connectivity each VRF established with the
Site-External

external routers on dedicated physical

Multi-Site
Overlay
interfaces/sub-interfaces
 Must use separate interfaces for
BGW BGW BGW BGW inter-site communication
No support for VXLAN encapsulated
VTEP VTEP VTEP VTEP

traffic on sub-interfaces
Site-Internal

Site 1

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 202
 VXLAN Multi-Site
Border Gateway and Shared Border (aka ‘GOLF’)
External router operates like a
traditional VXLAN EVPN VTEP
(Layer 3 only)
 Single MP-BGP EVPN peering
External
VRF-AVRF-B VRF-C
established with the external routers
Connectivity
Single MP-BGP EVPN routing
to exchange routes for all the VRFs
instance to exchange routes
Routed interface extending
‘underlay’ connectivity to
for all VRFs
 VXLAN Data-Plane between the
the external routers
Multi-Site BGWs and the external routers
Site-External

Overlay
 Same spine uplinks used for all
VXLAN encapsulated traffic (North-
South and East-West)
BGW BGW BGW BGW
VXLAN Data Plane
 Required because of the use of DCI link
VTEP VTEP VTEP VTEP
between BGW and WAN tracking
Edge Router
Site-Internal

 Various northbound hand-off options

depending on specific HW support:
Site 1
VRF-Lite, MPLS-VPN, LISP
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 203
VXLAN Multi-Site
Legacy Site Integration IR for BUM +
aggregated BUM Pair of Pseudo-BGWs
containment (EX/FX Switches)
BGW

VTEP VTEP VTEP VTEP

BGW BGW

Spine Spine Spine Spine Baremetal ADC

VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Greenfield Site Legacy Site

 Coexistence and/or migration use cases

• Extend Layer-2 and Layer-3 multi-tenant connectivity across sites
 Deploy a pair of Pseudo-BGWs in the legacy site
• Simplified configuration required on Pseudo-BGWs nodes
• Still offering native Multi-Site functions (Ingress Replication for BUM, BUM containment, etc.)
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 204
 Multi-Site and Legacy Site Integration
Default Gateway Deployment – Recommended
Default Gateway
migrated to the Border
Greenfield VXLAN BGW Gateways (VXLAN EVPN
EVPN Fabric offers L2 VTEP VTEP L3 VTEP VTEP
Anycast Gateway)
and L3 services for the BGW BGW
stretched IP subnets L2

Spine Spine Spine Spine

Legacy
infrastructure offers
only L2 services
L3 VTEP VTEP VTEP VTEP VTEP VTEP VTEP

L2

Distributed Anycast Greenfield Site Legacy Site

Gateway function

 Recommended approach is to migrate the default gateway from the

legacy aggregation devices to the Border Gateways (VXLAN EVPN
Anycast Gateway)
 Optimize routing between endpoints deployed across sites

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 205
 VXLAN Multi-Site and Legacy Site Integration
Starting from Legacy Networks Only (1)
Pair of Pseudo-BGWs
Pair of Pseudo-BGWs (EX/FX Switches)
(EX/FX Switches) BGW BGW

VTEP VTEP VTEP VTEP

Legacy Site 1 Legacy Site 2

 A pair of Pseudo-BGWs inserted in each legacy site to extend Layer-2 and Layer-
3 connectivity between sites
• Replacement of traditional DCI technologies (EoMPLS, VPLS, OTV, …)
 Slowly phase out the legacy networks and replace them with VXLAN EVPN fabrics
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 206
 VXLAN Multi-Site and Legacy Site Integration
Starting from Legacy Networks Only (2)
Convert the nodes to
Convert the nodes to full BGWs functions
full BGWs functions
VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP

‘Mixed’ Site 1 ‘Mixed’ Site 2

 Introduce VXLAN EVPN spines and additional VTEPs in each site

 Convert the Pseudo-BGWs to full BGW (may require vPC support on BGWs)
 Migrate endpoints between the legacy network and the new VXLAN EVPN fabric

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 207
VXLAN Multi-Site and Legacy Site Integration
Starting from Legacy Networks Only (3)

VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Greenfield Site 1 Greenfield Site 2

 Decommission the legacy networks and leave only the VXLAN EVPN
fabrics in place

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 208
VXLAN EVPN – Multi-Site
Multi-Site Core
• Border Gateway (BGW) to Border Gateway (BGW)
reachability required
• Reachability Back-to-Back (full-mesh) or via Layer-3
transport network
• Any Routing Protocol for BG reachability No Underlay Extension
• IPv4 Unicast Transport
VTEP
(Ingress
VTEP
Replication) VTEP VTEP
• BGP full-mesh or Route-Server (eBGP ”Route Reflector”)
for Overlay Control-Plane Multi-Site Border Gateway (BGW):
• Seamless insertion into existing VXLAN EVPN Fabrics
Spine Spine Spine
(Border Gateways require Nexus 9x00-EX/-FX)
Spine Spine Spine Spine Spine

• Layer-2 and Layer-3 extension to other Sites

• BGP- or VPC-based Border Gateway (BGW) Cluster (up
to 4 nodes when using BGP)
• All Border Gateways (BGW) are representing a common
Anycast VTEP
• Failure containment through Broadcast, Unknown Unicast
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

and Layer-2 Multicast limiter (off or rate-based)

• Co-Existence with VRF-Lite for External Connectivity
• Core and Fabric link tracking
Site 1 Site n

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 209
Multi-Site Advantages – ”The Multiple”

 Multiple Overlay Domains – Interconnected & Controlled

• Scaling and Segregating VXLAN EVPN Networks
 Multiple Overlay Control-Plane Domains – Interconnected & Controlled
• Limited Overlay Control-Plane Update Propagation
 Multiple Underlay Domains - Isolated
• Isolated Underlay Domains – No need for Extension
 Multiple Replication Domains for BUM – Interconnected & Controlled
• Individual BUM flooding domain with Traffic control

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 210
Inter-X Connectivity
Multi-Pod Multi-Fabric Multi-Site
Underlay Control Plane Unified Underlay Domain Separated Underlay Domains Separated Underlay Domains

Overlay Control Plane Separated Overlay Control-Plane Domains

Overlay Data Plane Single Data-Plane Separated Data-Planes Separated Data-Planes

Unified Underlay Domain (All

BUM Replication in DCI Dependency on DCI Choice (Unicast/Multicast)
Multicast or All Ingress Replication)

ARP Flood Suppression (DCI) yes yes yes

Unknown Unicast Flood

no yes yes
Suppression (DCI)

Broadcast Suppression/Limit
no yes yes
(DCI)

Layer-2 Loop Prevention Loop mitigation (Edge Protection) VPC at Border Loop mitigation (At DCI)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 211
Virtual Peer Link
(vPC) Update
Traditional vPC Recap
vPC Domain

vPC1 Physical Peer Link vPC2

Orphan Port Orphan Port

Peer Keepalive

vPC
Server Server

Server

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 213
 vPC for VXLAN and VXLAN EVPN
Spine Spine

Individual Identity vPC1 Individual Identity vPC2

10.1.1.11 10.1.1.12
Anycast VTEP (vip)
10.1.1.10
vPC1 vPC2

EVPN Route Type Attachment Next-hop

Type 5 vPC advertised by vip

(IP Prefix Routes)
Server Orphan advertised by vip Server

Type 2 vPC advertised by vip

(Host Routes) Orphan Port advertised by vip
Server

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 214
vPC for VXLAN and VXLAN EVPN
Spine Spine

Backup Routing over

Peer Link

vPC1 vPC2

Server Server

Server

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 215
vPC for VXLAN EVPN
Spine Spine

Per-VRF Peering for IP

Prefix Exchange

vPC1 vPC2

Subnet X Subnet Y
192.168.11.0/24 192.168.12.0/24

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 216
 vPC for VXLAN EVPN (advertise-pip)
Spine Spine

Individual Identity and Individual Identity and

VTEP vPC1 (pip) VTEP vPC2 (pip)
10.1.1.11 10.1.1.12
Anycast VTEP (vip)
10.1.1.10
vPC1 vPC2

Subnet X Subnet Y
192.168.11.0/24 192.168.12.0/24

EVPN Route Type Attachment Next-hop

Type 5 vPC advertised by pip

(IP Prefix Routes)
Server Orphan advertised by pip Server

Type 2 vPC advertised by vip

(Host Routes) Orphan Port advertised by vip
Server

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 217
 vPC without Peer-Link for (vPC2) VXLAN EVPN
Spine Spine

Virtual Peer Link over Fabric (Layer-3)

Virtual Peer Link
• Uses Spines for Redundancy, Resiliency
and Performance
• Doesn’t use VTEP IP address (loopback)

vPC1 vPC2
Orphan Port Orphan Port

Peer Keepalive

vPC
Server
Peer Keepalive remains Server
• Out-of-Band (mgmt0 or dedicated link)*
• In-Band (dedicated Loopback)

Server

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 218
vPC without Peer-Link for (vPC2) VXLAN EVPN
Spine Spine

Virtual Peer Link

vPC1 vPC2

Subnet X Orphan Port Orphan Port Subnet Y

192.168.11.0/24 192.168.12.0/24

Peer Keepalive

vPC
Server Server

Server

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 219
vPC without Peer-Link for (vPC2) VXLAN EVPN
• Introduction in NX-OS 9.2(3) • Smaller VTEP Scale per Fabric
• Part of Essentials License • initial release at ~1/3
• “always PIP” mode results in 3 VTEPs
• Supported FX/FX2 Platforms per vPC domain
• EX-based Platform in future • Compensated with upcoming VTEP scale
increase (9.3(x))
• PIM ASM and Ingress-
Replication for BUM • Leaf and Border deployments
only
• PIM BiDir under consideration
• no BGW for Multi-Site support
• TRM Support
• No FEX

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 220
VXLAN Tenant Routed
Multicast (TRM)
Same Subnet Forwarding no IGMP Snooping
Traditional Forwarding in VXLAN Overlays

S • ”Single Copy” in Core – Treated as BUM

SRC
• Same Subnet Only
TOR1 TOR2
SRC-10 Leaf • No Pruning on Local
Leaf Interface or Remote VTEP
224.10.10.10 Interface
10.10.10.100

R Spine

RCVR VLAN 101 (Green)

VLAN 101 (Green)
RCVR-10 Spine
10.10.10.10

TOR3 TOR4
Leaf Leaf

VXLAN EVPN
VLAN 101 (Green) VLAN 101 (Green)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 222
Same Subnet Forwarding with IGMP Snooping
Traditional Forwarding in VXLAN Overlays

S • ”Single Copy” in Core – Treated as BUM

SRC
• Same Subnet Only
TOR1 TOR2
SRC-10 Leaf • Pruning on Local
Leaf Interface
224.10.10.10
10.10.10.100 • VXLAN is ”pruned off” if no interest Receiver exists
behind any Remote VTEP
R Spine

RCVR VLAN 101 (Green)

VLAN 101 (Green)
RCVR-10 Spine
10.10.10.10
R
TOR3 TOR4 RCVR

RCVR-11
Leaf Leaf

10.10.10.11

VXLAN EVPN
VLAN 101 (Green) VLAN 101 (Green)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 223
Different Subnet Forwarding – Router on-a-Stick
Traditional Forwarding in VXLAN Overlays

S
SRC

TOR1 TOR2
SRC-10 Leaf Leaf
224.10.10.10 10.10.10.254
10.10.10.100 10.20.20.254

Spine

VLAN 101 (Green)

Spine

R
TOR3 TOR4 RCVR

RCVR-21
Leaf Leaf

10.20.20.21

VXLAN EVPN
VLAN 101 (Green) VLAN 101 (Green)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 224
Different Subnet Forwarding – Router on-a-Stick
Traditional Forwarding in VXLAN Overlays

S
SRC

TOR1 TOR2
SRC-10 Leaf Leaf
224.10.10.10 10.10.10.254
10.10.10.100 10.20.20.254

Spine

VLAN 101 (Green)

Spine

R
TOR3 TOR4 RCVR

RCVR-21
Leaf Leaf

10.20.20.21

VXLAN EVPN
VLAN 101 (Green) VLAN 101 (Green)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 225
Different Subnet Forwarding – Router on-a-Stick
Traditional Forwarding in VXLAN Overlays
• Multiple Copy in Core – Treated as BUM
S
SRC
• Different Subnet possible – RPF Challenges
TOR1 TOR2
• Pruning on Local Interface
SRC-10 Leaf Leaf
224.10.10.10 • VXLAN is NOT pruned if interest Receiver exists behind
10.10.10.254
10.10.10.100 one Remote VTEP 10.20.20.254

Spine

VLAN 101 (Green)

Spine

R
TOR3 TOR4 RCVR

RCVR-21
Leaf Leaf

10.20.20.21

VXLAN EVPN
VLAN 101 (Green) VLAN 101 (Green)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 226
Functional Components
Tenant Routed Multicast (TRM)
Spine Spine
Site-External DCI
(IP Routing and Increased
MTU Support)
•
•
• VXLAN EVPN
•
VTEP VTEP VTEP VTEP
•
DR DR DR DR
•
•
•
•
Baremetal Baremetal Baremetal
• Baremetal Baremetal

•
SRC-10 RCVR-10 RCVR-20 • RCVR-30 RCVR-11
224.10.10.10 10.10.10.10 20.20.20.20 30.30.30.30 10.10.10.11
10.10.10.100

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 227
Same Subnet Forwarding w/IGMP Snooping
TRM Forwarding (Layer-2 Only Mode)
•
S
•
SRC

TOR1 • TOR2
SRC-10
224.10.10.10
Leaf
• Leaf

10.10.10.100
•

R Spine

RCVR VLAN 101 (Green)

VLAN 101 (Green)
RCVR-10 Spine
10.10.10.10
R
TOR3 TOR4 RCVR

RCVR-11
Leaf Leaf

10.10.10.11

VXLAN EVPN
VLAN 101 (Green) VLAN 101 (Green)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 228
Different Subnet Forwarding
TRM Forwarding (Layer-3 Mode)

TOR1 TOR2
Leaf Leaf

Spine

VLAN 101 (Green)

VLAN 101 (Green)
L3VNI 50001
VLAN 202 (Blue)
VLAN 202 (Blue) Spine

TOR3 TOR4
Leaf Leaf

VXLAN EVPN
VLAN 101 (Green) VLAN 101 (Green)

VLAN 202 (Blue) VLAN 202 (Blue)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 229
Different and Same Subnet Forwarding
TRM Forwarding (Layer-3 Mode)
•
S
•
SRC

TOR1 • TOR2
SRC-10
224.10.10.10
Leaf
• Leaf

10.10.10.100
•

R
Spine

L3VNI 50001
RCVR

RCVR-20 Spine
10.20.20.20 R
R TOR3 TOR4 RCVR

RCVR-21
Leaf Leaf
RCVR

10.20.20.21
RCVR-10
10.10.10.10
VXLAN EVPN

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 230
Local and Remote Forwarding
TRM Forwarding (Layer-3 Mode)

S
SRC

TOR1 TOR2
SRC-10 Leaf Leaf
224.10.10.10
10.10.10.100 TTL Decrement

R TTL Decrement Spine

(routed) L3VNI 50001

RCVR

RCVR-20 Spine
10.20.20.20 R
R TOR3 TOR4 RCVR

RCVR-21
Leaf Leaf
RCVR

10.20.20.21
RCVR-10
10.10.10.10
VXLAN EVPN

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 231
Overlay Rendezvous Point
TRM Forwarding (Layer-3 Mode)
•
S
•
SRC

TOR1 • TOR2
SRC-10 Leaf • Leaf
224.10.10.10
10.10.10.100 TTL Decrement
•
R TTL Decrement Spine
•
(routed) L3VNI 50001
•
RCVR

RCVR-20 Spine
•
10.20.20.20 R
R TOR3 TOR4 RCVR

RCVR-21
Leaf Leaf
RCVR

10.20.20.21
RCVR-10
10.10.10.10
VXLAN EVPN

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 232
 Nicolas Delecroix
Nexus 9K Standalone Technical Marketing Engineer

Programmability
Extensibility: Guest Shell
and Docker
Securely Run Custom On-Box Linux Apps
$ dohost
NX-OS Guest Shell: Secure Linux Container 64
JSON Bit
CLI
Open-Source Your Custom Apps
bootflash: Apps (C, Python, Go…)

CentOS 7.0 rootfs

Network
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Linux + Nexus 9K = ♥︎
[root@guestshell ~]# ifconfig Eth1-42
Eth1-42: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 13.0.0.42 netmask 255.255.255.0 broadcast 13.0.0.255
ether 54:7f:ee:8e:27:bc txqueuelen 100 (Ethernet)
RX packets 3790 bytes 258373 (252.3 KiB)
RX errors 0 dropped 3553 overruns 0 frame 0
TX packets 772 bytes 201535 (196.8 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Get data plane traffic in addition to control plane traffic:

monitor session 1
source interface Ethernet1/42 rx
destination interface sup-eth0
no shut

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 236
 Linux  NX-OS Network Synchronization
[guestshell@guestshell ~]$ ifconfig Eth1-49
interface Ethernet1/49 Eth1-49:
mtu 9216 flags=4163<UP,BROADCAST,RUNNING,MULTICAST>
ip address 10.0.1.2/30 mtu 9216
no shutdown inet 10.0.1.2 netmask 255.255.255.252
broadcast 10.0.1.3

switch# sh vrf [guestshell@guestshell ~]$ ip netns list

VRF-Name VRF-ID State vpn1
default 1 Up -- management
management 2 Up -- default
vpn1 3 Up -- [guestshell@guestshell ~]$

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 237
 Linux  NX-OS Network Synchronization
[guestshell@guestshell ~]$ ifconfig Vlan10
interface Vlan10 Vlan10: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>
no shutdown mtu 1500
ip address 192.168.1.1/24 inet 192.168.1.1 netmask 255.255.255.0
broadcast 192.168.1.255

switch# sh ip route
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

10.0.0.0/30, ubest/mbest: 1/0

*via 10.0.0.6, Eth1/49, [110/2], 00:05:34, ospf-UNDERLAY, intra

[guestshell@guestshell ~]$ route

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.0.0.0 10.0.0.6 255.255.255.252 UG 51 0 0 Eth1-49
TECDCN-2002
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 238
Richly Populated Repositories for 3rd Party Apps
switch# guestshell
[guestshell@guestshell ~]$ cd /etc/yum.repos.d/
[guestshell@guestshell yum.repos.d]$ ls -l
total 15
-rw-r--r-- 1 root root 1664 Nov 3 19:25 CentOS-Base.repo
-rw-r--r-- 1 root root 1309 Nov 3 19:25 CentOS-CR.repo
-rw-r--r-- 1 root root 649 Nov 3 19:25 CentOS-Debuginfo.repo
-rw-r--r-- 1 root root 1331 Nov 3 19:25 CentOS-Sources.repo
[guestshell@guestshell yum.repos.d]$

[guestshell@guestshell ~]$ chvrf management yum repolist all

Loaded plugins: fastestmirror
[...]
base/7/x86_64 CentOS-7 - Base enabled: 9007

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 239
The Guest Shell is Secure

• Namespaces are separate, resource usage is controlled,

access is controlled.
• No visibility into Cisco proprietary software (cannot read,
write, or execute NX-OS binaries).
• No visibility into Cisco proprietary disk partitions.
• No access to internal, Cisco proprietary drivers.
• No ability to load kernel drivers.

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 240
Linux Apps Can Interact with the External World
Nexus 9K
Your Custom Applications Existing 3rd Party Linux
(Python, C++ etc.) Applications

Guest Shell Linux Networking Stack

NX-OS CLI

L2 L3 Interfaces Platform More…

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 241
Docker Engine
NX-OS 9.2(1) – July 2018

• Available on all Nexus 9K models,

and on Nexus 3K models equipped
with 8G+ of memory.

• Standard Docker engine with all the

commands supported: Nexus 9K
docker run/pull/push/kill/info
etc.

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 242
 Standardization, Flexibility, and Efficiency
Guest Shell Docker Engine
Number of container instances One Many
Access to storage and network Yes Yes
Linux distribution type CentOS Any
Container manipulation NX-OS CLI (# guestshell *) Standard Linux docker tool
primitives
Definition of the container Must be done on a Nexus 9K Can be done from any
image content computer supporting Docker
Repository of existing None Docker Hub
container images
Container orchestration None Docker Swarm or Kubernetes

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 243
Demo: Docker on N9K
switch(config)# feature bash-shell
switch(config)# run bash sudo su
bash-4.3#
# Start Docker
bash-4.3# service docker start

# Check the status

bash-4.3# service docker status
dockerd (pid 3597) is running...

# Persist on reload of switch:

bash-4.3# chkconfig --add docker

# Start an alpinelnx docker container:

bash-4.3# docker run --name=alpine -it alpine
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
4fe2ade4980c: Pull complete
Digest: sha256:621c2f39f8133acb8e64023a94dbdf0d5ca81896102b9e57c0dc184cadaf5528
Status: Downloaded newer image for alpine:latest
/ #
/ # grep PRETTY_NAME /etc/os-release
PRETTY_NAME="Alpine Linux v3.8"
TECDCN-2002
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 245
/ #
 What Apps are Interesting to Host on N9K?
• Monitoring agents • Automation agents (Chef, Puppet,
• Open-source agents: OpenTSDB, SaltStack...)
Ganglia, Nagios, etc. • Intrusion Detection
Monitor both standard Linux • DNSFlow agent to detect phishing
components (CPU, memory, activity
interface counters), and NX-OS • Custom Intrusion Detection agents
(routes, buffers,...)
• Automatic configuration backup to
• Custom agents: ECMP load a private Git repository
balancing, PTP accuracy…

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 246
Automation
 Legacy Automation is Challenging
• Expect scripts written in TCL.

• Combined with regular expressions to parse

the CLI prompt.
• Those regexps can become complex.

• Those regexps might have to change if

Cisco changes the CLI output (we try not to!)
• We need to deal with CLI prompts and
timeouts, too.
Source: "An Angry Engineer" https://www.youtube.com/watch?v=c_4SlcsUSYQ

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 248
248
 Telnet NX-API NX-API
SNMP NETCONF / RESTCONF / gRPC
or CLI REST
Client YANG Clients
SSH Client Client
Management Server

Your NX- Native NX-OS Nexus 9K

SDK Apps Features

SNMP NETCONF / RESTCONF / gRPC

NX-SDK APIs NGINX Server Agent YANG Agents

NX-OS Infra
(CLI Mgr, Syslog Mgr, ...) CLI

Data Management Engine (DME)

Transaction Commit Object Store Status: Success or Raise Fault

BGP Data VLAN Data QoS Data

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 249
NX-API
 {
"jsonrpc": "2.0",
"method": "cli",
CLI "params": {
"cmd": "show version",
Request "version": 1
Management },
Server }
"id": 1

{
"jsonrpc": "2.0",
"result": {
"body": {
"bios_cmpl_time": "03/02/2017",
JSON "bootflash_size": 7906304,
"kickstart_ver_str": "7.0(3)I7(3)",
Response "chassis_id": "Nexus 9508",
... Nexus 9K
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 251
Demo: NX-API
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
 Get Started With Just Two Commands
Ready-to-use Docker container with a pre-build Python environment,
and NX-API apps ready to run:
[devops@server ~]$ docker run -it ndelecro/nx-os-programmability
Status: Downloaded newer image for docker.io/ndelecro/nx-os-
programmability:latest
root@a3d1f69d8067:~#

TECDCN-2002
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 255
[devops@server ~]$ docker run -it ndelecro/nx-os-programmability
Status: Downloaded newer image for docker.io/ndelecro/nx-os-programmability:latest
root@a3d1f69d8067:~# cd NX-API_CLI/VXLAN_BGP_EVPN/
root@a3d1f69d8067:~/NX-API_CLI/VXLAN_BGP_EVPN# ./1.Create_L2VNI.py vteps 42 42000 239.1.1.1 e1/41

TECDCN-2002
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 256
[devops@server ~]$ docker run -it ndelecro/nx-os-programmability
Status: Downloaded newer image for docker.io/ndelecro/nx-os-programmability:latest
root@a3d1f69d8067:~# cd NX-API_CLI/VXLAN_BGP_EVPN/
root@a3d1f69d8067:~/NX-API_CLI/VXLAN_BGP_EVPN# ./1.Create_L2VNI.py vteps 42 42000 239.1.1.1 e1/41
****** VTEP 93180-EX-1 ******
vlan 42
vn-segment 42000

int nve1
member vni 42000
mcast-group 239.1.1.1
suppress-arp

evpn
vni 42000 l2
rd auto
route-target import auto
route-target export auto

int e1/41
switchport access vlan 42

****** VTEP 93180-EX-2 ******

vlan 42
...
TECDCN-2002
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 257
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Ansible
Writing Your Own Automation Can be Challenging
• NX-API brings a modern transport, and a structured output. But...
• Need to have development skills and allocate time to write, maintain and
test the code.
• Need to deal with authentication and transport.
• Ideally, be able to group switches by category.
• Not much leverage of existing, tested apps besides the low-level API.
• Not straightforward to re-use existing CLI (Jinja2) templates.
• Not idempotent by default, need to implement idempotency yourself.
• No parallelization by default, need to implement threading yourself.
• So let’s leverage existing tools on top of NX-API!
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Ansible Overview
• Ansibles automates most DC assets with a
Management Server minimal learning curve.
• Both switches and servers can be
Ansible Controller
managed.
Inventory Modules • Human-readable  very little scripting
skills required.
• Agent-less  easy to adopt.
Playbooks Config
• NX-OS: Ansible modules abstract the CLI.

• Advanced features:
Configure
• Variables • Events

• Conditionals • Loops
Targets
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 261
Management NX-API NX-API NETCONF / RESTCONF / gRPC
SSH SNMP
Server CLI REST YANG

Your NX- Native NX-OS Nexus 9K

SDK Apps Features

SNMP NETCONF / RESTCONF / gRPC

NX-SDK APIs NGINX Server Agent YANG Agents

NX-OS Infra
(CLI Mgr, Syslog Mgr, ...) CLI

Data Management Engine (DME)

Transaction Commit Object Store Status: Success or Raise Fault

BGP Data VLAN Data QoS Data

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
 Ansible Architecture • Inventory: target
systems for
automation.
Targets • Playbook: a series
of plays
Inventory (automation tasks).
• Modules:
accomplish specific
tasks in Ansible
(e.g. install
SSH packages,
Playbook Config Python configure NX-OS,
NX-API etc.)
• Ansible Config:
NETCONF determines how
your Ansible setup
behaves (how many
Modules concurrent
connections, etc.)
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 263
 Ansible Playbook = Sequence of Tasks to Execute
Example: Deploy NTP on all the Servers

Set of target devices (defined in a separate file)

[devops@server ~]$ cat ntp.yml Playbooks are YAML files
---
- hosts: all sudo For all those target devices,
become: yes
execute the tasks below
tasks:
- name: Ensure NTP is installed
yum: name=ntp state=present Arguments to
- name: Ensure NTP is running the module
service: name=ntpd state=started enabled=yes
[devops@server ~]$

Ansible module that will do the actual work Blue: Ansible keyword or module
White: a value that you define
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Ansible Inventory

[devops@server ~]$ cat /etc/ansible/hosts [devops@server ~]$ cat \

[nxos_vteps] /etc/ansible/group_vars/nxos_vteps
93180-EX-1 ---
93180-EX-2 ansible_network_os: nxos
92160-1 ansible_connection: network_cli
92160-2 ansible_user: devops
93180-FX-2 ansible_ssh_pass: automate

[nxos_spines]
9364-1
9364-2 Best practice: use Ansible Vault for
password encryption.
[older_routers]
router-A
router-B

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 265
Demo: Ansible
VXLAN BGP EVPN Automation
[devops@server ~]$ cat vxlan.yml
---
- name: Create L2VNI
hosts: nxos_vteps

tasks:
- name: Create VLAN and map to to VNI
nxos_vlan:
vlan_id: 2200
mapped_vni: 20200

- name: Add L2VNI to Overlay

nxos_vxlan_vtep_vni:
interface: nve1
vni: 20200
...

Blue: Ansible keyword or module name

White: a value that you define TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 267
VXLAN BGP EVPN Automation
[devops@server ~]$ cat vxlan.yml
...
tasks:
- name: Create VLAN and map to to VNI
nxos_vlan: vlan 2200
vlan_id: 2200 vn-segment 20200
mapped_vni: 20200

- name: Add L2VNI to Overlay interface nve1

nxos_vxlan_vtep_vni: no shutdown
interface: nve1 host-reachability protocol bgp
vni: 20200 member vni 20200
multicast_group: 239.239.239.100 suppress-arp
suppress_arp: true mcast-group 239.239.239.100

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 268
[devops@server ~]$ ansible-playbook vxlan.yml
PLAY [Create L2VNI] ************************************************************

TASK [Create VLAN and map to to VNI] *******************************************

ok: [93180-FX-2]
changed: [92160-1]
ok: [92160-2]
ok: [93180-EX-1]
changed: [93180-EX-2]

TASK [Add L2VNI to Overlay] ****************************************************

changed: [92160-1]
...

PLAY RECAP *********************************************************************

92160-1 : ok=5 changed=4 unreachable=0 failed=0
92160-2 : ok=5 changed=3 unreachable=0 failed=0
93180-EX-1 : ok=5 changed=3 unreachable=0 failed=0
93180-EX-2 : ok=5 changed=4 unreachable=0 failed=0
93180-FX-2 : ok=5 changed=3 unreachable=0 failed=0

[devops@server ~]$ TECDCN-2002

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 269
What’s the Cleanest Way to Handle This Error?
[devops@server ~]$ cat vxlan.yml Config prior to the playbook
... execution:
tasks:
- name: Create VLAN and map to to VNI interface nve1
nxos_vlan: no shutdown
vlan_id: 2200 host-reachability protocol bgp
mapped_vni: 20200 member vni 20200
admin_state: up suppress-arp
ingress-replication protocol bgp
- name: Add L2VNI to Overlay
nxos_vxlan_vtep_vni:
interface: nve1
vni: 20200
multicast_group: 239.239.239.100
suppress_arp: true

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 270
 [devops@server ~]$ cat rollback.yml
...
Checkpoint and Rollback! tasks:
- name: Create checkpoint
nxos_rollback:
checkpoint_file: backup.cfg

- name: VXLAN config

block:
- name: Create VLAN and map to the VNI
nxos_vlan:
vlan_id: 2200
mapped_vni: 20200
admin_state: up
- name: Add L2VNI to Overlay
nxos_vxlan_vtep_vni:
interface: nve1
vni: 20200
multicast_group: 239.239.239.100
rescue:
- name: Rollback to checkpoint
nxos_rollback:
rollback_to: backup.cfg
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 271
[devops@server ~]$ ansible-playbook rollback.yml
TASK [Create checkpoint]
*************************************************************************************************
changed: [93180-FX-2]

TASK [Create VLAN and map to the VNI]

*************************************************************************************************
changed: [93180-FX-2]

TECDCN-2002
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 272
[devops@server ~]$ ansible-playbook rollback.yml
TASK [Create checkpoint]
*************************************************************************************************
changed: [93180-FX-2]

TASK [Create VLAN and map to the VNI]

*************************************************************************************************
changed: [93180-FX-2]

TASK [Add L2VNI to Overlay]

*************************************************************************************************
fatal: [93180-FX-2]: FAILED! => {"changed": false, "clierror": "Cannot associate a multicast group
or vrf to an ingress replication vni.\n\n", "code": "400", "msg": "CLI execution error", "output":
[{"body": {}, "code": "200", "msg": "Success"}, {"body": {}, "code": "200", "msg": "Success"},
{"body": {}, "code": "200", "msg": "Success"}, {"clierror": "Cannot associate a multicast group or
vrf to an ingress replication vni.\n\n", "code": "400", "msg": "CLI execution error"}], "url":
"http://93180-FX-2:80/ins"}

TECDCN-2002
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 273
[devops@server ~]$ ansible-playbook rollback.yml
TASK [Create checkpoint]
*************************************************************************************************
changed: [93180-FX-2]

TASK [Create VLAN and map to the VNI]

*************************************************************************************************
changed: [93180-FX-2]

TASK [Add L2VNI to Overlay]

*************************************************************************************************
fatal: [93180-FX-2]: FAILED! => {"changed": false, "clierror": "Cannot associate a multicast group
or vrf to an ingress replication vni.\n\n", "code": "400", "msg": "CLI execution error", "output":
[{"body": {}, "code": "200", "msg": "Success"}, {"body": {}, "code": "200", "msg": "Success"},
{"body": {}, "code": "200", "msg": "Success"}, {"clierror": "Cannot associate a multicast group or
vrf to an ingress replication vni.\n\n", "code": "400", "msg": "CLI execution error"}], "url":
"http://93180-FX-2:80/ins"}
TASK [Rollback to checkpoint]
*************************************************************************************************
changed: [93180-FX-2]

PLAY RECAP
*************************************************************************************************
93180-FX-2 : ok=5 changed=3 unreachable=0 failed=1

[devops@server ~]$ TECDCN-2002

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 274
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 275
 Upgrade or Downgrade NX-OS at Scale
---
- name: Install NX-OS
hosts: my_switches

tasks:
- name: Enable scp-server feature
nxos_feature:
feature: scp-server
state: enabled

- name: Copy NX-OS image to the switch

nxos_file_copy:
local_file: /devops/img/nxos.7.0.3.I7.5a.bin

- name: Install NX-OS

nxos_install_os:
system_image_file: nxos.7.0.3.I7.5a.bin
register: install_state

- debug:
var: install_state TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
[devops@server ~]$ ansible-playbook install_nxos.yml

PLAY [Install NX-OS] **************************************************************

TASK [Gathering Facts] ************************************************************

ok: [93180-EX-1]

TASK [Enable scp-server feature] **************************************************

changed: [93180-EX-1]

TASK [Copy NX-OS image] ***********************************************************

ok: [93180-EX-1]

TECDCN-2002
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 277
TASK [Install NX-OS]
************************************************************************************************
changed: [93180-EX-1]

TASK [debug] ***********************************************************************************

ok: [93180-EX-1] => {
"install_state": {
"changed": true,
"failed": false,
"install_state": [
"Compatibility check is done:",
"Module bootable Impact Install-type Reason",
"------ -------- -------------- ------------ ------",
" 1 yes disruptive reset Incompatible image for ISSU",
"Images will be upgraded according to following table:",
"Module Image Running-Version(pri:alt) New-Version Upg-Required",
"------ ---------- ---------------------------------------- -------------------- ------------",
" 1 nxos 9.2(1) 7.0(3)I7(3) yes",
" 1 bios v07.64(05/17/2018):v07.45(12/04/2015) v07.61(04/06/2017) no",
"--------------------------------------",
"Module 1: Refreshing compact flash and upgrading bios/loader/bootrom."
]
}
}

PLAY RECAP *************************************************************************************

93180-EX-1 : ok=5 changed=1 unreachable=0 failed=0
[devops@server ~]$
TECDCN-2002
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 278
 NX-OS Ansible Modules
Over 70 NX-OS Modules in Ansible 2.7

• AAA • Interfaces • Snapshot

• ACL • NTP • Static Routing

• BGP • NX-API • Upgrade

• Checkpoint / Rollback • NX-OS Facts • VLAN

• CLI • OSPF • vPC

Note: can be used with
Jinja2 templates • PIM • VRF

• HSRP / VRRP • Port-Channel / vPC • VXLAN Flood & Learn

• Patching • VXLAN BGP EVPN

• IGMP / IGMP Snooping

Most extensive support for networking software in the industry.

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 279
https://github.com/ndelecro/nx-os-programmability/tree/master/Ansible/2.5
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
 Get Started With Just Two Commands
Ready-to-use Docker container with Ansible installed and configured, and
NX-OS playbooks available:
[devops@server ~]$ docker run -it ndelecro/nx-os-programmability
Status: Downloaded newer image for docker.io/ndelecro/nx-os-programmability:latest
root@a3d1f69d8067:~# ansible-playbook ~/Ansible/vxlan_nxapi.yml

PLAY [Create L2VNI]

*************************************************************************************************

TASK [Create VLAN and map to to VNI]

*************************************************************************************************
changed: [93180-FX-2]
changed: [93180-EX-1]

PLAY RECAP
*************************************************************************************************
93180-EX-1 : ok=1 changed=1 unreachable=0 failed=0
93180-FX-2 : ok=1 changed=1 unreachable=0 failed=0

root@a3d1f69d8067:~#
TECDCN-2002
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 281
NETCONF/YANG
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 283
Read all VRFs with OpenConfig

<get-config>
<source>
<running/>
</source>
<filter>
<network-instances xmlns="http://openconfig.net/yang/network-instance">
<network-instance/>
</network-instances>
</filter>
</get-config>

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 284
<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="2">
<data>
<network-instances xmlns="http://openconfig.net/yang/network-instance">
<network-instance>
<config>
<description/>
<enabled>true</enabled>
<name>default</name>
<type>L3VRF</type>
</config>
<name>default</name>
</network-instance>
<network-instance>
<config>
<description/>
<enabled>true</enabled>
<name>Testing1</name>
<type>L3VRF</type>
</config>
... TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 285
 Telnet NX-API NX-API
SNMP
or CLI REST
Client
Client YDK
SSH Client
Management Server

Your NX- Native NX-OS Nexus 9K

SDK Apps Features

SNMP NETCONF / RESTCONF / gRPC

NX-SDK APIs NGINX Server Agent YANG Agents

NX-OS Infra
(CLI Mgr, Syslog Mgr, ...) CLI

Data Management Engine (DME)

Transaction Commit Object Store Status: Success or Raise Fault

BGP Data VLAN Data QoS Data

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 286
Choosing an Automation
Strategy Discussion
Prototyping and
Validating with N9Kv
Nexus 9000v (N9Kv)
• N9Kv is:
• A simulation platform that runs N9K NX-OS as a virtual machine.
• Abstracting N9K hardware: supervisor, one line card, interfaces.
• Using a software data-plane.
• Supported in VIRL.

• N9Kv is not:
• A replacement for N9K physical platform.
• Designed to be used as a switch in a production data center.

• Supported hypervisors: ESX, KVM, VirtualBox, Fusion.

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 289
n9kv-1# sh mod
Mod Ports Module-Type Model Status
--- ----- ------------------------------------- --------------------- ---------
1 128 Nexus 9000v Ethernet Module N9K-9000v active *

Mod Sw Hw Slot
--- ----------------------- ------ ----
1 9.2(2) 0.0 NA

Mod MAC-Address(es) Serial-Num

--- -------------------------------------- ----------
1 00-50-56-b9-96-72 to 00-50-56-b9-96-f9 93B0EQCTFQB

Mod Online Diag Status

--- ------------------
1 Pass

* this terminal session

n9kv-1#
TECDCN-2002
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 290
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
NX-OSv Is Great for Prototyping and Testing
• DevOps
• Rapid prototyping of 3rd party tools & applications
• Rapid development & testing of platform-independent features

• Configuration policy impact analysis

• Validate network configuration prior to deployment in the actual network
• Analyze control plane behavior
• Lab as a Service
• No physical test beds required

• Learning Tool

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 293
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Inline Upgrade and Downgrade
NX-OS 9.2(1) – July 2018

• Prior to 9.2(1), the N9Kv upgrade or downgrade process is:

• Download the new OVA or VMDK image.
• Perform the whole VM bring up again in ESX, VirtualBox etc.
• Migrate the whole NX-OS configuration to the new VM.
• Re-configure the ports mapping in case the VM is part of a broader network
topology.
• With 9.2(1), N9Kv can now be upgraded and downgraded like a physical
Nexus 9K, while keeping the same VM.
• The upgrade or downgrade is disruptive. NX-OS reloads as part of the
process, and during the reload the control plane and data plane are down.

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 295
n9kv-1# sh mod | i Nexus
1 128 Nexus 9000v Ethernet Module N9K-9000v active *

n9kv-1# sh ver | i image

NXOS image file is: bootflash:///nxos.9.2.1.bin

n9kv-1# dir bootflash:

1420 Dec 04 18:05:32 2018 20181204_180528_poap_30696_init.log
0 Jan 07 10:10:30 2019 bootflash_sync_list
4096 Dec 11 10:35:50 2018 home/
4096 Jan 07 09:57:10 2019 lost+found/
1308795904 Jan 07 09:39:55 2019 nxos.9.2.1.bin
1322543104 Nov 05 06:29:19 2018 nxos.9.2.2.bin
4096 Dec 04 18:04:54 2018 virtual-instance/

Usage for bootflash://sup-local

2893565952 bytes used
576544768 bytes free
3470110720 bytes total

n9kv-1# install all nxos nxos.9.2.2.bin

Installer will perform compatibility check first. Please wait.
Installer is forced disruptive

Verifying image bootflash:/nxos.9.2.2.bin for boot variable "nxos".

[####################] 100% -- SUCCESS
TECDCN-2002
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 296
...
 N9000v Feature Support Shipping Planning

7.0(3)I7(x) 9.2(1) 9.3(1)

Guest Shell

NX-API / NX-API REST / YANG

POAP
BGP v4

VXLAN BGP EVPN

VXLAN BGP EVPN Multi-Site

VXLAN BGP EVPN Multi-Site with TRM

OSPF v4/v6

vPC

Number of Interfaces 64 128 576

Memory Footprint 8G 8G 8G
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 297
 N9000v Feature Support Shipping Planning

7.0(3)I7(x) 9.2(1) 9.3(1)

Guest Shell

NX-API / NX-API REST / YANG

POAP
BGP v4

VXLAN BGP EVPN

VXLAN BGP EVPN Multi-Site

VXLAN BGP EVPN Multi-Site with TRM

OSPF v4/v6

vPC

Number of Interfaces 64 128 576

Memory Footprint 8G 8G 8G
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 298
 Matthias Wessendorf
Technical Marketing Engineer
Data Center Network
Manager (DCNM)
DCNM Overview &
Functions
 DCNM What is it?
• NX-OS Mode Multi-Fabric manager for
IP Fabrics
• VXLAN Fabric Control
• Classic ‘FCAPs’ LAN manager
• Network Monitor
• FabricPath, STP/VPC networks
• SAN/Storage Manager for Nexus and
MDS platforms
• Flow Controller for IP Media Solutions
• Vertical market for Broadcast media
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 301
DCNM Functional Areas...

VXLAN-EVPN
Programmable Fabric LAN-Classic Management
Nexus 5K, 7K-9K Nexus 1K, 2K, 3K, 5K-7K-9K
Automation and Control

IP Media Networking SAN

Broadcast/Web
[PMN] [MDS and Nexus]
Media
Customers

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 302
DCNM Functionality Dashboard
Trend Analysis Host/Endpoint
Inventory & Health Configuration Automation and VM Analytics Monitoring

• Discovery & Fabric • Image Management • VXLAN Fabric Builder • VM Net Trace • VM Lifecycle
Builder • Backup / Restore • Classic Underlay • Monitor Graphs • Network Location
• CPU/Mem/Temp • Templates (POAP) • Interface Monitoring • Fabric-Wide View
• Traffic • Overlay (VRF/VNI)
• Health-Monitor • REST APIs
• Link View • Brownfield Migration
• VM-connectivity
Visualization and Alert/ Storage IP Media Net
Troubleshooting Notifications Management Controller

• Integrated Topology • Trap & Syslog • Classic FC/FCOE

• Search • Events & Forwarding • SAN Analytics • Digital Media Flow
• VXLAN-OAM Control
• Endpoint Topology View
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 303
DCNM for New or
Existing Fabrics
Leveraging DCNM for New and Existing Fabrics
Install / Use DCNM
(Virtual Appliance)

Bootstrap Devices
Discover Existing
[POAP]
Fabrics / Networks
STP/VPC
Fabric Builder
Create New VXLAN
DFA / FabricPath
Fabrics

Maintain and Operate New VXLAN Fabrics

& Existing VXLAN or FabricPath or VLAN Fabrics

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 305
VXLAN Fabric Builder Auto-Deployment

VXLAN Fabric
Auto-Deployment

Managed-
Turn-Key VTEP Simplified
Fabric
Deployment Deployment
Operations
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 306
POAP for Classic Deployments
Best
Practice
Template

+ -VXLAN
-Custom

POAP Bootstrap

Reliable Network Day 0 Deployment

Deployment

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 307
 Day 1+ Operations
Manage, Monitor, Visualize, Search
Challenge: Manage & Grow Underlay with minimal overhead & keep consistent intent
Deployed Fabric Manage Monitor / Visualize /
Search / Update
Underlay:
• SDN Networks [VTEPs]
• Image Update [ISSU]
• View Fabric Topology
• Monitor Health, Events,
Performance
[cpu/mem/iface/syslog]
• Add Devices/Expand

Cisco Advantage:
• Turnkey Management
• Integrated Views
• Comprehensive Fabric Views

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 308
Day 1+ Operations
Overlay Visibility, Growth
Challenge: Manage / Monitor SDN Overlay’s across a large fabric
Overlay Tasks: Monitor / Visualize / Search
• Visualize Overlays [VXLAN, VLAN, etc..]
• Add, Manage SDN Networks
• Shows VM Networking Path [vCenter]
• Find, Track VMs, Workloads [EPL]
• Find VN’s and VNI’s [VXLAN]
• View VXLAN E2E Connectivity [OAM]
• Identify Errors
• Validate Compliance

Cisco Advantage:
• Seamless Overlay/Underlay Correlation
• Easy to find workloads, VN’s, VNI’s on vast
fabric
• Easy to See Host-Network chain

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 309
Day 1+ Operations
Verify Compliance
Challenge: Ensure Deployment [Underlay, Overlay, Access] is Correct
Compliance Tasks: Detect and Fix

• Monitor Fabric
• Compare device configuration
against Fabric policy
• Remediate [revert or change Policy]
On-Demand
remediation

Cisco Advantage:
• Constant Monitoring
• Compliance engine brings fabric back to
intended configuration
• No un-anticipated behavior
Compliance engine remediates to intended configuration

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 310
Here is What That Looks Like in DCNM...

DCI

Existing DFA/FP Fabric New VXLAN Fabric

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 311
Getting Started with
DCNM
VXLAN User Experience with DCNM
• DCNM Differentiates Underlay vs. Overlay
• Use Fabric Builder
OR
• Use POAP templates for Underlay configuration
• VXLAN Best-Practice Templates from cisco.com
• Basic Manageability for “Classic” configurations

• Deploy Network/VRFs using “LAN Fabric Provisioning” for Overlay

• View VXLAN details via Topology Views, Search & Multi-Site

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 313
Discovering the Data Center
• DCNM Data Sources Include: SAN, LAN, VMware, & Storage Arrays

POAP will automatically start discovery, so you won’t need to do this if

you bootstrap via Fabric Builder or Classic POAP
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 314
VXLAN Underlay Bring-Up – DCNM Starting Point
Configure DCNM
Use Virtual Use Fabric Builder
Fabric
[regular POAP for
Appliance (VA) Management
(OVA / ISO Setup)
‘classic’ mode]
-VA Includes Fabric
--Generate POAP
Infrastructure -Management IP
definitions
-Fabric Management
subnet

Deploy Fabric
-Switch VTEP
Fabric
Configures
Automatically during Underlay
POAP Installed
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 315
 NEW in
DCNM 11

VXLAN Fabric Builder…

1 Pre-Stage, Minimal Input 3 Discover & Bootstrap

2 Assign Role & Deploy 4 Inspect New Fabric

Auto-VPC

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 316
 VXLAN Fabric Compliance
NEW in
DCNM 11

a.k.a. “Fabric Doctor”

1 3 Deploy Changes
Out-of-Sync
Detected

2 Preview Compliance
Remediation
4 Fabric Repaired!

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 317
POAP Dashboard for non-VXLAN Deployments

Control DHCP, File Server, and POAP Template Definitions from Here

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 318
POAP Dashboard
Boot/Discovery
Status

Mapped Serial Day 0 Generated Selected POAP

Number Configuration Template

Fabric Plan can

Automatically create
multiple mappings
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 319
POAP Templates – Best Practice or Custom

Choose Template

Copy / Paste
Settings

Template Use our best-practice templates for VXLAN-EVPN or

Determines the
Tabs customize your own.

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 320
Let’s Focus on New
VXLAN Functions
Top Down Deployment
• Deploys Configuration Profile for
VRF, Segment or Interface to -VRF
Switch without a Trigger -VNI

• Pushed from DCNM GUI or REST -VLAN

-(Interface)
API
• Doesn’t Require Switch Auto-
configuration
• Select an existing fabric or add a
new fabric and then define Fabric
Settings. Support the VXLAN-
EVPN for N9K.

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 322
Adding A New VXLAN Fabric

1. Create a New VXLAN Fabric ‘on the fly’

2. Use or adjust default settings
3. Update Pools for this fabric as necessary

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 323
Creating A New Network

1) Use Default VNI or

adjust
1) Choose VRF or add new

2) Add G/W to Define Net

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 324
Deploying The Network

1) Select Network

Staged Deployment is Blue

Yellow is “Deploying” 2 ) Choose which Switches to Deploy to
Green indicates Success
Red indicates failure
3) Deploy

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 325
 Deploying The Network – Selecting Switches
Double-Click Deploy to this switch Ready to Deploy
[De-Select to remove Network]

Select Ports if desired

(Not necessary if default is ‘trunk’)

In Progress Deployed

• Double Click the Switches where you want the network

• Select “Apply to Switch”
• Select Deploy
• Green indicates success

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 326
Controls
Deploy Details
Show / Troubleshoot Deployment

Preview

Add Switches to
Fabric

Refresh

Auto-Refresh
on/off

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 327
External Fabric Connectivity Provisioning
Border Node Deployments
• Setting up base and setup
configuration
• Deploying VRFs

• Deploying VRF_LITE using sub-

interfaces with pool management of
dot1q IDs
• IPv4 & IPv6 support
• VPC Support

• Deploying Networks for vanilla VLAN

hand-off

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 328
 VXLAN Multi-Site Deployment

Fabric 1 Fabric 2

Border Leaf
B Extensions B
VNI 34112 VNI 34112
VNI 26214 VNI 26214

Multi-Fabric Simplified/Coordinated Managed-Fabric

Management Operation Operations

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 329
EVPN Multi-Site Deployment
Support for Border Gateways
• Multi-Site Underlay & Overlay inter-
fabric connection setup
• B2B and Route-Server based topology
support

• Multi-Site Overlay extension

• Networks & VRFs

• Simultaneous VRF-LITE & Multi-Site

support

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 330
Troubleshoot VXLAN Using OAM
Show Fabric
Reachability

Helps Troubleshoot
Problems

Switch to Switch or
Host to Host

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 331
But What About
Fabricpath or VLAN
“Fabrics”?
Fabricpath SDN or Standard VLAN Deployment
Use Fabric Settings and Auto-Configuration Menus

DCNM shows what Fabric Types are available

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 333
Multiple Fabrics Architectures via One DCNM

DCI

DCNM gives you a consistent operations experience and single pane of glass

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 334
VXLAN / Fabric Recap...
• Multiple Fabrics on the same pane of glass
• Best Practice Templates for Underlay Provisioning
• Easy Overlay Deployment
• Manage Classic Configurations and new Fabrics
• VPC / STP networks
• Fabricpath

• View VXLAN and Fabricpath on the Topology [Search Details]

• VXLAN-OAM shows fabric reachability
• Endpoint Locator tracks VM lifecycle – “Where’s my VM”?

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 335
Cool Features for LAN
Fabrics

(Let’s Explore)
Features in DCNM

Top-Down Provisioning

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 337
Topology Views

Real-Time
Search

Detected VTEP

Health Score
(color)
Link Pop-Up Pop-Up Switch
Dashboard

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 338
 Topology Views- VMM Integration

Display Host Details

Display Connectivity
Details

Display connected
Physical Hosts

Display Port-Group
Display DVS/Vswitch
Details

Display VMs

Filter by VMM

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 339
 Endpoint Locator
• How many hosts on vlan 10 on eth1/1
on Leaf10 at 11/01/2017 between 2am -
3am?
• How many networks and VRFs are
active on leafs 1-10?
• Network activity heat-map

Graphical view of host life-

cycle on the network

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 340
Switch Dashboard Interfaces
Interface Page Programmable Show Commands

Interface History

Controls Configuration by Policy [Micro Template]

Policy
Add I/F Edit shut / no shut Show
History

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 341
Template Library

VXLAN Policy Micro-

POAP General CLI Show [cli]
Profile Templates

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 342
Monitoring

Fabric Link details

Monitor Access
Ports and VPCs

Stack Traces
and show near-real-
time sample

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 343
Built-In REST API-Docs Using Swagger
https://<dcnm-server-IP>/api-docs
DCNM GUI uses
REST API
Inspect with
Browser Tools
[e.g. . Google
Developer Tools]

Automate

DCNM REST APIs for Automation are built-in

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 344
Exploring .. Summary Dashboards

Add & remove Dashlets

on demand

Customize for your

environment

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 345
 Switch

Exploring Topology Pop-Over

• Dynamic Arrangement
• Multi-Fabric/Overlay
• Arrange by Tier Activate Beacon
• [Core, Ag, Access Leaf, Spine etc..]
Switch Color
• Metadata Tags
Shows Health
• Show FEX links
Metadata Tags
• Device Pop-Over System & User-Defined

• Side-By Side View

Switch Details
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Exploring Topology Side-By-Side Views

• Dynamic Arrangement
• Multi-Fabric/Overlay
• Arrange by Tier
• [Core, Ag, Access
Leaf, Spine etc..]
• Metadata Tags
• Show FEX links
• Device Pop-Over
• Side-By Side View

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 347
 Azeem Suleman
Principal Engineer
Application Centric
Infrastructure (ACI)
ACI: An Innovative Approach to Policy Based
Segmentation
Provided Provided Provided
Contract Contract Contract

F/W WEB ADC APP DB

OUTSIDE
ADC

What is an application network policy?

1. Group: A set of virtual or physical workloads with the same policy

2. Contracts: A set of rules governing communication between groups

3. Service Chains: A set of network services between groups

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 349
 Application Policy Infrastructure Controller (APIC)

Root

Policy Virtual
Infra Fabric
Universe Network

Tenants VLANs Nodes Hypervisors

Applications

View and Manage the Fabric as a System

• Enables Shared Visibility
Data Base Enabled and Defined Networking by Logical Views
• Enables Shared Context to Simplify Operations

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 350
ACI: The Elements
APICs L-Size (Recommended VMware VMs
3 Recommended for Production Physical for 1000+ physical leaf ports) Virtual
(Recommended for 2-4 leaves,
2 VMs + 1 Physical APIC)
At least 1 physical APIC required M-size (Recommended
for <1000 physical leaf ports)

Virtual
SPINES Modular Fixed Nexus vPod
9300 (vSpine)
Nexus 9500 (9332C, 9364C)
(w/9700 LCs)

LEAVES Virtual
Virtual/Container networking integration Fixed Nexus 9300 vPod
included (except vPod mode) (100M/1/10/25/40/50/100/400G) (vLeaf)

Premier Insights &

Assurance Add-ons
vPod
LICENSING Advantage Multisite
Remote-Leaf Mgmt Cluster +
Per AVE License
FC
FCoE
Storage Encryption
Essentials

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 351
 The DC network before The DC network NOW
Classic modular switching
ACI

Supervisors (1 or 2)
APICs
(1,3 or more)
Fabric Modules (3- 5)
Up to 18 RUs Scale-up

SPINE
(2 to 6)
Linecards (Copper, Fiber,1/10G)
Zero-touch VXLAN
No STP

LEAVES
(1 to 200 or more*) Scale as you need
Single VXLAN Network**
Evolution from Nexus 5000 and Nexus 7000

Single chassis (e.g. Nexus 7000) * > 200 Leaves with MultiPod/Multi-Site
** Other topologies available (e.g. 3-tier, etc.)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 352
Cisco, as Open as • Standard APIs across portfolio
You Want it to Be • Programmable hardware
• Large ecosystem of partners
• Extensible for homegrown tools

Be Open BYO Automation

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Simplify the Entire IT • Operational simplicity
Operations Lifecycle • From build to ongoing support
• Integrated automation
• Policy based management
Fully • Open API’s
Packaged
Systems Off The Shelf

ACI

Be Open

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
 ACI: How difficult is it to bring it up?
Let’s start with a single site

LLDP
Exchange
Spine Layer Nexus 9000
1 Connect all leaves to spines
Connect APIC(s) to any leaf or leaves

• ISIS protocol is run on links

TEP through
DHCP between spines / leaves
Leaf Layer Nexus 9000
ISIS Protocol
Adjacency

Certificate

2
Validation
Console into to each of the APICs
DME Start
Follow the initial configuration wizard

• Spine – OOB, Inband management and 1 console per Sup for 95xx
• Leaf – OOB, Inband management and 1 console
• APIC – CIMC and dual home connection, standby APIC (if possible)
• Fabric Name, Fabric ID, Infra TEP Pool /22, Infra VLAN(3967), BD Multicast Range, NTP, AAA
• Export backups / snapshots periodically
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 355
ACI: How difficult is it to bring it up?
What tasks & configuration did ACI just saved me from doing manually on every switch
BEFORE NOW
External to Internal Route
redistribution (MBGP)
Multicast and Control Plane
(MBGP)

Overlay Network (VXLAN)

Underlay Routed Network (IS-IS)

Switch management (Inband or

Out-of-Band options)
SSH to every switch, Assign IP Address, Enable
Telnet/SSH, Add users on every switch/Create ACLs ACI Automated tasks
(optional)
From HOURS to seconds!
(Times X Switches & Y VNIs)
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 356
Key Concepts &
Recommendations
What is Tenant

Tenant A Tenant B
A Tenant is a container for all
network, security,
troubleshooting and L4 – 7
service policies.

Tenant resources are isolated

from each other, allowing
management by different
administrators.

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 358
Where VRF is defined

Tenant A Tenant B
VRF 1 VRF 1 VRFs (contexts) are defined
within a tenant to allow
isolated and potentially
overlapping IP address
space.
VRF 2 VRF 2

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 359
What is Bridge Domain (BD)

Tenant A Tenant B Within a VRF (Context), one

VRF 1 VRF 1
or more bridge domains must
be defined.
Bridge Domain 1 Bridge Domain 1

Bridge Domain 2 Bridge Domain 2 A bridge domain is a L2

forwarding entity within the
VRF 2 VRF 2 fabric, used to define L2
Bridge Domain 3 Bridge Domain 3
forwarding domain and to
constrain broadcast and
Bridge Domain 4 Bridge Domain 4
multicast traffic.

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 360
Bridge Domain
Forwarding L2 unknown unicast based on spine-proxy mapping database
or flood and learn over VXLAN

THIS DOESN’T CONTROL THE BEHAVIOR OF L3 traffic, i.e. it doesn’t

control the forwarding of L3 "unknown" unicast

THIS doesn’t turn on or off the mapping database for MAC addresses.
MAC addresses are always learned in the mapping database

L3 Multicast (IANA Range). Known multicast traffic will have IGMP/MLD

snooping entry and forwarded to appropriate ports
Unknown multicast will get FLOOD to ports in BD or
In optimize flood case send only to router ports detected by PIM hellos

This option is only relevant if you do hardware-proxy forwarding and if “Unicast routing” is
enabled. ARP packets are flooded in the BD

Learn only IP addresses from configured subnets – It’s disruptive

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 361
 Bridge Domain Recommendations
Enforce
Subnet
L2 Unknown ARP Unicast Subnet
Scenario Check for
Unicast Flooding Routing Configured
IP
Learning
IP Routed Traffic. No FW + LB, No Floating IP Hardware Proxy Disabled Enabled Yes (if Yes
required. No Silent Hosts required)
IP Routed Traffic. No FW + LB. Silent Hosts. Hardware Proxy Disabled Enabled Yes Yes
Non IP, switched traffic. Silent Hosts. Flood N/A Disabled No N/A
Hosts with IP address may float between Hardware Proxy Enabled Enabled Yes Yes
MAC. FW + LB. NIC Teaming
Migration – Extending L2 from ACI with L3 Hardware Proxy Enabled Enabled If required If required
GW still on legacy network

• Start with HW-Proxy ON unless if deemed necessary

• ARP Flooding on for L4-7 devices failover process
• Data plane learning on (except for Service Redirect)
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 362
 Reaches

L2 Forwarding
Forward to
Yes Does spine knows No
Remote leaf Drop
remote Leaf Dst Mac
that floods it

Summary
Forward to Forward to Flood within
local port remote leaf BD Spine Proxy

Flood Hardware Proxy

Is Dst MAC on What is BD config?

Local Leaf? (L2 Unknown Uni)
Yes No

Does Leaf know

Dst MAC?
Yes No See later pres

L2 or L3 ?
L2

Packet coming in to Leaf

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI: What changes?
Easy as 1-2-3-4-5
BEFORE NOW
1
Physical Networks/VRFs Create Tenants
We would purchase separate networks and assign You can ”partition” your ACI Fabric & have up to 3000 Tenants even
different IP subnets to each (Prod, Test, etc.) using the same IP subnets with no conflict

Tenant Test Tenant Prod

1.1.1.0 1.1.1.0

Test Production
2.2.2.0 IP Change 1.1.1.0

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 364
What is End Point Group (EPG)

Tenant A Tenant B
EPGs exist within a single
VRF 1 VRF 1 bridge domain only – they do
Bridge Domain 1 EPG Bridge Domain 1 EPG not span bridge domains.
Bridge Domain 2 EPG Bridge Domain 2 EPG
EPGs defines the policy
enforcement entities/classes.
VRF 2 VRF 2
Class-based policies are
Bridge Domain 3 EPG EPG Bridge Domain 3 EPG EPG
applied between EPGs
Bridge Domain 4 EPG Bridge Domain 4 EPG

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 365
Application Policy Logical Construct
Tenant

VRF 1 VRF 2

Network
Bridge Domain 172 Bridge Domain 10 Bridge Domain 100
Subnet 172.1.1.0/24
Subnet 10.1.1.0/24 Subnet 10.1.1.0/24
Subnet 172.1.2.0/24 Subnet 10.1.2.0/24
… …
Subnet 172.20.1.0/24

EPG web
EPG WEB EPG DB Policy “HTTP”

App
Policy “HTTP” EPG db
Policy “SQL”
EPG APP Policy “SQL”
EPG app

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 366
Application Policy Logical Construct
Mapping the Configuration to the Packet
Coke-Tenant
• ACI Fabric leverages VXLAN Encapsulation to build VRF 1
network overlay
Bridge Domain 1 EPG
• VXLAN Source Group is used as a tag/label to identify the
specific end point for each application function (EPG) Bridge Domain 2 EPG

• Policy is enforced between an ingress or source application

tier (EPG) and an egress or destination application tier
(EPG) VRF 2
• Policy can be enforced at source or destination Bridge Domain 3 EPG

Bridge Domain 4 EPG

VXLAN Header:
Flags Flags/DRE Source Class ID == EPG VNID == BD/VRF M/LB/SP

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 367
End Point Group (EPG) Definition
• An Endpoint group (EPG) is a set of devices (end points) that share the
same policy requirements.
• Classification can be based on:
Application
• VLAN Profile
• VxLAN
• MAC Address EPG EPG
• IP Address
• VM Properties etc.
EP EP EP EP

Virtual Port, Physical Ports, External L2 VLAN, External L3 subnet

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 368
End Points (EPs)
• EPs are devices which attach to the network either virtually or physically, e.g.:
• Virtual Machine
• Physical Server (running Bare Metal or Hypervisor)
• External Layer 2 / 3 device
• Firewall / Load balancer etc. ACI Endpoint
Traditional Endpoint
- MAC or MAC/IP  IPv4 is /32
L2 – MAC Table L3 – ARP Table Route
- MAC Address - IP / MAC - VLAN / VxLAN  EPG (pcTag)
- VLAN - Interface
- Interface - Interface - VRF
- VRF - Flags  Local, vPC, static, etc.

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 369
 Endpoint Classification
Web App DB

Outside QoS QoS QoS

(Tenant
VRF) Filter Service Filter

Classification:
Classification: Classification: Classification:
L3_Out : Network/Mask

• EPG Classification on L3 Outside • EPG Classification on an access/server port is based on different attributes
is based on IP address • Port + VLAN, Port + VXLAN, Network/Mask
Network/Mask
• IP/MAC, VM Attributes for AVS attached VM’s
• IP & MAC Host Address

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 370
 Understanding Networks and Groups Abstractions
Legend
Tenant Child/Parent Object
(fvTenant) Relationship (Pointer)

Application Bridge
Outside VRF Contract
Profile Domain
(fvCtx) (vzBrCP)
Filter
Network (fvAp) (fvBD)

Subnet
(fvSubnet)
Subject
Endpoint
Group
(fvAEPg)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 371
By default …

endpoints in different EPGs can NOT

communicate at all

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 372
By default …

endpoints inside an EPG can

communicate freely

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 373
Intra EPG default can be changed

… today, to block intra-EPG communication

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 374
Every EPG belongs to a VRF and

an Application Network Profile

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 375
 Application Network Profile
A group of EPGs related to each other to represent an application

Health scores, statistics, logs

and audit data automatically
correlated and rolled up at
Application Profile level

EPG, uEPG, domain associations,

contract relations and L4-7
Configuration

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 376
 ACI: What changes?
Easy as 1-2-3-4-5
BEFORE NOW
Unclear network connectivity 2
Create Application Profiles
Show VLAN would show all and every VLAN per-Switch An Application Profile is a graphical representation of our network
without understanding how they connect between each configuration. Think of it as a “folder of VLANs” at the Fabric level.
other A Tenant may have multiple Application Profiles

Application Profile Learning App

Switch 1 Switch 5

Switch 2 Switch 6

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
 ACI: What changes?
Easy as 1-2-3-4-5
BEFORE NOW
Create VLANs per Switch 3
Create End Point Groups (EPGs)
Add VLANs per Switch, name each of them and then configure We will create an EPG and name it just as we would with a VLAN. You
trunks to extend connectivity. Additionally configure HSRP/VRRP may also add one Bridge Domain per EPG with an IP address (just like
for Gateways at the distribution/core layer an SVI) in case you want ACI Anycast Gateway functionality

Switch(config)#feature hsrp
Switch(config)#interface vlan 1
Switch(config-if)#ip address 1.1.1.253 255.255.255.0
Switch(config-if)#no shut
Collapsed HSRP/VRRP Switch(config-if)#hsrp 1 Spine Layer
Core Switch(config-hsrp)#ip 1.1.1.1 255.255.255.0 Tenant Production
Gateways Switch(config-hsrp)#priority 100 Application Profile SAP
Switch(config-hsrp)#preempt
Switch(config-hsrp)#no shut VXLAN
802.1q Switch(config)#interface vlan 2
Switch(config-if)#ip address 2.2.2.253 255.255.255.0
Anycast GW Leaf Layer
Switch(config-if)#no shut
Switch(config-if)#hsrp 2
Access Layer Switch(config-hsrp)#ip 2.2.2.1 255.255.255.0
Switch(config-hsrp)#priority 100
Switch(config)#vlan 1
Switch(config-hsrp)#preempt
Switch(config-hsrp)#no shut
BD 1.1.1.1 BD 2.2.2.1
Switch(config-vlan)#name Netweaver
Switch(config)#vlan 2
Switch(config-vlan)#name HANA
EPG EPG
Switch(config)#int e1/1
Switch(config-if)#switchport mode trunk
Netweaver HANA
Switch(config-if)#switchport trunk allowed vlan 1-2

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 378
Contracts
• Contracts are semantics to specify EPG to EPG communication in ACI
• Communication policy includes filters (ACLs), QoS and Service Graphs
• Contract filters are similar to Access Control Lists
• Contracts can be defined between EPGs or between L3out External EPGs and regular
EPGs
Contract - MyContract

Subject

Web-Prod Filters DB-Prod

QoS

Service
Graph

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 379
EPGs Provide and/or Consume Contracts

• EPGs will have associations to provide

and/or to consume a contract
• An EPG can provide and/or consume
multiple contracts.
• Contracts can be used between EPGs in the
A contract between VRFs is
same Application Profile, across Application
required to enable route-leaking.
Profiles, VRFs and even tenants.
• Contracts also define route-leaking
between VRFs

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 380
Contract Scope Defines Where They Will Be
Applied

• Contract “scope” limits the type of relations between

EPGs.

• Application Profile:
• Contract is applied between EPGs if they are of the
same AP

• VRF:
• Contract applied between EPGs if they are part of the
same VRF.

• Tenant:
• Contract applied if EPGs are in the same tenant, even
if different VRF

• Global:
• Contract can be exported, and is applied even if EPGs
may be part of different tenants

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 381
Contract Filters Define L2-4 Traffic
Filters can be re-used by many contracts

A filter may have multiple entries, each matching

specific protocols, ports or port ranges.

Filter definition allows all parameters for L2-4 filtering

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 382
Contract Filters Define L2-4 Traffic (contd.)

The Established flag is designed to allow

TCP traffic of existing connections: ACK or
RST

The stateful option can be used with AVE in order to

create a reflexive ACL to allow a specific TCP
connection reverse traffic

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 383
Configure Contracts for all EPG in a VRF (vzAny)
• vzAny represents the collection of EPGs that Tenant
belong to the same VRF, including L3 VRF1
external. BD1

• Instead of associating contracts to each EPG1

individual EPG you can configure a contract vzAny
to the vzAny BD2

EPG2
• With cross-VRF contracts, vzAny can be a
consumer, not provider
Tenant Shared Tenant Shared
Tenant ONE Tenant ONE
Services Services

EPG1
EPG1 VRF1 VRF Services
VRF1 VRF Services
EPG shared
EPG shared
vzAny service
vzAny service EPG2
EPG2

NOT “SUPPORTED”
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 384
ACI: What changes?
Easy as 1-2-3-4-5
BEFORE NOW
4
Create Contracts
Create ACLs per Switch/Port We will create a Contract to specify how 2 EPGs may talk between
Specify the type of traffic you want each switch to allow each other. This contract will be pushed to the whole fabric (physical,
virtual, etc.) consistently. NO complex IP + Ports to specify like ACLs

HSRP/VRRP Collapsed Core Spine Layer

Gateways
Contract SAP_POLICY
VXLAN
802.1q Filters
permit icmp Leaf Layer
Anycast GW
permit tcp eq 80
Access Layer (Bidirectional)

Switch(config)#ip access-list extended name SAP_POLICY BD 1.1.1.1 BD 2.2.2.1

Switch(config-acl)#10 permit icmp any any
Switch(config-acl)#20 permit tcp any any eq 80
Switch(config-acl)#30 permit tcp any eq 80 any
EPG ICMP
EPG
Switch(config)#int e1/1
Netweaver HANA
Switch(config-if)#ip access-group SAP_POLICY in FCoE
Switch(config-if)#ip access-group SAP_POLICY out

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 385
 Types of Fabric Routes Ensure BGP RR
is configured to
enable MP-BGP
MP-BGP
overlay-1

L3Out-1 L3Out-2

E1 E2 ext
ext
1 BD-B1 BD-B2 2
subnet subnet subnet subnet
ext-S1 int-S1 int-S2 ext-S2

• Internal Routes: Subnets defined under BD are internal routes and create static pervasive routes
within the fabric.
• External Routes: Routes learned via a routing protocol or static routes configured under an L3Out.
These routes are redistributed into MP-BGP and advertise to all leafs that contain the VRF
• Transit Routes – Routes advertised between L3Outs.

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 386
 Types of Fabric Routes – Internal Routes
Subnet: int-S2 3 Subnet int-S2 installed on border leaf
Scope: when creating contract
MP-BGP between EPG
 Private to VRF E2 and external overlay-1
EPG ext2
 Advertise Externally
 Share Between VRFs
L3Out-1 L3Out-2
1

E1 E2 ext
ext
1 BD-B1 BD-B2 2 2
subnet subnet subnet C1 subnet
ext-S1 int-S1 int-S2 ext-S2

There are three requirements to advertise Internal Routes out an L3Out:

1. The BD must be associated with the L3Out*
The association adds prefix entry to route map controlling advertised routes
2. A contract must exists between an EPG within the BD and an external EPG on the L3Out.
The contract creates internal BD route on border leaf (cannot advertise route until it exists locally)
3. The subnet must have a public scope (Advertise Externally)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 387
 Types of Fabric Routes – External Routes
ext-S1
ospf, eigrp, static route RD: L1:V1
redistributed into bgp and RT: ASN:V1 MP-BGP
exported into mp-bgp
overlay-1
ext-S1

L3Out-1 L3Out-2

VRF-V1 VRF-V1 VRF-V1

ext import RT from mp- ext
ext-S1 ext-S1
1 bgp and install route 2
subnet via:L1(bgp) into vrf as bgp learn via:L1(bgp) subnet
ext-S1 ext-S2
• External Routes from ospf, eigrp, or static are redistributed on the border leaf into the local bgp process.
• The bgp route is exported into MP-BGP with a route-target (RT) of the corresponding VRF. Each leaf in
the fabric with the VRF present will import the RT and install the route. External routes on the non -
originating border leaf will be seen as bgp learned routes.
• External Routes are controlled via Import Route Control flag

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 388
 Types of Fabric Routes – Transit Routes

MP-BGP
overlay-1
ext-S1
ext-S1

L3Out-1 L3Out-2

VRF-V1 VRF-V1 VRF-V1

ext ext
ext-S1 ext-S1
1 2
subnet via:L1(bgp) via:L1(bgp) subnet
ext-S1 ext-S2
• In this example, external route ext-S1 is a Transit Route when advertised out L3Out-2.
• If OSPF or EIGRP on L3Out-2, ext-S1 is redistributed from BGP into the IGP and advertised.
• Transit Routes are controlled via Export Route Control flag

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 389
L3 External Subnet Review
• External Subnets for the External EPG (Security Import)
Used to classify dataplane packets into external EPG for policy enforcement

• Export Route Control - filter Transit Routes advertised out of the fabric.

• Import Route Control - filter External Routes received on an L3Out

• Shared Security Import

used to classify dataplane packets into external EPG for policy enforcement for shared/leaked prefixes

• Shared Route Control

Allows external route to be leaked into another VRF

• Aggregate Export - allows prefixes to be aggregated together in export direction (0/0 or ::/0 only)
• Aggregate Import - allows prefixes to be aggregated together in import direction (0/0 or ::/0 only)
• Aggregate Shared Route - allows prefixes to be aggregated together for shared route control

Cisco APIC and Transit Routing

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_KB_Transit_Routing.html

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 390
ACI: What changes?
Easy as 1-2-3-4-5
BEFORE NOW
5
Configure IP Routing Create L3 Out
Configure the routing protocol you may need on each Specify on which leaf and port of the fabric you want to enable external
switch/router to learn routes coming from the outside routing. Those routes will be imported inside the ACI Fabric with BGP (auto-
configured) and Spines will serve as Route Reflectors. L3 Outs need a
contract to communicate to EPGs and BDs need to be associated to L3 Outs
OSPF Router

HSRP/VRRP Collapsed Core

Spine Layer
L3Out Internet
Gateways BGP RRs
Leaf 1 Int 1/15
ospf area 0
802.1q network p2p
mtu ignore Leaf 1
Leaf Layer
IP 221.221.221.2/24 1/15 E1/15 221.221.221.2/24

Access Layer Contract Internet (EPGL3Out)

permit any (bidirectional) OSPF L3 Out
221.221.221.1
Switch(config)#router ospf 1
Switch(config)#interface e1/1 BD 1.1.1.1 Router
Switch(config)#ip address 221.221.221.2 255.255.255.0
Switch(config-if)#ip ospf network point-to-point
Switch(config-if)#ip router ospf 1 area 0
EPG
Switch(config-if)#ip ospf mtu ignore
Netweaver
Internet

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 391
Global Settings
Best Practices Summary

• Disable Remote EP Learn

• Prior 3.0: Fabric  Access Policies  Global Policies  Fabric Wide Setting Policy
• After 3.0: System  System Settings  Fabric Wide Setting

• Enforce Subnet Check (will only work EX and FX based leafs)

• Prior 3.0: Fabric  Access Policies  Global Policies  Fabric Wide Setting Policy
• After 3.0: System  System Settings  Fabric Wide Setting

• IP Aging should be enabled

• Prior 3.0: Fabric  Access Policies  Global Policies  IP Aging Policy
• After 3.0: System  System Settings  Endpoint Control  IP Aging

• MCP (per VLAN) should be enabled

• Prior 3.0: Fabric  Access Policies  Global Policies  MCP Instance Policy default
• After 3.0: Fabric  External Access Policies  Policies  Global  MCP Instance Policy default

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 392
 ACI Fabric Endpoint Learning Evolution

Endpoint Learning Optimization Options

Unicast GARP-based Limit IP L4-L7 Endpoint Disable Remote Enforce

IP Aging
Routing EP Move Learning Virtual Dataplane EP Learn Subnet
Detection To Subnet IPs Learning (on border leaf) Check

• Avoid using default policy if possible

• Create multiple separate policy so easier to make changes
• Make sure proper naming conventions are used
ACI Fabric Endpoint Learning Whitepaper
https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739989.html

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 393
 Forwarding Flow
Drop and ARP
Forward to Does spine knows glean
remote leaf Dest IP in coop ? For destination
No
Yes IP
Summary
Flooded frame
Forward to
Yes
Does spine knows
No
Drop
remote leaf DMAC in coop ?
reached re mote
leaf
Forward to
Flood within Border Leaf
Forward to Forward to Forward to Forward to Drop
Spine Proxy Spine Proxy Per routing-
local port remote leaf BD (GIPo) local port remote leaf
table

Yes No
Hardware Does Leaf know
Flood
Proxy Dst IP as
L3OUT Routes?

Yes Does Leaf have

No
BD Subnets for Dst IP
Is Dst MAC on What is BD config? Is Dst IP on
(route to Acast Spine)
Local Leaf? (L2 Unknown Uni) Local Leaf?
Yes No Yes No

Does Leaf know

Does Leaf know Dst IP as
Yes Dst MAC? Yes EndPoint?
No No

L2 or L3 ?
L2 L3
(DMAC != ACI MAC) (DMAC == ACI_MAC)

Packet coming to the

Leaf © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
 ACI Policy Model Evolution
MAP EP to EPG

VLAN/VXLAN=EPG IP Based EPG MAC Based EPG VM Attributed based EPG

Contract Creation and Enforcement

Enforced vzAny Intra-EPG Ingress Contract Contract Intra-EPG Blacklist

Or Unforced Isolation Policy Preferred Group Inheritance Contract (3.2)
(2.2) (2.3) (3.0)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 395
 ACI: How do I start?
Easy as 1-2-3-4-5
5) Once all servers are migrated
Your existing network Your new ACI Fabric to the ACI Fabric, you may
remove your old gear
Internet/WAN If you add more leaves or spines,
Contract
VLAN 1  EPG 1 APIC will auto-discover and auto-
VLAN 2  EPG 2
configure them. It is that SIMPLE!
Nexus 7000
(or L2/L3 Boundary)
Nexus 9000
Spine Layer

802.1q
VXLAN
Nexus 5000 Nexus 9000
(or L2 Access/ToR) Anycast GW Leaf Layer

APIC Cluster

EPG 1
EPG 2
VLAN 1 1.1.1.0/24 1 1.1.1.0/24 2.2.2.0/24

Nexus 7K/5K and legacy Simplify & secure Integrate

Non-disruptive At your own pace
networking migration your DC network virtual & cloud

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 396
ACI Software Release
Guideline

Long Lived Releases

1 T wo Lo ng Lived R el eases At Any Given P o int o f T ime

2 Active Maintenance Wil l Be P rimaril y Focused On Long Lived R el ease

3 Target Duration Of Long Lived R el ease Support: Up to 18 Months From FCS

4 Direct Upgrade From One Long Lived To Next Long Lived R el ease Wil l Be Supported

5 Long Lived R el eases Are R ecommended For Networks That Wil l Not be Upgraded Frequentl y

Short Lived Releases

1 No Active Maintenance Beyond Six Months From FC S

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 397
 ACI Software Release Cadence
Major Releases =>

ACI ACI ACI ACI ACI ACI ACI

2.3 3.0 3.1 3.2 4.0 4.1 4.2

Maintenance Releases =>

ACI ACI ACI ACI ACI ACI ACI
2.2(2) 2.3(2) 3.0(2) 3.1(2) 3.2(2) 4.0(2) 4.1(x)

Target

Long Lived ACI ACI ACI

Releases 2.2(x) 3.2(x) 4.2(x)

Target – One Release Every Four Months.

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 398
ACI Anywhere

• Operational Simplicity: Same Containers Hypervisor

“look and feel” as On-

Premise
ACI Anywhere
• Automated Policy
Translation: Consistency
across the entire data center Cloud
Exchange
• Common Governance: Data
Center
End-to-end discovery,
visibility and troubleshooting
On Premises
Cloud
IOT Edge
 Cisco ACI Multi-Site Orchestrator

Cisco Virtual ACI (Virtual Edge)

Cisco ACI Cisco ACI Physical Remote Leaf

Key Components
Cisco Virtual ACI (Virtual Pod)

Cisco Cloud ACI

ACI Anywhere – Accelerate Multicloud
“Evolving our multicloud journey by extending ACI everywhere”

NEW !
ACI Multi-POD ACI Remote-Leaf Cloud ACI
Multiple Networks Physical Remote Leaf ACI Extensions to
(Pods) in a single extends an Availability AWS and Azure
Availability Zone Zone (Fabric) to Public Cloud
(Fabric) ACI 3.0 remote locations ACI 4.0

ACI 2.0 ACI 3.1 ACI 4.1

ACI Multi-Site Virtual ACI
Multiple Availability Virtual POD extends an
Zones (Fabrics) in a Availability Zone (Fabric)
Single Region ’and’ to remote locations on
Multi-Region Policy standard VMs
Management

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 401
 ACI Multi-Site
Multi-Site Orchestrator Site N
(MSO)
3 VM Cluster

VM VM VM VM VM VM VM
Any Routed IP Network

Site1 Site 2

VM VM VM VM VM VM VM VM VM VM VM VM VM VM

No Multicast <= 1s RTT Required (MSO  APIC) Single central management (MSO)
Phased Changes (Zones) Up to 12 Sites, distributed gateway Automated L2 DCI VXLAN extension

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 402
ACI Multi-Site
Software and Hardware Requirements

• Support all ACI leaf switches (1st Gen, EX, FX, FX2) Any Routed Network

• Modular Spine with EX/FX line card to

connect to the inter-site network
Can have only a subset
1st Gen 1st Gen -EX -EX of spines connecting to
• 9364c or 9332x fixed spine supported for the IP network
Multi-Site from ACI 3.1 release (shipping)
• 1st generation spines (including 9336PQ)
not supported
• Can still leverage those for intra-site leaf
to leaf communication

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 403
ACI Multi-Site
Software and Hardware Requirements

• Support all ACI leaf switches (1st Gen, EX, FX, FX2) Any Routed Network

• Modular Spine with EX/FX line card to

connect to the inter-site network
Can have only a subset
1st Gen 1st Gen -EX -EX of spines connecting to
• 9364c or 9332x fixed spine supported for the IP network
Multi-Site from ACI 3.1 release (shipping)
• 1st generation spines (including 9336PQ)
not supported
• Can still leverage those for intra-site leaf
to leaf communication

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 404
ACI Anywhere Shipping

Encrypted DCI Connectivity Multi-Site

IP / WAN
CloudSec

MACSEC MACSEC
Today Future

VM VM VM

Site A Site B Site C

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 405
 Remote Location A
ACI Remote Leaf RL

VM VM VM VM VM VM VM
Any Routed IP Network
Satellite DC

Remote Location B
RL

Pod 1 VM

VM VM VM VM VM VM VM
Brownfield

Remote Location C
RL

VM VM VM VM VM VM VM
Telco/Co-lo
VM VM VM VM VM VM VM

Zero Touch Auto <= 300 ms RTT Required Single central management
Discovery of Remote Leaf Up to 20 Remote Locations Automated L2 VXLAN extension

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 406
ACI Remote Leaf Requirements
Hardware & Software

ACI Main DC Remote Location

Supported Spines Supported Leaf
• N93180YC-EX
Fixed Spine • N93108TC-EX
• N9364C • N93180LC-EX
• N9332C (ACI 4.0) • N93180YC-FX
Modular Spine (C9504/C9508/C9516) • N93108TC-FX
• N9732C-EX • N9348GC-FXP
• N9736C-FX • N9336C-FX2

All hardware from –EX onwards is supported

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 407
 ACI Anywhere: ACI Virtual Edge
Decoupled From Hypervisor Kernel APIs
Multi-Site Orchestrator

Data Center 1
Data Center 2
(ACI Site 1) IPN
IP Network
(ACI Site 2)

Nexus 9000 Nexus 9000

(DC Network) VXLAN (DC Network)
L2 Extension

WAN
Local Router

VM
ACI Virtual Edge VM

Nexus 9000

VM VM VM VM VM VM VM (Remote Leaf Network)

Policy Consistency Across Enable Migration From Maintain Existing Operational

Multiple Hypervisros Legacy To ACI Models

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 408
 ACI Anywhere: ACI Virtual Pod
Extend ACI To Bare-metal Clouds, Remote Data Centers, and Legacy Infrastructure
Multi-Site Orchestrator

Data Center 1
Data Center 2
(ACI Site 1) IPN
IP Network
(ACI Site 2)

Nexus 9000 Nexus 9000

(DC Network) VXLAN (DC Network)
L2 Extension

ACI Virtual Pod 1 ACI Virtual Pod 2

WAN
Local Router
Pod 1 Pod 2
VM
ACI Virtual Edge VM

ACI Virtual Edge ACI Virtual Edge

Nexus 9000

VM VM VM VM VM VM VM
VM VM VM VM VM VM VM (Remote Leaf Network)

Virtual Spine/Leaf Functionality Single central management

Up to 64 AVEs per vPod
w/AVE integration Automated L2 VXLAN extension
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 409
ACI vPod Requirements
Hardware & Software

On-Premises Data Center vPoD Data Center

Supported Spines • VMware vCenter running 6.0 or later
Fixed Spine • 2 hosts for Management cluster
• N9364C
• N9332C • Management cluster may exist on the
same AVE ESXi nodes
Modular Spine (C9504/C9508/C9516)
• N9732C-EX with N9K-C950x-FM-E(2) • ESXi 6.0 or 6.5
• Each vSpine (x2) & vLeaf(x2) VM consumes 4vCPU,
• N9736C-FX with N9K-C950x-FM-E(2) 16 GB RAM and 80 GB storage
APIC Controller Software • Each AVE (one per ESXi host) VM consumes
2vCPU, 8 GB RAM and 8 GB storage
• ACI 4.0+ onward release

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 410
Challenges in building a Multi Cloud environment

Build an automated and Maintain consistent policy, Provide a single pane of

secure interconnect between security and analytics for glass to manage policies
on-Premises and Cloud workloads deployed across across on-premise and cloud
datacenters with ease of on-premises and cloud locations
provisioning and monitoring locations
at scale

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 411
 ACI Anywhere: Public Cloud Extensions
Seamlessly Connect Multiple Data Centers
Multi-Site Orchestrator

Data Center 1
Infra VPC
(ACI Site 1) IPN
IP Network

Nexus 9000
(DC Network) VXLAN User VPC User VPC
Encrypted L2 Extension

Encrypted
WAN
Local Router

VM EPG
Web ACI Virtual
Contract
APPEdge
EPG
Contract VMEPG
DB
SG
Web
SG Rule
SG
APP
SG Rule
SG
DB

Nexus 9000
VM VM VM VM VM VM VM

VM VM VM VM VM VM VM (Remote Leaf Network)

Discovery Policy CSR-1Kv/Direct- Single Point Operational

& Visibility Translation Connect integration Of Orchestration Consistency
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 412
Virtual Private Network (VPN)
Multisite Orchestrator

Public Cloud Site B

Site A On-Premise
User VPC-1
IPSec VPN Tunnel (Underlay)
VGW
CSR1000V
BGP-EVPN Session (Control Plane)

VXLAN Tunnel (Data Plane)

Customer
Premise AWS
AWS Instances
Internet
Router Internet
Gateway

Infra VPC VGW

VM VM VM AWS Instances

User VPC-2
AWS Region

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 413
 For your info
& reference

Policy Mapping - AWS

User Account Tenant
Virtual Private Cloud VRF

VPC subnet BD Subnet

Tag / Label EP to EPG Mapping

Security Group EPG

Network Access List Taboo
Security Group Rule Contracts, Filters
Outbound rule Consumed contracts
Source/Destination: Subnet or IP or Any or ‘Internet’
Protocol
Port
Inbound rule Provided contracts
EC2 Instance

Network Adapter End Point (fvCEp)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 414
 For your info
& reference

Policy Mapping - Azure

Resource Group Tenant
Virtual Network VRF
Subnet BD Subnet

Application Security Group EPG

(ASG)

Network Security Group

(NSG) Filters

Outbound rule Consumed contracts

Source/Destination: ASG or Subnet or IP or Any or ‘Internet’
Protocol
Port

Inbound rule Provided contracts

Virtual Machine

Network Adapter

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 415
 For your info
& reference

Cloud Hierarchy

AWS Azure GCP

OU Organization Organization
Account subscription Project
Region Region VPC
VPC VNet Region
AZ Availability sets/zones Subnet
Subnet Subnet Zone

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 416
 For your info
& reference

Virtual Networking Comparison

AWS Azure GCP
VPC is regional VNet is regional VPC is global
Dedicated HW option Some large mem VM Sole tenant
IPv4/IPv6 IPv4/IPv6 via LB IPv4/ IPv6 proxy
Internet access by Internet access by
Public private subnets
default default
CIDR can’t change/new Add, remove, expand or
CIDR can expand only
blocks can be added shrink
Subnet in AZ Subnets are regional Subnets are regional

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 417
Use case #1: Hybrid-Cloud Deployment

Multi-Site Orchestrator

Region(s) On-Premises Region(s)

Hybrid Cloud supported with AWS in Q1-CY19 and Azure in Q2-CY19

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 418
Use case #2: Cloud First with Multiple Regions

Cloud APIC

US-West Region London Region Seoul Region

One ACI Policy Domain with Multiple AWS Regions

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 419
Use case #3: Multi-Cloud

Multi-Site Orchestrator

Region(s) Region(s) Region(s)

Multi-Cloud with AWS and Azure Cloud Sites supported in 2H-CY19

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 420
Operations
 APIC Management Information Model Reference

From APIC GUI

https://apic/doc/html/
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 422
 Visore – Web Based MO Query and Browser Tool
https://<IP>/visore.html

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 423
Network Monitoring and Troubleshooting Tools

Physical Network Abstracted Network

• ping • properties (EP / TEP / contract)
• health scores / faults / events / audit
• traceroute
• iping, itraceroute
• show (interface / table / etc)
• atomic counters
• syslog • statistics
• SPAN • diagnostics (on-demand)
• SPAN
• ELAM

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 424
Capacity Dashboard

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 425
Configuration Rollback

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 426
Endpoint Tracker

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 427
Traffic Map
Help visualize and quickly spot high traffic density and underutilized
nodes in the Cisco ACI™ fabric.

A grid is presented with a list of node IDs or vPC pairs on each axis.
Traffic flow between a given pair of nodes or between a vPC pair is
presented using color-coded cells on the heat map.

Traffic density is presented in a range of colors, from lightest

(yellow), to shades of orange, to red (highest). Traffic statistics are
collected using atomic counters.

• You can order by name or by traffic.

• Traffic can be seen by:
- Sent packets
- Received packets
- Dropped packets
- Excess packets

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 428
What is Ftriage
ACI Debugging

• Fabric triaging tool. Python utility, runs on APIC in admin mode

• Logs into switch nodes to capture requested data with
commands/query/ELAM
• Driven by specific user inputs which are validated first
• Runs Elam to determine packet data path
• Traces packet hop by hop till the point where it exits the fabric or gets
dropped
• Drop reason is provided along the node, interface info

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 429
 3.1
Tabular Views To See Individual
Faults
Port / Interface State
Changes

Fault List
Link Flapping Table

3.2
User Can
Improved Change
Folder Top GUI
Structure For
Header In Order To Admin  Security Management  GUI Alias
Fabric / Access Policies And
Distinguish L4-L7
Between Multiple
Apics
Fabric  Fabric Fabric  External Access
Fabric  Fabric
APIC Alias Fabric  Access
Policies
Policies
TECDCN-2002
Policies
Policies © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 430
 3.1

Simplified Workflows To
Guide Users Step By Step
In A Single View

Infra Workflows
Configuration

3.2

Users Can Copy / Paste Port

Configuration Within Leafs

Fabric  Pod  Leaf  Interface  Mode

(Configuration)
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 431
 3.1

Show Status Of The Interface

As Well As Statistics

Operations Operational Status & Stats

3.2

Customize Topology By Zone

Topology by Zone
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 432
Usability Enhancements 4.1
Unified Reskin

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 433
Usability Enhancements 4.1
Alert List

• Alert to detect if OSPF connectivity

is down (MPoD) configuration

• Alert to detect process crash and

acknowledge old crashes
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 434
Usability Enhancements 4.1
Interface Status & Favorite Tab

• This status is shown only for physical interfaces prior to 4.1,

now all other interfaces like VPC / PC will have the status too

• User can mark any tab as their favorite tab and they will be navigated to that tab
every time the policy load

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 435
ACI 4.2: Usability Enhancements 4.2
Simplify L3Out in 3 Steps

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco ACI App Center Apps
Programmable Infrastructure: Open API’s for Value Added Applications

https://aciappcenter.cisco.com/

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 437
Insights & Assurance
Network Insights & Network Assurance
OpStack
Architecture and Planning
NAE Policy Explorer
• Network Policy exploration
• Ad-hoc connectivity and segmentation
discovery

Network Operations
Network Assurance Engine (NAE)
• Policy/ Control/Data plane Assurance
• Incident and Problem Management
• Compliance and Audit

Network Insights * Network Administration and Maintenance

• Fabric Health monitoring
• Fabric wide resource monitoring
• Anomaly detection

* Available for both ACI and Nx9K fabrics

NAE Policy Explorer (NAE PE)
Easy Search Function
using Network &
Application Semantics
(VRF, Leaf – EP, EPG)

Natural Language Query

(What, Can, Through, ..)

App on APIC
Note: NAE PE capability
will also be available in
NAE itself in the future

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 440
Network Assurance Engine: How It Works

Data Comprehensive Intelligent

Collection Network Modeling Analysis
Captures all non-packet data: Mathematically accurate models 5000+ domain knowledge-based
intent, policy, state across spanning underlay, overlay and error scenarios built-in, codified
data center network virtualization layers remediation steps

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 441
User Interface: Centered Around “Smart Events”
Incidence and
Change Compliance and
Problem
Management Visualization
Management

Smart Events: What, Where, Why, and How

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 442
Network Insights Applications
Providing Network Health Visibility & Enabling Proactive Insights

New Apps

Network Availability Network Health

Network Insights Advisor Network Insights Resources

NIA NIR

Proactive Software Recommendations/Notifications Physical/Logical Network Capacity & Utilization

Issue Vulnerability Detection & Remediation Data & Control Plane & Environmental Health

Enhance Availability, Uptime & Network Wide Visibility © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 443
Network Insights Applications
Apps
DCNM APIC

Platform
App Hosting Framework App Hosting Framework
App Store App Store

Data collection and ingestion Data correlation and analysis Data visualization and action

Visibility Insights Proactive Troubleshooting

Learn from your network and See problems before Find root cause faster with
recognize anomalies your end users do granular details

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 444
Network Insights Resources
Understand What’s Running In Your Network

System and Resources

Network
Event Analytics
Insights
Resources

Flow Analytics

Deep Insights Into Network Health

(Control Plane, Data Plane, Capacity, Utilization and Environmental Health)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 445
Network Insights Resources
Understand What’s Running In Your Network
Resource
Event Analytics Dashboard Analytics

Data Collection

Anomaly
Detection

Remediation

Event Analytics Dashboard Displays Faults, Events, And Audit Logs In A Time Series Fashion.

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 446
Network Insights Resources
Understand What’s Running In Your Network

Flow Anomalies Flow Analytics Dashboard

Packet Drops

Latency

End Point Move

Flow Analytics Dashboard Displays Key Indicators Of Infrastructure Data Plane Health.

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 447
Network Insights Advisor
Before Network Insights
After Network Insights Advisor
Advisor
Time to resolve issue in Hours

Notice/Anomaly Detection 4

360 Case Accepted, Outputs Attached 2

840 Hrs 240 Tech Support Analysis 3

<1 Day
(~35 days) 120 Back and Forth Communication
120 Remediation, Close case 1

Downtime/Outages to the Network cost Millions

Success NIA immediately flags anomalies NIA helps prevent Significant OPEX, CAPEX and
metrics and optimizes your network downtimes/outages time savings

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 448
Network Insights Advisor

Dashboard ”Give me a summary of issues”

!

Advisories, Notifications, PSIRTs

• Provide Timely updates about your
system
• Track Bugs and PSIRTs

Anomalies
• Compliance, Consistency, unplanned
events

(Fabric) Fabric wide analysis

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 449
Network Insights Resources For your info
& reference

Common Use Cases Dashboard -- “Tell me now if I’ve got a problem!”

• Anomalies
!

System
• Resource Utilization (fabric wide)
• Trend Monitoring
(rising/falling)
• Fabric Capacity
• Environmental
Operations
• Statistics
• Flow Analytics
• Event Analytics

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 450
Network Insights Advisor For your info
Notify About Anomalies & reference

Known Anomalies

3 Alert / Inform Detected:

CSCDT2396 SAL1820SDRE

Weekly
Sync Recommend:
Upgrade S/W to NXOS
7.0(3)I7(3) in SAL1820SDRE

NIA

2 Detect
Insight
DB Fabric
Monitor 1
4 Implement

Detect Alert Remediate

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
 Network Insights Advisor For your info

Notify Me About New Releases & reference

Notifications
s Detected:
PSIRT: SAL1820SDRE
p
3 Alert / Inform
Recommend:
Upgrade S/W to NXOS
7.0(3)I7(3) in SAL1820SDRE

NIA

Push
Insight
Notification
DB Fabric
Monitor 1
4 Implement

p PSIRT
2 Identify Switches
s S/W p p p

Notify

Detect Alert Remediate

Detect Alert Remediate
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Main Dashboard For your info
& reference

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 453
Browse - Anomalies For your info
& reference

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 454
Detail – Process Details For your info
& reference

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 455
System - Resource Utilization For your info
& reference

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 456
For your info
& reference

Resource
Utilization
Details

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 457
Operations – Flow Analytics For your info
& reference

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 458
Flow Details Drill-Down For your info
& reference

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 459
Operations – Event Analytics For your info
& reference

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 460
For your info
& reference

Flow Details
Drill-Down

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
 Nicolas Delecroix
Technical Marketing Engineer
ACI Programmability
The APIC REST API is the Core of ACI Programmability

GUI CLI Python

REST API

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 463
 ACI Object Model

• Objects within APIC are structured in

tree-based hierarchy

• Objects are referred to as Managed

Objects (MO)

• Every object has a parent, with

exception of top:Root (top of tree)

• Relationships exist between objects

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 464
 How to Identify Objects
Distinguished Name
fvAp fvAEPg

polUni fvTenant vzFilter vzEntry

vzBrCP vzSubj

topRoot fabricPathEpCont fabricPathEp

fabricTopology fabricPod

fabricNode

vmmProvP vmmDomP vmmCtrlrP

EPG in tenant “Cisco” under application “DNS” Interface Eth1/4 on leaf 102 in pod 1
uni/tn-Cisco/ap-DNS/epg1 topology/pod-1/paths-102/pathep-[eth1/4]

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 465
 The REST API Exposes the Object Model

http(s):// host:port /api /{mo|class} /{dn|classname} .{xml|json} ?[options]

http or Specify Distinguished Specify filters,

APIC host API Encoding for
https Managed name or Object selectors or
and port Operator response
protocol Object or Class Class modifiers to query,
Operator joined using
ampersand (&)
Read properties for an EPG by Distinguished Name:

http://apic/api/mo/uni/tn-Cisco/ap-Software/epg-Download.xml

Find all 10G ports on fabric:

http://apic/api/class/l1PhysIf.xml?query-target-filter=eq(l1PhysIf.speed,"10G")

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 466
 Create / Update Operations

http(s):// host:port /api /{mo|class} /{dn|classname} .{xml|json} ?[options]

<fvTenant name="NewTenant">
<fvAp name="NewApplication">
<fvAEPg name="WebTier">
<fvRsPathAtt encap="vlan-1” mode=“regular”
tDn="topology/pod-1/paths-17/pathep-[eth1/1]"/>
</fvAEPg>
</fvAp>
</fvTenant>

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 467
Query Target Filters

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 468
Object Store Browser

• APIC has a built-in object browser to

navigate the object tree and inspect
the state of objects.
• Point the web browser to Visore:
http://<apic>/visore.html
• Search for a particular object or dn
(fvTenant, topSystem, topology/pod-
1/node-101)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI Python SDK
ACI Python SDK, AKA Cobra
• Cobra is a native Python API library that abstracts the APIC REST API
• Supports lookups, creations, modifications, deletions
• Objects in Cobra are a 1:1 representation of objects in the ACI object model
• All data has client side consistency checks performed
• Packaged as .egg, install with easy_install
from cobra.model.fv import Tenant
from cobra.model.pol import Uni
from cobra.mit.request import ConfigRequest

uniMo = Uni('')
t = Tenant(uniMo, 'Tenant1') # We create a tenant as a child of the universe
c = ConfigRequest() # Create a ConfigRequest to contain our new object
c.addMo(t) # Add our tenant to the ConfigRequest
moDir.commit(c) # Commit our configuration request

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 472
Simple 3-Tier App with Cobra

from cobra.model.fv import *

from cobra.model.pol import Uni Underlay and overlay
automation with a single API
uniMo = Uni('') model
t = Tenant(uniMo, 'Tenant1')
ap = Ap(t, 'Exchange')
epg1 = AEPg(ap, 'OWA')
epg2 = AEPg(ap, 'FrontEnd')
epg3 = AEPg(ap, 'MailBox')
ep = RsPathAtt(epg1, tDn =‘topology/pod-1/paths-17/paths-[eth1/1]’,
mode=‘regular’, encap =‘vlan-10’)
c = ConfigRequest()
c.addMo(t)
moDir.commit(c)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 473
We auto-generated the ACI Python API…

Can we auto-generate our app code, too?

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 474
• Use the GUI to
perform actions
• GUI creates REST
• API Inspector
shows REST
• Arya auto-
generates code
from REST
• So you can
automate tasks,
without having to
write any code 
• Available at
http://github.com/
datacenter/ACI
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 475
Ansible for ACI
Ansible for ACI

GUI CLI Python

REST API

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 477
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 478
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
 You can create
arbitrarily
complex/rich items.
This example shows
how to use a single
play to create
provider or consumer
ACI contracts. No
need to create two
plays (one for
consumer contracts,
one for provider)!

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 481
Query ACI

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Conclusions
Evolving Network Designs – Routed Fabrics

VXLAN Bridging / Routing

• VXLAN Flood & Learn

• VXLAN EVPN
• Separate Management Tools
(e.g. Nexus Fabric Manager)

ACI
• VXLAN Routing APIC

• Policy Controller (APIC)

• Consistent policy across physical
and virtual network
• Multi-hypervisor (VMware,
MSFT, OVS)
• Endpoint agnostic (bare metal,
VM, container)

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 484
 API
Network Stack
Orchestration/ Policy
Multi Site
Federation

Insights Assurance Telemetry Data

Fabric Level API’s

Network Semantics
DCNM APIC (DCNM, APIC)
Application Semantics
(APIC)

Fabric (NX-OS) Fabric (ACI)

Device Level API’s

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 485
 Cisco Webex Teams

Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session

How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space

cs.co/ciscolivebot#TECDCN-2002

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 486
Complete your online
session survey
• Please complete your Online Session
Survey after each session
• Complete 4 Session Surveys & the Overall
Conference Survey (available from
Thursday) to receive your Cisco Live T-
shirt
• All surveys can be completed via the Cisco
Events Mobile App or the Communication
Stations

Don’t forget: Cisco Live sessions will be available for viewing

on demand after the event at ciscolive.cisco.com

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 487
Continue Your Education

Demos in Meet the Related

Walk-in
the Cisco engineer sessions
self-paced
Showcase labs 1:1
meetings

TECDCN-2002 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 488
Thank you