Implementing Next Generation

Performance Routing – PfRv3

BRKRST-2362

Jean-Marc Barozet
Technical Leader
IWAN Solution Group
Agenda
• Business Trends
• PfRv3 Principles
• Monitoring Details – The Life of a Packet
• Path Enforcement – Route Override
• Enterprise Deployment
• IWAN Management
• Key Takeaways

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Business Trends
Hybrid WAN: Leveraging the Internet
Secure WAN Transport and Internet Access

Secure WAN
Transport

Private
MPLS (IP-VPN) Cloud
Virtual
Private
Cloud
Branch

Internet Public
Direct Internet Cloud
Access

Secure WAN transport for private Increased WAN transport

capacity, cost effectively!
and virtual private cloud access
Improve application
Leverage local Internet path for performance (right flows to
BRKRST-2362
public cloud and InternetCisco
© 2015 Cisco and/or its affiliates. All rights reserved.
access
Public right places)
Cisco Intelligent WAN (IWAN)
AVC
Private
MPLS Cloud
Virtual
ISR-AX Private
Cloud
3G/4G-LTE ASR1000-
AX
Branch
Internet Public
WAAS
PfRv3 Cloud
Akamai

Management & Orchestration

Transport Intelligent Application Secure

Independence Path Control Optimization Connectivity

 IPSec WAN Overlay  Optimal application routing  Performance monitoring  NG Strong Encryption
 Consistent Operational  Efficient use of bandwidth  Optimization and Caching  Threat Defense
Model
DMVPN Performance Routing AVC, WAAS, Akamai Suite-B, CWS, ZBFW
BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Intelligent Path Control with PfR
Leveraging the Internet
Voice/Video/Critical take
the best delay, jitter, and/or
loss path

MPLS
Private Cloud

Branch Virtual Private

Cloud
Other traffic is load Internet
balanced to maximize
bandwidth Voice/Video/Critical will be
rerouted if the current path
• PfR monitors network performance and routes applications degrades below policy thresholds
based on application performance policies
• PfR load balances traffic based upon link utilization levels
to efficiently utilize all available WAN bandwidth
BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
PfRv3 – How it Works
Traffic
ISR G2 Classes Performance
ASR1K MC Learning MC Measurements
MC
Active TCs TC Path
BR BR BR BR BR BR

Define your Traffic Policy Learn the Traffic Measurement Path Enforcement

Define path optimization Traffic flowing through the Report the measured TC Master Controller directs
policies on the Hub MC Border Routers (BRs) that performance metrics to BR path changes to keep
load balancing, match a policy are learned the Master Controller for traffic within policy
path preference, application Traffic Classes policy compliance Route Enforcement
metrics Unified Performance Unified Performance module in feature path
DSCP Based Policies Monitor Monitor
Application Based Policies

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Intelligent Path Control

Performance Routing Evolution

Available Now
IWAN 2.0
Today

IWAN 2.0

PfRv3
PfRv2 • Centralized provisioning
• AVC Infrastructure
PfR/OER • Policy simplification
• VRF Awareness
• App Path Selection
• Internet Edge • Blackout ~ 2s
• Blackout ~6s
• Basic WAN • Brownout ~ 2s
• Brownout ~9s
• Provisioning per site per • Scale 2000 sites
• Scale 500 sites
policy • Hub config only
• 10s of lines of config
• 1000s of lines of config

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Configuration – Improvement and Simplification
PfRv2 PfRv3 MC Configuration on Hub site only
pfr master
policy-rules IWAN-pfr-map
domain <MYNAME>
max-range-utilization percent 30
mc-peer domain 1 eigrp Loopback0
target-discovery
logging
!
vrf <name>
border 10.0.0.30 key-chain PFR-KEYCHAIN
interface GigabitEthernet0/0.32 internal
interface GigabitEthernet0/0.31 internal
interface GigabitEthernet0/0.30 internal
interface Tunnel2 external
master hub
link-group internet

!
interface Tunnel1 external
link-group mpls

learn
traffic-class filter access-list DENY_GLOBAL_LEARN_LIST
source-interface Loopback0
list seq 10 refname LEARN_VOICE_VIDEO
traffic-class access-list VOICE_VIDEO filter BRANCH_PREFIX
count 2000 max 10000
throughput
list seq 20 refname LEARN_CRITICAL
password IWAN
traffic-class access-list CRITICAL filter BRANCH_PREFIX
count 2000 max 10000
throughput
list seq 30 refname LEARN_BEST_EFFORT
traffic-class prefix-list BEST_EFFORT_PREFIX
load-balance
count 2000 max 10000
throughput
periodic 90
probe packets 20
!
class VOICE sequence 10
!
pfr-map IWAN-pfr-map 10
match pfr learn list LEARN_VOICE_VIDEO
set periodic 90
set delay threshold 200
set mode monitor fast
match dscp ef policy voice
set resolve delay priority 1 variance 5
set resolve loss priority 2 variance 5
set resolve jitter priority 3 variance 5
set loss threshold 50000
set jitter threshold 30
path-preference mpls fallback inet
set probe frequency 8

!
set link-group mpls fallback internet

pfr-map IWAN-pfr-map 20
match pfr learn list LEARN_CRITICAL
class VIDEO sequence 20
set periodic 90
set delay threshold 100
set mode monitor fast
set resolve delay priority 1 variance 20
set resolve loss priority 5 variance 10
match dscp af41 policy real-time-video
set probe frequency 8

!
set link-group mpls fallback internet

pfr-map IWAN-pfr-map 30
match pfr learn list LEARN_BEST_EFFORT
set periodic 90
match dscp cs4 policy real-time-video
set mode monitor passive
!
router eigrp IWAN
!
service-family ipv4 autonomous-system 1
path-preference mpls fallback inet
!
sf-interface default
shutdown
exit-sf-interface
!
class APPLICATION sequence 30
sf-interface Loopback0
no shutdown
hello-interval 200
hold-time 600
exit-sf-interface
match dscp af31 policy low-latency-data
!
topology base
exit-sf-topology
neighbor 10.0.0.91 Loopback0 remote 100
exit-service-family
!
!
ip prefix-list BEST_EFFORT_PREFIX seq 10 permit 0.0.0.0/0
!
ip prefix-list BEST_EFFORT_PREFIX seq 5 deny 10.0.0.0/16
ip prefix-list BEST_EFFORT_PREFIX seq 10 permit 0.0.0.0/0

• 500 sites • 2000 sites

!
!!
ip access-list extended CRITICAL
permit ip any any dscp af31
!
ip access-list extended VOICE_VIDEO
permit ip any any dscp ef
permit ip any any dscp af41
permit ip any any dscp cs4
!

• ~90 lines of configuration • <20 lines of configuration, all under “domain”

• All MCs • On the hub only
BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
PfRv3 Principles
IWAN Domain – Concept DC1 DCn

DCI
IWAN POP1 WAN Core IWAN POP2
• A collection of sites sharing the
same set of policies MC1
Transit Transit
MC2

• Each site runs Performance

Routing components BR1 BR2 BR3 BR4

• They exchanges services

through the Enterprise Domain
Peering framework IWAN
Domain
Controller DMVPN DMVPN
• Centralized configuration from a MPLS INET
Domain Controller
• Overlay network per Transport
for flexibility and simplification
R10 R11 R12 R13

Branch Branch Branch

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
PfRv3 Components

MC/BR
BRANCH
MPLS Dual CPE
MC BR BR

BR
Transit BRANCH
INET
MC/BR Single CPE

The Decision Maker: Master Controller (MC) The Forwarding Path: Border Router (BR)
 Apply policy, verification, reporting  Gain network visibility in forwarding path
 No packet forwarding/ inspection required (Learn, measure)
 Standalone of combined with a BR  Enforce MC’s decision (path enforcement)
 VRF Aware  VRF aware

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
PfRv3 Sites BRANCH SITE
Site11 – Site ID = 10.2.11.11
TRANSIT SITE
Hub – Site ID = 10.8.3.3
MC/BR
BRANCH
MPLS Dual CPE
MC BR BR

TRANSIT
BR
INET BRANCH
MC/BR Single CPE

• Controlled by a local Master Controller (MC) BRANCH SITE

• Site ID – the IP address of the MC loopback Site10 – Site ID = 10.2.10.10

• One/Multiple BRs Site Type

• Each BR one/multiple links  Transit Sites – Enterprise POPs or Hubs
 Branch Sites - Stub

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Transit Site – Hub Master Controller

MC/BR
BRANCH
MPLS Dual CPE
DC/MC BR BR

BR
TRANSIT BRANCH
INET
MC/BR Single CPE

domain IWAN • One of the MC is assigned the Domain Controller role

vrf default
master hub • Central point of provisioning for Domain policies
source-interface Loopback0
• DC + MC = Hub Master Controller

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Branch Site

MC/BR
BRANCH
MPLS Dual CPE
DC/MC BR BR

BR
TRANSIT BRANCH
INET
MC/BR Single CPE

domain IWAN
vrf default
• Service Exchange master branch
source-interface Loopback0
• Policies and Monitor configurations hub 10.8.3.3
• Site Prefixes border
source-interface Loopback0
master local
BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Policy/Monitor Distribution

MC/BR
BRANCH
Policies MPLS Dual CPE
Monitors
DC/MC BR BR

BR
TRANSIT BRANCH
INET
MC/BR Single CPE

• Domain policies and monitor instances are configured on the Hub MC.
• Policies are defined per VRF
• Then distributed to branch sites using the peering infrastructure
BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Policy/Monitor Distribution

Policies
MC/BR
Monitors
BRANCH
MPLS Dual CPE
DC/MC BR BR

BR
TRANSIT BRANCH
INET Policies
MC/BR Single CPE
Monitors

• Domain policies and monitor instances are configured on the Hub MC.
• Policies are defined per VRF
• Then distributed to branch sites using the peering infrastructure
BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
 Performance Policies - DSCP or App Based

• Policies:
domain IWAN
– DSCP or Application Based Policies
vrf default
(NBAR2)
master hub
– DSCP marking can be used with
load-balance
NBAR2 on the LAN interface
class MEDIA sequence 10 (ingress on BR)
match application telepresence-media policy real-time-video
match application ms-lync policy real-time-video
• Default Class is load balanced
path-preference MPLS fallback INET
class VOICE sequence 20
match dscp ef policy voice
path-preference MPLS fallback INET
class CRITICAL sequence 30
match dscp af31 policy low-latency-data

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
 Built-in Policy Templates
Pre-defined Threshold Definition
Template
Voice priority 1 one-way-delay threshold 150 (msec)
priority 2 packet-loss-rate threshold 1 (%)
priority 2 byte-loss-rate threshold 1 (%)
priority 3 jitter 30 (msec)
Real-time-video priority 1 packet-loss-rate threshold 1 (%)
priority 1 byte-loss-rate threshold 1 (%)
priority 2 one-way-delay threshold 150 (msec) Pre-defined Threshold Definition
priority 3 jitter 20 (msec) Template
Low-latency-data priority 1 one-way-delay threshold 100 (msec) Bulk-data priority 1 one-way-delay threshold 300 (msec)
priority 2 byte-loss-rate threshold 5 (%) priority 2 byte-loss-rate threshold 5 (%)
priority 2 packet-loss-rate threshold 5 (%) priority 2 packet-loss-rate threshold 5 (%)
Best-effort priority 1 one-way-delay threshold 500 (msec)
priority 2 byte-loss-rate threshold 10 (%)
priority 2 packet-loss-rate threshold 10 (%)
scavenger priority 1 one-way-delay threshold 500 (msec)
priority 2 byte-loss-rate threshold 50 (%)
priority 2 packet-loss-rate threshold 50 (%)
BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Discovery – WAN Interface
Branch BRs discover Path Names
and External Interfaces
Define the Path Names on the
Hub Border Routers
MC/BR
BRANCH
MPLS Dual CPE
MC BR
BR
Hub
Master MC

BR
TRANSIT HUB BRANCH
INET MC/BR Single CPE

• Using Smart Probes

• Hub generates Discovery packets over all available paths
• Discovery packets intercepted on spokes
• External interfaces and path names automatically discovered and added
BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Performance Monitoring – User Traffic

Passive Monitoring
MC/BR SITE2
Dual CPE
MPLS

MC BR BR

SITE1

BR SITE3
Single CPE
INET
MC/BR

Performance Monitor
Bandwidth on egress • Collect Performance Metrics
Per Traffic Class • Per Channel
(dest-prefix, DSCP, AppName) - Per DSCP
- Per Source and Destination Site
- Per Interface
BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
Performance Monitoring

Smart Probing
MC/BR SITE2
Dual CPE
MPLS

MC BR BR

SITE1

BR SITE3
Single CPE
INET
MC/BR

Integrated Smart Probes Performance Monitor

• Traffic driven – intelligent on/off • Collect Performance Metrics
• Site to site and per DSCP • Per Channel
- Per DSCP
- Per Source and Destination Site
- Per Interface
BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
Performance Violation

MC/BR
SITE2
MPLS Dual CPE

MC BR BR

SITE1

BR
SITE3
INET
MC/BR Single CPE

Threshold Crossing Alert (TCA)

• Sent to source site
• loss, delay, jitter, unreachable

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
Performance Violation – NetFlow Export

MC/BR
SITE2
MPLS Dual CPE

MC BR BR

SITE1

BR
SITE3
INET
MC/BR Single CPE

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
Policy Decision

MC/BR
SITE2
MPLS Dual CPE

MC BR BR

SITE1

BR
SITE3
INET
MC/BR Single CPE

• Reroute Traffic to a Secondary Path

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
Policy Decision

MC/BR
SITE2
MPLS Dual CPE

MC BR BR

SITE1

BR
SITE3
INET
MC/BR Single CPE

• Reroute Traffic to a Secondary Path

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
Key Points and … Limitations
• Key Points
– DSCP or Application (NBAR2) based policies
– Control traffic in the global Routing Table and per VRF
– Passive Monitoring based on user traffic
– Leverage smart probing when needed
– Cooperation between pair of sites
– Exports of PfR information with NetFlow v9 to collectors
• Limitation:
– No IPv6 support yet (Roadmap) – But infrastructure is IPv6 ready
– NBAR2 with Asymmetric Routing is not yet supported

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Monitoring Details – The Life of a Packet
PfRv3 Monitoring
Based on Performance Monitor
• PfRv3 defines multiple Performance
Monitor Instances (PMI)
• Applied on all External interfaces
Monitor1 – Site Prefix Learning
(egress direction)
MC
Monitor2 – Aggregate Bandwidth per
Traffic Class (egress direction)
Exports to MC

1 1

2 2
Site
BR BR
3 3

Monitor3 – Performance
measurements (ingress direction)

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Performance Monitor
Multiple Monitors with Unique Key Fields
Performance
Monitor 1

Traffic Performance
BR Monitor 2

Non-Key Fields
Key Fields Packet 1 Non-Key Fields Key Fields Packet 1
Packets
ipv4 destination prefix 10.1.10.0 Packets Source Prefix 10.8.0.0
Bytes
ipv4 destination mask /24 Bytes Source Mask /16
Destination Site ID 10.2.10.10 Input VRF VRF1 Timestamps
Timestamps
Application Name Skype
DSCP AF41

Interface Output Tunnel100 Performance Monitor Cache #2

Source Source Input
… Pkts
Prefix Mask VRF

10.8.0.0 /16 VRF1 … 11000

Performance Monitor Cache #1
Dest Dest. App Output
Dest Site DSCP … Pkts
Prefix Mask Name I/F

10.1.10.0 /24 10.2.10.10 Skype AF41 Tunnel100 … 1100

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Source Site – Egress Traffic
Collecting Traffic Class IWAN POP

Hub MC
10.8.3.3/32
MC1 Dst-Site- App DSCP Dst-Site- State BW BR Exit MC1
Pfx Id
10.1.10.0 APP1 AF41 ? UK 24 BR1 Tu10
BR1 BR2
Source Destination DSCP App 2

10.8.1.200 10.1.10.200 AF41 APP1

• Traffic going outside a source site MPLS INET

– Initially based on Routing Information

• Captured by a Performance Monitor on
the external interface on egress R10 R11 R12 R13

• BR reports to its local MC 10.1.10.0/24 10.1.11.0/24

10.1.12.0/24
10.1.13.0/24
• Monitor interval is 30 sec

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Monitor2 Details
PMI: [Egress-Aggregate] - #2
Aggregate Bandwidth Per TC
Trigger NBAR: no/yes

• Aggregate Bandwidth Per TC Key Fields

• ipv4 destination prefix,
• ipv4 destination mask,
• Bandwidth is measured egress direction. • pfr site destination prefix ipv4
• pfr site destination prefix mask ipv4
• [application id]
• Using monitor #2 • ip dscp
• interface output

• Collect bandwidth per Traffic Class Non-Key Fields

• counter bytes long
MC • counter packet long
• timestamp absolute monitoring-interval start

PMI: Performance Monitor Instance

Exports to MC
Monitor-interval
30 sec default
2
SITE BR BR

Traffic Flow

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
 What is a Traffic Class?
IWAN POP
Dst-Site-Pfx App DSCP Dst-Site-id State BR Exit
Hub MC
10.1.10.0 APP1 AF41 ? UK ? ? 10.8.3.3/32
MC1
10.1.10.0 N/A EF ? UK ? ?
10.1.10.0 N/A AF31 ? UK ? ?
BR1 BR2
10.1.10.0 N/A 0 ? UK ? ?
10.1.11.0 N/A EF ? UK ? ?
10.1.11.0 N/A AF31 ? UK ? ?
10.1.11.0 N/A 0 ? UK ? ?
10.1.12.0 N/A 0 ? UK ? ? MPLS INET
…

• What is a Traffic Class (TC)?

– Destination Site Prefix
R10 R11 R12 R13
– DSCP value
– Application Name 10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
• Each Master Controller owns a TC Database

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
We need the Destination Site ID – Why??

MC1
BR
MPLS ?
Dst-Site-Pfx App DSC Dst-Site-id State BR Exit
P
10.1.10.0 APP1 AF41 ? UK ? ?

BR
SITE
INET

• Performance Measurement is done on the destination site

• Source site needs to get Performance Metrics from the destination site

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Site Prefix Exchange – Dynamic
Based on User Traffic IWAN POP

Hub MC
10.8.3.3/32
• Source Prefix and Mask collected from MC1

Performance Monitor Site 10

10.1.10.0/24
BR1 BR2
• Monitor interval is 30 sec
• BR send to its local MC
• MC send information to all peers via Peering MPLS INET

Site 10
10.1.10.0/24 Site 10 Site 10
Source Destination DSCP App
1 10.1.10.0/24 10.1.10.0/24
10.1.10.200 10.8.1.200 AF41 XYZ
R10 R11 R12 R13

10.1.12.0/24
R10 Site-Pfx Mask 10.1.10.0/24 10.1.11.0/24
MC 10.1.13.0/24
10.1.10.0 /24

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Site Prefix Exchange – Dynamic
Based on User Traffic IWAN POP

Hub MC
10.8.3.3/32
Site Prefix List MC1

Hub 10.8.0.0/16
R10 10.1.10.0/24
BR1 BR2
R11 10.1.11.0/24
R12 10.1.12.0/24
R12 10.1.13.0/24

MPLS INET
• Every MC in the domain owns a Site Prefix
database
• Gives the mapping between site and prefixes
R10 R11 R12 R13

10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Monitor1 Details
Site Prefix Learning
PMI: [Egress-prefix-learn] - #2

• Site Prefix collected with Performance Monitor

Trigger NBAR: no

• Using monitor #1 Key Fields

• ipv4 source prefix,
• ipv4 source mask,
• Routing vrf input

Non-Key Fields
• counter bytes long
• counter packet long
MC • timestamp absolute monitoring-interval start

Exports to MC
Monitor-interval
30 sec default
1
Site10
10.1.10.0/24 BR BR

Traffic Flow

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
Site Prefixes – Static Configuration
TRANSIT SITE
• This allows configuring site-prefix manually instead of
learning. Hub MC
10.8.3.3/32
MC1
• This configuration should be used at the site if the
site is used for transit. BR1 BR2
– For example, Site A reaches Site B via Hub-Site, where
Hub-Site is transit site. The configuration is used to
prevent learning of Site A prefix as Hub-Site prefix when it
is transiting from Hub.

domain IWAN Source Destination DSCP App

vrf default 10.1.10.200 10.1.11.200 AF41 AppXY
master hub
source-interface Loopback0
site-prefixes prefix-list DC1_PREFIX
!
ip prefix-list DC1_PREFIX seq 10 permit 10.8.0.0/16
!

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
 Source Site – Egress Traffic
Traffic Class Controlled IWAN POP

Hub MC
10.8.3.3/32
Site Prefix List MC1
Hub 10.8.0.0/16
R10 10.1.10.0/24
MC1 BR1 BR2
R11 10.1.11.0/24
R12 10.1.12.0/24 Source Destination DSCP App 2

R12 10.1.13.0/24 10.8.1.200 10.1.10.200 AF41 APP1

MPLS INET
Dst-Site-Pfx App DSCP Dst- State BW BR Exit
MC1 Site-id
10.1.10.0 APP1 AF41 R10 CN 24 BR1 Tu100

• Now the MC has the destination site R10 R11 R12 R13
information 10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
• Traffic Class is controlled

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Source Site – Traffic Class Database
IWAN POP

Hub MC
10.8.3.3/32
Dst-Site-Pfx Dst-Site-id App DSCP State BR Exit MC1

10.1.10.0 R10 APP1 AF41 CN BR1 Tu100

10.1.10.0 R10 N/A EF CN BR1 Tu100

BR1 BR2
10.1.10.0 R10 N/A AF31 CN BR1 Tu100

10.1.10.0 R10 N/A 0 CN BR2 Tu200

10.1.11.0 R11 N/A EF CN BR1 Tu100

10.1.11.0 R11 N/A AF31 CN BR1 Tu100

10.1.11.0 R11 N/A 0 CN BR2 Tu200 MPLS INET

10.1.12.0 R12 N/A 0 CN BR2 Tu200

• Same process for all traffic to between all

sites. R10 R11 R12 R13

• Traffic Class database contains bandwidth, 10.1.10.0/24 10.1.11.0/24

10.1.12.0/24
10.1.13.0/24
destination sites, BR and external path used

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
 Destination Site – Ingress Traffic
Channel Performance IWAN POP

Hub MC
• Traffic flow captured on the destination MC1
10.8.3.3/32

site
• Performance Monitor collects BR1 BR2

Performance Metrics
• Per Channel
• Default Monitor interval is 30 sec MPLS INET

Source Destination DSCP App

10.8.1.200 10.1.10.200 AF41 APP1
3

R10 R11 R12 R13

Channel Dst-Site-id Path DSCP BW Delay Jitter Loss
R10 MC
5 Hub Tu1 AF41 24 51 2 1 10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
What is a Channel?
Between Sites
• A Channel is a unique combination of:
– Interface
– Peer-Site id
– DSCP
• Created
– Based on real traffic observed on the BRs
– Added every time there is a new DSCP or a new interface or a new site
added to the prefix database.
– Smart Probe is received
• On all exits (Present Channel, Backup Channel)

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Monitor3 Details
PMI [Ingress-per-DSCP] - #3
Ingress Performance Monitor
Key Fields
• Performance is measured ingress direction: • pfr site source id ipv4
• pfr site destination id ipv4
– Actual Traffic • ip dscp
• Interface input
– Smart Probes if no traffic • policy performance-monitor classification hierarchy
Monitor-interval
• Using monitor #3 Non-Key Fields
• transport packets lost rate
30 sec default

• transport bytes lost rate

• Monitors performance metrics per DSCP and • pfr one-way-delay
per Site • network delay average
• transport rtp jitter inter arrival mean
• Export only when there is a performance issue • counter bytes long
• counter packets long
(performance threshold crossed) • timestamp absolute monitoring-interval start

3
SITE BR1 R10

Traffic Flow

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Channel Details
IWAN POP

Hub MC
10.8.3.3/32
MC1

BR1 BR2
Present Channel 3
Backup Channel 4
• Site 10
• Site 10
• DSCP AF41
• DSCP AF41
• DMVPN1
• DMVPN2
MPLS INET

R10 R11 R12 R13

10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
Channel Details
IWAN POP

Hub MC Between Any Pair

10.8.3.3/32
MC1
of Sites that has
traffic!
BR1 BR2

Present Channel 13 MPLS INET

Backup Channel 14
• Site 11
• Site 11
• DSCP EF
• DSCP EF
• DMVPN1
• DMVPN2

R10 R11 R12 R13

10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
Smart Probing
Help for Measurement Over Channels

INET
MC MC

3
Site10 Site11
BR MPLS BR
10.1.10.0/24 3 10.1.11.0/24

Traffic Flow

• Without actual traffic

– BR sends 10 probes spaced 20ms apart in the first 500ms and another similar 10 probes in the next 500ms, thus
achieving 20pps for channels without traffic.
• With actual traffic
– Lower frequency when real traffic is observed over the channel
– Probes sent every 1/3 of [Monitor Interval], ie every 10 sec by default
• Measured by Unified Performance Monitoring just like other data traffic

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
Channel Unreachable
• PfRv3 considers a channel reachable as long as the Exports to MC
BR
site receives a PACKET on that channel
1

• A channel is declared unreachable in both direction if

Channel
– There is NO traffic on the Channel, probes are our only way DSCP EF
of detecting unreachability. So if no probe is received within
1 sec, we detect unreachability.
– When there IS traffic on the channel, if we don’t see any
packet for more than a second on a channel we detect
unreachability.

• A channel is put to reachable if following happens 3

– Traffic is received from the remote side.
– Unreachable TCA not received for two monitor intervals BR

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
Unreachable
Channel State Examples

R84#sh domain IWAN border channels R84#sh domain IWAN border channels

Channel id: 1 Channel id: 3

Channel create time: 12:52:35 ago Channel create time: 12:50:55 ago
Site id : 255.255.255.255 Site id : 10.2.11.11
DSCP : default[0] DSCP : default[0]
Service provider : MPLS Service provider : MPLS
Number of Probes sent : 0 Number of Probes sent : 843019
Number of Probes received : 0 Number of Probes received : 838980
Last Probe sent : 12:52:35 ago Last Probe sent : 00:00:00 ago
Last Probe received : - ago Last Probe received : 00:00:00 ago
Channel state : Initiated and open Channel state : Initiated and open
Channel next_hop : 0.0.0.0 Channel next_hop : 10.0.100.11
RX Reachability : Initial State RX Reachability : Reachable
TX Reachability : Reachable TX Reachability : Reachable
Channel is sampling 0 flows Channel is sampling 0 flows
Supports Zero-SLA: Yes Supports Zero-SLA: Yes
Muted by Zero-SLA: No Muted by Zero-SLA: No
Probe freq with traffic : 1 in 10000 ms Probe freq with traffic : 1 in 10000 ms

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
Smart Probing – Zero SLA Support

INET
MC MC

3
Site10 Site11
BR MPLS BR
10.1.10.0/24 3 10.1.11.0/24

Traffic Flow

• No SLA on the secondary path, but still probing all channels (DSCPs)
– 10 DSCP => 10 Smart Probes over the secondary path
• Waste of bandwidth, especially for metered interfaces (4G/LTE)

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
Zero-SLA
Principles
• Zero-sla added on the WAN interface path configuration option.
• PfR will only probe the default channel (DSCP 0).
– It will mute all other smart-probes besides the default channel.
– The default channel runs as DSCP 0, TCA and ODE are DSCP CS6.
– Extrapolate metrics on this to all other DSCPs on this channel
• Site Capability Exchange:
– Site-capability exchange: allow for backwards compatibility and coexistence with legacy
sites until they are upgraded.
– The site-capability is fully extensible so that additional domain configuration features can
be added, not just PfRv3 capabilities.

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
 Zero SLA Configuration
R83#sh domain IWAN master site-capability
interface Tunnel200 Device Capability IWAN POP
description --- INET ---
domain IWAN path INET zero-sla -----------------------------------------------------------
| Capability | Major | Minor |
! -----------------------------------------------------------
| Domain | 2 | 0 |
-----------------------------------------------------------
| Zero-SLA | 1 | 0 |
-----------------------------------------------------------
Site id :10.2.10.10 BR2

----------------------------------------------------------- Tunnel 200

| Capability | Major | Minor |
-----------------------------------------------------------
| Domain | 2 | 0 |
-----------------------------------------------------------
| Zero-SLA | 1 | 0 | INET
-----------------------------------------------------------
Site id :10.2.12.12
DMVPN-2

[SNIP

R83#

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
 Zero SLA Configuration
R83#sh domain IWAN master site-capability
interface Tunnel200 Device Capability IWAN POP
description --- INET ---
domain IWAN path INET zero-sla -----------------------------------------------------------
| Capability | Major | Minor |
! -----------------------------------------------------------
| Domain | 2 | 0 |
-----------------------------------------------------------
| Zero-SLA | 1 | 0 |
-----------------------------------------------------------
Site id :10.2.10.10 BR2

Borders:
----------------------------------------------------------- Tunnel 200
| Capability | Major | Minor |
IP address: 10.8.4.4
-----------------------------------------------------------
Version: 2
| Domain | 2 | 0 |
Connection status: CONNECTED (Last Updated 1d00h ago )
-----------------------------------------------------------
Interfaces configured:
| Zero-SLA | 1 | 0 | INET
Name: Tunnel100 | type: external | Service Provider: MPLS | Status: UP | Zero-SLA: NO
-----------------------------------------------------------
Number of default Channels: 0
Site id :10.2.12.12
DMVPN-2
Tunnel if: Tunnel0
[SNIP
IP address: 10.8.5.5
Version: 2
R83#
Connection status: CONNECTED (Last Updated 00:04:59 ago )
Interfaces configured:
Name: Tunnel200 | type: external | Service Provider: INET | Status: UP | Zero-SLA: YES
Number of default Channels: 0

Tunnel if: Tunnel0

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Path Enforcement – Route Override
 Performance Violation
Enterprise HQ
• Performance notification exported ONLY Hub MC
when there is a violation on a specific MC1
10.8.3.3/32

channel
– Generated from ingress monitor attached on BR1 BR2
BRs to the source site MC
– Based on Monitor interval (30 sec default, R10
configurable) TCA Delay
– Via all available external interfaces. DSCP AF41
MPLS INET
Path MPLS

3
Channel Dst-Site-id DSCP Path BW Delay Jitter Loss
R10 R10 R11 R12 R13
5 Hub AF41 Tu1 24 250 2 1
10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
Performance Violation – Detected on Dst Site
Enterprise HQ

Hub MC
10.8.3.3/32
Dst-Site-Pfx Dst-Site-id App DSCP State BR Exit MC1

10.1.10.0 R10 APP1 AF41 CN BR1 Tu1

10.1.10.0 R10 N/A AF41 CN BR1 Tu1

BR1 BR2
10.1.10.0 R10 N/A AF31 CN BR1 Tu1

10.1.10.0 R10 N/A 0 CN BR2 Tu2

10.1.11.0 R11 N/A EF CN BR1 Tu1

10.1.11.0 R11 N/A AF31 CN BR1 Tu1

10.1.11.0 R11 N/A 0 CN BR2 Tu2 MPLS INET

10.1.12.0 R12 N/A 0 CN BR2 Tu2

3
R10 R10 R11 R12 R13
TCA Delay
10.1.12.0/24
DSCP EF 10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
Path MPLS

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Policy Decision – Reroute TC
Enterprise HQ
• MC computes a new path for each Hub MC
10.8.3.3/32
impacted TC MC1

• MC tells the BRs to enforce the new BR1 BR2

path

MPLS INET

R10 R11 R12 R13

10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Reroute TC – Path Enforcement
Enterprise HQ
• Dataplane forwarding Hub MC
10.8.3.3/32
MC1
• Activated on all but external interfaces
• Lookup per packet - output-if/next hop BR1 BR2

retrieved
– Packet Forwarded
– If no entry – Uses FIB entry MPLS INET

• TC flows redirected to the new path over

the auto mGRE tunnels between the BRs
3
• No change in the routing table R10 R11 R12 R13

10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Enterprise Deployment
Intelligent WAN Deployment Models
Dual MPLS Hybrid Dual Internet

Internet
Public Enterprise Public

MPLS MPLS
MPLS + Internet
Internet
Branch Branch Branch

Highest SLA guarantees More BW for key applications Best price/performance

Tightly coupled to SP Balanced SLA guarantees Most SP flexibility
Expensive Moderately priced Enterprise responsible for
SLAs

Consistent VPN Overlay Enables Security Across Transition

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
Hybrid WAN Designs
Traditional and IWAN
TRADITIONAL
Active/Standby HYBRID IWAN HYBRID
WAN Paths Active/Active
WAN Paths
Primary With Backup

Two Ipsec Data Center Data Center

Technologies One IPsec Overlay
GETVPN/MPLS ASR 1000 ASR 1000 ASR 1000 ASR 1000 DMVPN
DMVPN/Internet
ISP A SP V ISP A SP V

Two WAN
Routing Domains
DMVPN GETVPN DMVPN DMVPN
MPLS: eBGP or Static One WAN
Internet MPLS Internet MPLS
Internet: iBGP, EIGRP Routing Domain
or OSPF
iBGP, EIGRP, or OSPF
Route Redistribution
Route Filtering Loop
Prevention

ISR-G2
Branch ISR-G2 Branch

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
IWAN Layers

Intelligent Path
AVC PfR QoS Selection

Overlay routing
Overlay Routing Protocol (BGP, EIGRP) over tunnels

Transport Independent Design (DMVPN) Transport Overlay

ZBFW
MPLS Routing Internet Routing CWS
Infrastructure Routing

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
IWAN Deployment – DMVPN
DCI
IWAN POP1 WAN Core IWAN POP2
• IWAN Prescriptive Design – Transport
Independent Design based on DMVPN MC1 MC2

– Branch spoke sites establish an IPsec tunnel to

and register with the hub site
R84 R85 R94 R95
– Data traffic flows over the DMVPN tunnels
– WAN interface IP address used for the tunnel
source address (in a Front VRF)
– One tunnel per user VRF ATBT ISLAND
MPLS INET
• Over the Top Routing
– BGP or EIGRP are typically used for scalability
– IP routing exchanges prefix information for each
site
R10 R11 R12 R13
• Per-tunnel QOS is applied to prevent hub site 10.1.10.0/24
10.1.12.0/24
10.1.11.0/24
oversubscription to spoke sites 10.1.13.0/24

http://docwiki.cisco.com/wiki/PfR3:Solutions:IWAN
BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Best Practice: VRF-Aware DMVPN
Keeping the Default Routes in Separate VRFs with Front Door VRF
• Enable FVRF DMVPN on the Spokes
default

EIGRP
• Allow the ISP learned Default Route in the default VRF: INET-PUBLIC
VRF INET-PUBLIC and use for tunnel INSIDE
Internet Edge
establishment Block

default
• Global VRF contains Default Route learned
via tunnel. User data traffic follows Tunnel to VPN-DMZ
default
INSIDE interface on firewall OUTSIDE

• Allows for consistent implementation of default

Internet
corporate security policy for all users

VRF: INET-PUBLIC default

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
DMVPN Configuration – F-VRF
IWAN POP

MC1

R84 R85
vrf definition IWAN-TRANSPORT-1 10.0.100.84 10.0.200.85
!
Front-door VRF definition for
address-family ipv4
MPLS Transport
exit-address-family
!
MPLS INTERNET

10.0.100.10 10.0.200.10

R10

10.1.10.0/24

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
 DMVPN Configuration – IPSec
! IWAN POP
! <removed IKEv2 proposal, will use smart default)
!
! MC1
crypto ikev2 keyring DMVPN-KEYRING-1
peer ANY
address 0.0.0.0 0.0.0.0
pre-shared-key c1sco123 R84 R85
! 10.0.100.84 10.0.200.85
!
crypto ikev2 profile FVRF-IKEv2-IWAN-TRANSPORT-1
match fvrf IWAN-TRANSPORT-1
match identity remote address 0.0.0.0
authentication remote pre-share Maximise window size to
authentication local pre-share eliminate future anti-replay issue MPLS INTERNET
keyring local DMVPN-KEYRING-1
!
crypto ipsec security-association replay window-size 512
!
crypto ipsec transform-set AES256/SHA/TRANSPORT esp-aes 256 esp-sha-hmac 10.0.100.10 10.0.200.10
mode transport
!
crypto ipsec profile DMVPN-PROFILE-1
R10
set transform-set AES256/SHA/TRANSPORT
set ikev2-profile FVRF-IKEv2-IWAN-TRANSPORT-1
10.1.10.0/24

crypto ikev2 dpd 40 5 on-demand ! Set DPD timers for Branch

Configs ONLY!

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
DMVPN Hub Configuration – Interfaces & Routing
MPLS TRANSPORT – R84
IWAN POP
interface GigabitEthernet0/0/3
description MPLS-TRANSPORT Put Transport Interface into
MC1
vrf forwarding IWAN-TRANSPORT-1 MPLS Front-door VRF
ip address 172.16.84.4 255.255.255.0
!
interface Tunnel100 R84 R85
Instantiate DMVPN Tunnel
bandwidth 1000 10.0.100.84 10.0.200.85
ip address 10.0.100.84 255.255.255.0
no ip redirects
ip mtu 1400
ip pim nbma-mode
ip pim sparse-mode
ip nhrp authentication cisco123 MPLS INTERNET
ip nhrp map multicast dynamic
ip nhrp network-id 100
DMVPN Network ID: MPLS
ip nhrp holdtime 600
ip nhrp redirect Set DMVPN Ph3
ip tcp adjust-mss 1360
tunnel source GigabitEthernet0/0/3 Map to Physical Interface
tunnel mode gre multipoint
tunnel key 101
tunnel vrf IWAN-TRANSPORT-1
tunnel protection ipsec profile DMVPN-PROFILE-1 Tunnel endpoint is in Front-door VRF
!
ip route vrf IWAN-TRANSPORT-1 0.0.0.0 0.0.0.0 172.16.84.8
Default route for Tunnel endpoints

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
DMVPN Spoke Configuration – Interfaces & Routing
!
Interface GigabitEthernet0/1 Put Transport Interface into
vrf forwarding IWAN-TRANSPORT-1
MPLS Front-door VRF
ip address 10.0.100.10 255.255.255.0
!
interface Tunnel100
Instantiate DMVPN Tunnel
ip address 10.0.100.10 255.255.255.0
no ip redirects
ip mtu 1400
ip pim dr-priority 0
ip pim sparse-mode
ip nhrp authentication cisco123 DMVPN Network ID: MPLS
ip nhrp network-id 100
ip nhrp holdtime 600
ip nhrp nhs 10.0.100.84 nbma 172.16.84.4 multicast Multiple DMVPN Hub for
ip nhrp nhs 10.0.100.94 nbma 172.16.94.4 multicast Resiliency
ip nhrp registration no-unique
ip nhrp shortcut
ip tcp adjust-mss 1360 Set DMVPN Ph3 10.0.100.10 10.0.200.10
if-state nhrp
tunnel source GigabitEthernet0/1
tunnel mode gre multipoint R10
tunnel key 101 Tunnel endpoint is in Front-
tunnel vrf IWAN-TRANSPORT-1 door VRF 10.1.10.0/24
tunnel protection ipsec profile DMVPN-PROFILE-1
!
ip route vrf IWAN-TRANSPORT-1 0.0.0.0 0.0.0.0 172.16.101.8
Default route for Tunnel endpoints

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
IWAN Routing Protocols
Which protocol should I use?

• IWAN Profiles are based upon BGP and EIGRP for scalability and optimal Intelligent
Path Control
• Scalability:
– BGP (Path Vector) and EIGRP (Advanced Distance Vector) provide best scale over
large hub-and-spoke topologies like DMVPN
– OSPF (Link State) maintains a lot of network state which cannot be subdivided easily in
large DMVPN networks
• Intelligent Path Control:
– PfR can be used with any routing protocols by relying on the routing table (RIB).
• Requires all valid WAN paths be ECMP so that each valid path is in the RIB.
– For BGP and EIGRP, PfR can look into protocol’s topology information to determine both
best paths and secondary paths thus, ECMP is not required.

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
EIGRP IWAN Design Overview 10.9.0.0/16
IWAN POP2
Principles
IWAN POP1 10.8.0.0/16
• Single EIGRP process for Branch, WAN and MC1
Hub MC
10.8.3.3/32
POP/hub sites
• Extend Hello/Hold timers for WAN R84 R85
Delay TAG Delay TAG
• Adjust tunnel interface “delay” to ensure WAN 1000 DMVPN-1 2000 DMVPN-2

path preference
• MPLS primary, INET secondary Set Interface
Delay to
influence best MPLS INET
• Hubs path DMVPN-1 DMVPN-2

– Route tag filtering to prevent routing loops

across DMVPNs
Delay 1000
– Branch prefix summary route for EIGRP
spoke-to-spoke tunnels Stub R10 R11 R12 R13

10.1.12.0/24
10.1.10.0/24 10.1.11.0/24
• Spokes 10.1.13.0/24

– EIGRP Stub for scalability

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
IWAN Deployment – BGP 10.9.0.0/16
IWAN POP2

• A single iBGP routing domain is used IWAN POP1 10.8.0.0/16

Hub MC
MC1 10.8.3.3/32
• Extend Hello/Hold timers for WAN
• Hub: R84 R85
– DMVPN hub routers function as BGP route- Community 10:84 Community 10:85
iBGP iBGP
reflectors for the spokes
– BGP dynamic peer feature configured on the
route-reflectors
MPLS INET
– Summary route to spokes 10.6.34.0/23 10.6.36.0/23
– Set Community for site local prefixes
Local Pref
• Spokes: 200
– peer to a redundant pair of route-reflectors in LP 50
each DMVPN cloud R10 R11 R12 R13

– Inbound route-map to set local-preference based 10.1.12.0/24

10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
on community
– Ensure that preferred path is MPLS

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
PfRv3 and Parent Routes
• Make sure that all Border Routers have a route over each external path to the
destination sites
– PfR will NOT be able to effectively control traffic otherwise.
• PfRv3 always checks for a parent route before being able to control a Traffic
Class. Parent route check is done as follows:
– Check to see if there is an NHRP shortcut route
– If not – Check in the order of BGP, EIGRP, Static and RIB
– If at any point, an NHRP short cut route appears, PfRv3 would pick that up and
relinquish using the parent route from one of the routing protocols.
• PfR3 currently supports only one next-hop per multipoint interface. Routing has
to be done such that only one next-hop per destination prefix is in the routing
table per DMVPN tunnel interface.

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
 Transit Site – Hub MC TRANSIT SITE

domain IWAN Policies

vrf default Monitors
master hub
MC1
source-interface Loopback0
MC1 enterprise-prefix prefix-list ENTERPRISE_PREFIX
site-prefixes prefix-list DC1_PREFIX
R84 R85
path MPLS path INET
domain IWAN domain IWAN
vrf default vrf default
border border
master 10.8.3.3 master 10.8.3.3
BR1 source-interface Loopback0 source-interface Loopback0
BR2 ! ! MPLS INET
interface Tunnel100 interface Tunnel200
description -- Primary Path -- description – Secondary Path --
domain IWAN path MPLS domain IWAN path INET

• IWAN POP is the central hub for the

Enterprise Domain MC/BR MC/BR MC/BR BR

10.1.12.0/24
• MC1 – Hub MC 10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
• BR1 – Hub BR, DMVPN Hub for MPLS
• BR2 – Hub BR, DMVPN Hub for INET
BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
Single CPE Branch Sites
TRANSIT SITE

Hub MC
MC1 10.8.3.3/32

domain IWAN
vrf default R84 R85
master branch
source-interface Loopback0
hub 10.8.3.3
R10 border
R11 master local
source-interface Loopback0 MPLS INET

• Stub Site
• Combination of MC and BR on the same CPE R10 R11 R12 R13

10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Dual CPE Branch Sites
TRANSIT SITE

Hub MC
domain IWAN MC1 10.8.3.3/32
vrf default
master branch
source-interface Loopback0 R84 R85
hub 10.8.3.3
R12 border
master local
source-interface Loopback0

MPLS INET
domain IWAN
vrf default
R13 border
master 10.2.12.12
source-interface Loopback0
R10 R11 R12 R13

10.1.12.0/24
10.1.10.0/24 10.1.11.0/24
• Stub Site 10.1.13.0/24

• One of the BR is also the MC

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
 Enterprise Domain
Policy – DSCP or App Based

domain IWAN
vrf default
• When load balancing is enabled,
master hub
PfRv3 adds a “default class for
load-balance
match all DSCP (lowest priority
class VOICE sequence 10
compared to all the other classes)”
and we influence this traffic.
match dscp ef policy voice
path-preference MPLS fallback INET • When load balancing is disabled,
class VIDEO sequence 20 PfRv3 deletes this “default class”
match dscp af41 policy voice and as a part of that frees up the
path-preference MPLS fallback INET TCs that was learnt as a part of LB –
class CRITICAL sequence 30 they follow the routing table
match dscp af31 policy low-latency-data

MC1

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
Policy – Monitor Intervals

domain IWAN Monitoring Interval Fast

vrf default Reaction Time 2 sec
master hub
monitor-interval 2 dscp ef
monitor-interval 2 dscp af41
monitor-interval 2 dscp cs4
monitor-interval 2 dscp af31

MC1

• Advanced commands
– Carefully choose monitor interval for critical applications

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
 Deploying with user VRFs
TRANSIT SITE
vrf definition TEST1
!
address-family ipv4 MC1
exit-address-family 1 2 1 2
!
vrf definition TEST2 R84 R85
! interface Tunnel 101
address-family ipv4 vrf forwarding TEST1
exit-address-family tunnel key 101
! tunnel vrf IWAN-TRANSPORT-1
!
interface Tunnel 102 MPLS INET
vrf forwarding TEST2
tunnel key 102
tunnel vrf IWAN-TRANSPORT-1

R10 R11 R12 R13

• DMVPN Tunnel per VRF 1 2 1 2 1 2 1 2
10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
• Over the top routing per VRF
• SAF Peering per VRF
Enterprise Branch Sites
BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
Deploying with VRF – Hub MC
TRANSIT SITE
GLOBAL: 10.8.3.3
VRF TEST1: 11.8.3.3
interface Loopback1 MC1 VRF TEST2: 12.8.3.3
vrf forwarding TEST1
!
interface Loopback2 R84 R85
vrf forwarding TEST2

domain IWAN
vrf TEST1 MPLS INET
master hub
source-interface Loopback1
!
vrf TEST2
master hub
R10 R11 R12 R13
source-interface Loopback2
10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24

Enterprise Branch Sites

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
Deploying with VRF – Hub MC Policies
domain IWAN [Cont’d]
vrf TEST1 vrf TEST2
master hub master hub
load-balance load-balance
class VOICE sequence 10 class VOICE sequence 10
match dscp ef policy voice match dscp ef policy voice
path-preference MPLS fallback INET path-preference MPLS fallback INET
class VIDEO sequence 20 class CRITICAL sequence 30
match dscp af41 policy voice match dscp af31 policy low-latency-
path-preference MPLS fallback INET data
class CRITICAL sequence 30
match dscp af31 policy low-latency-
data

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
Deploying with VRF – Hub BR
TRANSIT SITE

domain IWAN GLOBAL: 10.8.3.3

vrf TEST1 VRF TEST1: 11.8.3.3
MC1 VRF TEST2: 12.8.3.3
border
master 11.8.3.3
source-interface Loopback1 R84 R85
! Tu101
vrf TEST2 Tu102
border
master 12.8.3.3
source-interface Loopback2
! MPLS INET
interface Tunnel101
description -- Primary Path –
vrf forwarding TEST1
domain IWAN path MPLS
!
interface Tunnel102
description -- Primary Path – R10 R11 R12 R13

vrf forwarding TEST2 10.1.12.0/24

10.1.10.0/24 10.1.11.0/24
domain IWAN path MPLS 10.1.13.0/24

Enterprise Branch Sites

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
Deploying with VRF – Branch MC/BR
TRANSIT SITE
GLOBAL: 10.8.3.3
VRF TEST1: 11.8.3.3
MC1 VRF TEST2: 12.8.3.3
domain IWAN
vrf TEST1
master branch R84 R85
source-interface Loopback1
hub 11.8.3.3
border
master local
source-interface Loopback1 MPLS INET
!
vrf TEST2
R10 master branch
source-interface Loopback2 Tu101
hub 12.8.3.3 Tu102
border
R10 R11 R12 R13
master local
source-interface Loopback2 10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24

Enterprise Branch Sites

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
Redundant MC – Anycast IP
TRANSIT SITE
In IWAN POP
Hub MC Backup Hub MC
MC1 10.8.3.3/32 MC2 10.8.3.3/32
• What happens when a MC fails?
• Traffic forwarded based on routing information
– ie no drop R84 R85

• What happens when the Hub MC fails?

• Branch MCs keep their configuration and
policies
MPLS INET
• Continue to optimize traffic
• A backup MC can be defined on the hub.
• Using the same IP address as the primary
• Routing Protocol is used to make sure BRs and
R10 R11 R12 R13
branch MC connect to the primary
10.1.12.0/24
• Stateless redundancy 10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
• Backup MC will re-learn the traffic

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
Dual POPs – Different Prefixes
DCI
IWAN POP1 WAN Core IWAN POP2
• Requirements:
MC1 MC2
– Separate datacenters/POPs 10.8.0.0/16 10.9.0.0/16
– Separate prefix advertised from each datacenters
to spokes R84 R85 R94 R95
• POP2 Hub MC EIGRP/BGP
10.8.0.0/16
EIGRP/BGP
10.8.0.0/16
EIGRP/BGP
10.9.0.0/16
EIGRP/BGP
10.9.0.0/16
– Configured as Branch

MPLS INET

R10 R11 R12 R13

10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
Horizontal Scaling Architecture
TRANSIT SITE
PfRv3 Multiple DMVPN Next Hop Limitation
Hub MC
MC1 10.8.3.3/32
• Limitations:
– PfRv3 manages traffic between Tunnel Interfaces, not
multiple tunnels within a single Tunnel Interface
BR1 BR2 BR3 BR4
– Spokes have multiple next hops on the same DMVPN
tunnel Interface Next Hop 1 Next Hop 2

– Channel definition:
• local site id + remote site id + DSCP + color(SP)
• No differentiation for multiple channels within a color(SP) MPLS INET

• Solution: PfRv3 DMVPN Multiple Next Hop support

– Need to add sub-color to differentiate channels
– New channel definition
• local site id + remote site id + DSCP + color(SP) + SP tag R10 R11 R12 R13
– BR1 with tag 1, BR2 with tag 2 10.1.12.0/24
10.1.10.0/24 10.1.11.0/24 10.1.13.0/24
• Targeted for Spring XE 3.15 / 15.5(2)T releases

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
Multiple POPs – Common Prefixes DC1

DCI
TRANSIT SITE1 WAN Core TRANSIT SITE2
• Requirements:
MC1 MC2
– 2 (or more) Transit Sites advertise the very same 10.8.0.0/16 10.8.0.0/16
set of prefixes 10.9.0.0/16 10.9.0.0/16

– Datacenter may not be collocated with the BR1 BR2 BR3 BR4
Transit Sites
– DCs/DMZs are reachable across the WAN Core
for each Transit Site
– Branches can access any DC or DMZ across DMVPN DMVPN
MPLS INET
either POP(hub). And, DC/DMZs can reach any
branch across multiple Transit Sites (hubs).
– Multiple BRs per DMVPN per site may be
required for crypto and bandwidth horizontal
scaling
R10 R11 R12 R13
• Targeted for Spring XE 3.15 / 15.5(2)T releases 10.1.10.0/24
10.1.12.0/24
10.1.11.0/24 10.1.13.0/24

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
 Monitoring Operations
R83#sh domain one master traffic-classes summary

APP - APPLICATION, TC-ID - TRAFFIC-CLASS-ID, APP-ID - APPLICATION-ID

SP - SERVICE PROVIDER, PC = PRIMARY CHANNEL ID,
BC - BACKUP CHANNEL ID, BR - BORDER, EXIT - WAN INTERFACE
UC - UNCONTROLLED, PE - PICK-EXIT, CN - CONTROLLED, UK - UNKNOWN

Dst-Site-Pfx Dst-Site-Id APP DSCP TC-ID APP-ID State SP PC/BC BR/EXIT

10.1.13.0/24 10.2.13.13 N/A ef 11 N/A CN MPLS 25/27 10.8.4.4/Tunnel100

10.1.12.0/24 10.2.12.12 N/A ef 9 N/A CN MPLS 20/19 10.8.4.4/Tunnel100

10.1.11.0/24 10.2.11.11 N/A af31 6 N/A CN MPLS 24/22 10.8.4.4/Tunnel100

10.1.13.0/24 10.2.13.13 N/A af31 8 N/A CN MPLS 26/28 10.8.4.4/Tunnel100

10.1.12.0/24 10.2.12.12 N/A af31 4 N/A CN MPLS 11/12 10.8.4.4/Tunnel100
10.1.11.0/24 10.2.11.11 N/A default 5 N/A CN INET 8/NA 10.8.5.5/Tunnel200

[SNIP]

Traffic
Total Traffic Class
Classes: 12 –Site:
Site11
6 -Internet:
Critical 0TC Id Controlled Path Information - Channels
R83#

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
 Check Traffic Classes Details
R83#sh domain one master traffic-classes

Dst-Site-Prefix: 10.1.10.0/24 DSCP: ef [46] Traffic class id:25

TC Learned: 00:12:44 ago
Present State: CONTROLLED Check Traffic Class
Current Performance Status: in-policy Voice for site 10
Current Service Provider: INET since 00:06:01
Previous Service Provider: INET for 181 sec
Active Path used
(A fallback provider. Primary provider will be re-evaluated 00:00:01 later)
BW Used: 24 Kbps
Present WAN interface: Tunnel200 in Border 10.8.5.5
Check Channels
Present Channel (primary): 84
used (Primary and
Backup Channel: 85
Destination Site ID: 10.2.10.10 Backup)

Class-Sequence in use: 10
Class Name: VOICE using policy User-defined Policies and
priority 2 packet-loss-rate threshold 5.0 percent
priority 1 one-way-delay threshold 150 msec
reason for last
priority 2 byte-loss-rate threshold 5.0 percent change
BW Updated: 00:00:14 ago
Reason for Route Change: Delay
BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
Check Channel after TCA
On Demand Export
(ODE)
MC1#sh domain IWAN master channels | beg 107
Channel Id: 107 Dst Site-Id: 10.2.11.11 Link Name: MPLS DSCP: ef [46] TCs: 1
Channel Created: 00:15:03 ago
Provisional State: Initiated and open Threshold Crossing
Operational state: Available Alert (TCA)
Interface Id: 11
Estimated Channel Egress Bandwidth: 0 Kbps
Immitigable Events Summary:
Total Performance Count: 0, Total BW Count: 0
ODE Stats Bucket Number: 1 [SNIP]
Last Updated : 00:05:45 ago
Packet Count : 2 TCA Statistics:
Byte Count : 116 Received:1 ; Processed:1 ; Unreach_rcvd:0
One Way Delay : 254 msec* Latest TCA Bucket
Loss Rate Pkts: 0.66 % Last Updated : 00:05:45 ago
Loss Rate Byte: 0.0 % One Way Delay : 266 msec
Jitter Mean : 38000 usec Loss Rate Pkts: NA
Unreachable : FALSE Loss Rate Byte: NA
Jitter Mean : NA
Unreachability: FALSE

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
PfRv3 Management
 PfRv3 Exporter Configuration
IWAN POP

Hub MC
MC1 10.8.3.3/32
domain IWAN
vrf default
BR1 BR2
master hub
collector 10.151.1.95 port 2055

MC1
MPLS INET

• Enable exporter on the Hub MC

– Collector IP address
– Default UDP port 9995
R10 R11 R12 R13
• Distributed through SAF to all MCs and BRs in 10.1.12.0/24
10.1.10.0/24 10.1.11.0/24
the domain 10.1.13.0/24

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
PfRv3 NetFlow Export
List of Templates
• Exports from MC
– TCA record
– Route Change record
– Immitigable Event Summary
• Bandwidth
– Exports from BRs
– Egress Measurement Template
– Ingress Measurement Template
• All records available at:
– http://docwiki.cisco.com/wiki/PfRv3:Reporting

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
Cisco IWAN Management
On-Prem Management Specialized Management Cloud-Based Management

Prime
Infrastructure
2.2
End-to-End Assurance of Application Application Aware Network Automates Deployment
Experience Performance Management and Lifecycle Management

• Single-pane view of IWAN • Integrates with Cisco AVC and PfR • Eliminates manual building of WANs
• IWAN deployment workflows • Monitor and analyze application traffic • Automated SD-WAN orchestration
• Plug and Play • End-to-end flow visualization • Centralized hybrid WAN management
• DMVPN, QoS, AVC deployment and • Flow & App-based Troubleshooting • Quick config updates and IOS upgrades
monitoring • Fix and Verify in Realtime • Leverages onePK and REST APIs
• PfR v3 in Q1 2015
• License includes IWAN App and APIC-
EM controller!
BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
LiveAction 4.1

1. Alert Workflow
1. PfR path change
visualization

2. Alert and report on PfR

Out of Policy events

3. Click on Alerts to get the

details

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
 PfRv3 TCA Alerts on DC1
Drill down to site pairs

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
 2. PfRv3 Dashboard

All Alerts

Alerts / performance
by Site

Alerts / performance
by Application Group

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 97
 Alerts / performance
by Service Provider

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 98
IWAN Automation and Orchestration Evolution
Capacity Planning, Troubleshooting,
Prime
Change control

Cisco IWAN Apps Mid Partners (future)

Traditional CY2015
PnP
Management IWAN PKI
Provisionin
Transport Automation Apps
Systems g

Intelligent Application
Security
Path Control Experience
Cisco Prime

Evolution
REST APIs
APIC-EM Services (Partial)

PKI NetFlow Network Events Inventory PnP APIC-

Svc Svc Svc Svc Svc Svc EM

Device Abstraction Layer

OnePK/OpenFlow CLI

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 99
Key Takeaways
Performance Routing – IOS and IOS-XE Releases
PfR/OER version 1 PfR version 2 PfR version 3
IOS 12.3(8)T, XE 2.6.1 IOS 15.2(3)T, IOS-XE 3.6 IOS 15.4(3)M, IOS-XE 3.13
Per Device provisioning Per Device provisioning PfR Domain
Passive monitoring with Traditional Target Discovery (TD) One touch provisioning
NetFlow (TNF) Automatic provisioning of jitter probes Auto Discovery of sites
Active monitoring with IP SLA Passive monitoring with Traditional NBAR2 support
Manual provisioning jitter probes NetFlow (TNF) Passive Monitoring (performance
1000’s lines of configuration (pfr-map Active monitoring with IP SLA monitor)
per site) 10’s lines of configuration Smart Probing
VRF Awareness
IPv4/IPv6 (Future)
<10 lines of configuration and
centralized
Blackout 6 seconds Blackout 6 seconds Blackout ~ 2 sec
Brownout 9 seconds Brownout 9 seconds Brownout ~ 2 sec
Limited scalability due to provisioning Scale 500 sites Scale 2000 sites
(~ tens of sites)

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 101
Performance Routing – Platform Support

Cisco CSR-1000

MC
Cisco ASR-1000 BR*

Cisco ISR 4000 MC

BR
4400
Cisco ISR G2 family 4300
3900-AX
2900-AX MC
1900-AX BR
890
MC
BR

* BR support coming
BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
Key Takeaways
• IWAN Intelligent Path Control pillar is based upon Performance
Routing (PfR)
– Maximizes WAN bandwidth utilization
– Protects applications from performance degradation
– Enables the Internet as a viable WAN transport
– Provides multisite coordination to simplify network wide provisioning.
– Application-based policy driven framework and is tightly integrated with
existing AVC components.
– Smart and Scalable multi-sites solution to enforce application SLAs while
optimizing network resources utilization.

• PfRv3 is the 3rd generation Multi-Site aware Bandwidth and Path

Control/Optimization solution for WAN/Cloud based applications.
– Available this summer on ASR1k, 4451-X, ISR-G2

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 103
More Information
• Cisco.com IWAN/PfR Page:
– http://www.cisco.com/go/iwan
– http://www.cisco.com/go/pfr

• PfRv3 Home Page

– http://docwiki.cisco.com/wiki/PfRv3:Home

• Leverage dcloud.cisco.com virtual labs

• LiveAction:
– 1 year free license for Cisco employees
– The New LiveAction 4.1: http://liveaction.com/new-liveaction-4-1/
– Download LiveAction v4.1.2: http://liveaction.com/download/links/
– LiveAction PfRv3 Demo: http://player.vimeo.com/video/103767237
– LA 4.1 IWAN Webinar, October 2nd 2014: http://liveaction.com/webinars/oct-02-
2014/?elq=d49b1ff6a68b43a7a4f8e81baa8bc801&elqCampaignId=172

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 104
Call to Action
• Visit the World of Solutions for
– Cisco Campus
– Walk in Labs
– Technical Solution Clinics
• Meet the Engineer
• Lunch time Table Topics
• DevNet zone related labs and sessions
• Recommended Reading: for reading material and further resources for this
session, please visit www.pearson-books.com/CLMilan2015

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 105
Complete Your Online Session Evaluation
• Please complete your online session
evaluations after each session.
Complete 4 session evaluations
& the Overall Conference Evaluation
(available from Thursday)
to receive your Cisco Live T-shirt.

• All surveys can be completed via

the Cisco Live Mobile App or the
Communication Stations

BRKRST-2362 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 106