IOS XR

Software Architecture

XR Training / lwigley

2005 Cisco Systems, Inc. All rights reserved.

Cisco Internal/Confidential

IOS XR Architecture Agenda

Core OS Capabilities
Protected process memory space
Preemptive multitasking
Dynamically Linked Libraries (DLLs)

Modular Software
High Availability
Process Restart
ISSU
MDR

LPTS

XR Training / lwigley

2006 Cisco Systems, Inc. All rights reserved.

Cisco Internal/Confidential

Protected Process Memory Space

Each process has a virtual memory space
Kernel/MMU maps virtual address to physical address (at page level)
Threads share the memory space

One process cannot corrupt anothers memory

Process can only access virtual space
In IOS all processes shared same virtual space

Communication between processes via controlled APIs

Limited use of shared memory

0x000000
0x100000
0x200000
0x300000

OSPF

0x400000

0x00000
0x10000

0x500000

0x20000

0x600000
0x700000
0x800000
0x900000
0xa00000

XR Training / lwigley

2006 Cisco Systems, Inc. All rights reserved.

Cisco Internal/Confidential

Preemptive Multitasking
Sleeping

Default priority is 10
Higher priority processes can interrupt

Waiting

In IOS, must wait for running process to finish

FIFO within same priority

Threads run while parent process is running
CRS-1/16 and DRP have two CPUs
Running

10

50

50

16

62

10

50

10

50

Ready

62

50
10 10

16

XR Training / lwigley

50

50

2006 Cisco Systems, Inc. All rights reserved.

50

16

16

10

10

16

50

10

16

50

62

50

16

10

16

62

10

Cisco Internal/Confidential

Preemptive Multitasking

High priority wakes up and is ready to run

Priority 62 process wakes up

10

50

16

62

Ready

50
10

50

10

10
50

(timer expired)

62

50
10 10

16

XR Training / lwigley

50

50

2006 Cisco Systems, Inc. All rights reserved.

50

16

16

10

10

16

50

10

16

50

62

50

16

10

16

62

10

Cisco Internal/Confidential

Preemptive Multitasking

High priority process gets immediate CPU time

No wait since lower priority process is running

10

50

16

62

No waiting!

50
10

50

10

10
50

62

50
10 10

16

XR Training / lwigley

50

50

2006 Cisco Systems, Inc. All rights reserved.

50

16

16

10

10

16

50

10

16

50

62

50

16

10

16

62

10

Cisco Internal/Confidential

Preemptive Multitasking

Priority 10 process back on ready queue

Process that was running goes back onto queue

10

50

50

16

62

10

50

10

50

62
62

50

10

16

XR Training / lwigley

50

50

2006 Cisco Systems, Inc. All rights reserved.

50

50

16

16

10

10

16

50
10 10

16

62

50

10

16

16

10

Cisco Internal/Confidential

Sporadic Scheduling 3.4.x

Special handling for known CPU hogs

Some processes known to consume many cycles
May be assigned with two priorities
Allowed to run at the higher priority part time
High Priority
runtime consumed

62,50

10

10

High Priority
runtime remaining

50

16

62,50

10

50

10

10
50

62

50
10 10

16

XR Training / lwigley

50

50

2006 Cisco Systems, Inc. All rights reserved.

50

50

16

16

10

10

16

50

10

16

50

62

50

10

16

16

10

Cisco Internal/Confidential

Example Process Priorities

RP/0/RP0/CPU0:RTPTME-CRS#show process
JID
TID Stack pri state
HR:MM:SS:MSEC
80
1
12K 63 Nanosleep
0:00:00:0006
57
1
28K 10 Receive
0:00:03:0058
57
2
28K 10 Nanosleep
0:00:00:0048
57
3
28K 10 Receive
0:00:02:0527
57
4
28K 10 Receive
0:00:04:0537
57
5
28K 10 Receive
0:00:05:0847
281
1
40K 10 Receive
0:01:19:0038
281
2
40K 10 Receive
0:00:00:0000
281
5
40K 10 Receive
0:00:00:0002
281
7
40K 10 Nanosleep
0:43:16:0203
71
2
36K 10 Sigwaitinfo
0:00:00:0002
69
4
56K 12 Intr
0:00:00:0000
78
7
36K 10 Condvar
0:00:00:0000
59
2
56K 10 Sem
0:02:13:0504
67
2
24K 10 Receive
0:00:00:0000
67
3
24K 10 Nanosleep
0:00:00:0000
59
1
56K 50 Receive
0:11:35:0873
59
2
56K 10 Sem
0:02:13:0508
559
5
56K 10 Sem
0:00:00:0000
59
6
56K 10 Receive
0:00:00:0000
59
7
56K 55 Receive
0:32:30:0824
54
1
64K 10 Receive
0:00:00:0034
XR Training / lwigley

2006 Cisco Systems, Inc. All rights reserved.

Highest

NAME
Priority
wd-mbi
dllmgr
dllmgr
dllmgr
dllmgr
dllmgr
qsm
qsm
qsm
qsm
pkgfs
pcmciad
syslogd_helper
eth_server
nvram
nvram
eth_server
eth_server
eth_server
eth_server
eth_server
bfm_server
Cisco Internal/Confidential

Dynamically Linked Libraries (DLLs)

Reduces overall memory usage
Only load active libraries into memory
Processes share the library code

libtrace

libinfra

libinfra

libxyzzy

libsyslog

libsyslog

libplatform

libplatform

libinfra

libnodeid

libnodeid

libsyslog

libospf_error

libospf_error

libplatform

libbgp_vrf

libbgp_vrf

libbgp_policy

libbgp_policy

libbgp_rib

libbgp_rib

DLLs unloaded when no longer needed

No virtual memory / swapping
libtrace

libnodeid

Still not used

BGP DLLs loaded

libospf_error

RP Memory before BGP Configured

XR Training / lwigley

libtrace

2006 Cisco Systems, Inc. All rights reserved.

RP Memory after BGP

Disk0

Cisco Internal/Confidential

10

Example - Loading BGP DLL

RP/0/5/CPU0:GSR#show dll | inc bgp
/disk0/c12k-rout-3.3.0/lib/cerrno/libbgp_error.dll 0xfcb95000 0x0000199c 0x00000088
1
/disk0/c12k-rout-3.3.0/lib/cerrno/libbgp_error2.dll 0xfcb9b000 0x00001a5c 0x00000088
1
/disk0/c12k-rout-3.3.0/lib/cerrno/libbgp_policy_ora_error.dll 0xfcba8000 0x00000088
2
/disk0/c12k-rout-3.3.0/lib/libbgp_policy_ora_v1_0_0.dll 0xfd136000 0xfd172000 0x00000988
1
/disk0/c12k-rout-3.3.0/lib/libbgp_damp_ora_v1_0_0.dll 0xfd18b000 0x0000efb4 0x00000868
1
RP/0/5/CPU0:GSR#show dll mem
---------------------------------------------------------------------------Total DLL Text - 18333980 bytes Total DLL Data - 12491748 bytes
Total DLL Memory - 30825728 bytes
RP/0/5/CPU0:GSR#conf t
RP/0/5/CPU0:GSR(config)#router bgp 100
RP/0/5/CPU0:GSR(config-bgp)#commit
RP/0/5/CPU0:GSR(config-bgp)#end
RP/0/5/CPU0:GSR#show dll mem
---------------------------------------------------------------------------Total DLL Text - 19744580 bytes Total DLL Data - 13162704 bytes
Total DLL Memory - 32907284 bytes
RP/0/5/CPU0:GSR#show dll | inc bgp
/disk0/c12k-rout-3.3.0/lib/libbgp_util_common.dll 0xfc04d000 0x00011664 0x0000094c
2
/disk0/c12k-rout-3.3.0/lib/libbgp_edm_util.dll 0xfc05f000 0x000084e8 0xfc068000 0x00000748
2
/disk0/c12k-rout-3.3.0/lib/libbgp_vrf_db.dll 0xfc069000 0x0000d1b0 0xfc077000 0x00003000
2
/disk0/c12k-rout-3.3.0/lib/libbgp_vrf_error.dll 0xfc0761b0 0x00000d4c 0xfc068748 0x00000088
1
/disk0/c12k-rout-3.3.0/lib/libbgp_af_lib.dll 0xfc07a000 0x00006488 0xfc03194c 0x00000688
2
/disk0/c12k-rout-3.3.0/lib/libbgp_vrf_import.dll 0xfc081000 0x000070c4 0xfc089000 0x00018000
2
/disk0/c12k-rout-3.3.0/lib/libbgp_debug.dll 0xfc0a1000 0x0000933c 0xfc0ab000 0x000009d8
2
/disk0/c12k-rout-3.3.0/lib/libbgp_vrf_config.dll 0xfc0ac000 0x0000d080 0xfc0ba000 0x00000908
1
/disk0/c12k-rout-3.3.0/lib/libbgp_sysdb_util.dll 0xfc0c3000 0x0000572c 0xfc068858 0x00000648
2
/disk0/c12k-rout-3.3.0/lib/libbgp_af.dll 0xfc0c9000 0x0000b438 0xfc0bd000 0x00000838
1
/disk0/c12k-rout-3.3.0/lib/libbgp_table.dll 0xfc0d5000 0x0001cef4 0xfc0be000 0x00000aac
1
/disk0/c12k-rout-3.3.0/lib/libbgp_nbr.dll 0xfc0f2000 0x0000759c 0xfc0ba908 0x000006e8
1
XR Training / lwigley

2006 Cisco Systems, Inc. All rights reserved.

Cisco Internal/Confidential

11

Process Restart

Microkernel includes minimal functionality

Non-kernel processes can be restarted
Critical to HA, ISSU, and MDR functions
Restarting many processes can be tricky
Dependent processes may also need to restart

XR Training / lwigley

2006 Cisco Systems, Inc. All rights reserved.

Cisco Internal/Confidential

12

Process Restart

Microkernel architecture enables restart of most processes

Green areas
cannot restart

BGP

OSPF

BGP RIP OSPF

BGP

OSPF

EIGRP

ISIS

LDP VPN ISIS

EIGRP

ISIS

RIP

VPN

RIP

VPN

SSH

Telnet
Server

SSH

Telnet
Server

LDP

ACLs

LDP

ACLs

IPv4 Forwarding

Telnet
Server

ACLs
IPv4 Forwarding

TCP/IP

Drivers

TCP/IP

Drivers

Timers

Scheduler

Timers

Scheduler

Monolithic
IOS

XR Training / lwigley

SSH

2006 Cisco Systems, Inc. All rights reserved.

Kernel

BSD based routers

IPv4 Forwarding
TCP/IP
Timers

Drivers
Scheduler

Microkernel
IOS XR

Cisco Internal/Confidential

13

Modular Software
Software is divided into packages groups of files
Allows modular install and upgrades
Upgrade may replace one file or everything
MGBL

SEC

MPLS

MCAST

Routing
Line card
Forwarding
Admin
Base
OS-MBI

XR Training / lwigley

2006 Cisco Systems, Inc. All rights reserved.

Cisco Internal/Confidential

14

Kernel Functions
Memory management
Allocate memory to processes
Enforce access controls

Monitor processor utilization

Messaging
Provide a messaging service between processes via APIs

XR Training / lwigley

2006 Cisco Systems, Inc. All rights reserved.

Cisco Internal/Confidential

15

In Service Software Upgrade (ISSU)

Changing software with no impact to transit traffic
What customers expect
Maintenance release upgrades without impact
Major release upgrades to/from any version without any impact

What we have today

Some SMUs with limited scope will ISSU on same RP
Other upgrades may require node or chassis reset
Major releases
Maintenance releases
Complex SMUs

XR Training / lwigley

2006 Cisco Systems, Inc. All rights reserved.

Cisco Internal/Confidential

16

Minimally Disruptive Restart (MDR)

Reduces forwarding disruption during software upgrade
MSC hardware keeps forwarding while MSC software resets
RP spoofs packets normally generated by LC CPU
FR LMI
ATM OAM
PPP
HDCL

XR Training / lwigley

2006 Cisco Systems, Inc. All rights reserved.

Cisco Internal/Confidential

17

LPTS Interoffice Mail for Data Plane

DRP

RP

LPTS

LC
XR Training / lwigley

LC
2006 Cisco Systems, Inc. All rights reserved.

LC
Cisco Internal/Confidential

18

LPTS Protects RPs and LC CPUs

Only forwards packets to registered ports

RP
LC
LC
LC
LC
LC
LC
LC

XR Training / lwigley

2006 Cisco Systems, Inc. All rights reserved.

Cisco Internal/Confidential

19

LPTS Example - BGP

RP/0/RP0/CPU0:RTPTME-CRS#show lpts pifib hardware entry loc 0/2/CPU0

L4 Protocol
: TCP
VRF ID
: default
Source IP
: 8.1.1.2
Port/Type
: Port:179
Source Port
: any
Is Fragment
: 0
Is SYN
: any
Interface
: any
V/M/C/L/T/F
: 1/0/1/IPv4_LISTENER/0/BGP-cfg-peer
DestNode
: 0/RP0/CPU0
DestAddr
: 30
Sq/Dq/Ct
: 24/6/0x24456
Accepted/Dropped : 0/0
Lp/Sp
: 3/255
# of TCAM entries : 1
Po/Ar/Bu
: 105/10000pps/100ms
State
: Entry in TCAM
Rsp/Rtp
: 13/13

XR Training / lwigley

2006 Cisco Systems, Inc. All rights reserved.

Cisco Internal/Confidential

20

Dynamic LPTS Example

New BGP Peer Configured (but not up)

router bgp 100
address-family ipv4 unicast
!
neighbor 192.1.1.2
remote-as 200
address-family ipv4 unicast
route-policy pass-all in
route-policy pass-all out

L4 Protocol
VRF ID
Source IP
Port/Type
Source Port
Is Fragment
Is SYN
Interface
V/M/C/L/T/F
DestNode
DestAddr
Sq/Dq/Ct
Accepted/Dropped
Lp/Sp
# of TCAM entries
Po/Ar/Bu
HPo/HAr/HBu
State
Rsp/Rtp
XR Training / lwigley

:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:

There will be another entry with 179 as the

source port, for situations where we open
the session.

TCP
default
192.1.1.2
Port:179
any
0
any
any
1/0/1/IPv4_LISTENER/0/BGP-cfg-peer
0/RP0/CPU0
30
24/6/0x24454
0/0
1/255
1
105/10000pps/100ms
105/10000pps/100ms
Entry in TCAM
2/2

2006 Cisco Systems, Inc. All rights reserved.

Cisco Internal/Confidential

21

Dynamic LPTS Example

BGP Session Up Values Change

RP/0/RP0/CPU0:RTPTME-CRS#show bgp sum
Neighbor
192.1.1.2

Spk
0

L4 Protocol
VRF ID
Source IP
Port/Type
Source Port
Is Fragment
Is SYN
Interface
V/M/C/L/T/F
DestNode
DestAddr
Sq/Dq/Ct
Accepted/Dropped
Lp/Sp
# of TCAM entries
Po/Ar/Bu
HPo/HAr/HBu
State
Rsp/Rtp
XR Training / lwigley

:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:

AS MsgRcvd MsgSent
200
19
47

TblVer
1

InQ OutQ Up/Down St/PfxRcd

0
0 00:08:36
0

TCP
default
192.1.1.2
Port:25689
179
0
any
any
0/0/0/IPv4_LISTENER/0/TCP-listen
0/RP0/CPU0
30
24/7/0x24456
23/0
1/255
1
123/25000pps/100ms
123/25000pps/100ms
Entry in TCAM
1/1

2006 Cisco Systems, Inc. All rights reserved.

Cisco Internal/Confidential

22

XR Training / lwigley

2006 Cisco Systems, Inc. All rights reserved.

Cisco Internal/Confidential

23