You are on page 1of 34

Automatic Test Packet

Generation
Hongyi Zeng , Peyman Kazemian , Nick McKeown
University, Stanford, CA, USA
George Varghese
UCSD, La Jolla
Microsoft Research, Mountain View, CA, USA
https://github.com/eastzone/atpg/wiki

Debugging the net
Troubleshooting the network is difficult.

2

Troubleshooting the network is difficult.
 Survey of NANOG mailing list
 Data set: 61 responders:
 23 medium size networks (<10K hosts),
12 large networks (< 100K hosts).
 Frequency: 35% generate >100 tickets per month.
 Downtime: 56% take over an half hour to resolve.
 Current tools: Ping (86%), Traceroute (80%).
 70% asked for better tools, automatic tests.
3

and pinpoint faults. congestion and performance. Goal: automatically generate test packets to test the network.Automatic Test Packet Generation Framework that test the liveness . 4 .

FIB’s . T Using the header space analyses we can collect the entire packet range from the net. topology 5 Header Space Analyses Equivalent classes & packets .ATPG use the header space framework.

Represent by port and header. generate a list of output packets. 6 .models a network. Transfer function T. Define how packet field are modified.

applying the T and  repeatedly till the packet reach her destination.ATPG use the header space framework. 7 . ordered list of rules the packet match so far as it travers the net. Transfer function  models the network by specifying which pairs of ports are connected.

 ATPG use only available test terminals. 8 .Test packet generation Assume there is a set of terminals in the network can send and receive test packets. Goal : generate a set of test packets to exercise every rule.  ATPG use headers that test terminal is permitted to send.

Test packet generation Test packet generator Test Pkt Header space analysis DB All pairs reachability T function parser Topology . FIB . ALC … 9 Test terminals .

Test packet Selection Determine the set of packet headers that can be sent from each terminal to another. Sent periodically the test . use localization algorithm.number test terminals D . Generate all pairs reachability table. Sample and compression.average number of Rules in box .network diameter R . 10 T . equivalent classes. If a error is detected .

Find all packet space T 11 T T .

0/16.2 0 A 2 0 1 1 B 2 0 C 1 Header dst_ip = 10. tcp = 80 12 dst_ip = 10.2/16 Ingress Port Egress Port Rule History .

2/16 dst_ip = 10.1/16 dst_ip = 10.2/16 dst_ip = 10.0/16.2/16 0 Ingress Port 1 1 2 0 dst_ip = 10. A tcp = 80 dst_ip = 10.2/16.1/16 13 2 Port Egress C 1 B 0 History Rule . tcp = 80 dst_ip = 10.2Header dst_ip = 10.

Header dst_ip = 10.  You can peak randomly or peak all the headers from a certain class to check a fault for a specific header.1/16 dst_ip = 10.2/16.2/16 dst_ip = 10. tcp = 80 14 dst_ip = 10.Sampling  Pick at least one packet from every equivalent class. tcp = 80 dst_ip = 10.0/16.2/16 dst_ip = 10.2/16 dst_ip = 10.1/16 Ingress Port Egress Port Rule History .

2/16 dst_ip = 10.0/16.2/16 dst_ip = 10.1/16 Ingress Port Egress Port Rule History . Header dst_ip = 10.2/16 dst_ip = 10.  The remaining test packet will be use later for fault localization. tcp = 80 15 dst_ip = 10.2/16.Compression  Using the Min-Set Cover algorithm. tcp = 80 dst_ip = 10.1/16 dst_ip = 10.

Fault Localization  Success and failure depend on the nature of the rule forwarding / drop.effect all the packets matching the rule.effect some of the packets matching the rule. Match fault .  Divide the fault in two categories: Action fault . 16 .

F .2/16suspect dst_ip = 10.2/16 rules 17 Result pass Minimize the suspect rules set Report the set fail fail .Fault Localization Find the set of suspect rules P is the set of all the rules traversed by success test packets.P is the set of suspect rules.0/16.  Minimize the suspect rules set P F We will make use of our reserved packets. tcp = 80 Find the dst_ip = 10. Header Ingress Port Egress Port Rule History dst_ip = 10. F is the set of all the rules traversed by failing test packets.

Can be solve by adding more test terminals to the network.False positive There is a possibly for false positive . 18 . Rule that are left as suspect although it behave correctly.

Test packet generator Test Pkt Fault Localization Header space analysis DB All pairs reachability T function parser Topology . ALC … Test terminals 19 . FIB .

 Congestion and available bandwidth:  Measure the latency . Forward rule and Drop rule : if packet exercises the rule. 20 .  Test performance of the network. Link rule : if packet passing the link without header modification. bandwidth.Use case for ATPG  Test functional correctness of a network.

Drop rule C Fwd A Broadcast A drop HeaderC C A Header R3 C A R1 R2 HeaderB B A Header R4 B B Fwd 21 .

 The data plane information Collected and parsed (Cisco IOS) .  The generator use Hassel header space analysis library to construct T functions.  The process uses JSON to communicate with the test agents.Implementation  ATPG written in python.  All pairs reachability is computed using multiprocess parallel processing  After Min set cover executes the output stored in SQLite database. 22 .

SDN controller 23 .Alternate implementation & SDN In SDN. the controller could directly instruct the switch’s to send test packets and to forward test packet back to the controller.

Testing in Emulated Network Replicated the Stanford backbone network in Mininet. Forwarding Error 24 Congestion Available Bandwidth Priority . real port configuration and real topology. Using open vSwitch .

Testing in Emulated Network 25 .

Forwarding Error bbra s1 goza 26 s2 coza s5 s4 poza boza .

Forwarding Error bbra s1 goza 27 s2 coza s5 s4 poza boza .

Congestion bbra s1 pozb 28 s2 poza yoza roza .

s1-bbra.bbra-roza .s2-bbra.s1-bbra.bbra-yoza pzob roza Pozb-s1.bbra-yoza roza yoza Roza-s2.bbra s1 poza pozb 29 s2 yoza roza Ingress Port Egress Port Rule History poza yoza Poza-s1.bbra-yoza pzob yoza Pozb-s1.s1-bbra.

30 . and conducts a ping test every 10 seconds.Production Network Deployment Deployed an experimental ATPG system in Stanford University. For test terminal they use 53 Wi-Fi access. Test agent download the test packets from a central web server every ten minutes .

Production Network Deployment During the experiment . a major outage occurred .ATPG has captured the problem. 31 .

32 .Evaluation Evaluated the system on Stanford backbone. 700000 4500 600000 4000 500000 3500 3000 400000 2500 300000 Packets Test Packets 2000 comp time 1500 200000 1000 100000 500 0 -100000 0 20 40 60 80 100 120 0 -500 0 20 40 60 80 100 120 testing all rules 10 times per second needs < 1% of link overhead.

. Sampling time • ATPG uses sampling to collect the forwarding state. Non deterministic boxes • internal state can be changed.Limitation 33 Dynamic . • random to route packets. Invisible rule • backup rules changes may be occurred without been notice.

Questions? 34 . • < 1% of link overhead.Conclusion • ATPG tests liveness and performance. • Works in real networks.