Ethernet Performance Measurement

Ethernet Performance Measurement
on an MPLS platform
Michiel Appelman 2011
Ethernet Performance Measurement

on an MPLS platform
By: Michiel A PPELMAN
Supervisor: Attilla DE G ROOT
January 14, 2011
Summary
The Amsterdam Internet Exchange (AMS - IX) is an organization to facilitate the exchange of IP trafc between its participants and is set up as a not for prot membership based association. The members consist of large ISPs, mobile operators, content providers and other communication companies. AMS - IX has been searching for a way to measure values like delay and jitter over this platform, after a need for a new service called Inter-IP eXchange (IPX) arose. Mobile operators are planning to use this new exchange structure to separate their communication-specic data from the regular internet. The IPX standard requires Service Level Agreements ( SLA s) to be met by all parties involved in providing end-to-end connections, including AMS - IX as an trafc exchange platform. To provide its customers with a reliable network and prove that the SLA will be met, AMS - IX must be able to collect the statistics of the network. Because the network uses a combination of Ethernet, MPLS and VPLS in exchanging trafc, traditional higher level Operations, Administration and Maintenance (OAM) utilities could not be used to measure its performance reliably. At the moment there is one new standard dening measurement of the various Key Performance Indicators (KPIs) over Ethernet and is developed by the ITU - T in coordination with the IEEE. Its referenced by Y.1731 and is commonly known as Ethernet OAM Performance Measurement. With this knowledge, a tting hardware solution supporting Performance Measurement was searched for among different vendors. Accedian Networks already provided AMS IX with some test Network Interface Devices ( NID s) to set a baseline. Other devices were looked at as well and a list of requirements was compiled. Eventually the test devices by Accedian were the only models which could pass all criteria and had a performance that would be of use on the AMS - IX platform. Accedian was the choice as a manufacturer and the rst devices for 1 Gigabit Ethernet (GigE) connections were ordered and installed. While testing these new devices it became clear that an important external part of the measurements is the synchronisation of the clocks between them. The slightest difference between clocks will result in a signicant variation in the one way delay values, the only KPI inuenced by this. Because the performance of the platform is of such a high level, differences in milliseconds are not acceptable. Using the Accedian NIDs the clocks can be set to within a couple of microseconds but a new external time source will be needed to provide them with a valid time. A search for this new NTP-server is still on-going. The data collected by the devices using ITU - T Y.1731 over the platform is graphed to provide the customers and the company itself with easy to interpret images. This is done by polling all the devices every minute through SNMP, retrieving their average values and graphing them. The images are then displayed in a web portal where users can select source and destination of the measurements. The stored values are consolidated over time and at the end of each month the data is processed and the values are i
then reported to customers through a 95th percentile. At the moment the measurements on the platform are all operational and the values are being collected. The Inter-IPX VLAN has yet to have customers connect to it but the data is enough to give AMS - IX itself a valuable insight in their platform. The Inter-IPX service will be provided in the coming months and SLAs will also be made available to other customers who wish to have such a contract. All this is enabled by the relatively new ITU - T Y.1731 standard and the NIDs manufactured by Accedian. The remaining tasks to make the whole measurement system complete, are the implementation of new and reliable time sources and the expansion of the devices to also include 10 GigE connections to the network. Doing so will provide a full picture of the network, from and to every switch. Both recommendations will be implemented in the rst quarter of this year.
ii
Contents
Summary 1 2 Introduction Background 2.1 AMS - IX . . . . . . . . 2.1.1 The platform 2.1.2 Organisation 2.2 Inter-IPX . . . . . . . i 1 3 3 3 4 4 6 6 6 7 8 8 8 9 9 9 10 10 11 11 12 12 12 16 16 16 18 20 20 20 23 23 24 25 27 28
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
Project Details 3.1 Service Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . Operations, Administration and Maintenance 4.1 Ethernet OAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 IEEE 802.3ah . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 IEEE 802.1ag . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.1 Maintenance Domains and Levels . . . . . . . . . . . . . . . . . . 4.3.2 Maintenance Association and Maintenance End Point . . . . . . . . 4.3.3 Inter-Domain Operations, Administration and Maintenance (OAM) 4.4 ITU - T Y.1731 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Performance Assurance Agent . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Research 5.1 Probe Hardware . . . . . . . . . . . . . . 5.1.1 Vendors . . . . . . . . . . . . . . 5.1.2 Comparison . . . . . . . . . . . . 5.2 Platform Behavior . . . . . . . . . . . . . 5.2.1 Multicast Frames . . . . . . . . . 5.2.2 Label Switched Paths . . . . . . . . 5.2.3 Redundancy . . . . . . . . . . . . 5.3 Timing . . . . . . . . . . . . . . . . . . . 5.3.1 Hardware Timestamping in NTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Implementation 6.1 Hardware . . . . . . . . . . . . . . . . . . . . . . . . 6.1.1 Network Interface Device (NID) Conguration 6.2 Data Collection . . . . . . . . . . . . . . . . . . . . . 6.2.1 Scripts . . . . . . . . . . . . . . . . . . . . . . 6.3 Timing Sources . . . . . . . . . . . . . . . . . . . . . iii
Conclusion 7.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
31 31 32 32
Appendices A Performance measurement infrastructure project A.1 Introduction and Purpose of the project . . . . . . . . . . . . . . . . . . . A.2 Stage 1: Measurements and reporting on Key Performance Indicators (KPIs) between a selected subset of the AMS - IX access routers. . . . . . . . . . . A.3 Stage 2: Measurements and reporting on KPIs between a selected subset of the AMS - IX access routers . . . . . . . . . . . . . . . . . . . . . . . . . . B Plan van Aanpak B.1 Inleiding . . . . . . . . . . . . . B.1.1 AMS - IX . . . . . . . . . . B.1.2 Inter-IPX . . . . . . . . . B.2 Doelstelling . . . . . . . . . . . B.3 Projectopdracht . . . . . . . . . B.4 Activiteiten . . . . . . . . . . . . B.5 Producten . . . . . . . . . . . . B.5.1 Plan van Aanpak . . . . B.5.2 Planning . . . . . . . . . B.5.3 Onderzoek Apparatuur B.5.4 Verslag en Presentatie . B.6 Randvoorwaarden . . . . . . . B.6.1 Randvoorwaarden . . . B.7 Kwaliteit . . . . . . . . . . . . . B.8 Kosten/baten overzicht . . . . B.8.1 Kosten . . . . . . . . . . B.8.2 Baten . . . . . . . . . . . B.9 Risico analyse . . . . . . . . . . B.9.1 Tijdnood . . . . . . . . . B.9.2 Extra BV . . . . . . . . . B.9.3 Expertise . . . . . . . . . B.10 Planning . . . . . . . . . . . . . C Conguration Example D Graph Generating Bash Script E Acronyms F Bibliography iv . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 34 35 36 37 37 37 37 37 38 38 39 39 40 40 40 40 40 41 41 41 41 41 41 41 42 42 45 47 49 51
List of Figures
1 2 3 4 5 6 7 8 9 10 11 12 Topology of the Amsterdam Internet Exchange (AMS - IX) platform The GPRS Roaming eXchange (GRX)-structure . . . . . . . . . . . . Structure of Ethernet OAM . . . . . . . . . . . . . . . . . . . . . . Inter-Domain structure in OAM . . . . . . . . . . . . . . . . . . . Overview of a Continuity Check Message Ethernet OAM frame. . . The Label Switched Paths (LSPs) over the AMS - IX platform . . . . Structure of an NTP packet. . . . . . . . . . . . . . . . . . . . . . Synchronisation process between a client and the NTP-server. . The path that an NTP-packet travels through the OSI-layers. . . . Graph generated by Round Robin Database (RRD)tool . . . . . . . Overview page KPI measurements . . . . . . . . . . . . . . . . . Overview of a One Way Delay Measurement Ethernet OAM frame. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 5 9 10 17 19 21 22 22 27 28 29
List of Tables
1 2 3 4 5 Comparison of different vendors . . . . . . . . . . . . . . Device conguration . . . . . . . . . . . . . . . . . . . . . OAM threshold values . . . . . . . . . . . . . . . . . . . . Object Identiers (OIDs) for various KPIs . . . . . . . . . . Comparison of Network Time Protocol (NTP) time sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 23 25 26 30
Introduction
Chapter 1
Introduction
This report is a product of a research project that was commissioned by the Amsterdam Internet Exchange (AMS - IX), a provider of an Internet Protocol (IP) trafc exchange platform. A signicant number of companies connect to this platform such as Internet Service Providers (ISPs), content providers, mobile operators, etc. They connect to one of the eight Point of Presences (POPs) in the Amsterdam area which enables them to establish peering sessions with other members and exchange trafc with them. Reason Mobile operators connect to each other through GPRS Roaming eXchange (GRX) providers to provide authentication to their customers when they are roaming on another network. A trend within the telecommunications industry is to consolidate all trafc into one transport mode, be it voice, data, video or any other format.[1] This is called the Next Generation Network (NGN) and operators are now looking at connecting with each other over a network which supports IP-trafc, called IP eXchange (IPX). AMS - IX role in this network would be to provide IPX-providers with a similar exchange platform as is now in place for GRX providers. This project was started while developing this new service that would require Service Level Agreements (SLAs) to guarantee delivery of this vulnerable trafc. The values of these SLAs need to be measured to provide the customers and AMS - IX itself with a clear overview of the performance of the network. Research Questions This research paper will need to provide answers to the following questions: 1. How can the performance of an OSI-layer 2 platform, such as provided by AMS - IX to its customers, be measured? 2. How can AMS - IX provide their customers with the measured values and include this data in an SLA? Structure Further project details including other requirements set by AMS - IX will be looked at in chapter 3. In the following chapter a more detailed view of the structure and operation of AMS - IX will be given. Chapter 4 discusses the techniques which will be used to measure the SLA values. This includes standards by the IEEE and ITU - T. The Y.1731 standard by the ITU - T will be discussed in detail in chapter 5 along with the devices that were considered in nding a 1
Introduction
Chapter 1
tting solution. The difculties with regards to the Multiprotocol Label Switching (MPLS) platform and timing between the devices are also brought up and solved where possible. The ndings of this research are then implemented in chapter 6. The devices which have been purchased are installed and congured to support the measurements of the dened Key Performance Indicators (KPIs). Also shown is how through various tools, the data collected by the devices is stored and then visualised in graphs, as well as a more deeper study in timing sources for the network. The conclusion summarises the results of the research and answers the afore mentioned questions but also provides some recommendations for tasks still to be nished.
Background
Chapter 2
2
2.1
Background
AMS - IX
The Amsterdam Internet Exchange (AMS - IX) provides an Multiprotocol Label Switching (MPLS)-platform to which a customer can connect their network and exchange data with the networks of others. AMS - IX has several Point of Presences (POPs) distributed around the Amsterdam-area where customers have the possibility to connect their routers to the platform. There are two datacenters between which the core of the network is divided, it consists of four switches with connections to every other access switch in the network (see gure 1).
Interxion AMS5 SARA NIKHEF euNetworks GlobalSwitch Telecity 2 Telecity 4 Equinix
10GE
10GE
10GE
10GE
10GE
10GE
10GE
pxc-eqx-118 PXC
PXC
PXC
PXC
PXC
PXC
PXC
PXC
MLX16
MLX16
MLX32
MLX32
MLX32 MLX32
MLX32 MLX32
MLX16
MLX16
MLX16
MLX16
MLX32 MLX32
MLX32 MLX32
MLX16
MLX16
MLX32
MLX32
MLX32
MLX32
MLX8 GE
MLX8 10/100/1000
MLX8 10/100/1000
MLX8 10/100/1000
MLX8
MLX8 10/100/1000 10/100/1000 10/100/1000
MLX8 GE
Figure 1: Topology of the AMS - IX platform
2.1.1
The platform
was founded by several Internet Service Providers (ISPs) who could save a lot of money if they exchanged trafc directly instead of paying their upstream-providers to reach each other. What they did was buy a shared infrastructure device on which they could all connect to each other. This concept of a shared platform holds true today although the scale has changed signicantly. Because of this, a separate, neutral entity was needed to maintain this piece of shared infrastructure. AMS - IX became this entity and its neutrality is ensured through an organisational structure which is described in section 2.1.2. The platform uses MPLS[2] for transporting data from one customer to the other and using Virtual Private LAN Service (VPLS), it appears as a single large broadcast domain[3] from a customers point of view, so they can setup their peering sessions.
AMS - IX
To peer is for two customers to exchange information about their networks through BGP . Although the customers can reach each other over the platform, this does not mean that they immediately can start exchanging trafc. They will have to talk to the
Background
Chapter 2
other customers to peer with. When they have established a session they can start sending and receiving trafc to each other. The optical link of a customer with a 10 Gigabit Ethernet (GigE) connection is being terminated on a so called Photonic Cross Connect (PXC). This device receives the light from the customer router and is able to direct that light to one of two Provider Edge (PE)router. This is all done in hardware without converting the light to an electrical signal. To do this the device essentially uses micromirrors to switch it from one ber output to the other. This system is able to provide a redundant connection to the network which can recover from a broken switch within 20 milliseconds and without any downtime for the customer. 2.1.2 Organisation
AMS - IX is split up in two legal entities: a corporation and an association. The association is the one and only stockholder of the corporation and consists of the members which are the parties connecting to the AMS - IX platform. Each member has an equal vote in the decision-making of the corporation. Twice a year the members get together for a general meeting to discuss the future of the company. Being a member and thus stockholder also means that if the corporation makes any prot this could be benecial because prices may go down.
2.2
Inter-IPX
Over time AMS - IX also started offering other services than just the ISP peering, one of which are Inter-GRX connections. A GPRS Roaming eXchange (GRX) is an exchange point for mobile operators to exchange roaming trafc and authenticate subscribers of network A when they are using network B, eg. when they are in a different country. This exchange is maintained by a GRX-provider, which connects several operators, mostly from the same continent. Not all mobile operators are connected to the same GRXprovider though, and so the need for a connection between those exchanges arises. A company providing that service is called an Inter-GRX provider, of which three exist in the world: Singapore, Ashburn (VS) and the one at AMS - IX, which is the largest. The GRX providers connect to each other using a separate VPLS -instance on the platform similar to a Virtual LAN (VLAN). Figure 2 illustrates the structure of the various GRXelements. A new standard that has been developed by the Global System for Mobile Communications Association (GSMA) is IP eXchange (IPX). The GSMA is the developer of standards like General Packet Radio Service (GPRS) and GRX and has been working on IPX since the trend of Next Generation Networks (NGNs) started, networks which use the same mode of transport regardless the payload.[1] More and more trafc between operators is being
Background
Chapter 2
Figure 2: The GRX-structure sent over Internet Protocol (IP), be it data, voice or video. The goal of IPX is to supersede and include GRX and provide the following advantages: 1. connections between every kind of service provider instead of mobile operators only; 2. end-to-end Quality of Service (QoS) agreements for IP trafc; 3. any service based on IP can be used between peers; 4. connections can be sold as upstream to other providers. To make the transition to IPX without much impact to the trafc the current GRX structure is essentially upgraded. The diagram from gure 2 would not change drastically after the change. AMS - IX started a trial period of the Inter-IPX service in the end of 2010 and eventually wants to provide this service to all potential customers.
Project Details
Chapter 3
Project Details
In chapter 1 two research questions were proposed and will need to be answered but there are some requirements that the answers of the questions will also need to meet.
3.1
Service Levels
As was shown in the previous chapter, there are some advantages to the new IPX standard. One of these is the fact that there will be QoS-agreements that have to be met. These agreements are commonly referred to as Service Level Agreements (SLAs) and form an important aspect to this project. Traditionally AMS - IX has provided their members with an Service Level Description (SLD), a document that contains some basic information about connecting to the platform. It denes terms like the demarcation point, availability, performance, and maintenance periods. Although it includes some performance targets, there are no penalties when failing to meet them and therefore is called a Description instead of an Agreement. The Inter-IPX service will require a penalty scheme because of the end-to-end QoS agreements that will need to be met at all times. This means that the targeted values that are present in the SLD will become maximum values in the new SLA that should not be exceeded. To ensure this doesnt happen AMS - IX will need to start measuring these various indicators. Before the beginning of the trial AMS - IX has been using three demo probes capable of doing so. The probes were tested on three different locations and produced by Accedian Networks. They were directly connected to the access switches, ran some standard measurements and helped baseline the platform to get some initial values for the SLA.
3.2
Assignment
This project will need to make sure that the Key Performance Indicators (KPIs) included in an SLA can be measured over time. If hardware is to be purchased all options will need to be looked at and compared. The data that results from the measurements needs to be stored and visualised in a clear matrix with the data from each probe opposite of each other. This way a clear image of the performance of the network is formed for both AMS - IX and its members in the earlier stage of testing. When expanding to more than twenty devices this will become less useful. Eventually a solution has to be provided that gives a clear insight in the measurement of the KPIs, can provide monthly reports to customers with an SLA and also performs reliably in the platform. The Inter-IPX working group within AMS - IX starting the trial will also be followed closely to map their requirements for this assignment. The projectleader is Luisa Garcia Montoya although this research project is supervised by Attilla 6
Project Details
Chapter 3
de Groot, a member of the Network Operations Center (NOC). A detailed description of the assignment provided by Chief Technology Ofcer (CTO) Henk Steenman has been included in Appendix A. The implementation and visualisation within the web-portal will need to be cordinated with the web developers, monthly reports will be drafted with the Member Relations department and the technical details will be discussed with the NOC. This project will be marked as complete when its possible to: 1. measure the dened KPIs over the platform; 2. show this data in the web portal; 3. present aggregated stats publicly; 4. show each customer data that is of interest to them, and 5. scale the solution up to a device per customer. The time schedule and agreements made are also included in the so called Plan van Aanpak which is included in appendix B (in Dutch).
3.3
Hardware Requirements
A potential hardware solution has to meet the following requirements: 1. It can measure: (a) One Way Average Delay; (b) Two Way Average Delay; (c) One Way Average Delay Variation also known as jitter; (d) Two Way Average Delay Variation also known as jitter; (e) Frame Loss, and (f) Uptime. 2. Values from these measurements can be stored locally or remotely; 3. There is a possibility to place the device between the customer router and the AMS - IX PE -router.
Operations, Administration and Maintenance
Chapter 4
Operations, Administration and Maintenance (OAM) is a technique used by network administrators to manage their long-distance networks, which can spread over multiple cities, countries or even continents. It enables them to be notied when one of the links in the network shows errors, high delays or any other issues. Doing this manually would be a huge and tiresome task. Based on the events detected by the OAM process other actions can be triggered eg. taking another routing path although this is not an internal function of OAM. These kind of functionalities are included in Asynchronous Transfer Mode (ATM) for example, and MPLS includes some tools to troubleshoot Label Switched Paths (LSPs). Higher up the stack, IP also includes some of these manual tools which are included with Internet Control Message Protocol (ICMP).
4.1
Ethernet OAM
Ethernet wasnt developed with long distances in mind, it is traditionally used as a Local Area Network (LAN) protocol. The links in this case are not of great length and easily manageable by a single person. When Ethernet-networks started to expand, more personnel was needed to look after these links, a company could have multiple locations with different engineers but the network was still in their control. The demand for OAM functionalities remained quite low. These days Ethernet is also being used to connect networks from different companies to each other. Instead of a LAN protocol it is more and more used for Wide Area Networks (WANs), replacing traditional protocols like ATM while users still expect the same highquality connections. Around the begin of the century the demand for the same OAM functionalities grew and were starting to be added to the Ethernet-protocol. In this chapter, four different standards will be discussed that try to fulll this demand.
4.2
IEEE
802.3ah
In November 2000 a new study group was founded after 300 representatives of different companies got together on a Call for Interest. The goal of this group was to add functionalities to Ethernet to use it as Ethernet in the First Mile (EFM). The First Mile essentially being the local loop to the customer. [4] It added various new features that were of importance to Service Providers (SPs) one of which were new types of media that could be used, eg. Passive Optical Networks (PONs) and legacy voice-grade copper wiring. It also provided discovery and monitoring OAM tools on link level to the Institute of Electrical and Electronics Engineers (IEEE) 802.3 standard for Carrier Sense Multiple Access (CSMA)/Collission Detection (CD) networks in 2008. [5]
Chapter 4
4.3
IEEE
802.1ag
Although IEEE 802.3ah adds some OAM features they are still limited to one link, ie. it cant provide monitoring for a whole path. This is why in 2007 the IEEE approved a new draft to be included in the 802.1Q standard. This section is known as Connection Fault Management (CFM) and provides end-to-end OAM functionality. [6] 4.3.1 Maintenance Domains and Levels 802.1ag denes a new hierarchy within the different networks that it covers. The top most level consists of a Maintenance Domain (MD) with a specied MD-level. There a 8 levels that can be chosen of which the lowest levels are for the companies on the lowest level of the network, eg. SPs and carriers. The top most levels are for end users and customers. This means that higher level data may be carried over lower levels but lower level data may not propagate to higher level domains. 4.3.2 Maintenance Association and Maintenance End Point Below MDs there exist Maintenance Associations (MAs): groups that consist of two or more Maintenance End Points (MEPs). A MEP is a device that resides at the edge of the domain, has its own unique MEP IDentier (MEPID) and knows all of its peers through a list dened in the MA. It maintains connections called Maintenance Entitys (MEs), with all the other MEPs provided that they share the same MA and MD-level. Figure 3 tries to illustrate this structure.
Figure 3: Structure of Ethernet OAM Over the ME-paths the MEPs keep each other informed about their status and the links in between. These paths are represented by the dotted lines in gure 4. Errors can be propagated and paths can be traced to check for any faulty links. Intermediate devices 9
Chapter 4
can also participate in these checks. These would be called Maintenance Intermediate Points (MIPs) and are passive in the OAM domain, ie. they only respond to incoming messages and never originate messages. 4.3.3
OAM
Inter-Domain OAM
measurements can be setup between any two MEPs, even if they are located on two different continents. But it is possible that this connection traverses several other networks which also use OAM. To distinguish between frames from both networks the previously mentioned MD-levels are used. Any lower level MD transparently passes on the frames from the higher levels until the frame reaches a MEP of the same level and is then processed. But its also the other way around: lower level frames are not allowed to enter the higher level MDs. For this to work a device which connects a higher level network to a lower level one would need to appear as a MIP to the higher level. This is a partial function of this OAM entity and is therefore also called a MIP Half Function (MHF). See also gure 4.
Figure 4: Inter-Domain structure in OAM
4.4
ITU - T
Y.1731
At the same time the IEEE was developing 802.1ag, the International Telecommunication Union - Telecommunication Standardization Sector (ITU - T) was also looking into Ethernet OAM functionalities. The two bodies made some agreements as not to overlap each other, while both standards would be using the same EtherType (0x8902) and frame format. They did not agree on the naming of the different parts of the OAM hierarchy however, and thus the ITU - T has a different naming convention. A MA is called a Maintenance Entity Group (MEG) by their standard and although the acronym MEP remains the same, in this case its short for MEG End Point. The Y.1731 standard also doesnt dene MDs and only considers the MEG-levels. Figure 3 includes both types of naming. [7] 10
Chapter 4
The ITU - T additionally added Performance Measurement (PM) to their standard, an important functionality for the mobile operators. These companies require to know how long it would take for a frame to get from one side of the network to the other. Measurements for delay, jitter and frame loss were being implemented and eventually this standard was made ofcial even before the IEEE approved 802.1ag in 2007.
4.5 Performance Assurance Agent

While testing the demo devices from Accedian that AMS - IX had been using, there also was a proprietary protocol that was looked at capable of measuring these indicators, developed by Accedian and called Performance Assurance Agent (PAA). According to the IEEE assignment database [8] EtherType 0x88fc contains the following: The protocol will use a common header multiplexing various subprotocols into a single ethertype. The format is as follow: Version Subtype Flag Offset 8 8 8 8 bits bits bits bits Version of the common header Subprotocol identifier Dependent subprotocol flag bits Offset to the subprotocol PDU
The actual subprotocol PDU location is computed by adding the Offset value to the address of the Offset field itself. For example an Offset of 0 point to the byte following the Offset field. Further details about the contents of the subprotocol Protocol Data Units (PDUs) are unfortunately not available.
4.6
Solution
The protocol of choice on the AMS - IX platform would be the two supporting the measurements of delays from end-to-end: ITU - T Y.1731 or PAA. Because the latter lacks documentation and therefore the way the measurements work can not be checked, it is deprecated. With Y.1731 AMS - IX will be able to measure their required KPIs and the standard is both open and more widely adopted.
11
Research
Chapter 5
Research
After knowing the basics of the theory and a good understanding of the demands set by AMS - IX it is possible to search for a tting solution. Possible vendors for measurement devices will be compared and further details about the OAM standards will be provided.
5.1
Probe Hardware
To know what kind of hardware to use it is important to understand which requirements it must ll. The more important ones were already discussed in paragraph 3.3 and in summary consisted of: One Way measurements; Two Way measurements; Frame Loss; Uptime; Storage of values and using it as an intermediate device. At the moment, only 1 GigE customers are taken in to consideration but eventually there might be an upgrade to include 10 GigE ports as well. This has to be taken in to account when searching for possible solutions. It is also important for these devices to sync to each other so that they all share the same time, this is an essential requirement for the One Way delay measurements. On the other hand, uptime is a value which can not be calculated using one device. This is a result of the requirement that has been set in a draft Inter-IPX SLA, that every customer has to have two connections on two different locations to the AMS - IX platform. This means that the results of both devices need to be compared and only when both of them are down outside of a scheduled maintenance window, it counts as downtime. For this reason, this indicator is not taken into account but will be calculated after the data has been collected and is to be presented to the customer. 5.1.1 Vendors
Knowing the requirements which have to be fullled by a certain device makes searching for one a more manageable task. In this section the devices that were looked at will be discussed.
12
Research
Chapter 5
Accedian Accedian is a vendor of service level assurance devices which are primarily built for Ethernet backhauls of large mobile carrier networks. For these operators measuring delays between their sites is essential to ensure high quality voice connections. They have models which support 1 GigE and others that also support 10 GigE, an interesting option when scaling is necessary. These devices had the advantage to already be in use by AMS - IX before starting this project. The NIDs were performing correctly and purchasing them was already a discussed option. They nevertheless had to be tested further and models from other vendors needed to be considered in case these did not t the requirements after all. Two 1 GigE MetroNIDs and one 10 GigE MetroNODE were provided to use in the platform and provide it with some baseline-values. This also enabled the engineers to see how the network would react to the data exchanged by the Network Interface Devices (NIDs). The acronym NID is used to describe the actual location of the device in the network: essentially it provides the customer with a network interface to connect to the platform. This is the demarcation point for the equipment of the provider. In section 5.2.3 it will explained why AMS - IX would not use the devices in this way, but the term NID will still be used to refer to these devices. The devices include the standard OAM Performance Measurement but also add the proprietary PAA protocol which was discussed in 4.5. Both provide one and two way measurements and counters for frame loss. The PAA protocol was considered as an option but after testing it became apparent that it would not scale to a large number of devices. Because of the fact that all destination Media Access Control (MAC)-addresses have to be manually congured it would make it a tedious task to congure a large amount NIDs. This is in contrast to the IEEE and ITU - T standards which auto-detect MAC-addresses of their peers. A NID has two active forwarding ports, one facing the network and the other facing a customer. This means that as an intermediate device it can be used to measure the whole path that the customer data takes.1 Models exist with copper or ber ports, or a combination of both. The model that would t AMS - IX is the EtherNID GE, it has a combination of two Small form-factor pluggable (SFP) ber ports and three copper ports of which one is the management port. Using this combination, both customers with ber as well as with copper connections can be connected to the device. Brocade The devices that are in place right now for switching data are from Brocade. This vendor has minimal support built-in for OAM Performance Measurement. The hardware has
1
With some exception on the AMS - IX platform, see 5.2.2.
13
Research
Chapter 5
the functionality to act as a MIP and as such can participate in any Link Traces originating from MEPs. They can also be congured as MEPs themselves but although this has been implemented, one way delay measurements are not supported because they can not synchronise their clocks properly. [9] Test Trafc Measurement (TTM) servers
RIPE
For some years AMS - IX has been working with Rseaux IP Europens (RIPE) to provide them with statistics regarding the quality of the Internet. They do this using servers hosted by a variety of SPs or other IT-related companies throughout the world. All these servers exchange ICMP ping-messages to measure their delays and frame loss to each other. AMS - IX has been hosting four of these devices in datacenters NIKHEF, SARA, GlobalSwitch and TeleCity. Expanding this setup to all eight datacenters would provide AMS - IX with an overview of their internal network as well. Although this would be very helpful, the administration of the servers and the data collected would be done by another company. This situation would not be desirable and also wouldnt make it possible to collect the measured values to calculate downtime periods. On top of that the devices have stopped working reliably in the network ever since AMS - IX started using the MPLS platform. This was caused by the different paths available to the device to get out of the network. It could take a different path each time and because it only sent out a message every 30 seconds, the Provider (P)-routers in the path sometimes had to relearn their MAC-address. This would mean that the frame would have to pass over the Central Processing Unit (CPU) for the switch to learn the MAC -address again and this would cause a much higher delay. Besides these problems the devices werent very scalable because they all needed a Global Position System (GPS) antenna, they couldnt be placed as an intermediate devices and also dont support 10 GigE connections. MRV Another supplier that is familiar within AMS - IX is MRV. Their products are used for Dense wavelength division multiplexing (DWDM) connections between datacenters, these devices multiplex different light-streams onto one ber strand using different wavelengths. This allows AMS - IX to multiply the capacity of a single ber strand by a factor eight. MRV recently started producing the OptiSwitch 940 which builds on their Carrier Ethernet portfolio. It is marketed as a demarcation device and so multiple customers can be aggregated on the device to one or more uplinks. The 940 model includes OAM for
14
Research
Chapter 5
the rst time to measure one and two way delays, in addition to frame loss. The downside is that the series at this moment lack a stable clock synchronisation method.[10] In a later hardware revision it will be able to use Synchronous Ethernet (SyncE) for timing but that would also require additional hardware in the management network. There also exists an OptiSwitch 930 model which suffers from the same limitation but does support 10 GigE interfaces. RAD Besides the known names in the business there are also two new vendors which provide similar solution. One of them is RAD, a manufacturer of products to support carriers and mobile operators. They provide Ethernet OAM functionality with their ETX devices, one of which is the ETX-202A. This devices allows connections up to 1 GigE and provides them with one and two way delays measurement, frame loss and Simple Network Management Protocol (SNMP) polling for the collected values. As with the other devices, an important factor is the synchronisation between clocks. The supplier of the RAD devices couldnt give a specic value for the average variation between the 202A devices but did mention the 204A model, which similar to the MRV OS940 could sync through SyncE. This would require the additional networking hardware as well. To support 10 GigE connections the supplier could provide AMS - IX with the RAD 1002 model which is a similar demarcation device. Unfortunately this relatively new product is not yet able to support Ethernet Performance Measurement. Omnitron iConverter Another new vendor capable of providing AMS - IX with OAM devices is Omnitron. Its primary product line is a range of media-type converters called iConverter. An important feature that these devices also support is Ethernet Performance Measurement. The GM3 model supports the same measurements for one and two way delay and frame loss while also making it possible to poll these values through SNMP. The devices are used as a media-type converter and thus always sit in between the customer and the platform. GigE customers are supported but the vendor lacks support for 10 GigE connections. Apart from this limitation the clock synchronisation is not specic: the vendor could not specify a variation value for Network Time Protocol (NTP) and didnt provide any information about other synchronisation methods.
15
Research
Chapter 5
5.1.2
Comparison
With all specications considered a matrix can be compiled comparing the features of the various vendors. Table 1 gives an overview of the devices that were looked at.
Accedian EtherNID Brocade internal RIPE TTM boxes MRV OS940 RAD 202A iConverter GM3 10GigE yes* yes no yes no no One Way yes no yes yes yes yes Two Way yes yes yes yes yes yes Loss yes yes yes yes yes yes Intermediate yes yes no yes yes yes
SNMP
yes yes no yes yes yes
Sync Accuracy <20 microseconds none 100 nanoseconds none yet** N/A N/A
Table 1: Comparison of different vendors

* Available on another model called MetroNODE. ** Should be available in a next hardware revision.
From this table it can be concluded that only two options pass all criteria: Accedian and MRV. Both vendors would fulll the requirements passed by AMS - IX although the MRV OptiSwitch lacks an important detail: clock synchronisation. At this moment this is not optimal and would later be improved by adding SyncE support but this would require more investment in adding hardware to the management network that also support Synchronous Ethernet. Accedian on the other hand claims a synchronisation between clocks using their NTP implementation of less than 20 microseconds. Thanks to the demo models that were in use this claim could and has been veried. These observations allow for the conclusion that the Accedian EtherNIDs and MetroNODEs would be the preferable solution to measure the various KPIs over the AMS - IX platform.
5.2
Platform Behavior
Testing continued with the Accedian NIDs to see how the network would react. Some unexpected behaviour was experienced while running the Performance Measurements considering the paths that the OAM PDUs would take over the platform. To understand this a more detailed view of the OAM frames is need. 5.2.1 Multicast Frames
In chapter 4.1 Ethernet OAM has already been shown to add some important features to the lower levels of the Open System Interconnection (OSI)-model. The protocol has been developed as a network layer independent function and as such does not rely on the TCP / IP -stack. The OAM data gets an Ethernet-header with its own specic EtherType 16
Research
Chapter 5
(0x8902) before being put on the wire. Of course this Ethernet Frame also needs a destination MAC-address to send it to which varies depending on the type of message being sent. The conguration of an OAM Maintenance Domain does not require to manually set the MAC -addresses of all the peers. These can be auto-discovered through sending out a multicast Ethernet frame which announces the MEG End Point (MEP) MAC-address to all the peers. These message also function as a keep-alive message and are thus called Continuity Check Messages (CCMs). They are sent out at a certain interval and are not replied to by the other MEPs in the MEG. If a MEP stops receiving CCMs it marks the peer MEP as failed after 3.5 times the interval. Besides these two functions, the CCM PDU s also provide the counters for the frame loss measurements, an essential KPI .
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Multicast destination MAC-address 0180.C200.003y Source MAC-address EtherType 0x8902 OpCode (CCM=1) MEL Version Flags Sequence Number (0) TLV Offset (CCM=70) Sequence Number (cont.) MEPID
MEG ID (48 octets)
TxFCf RxFCb TxFCb Reserved (0) End TLV

FCS (cont.) FCS
Figure 5: Overview of a Continuity Check Message Ethernet OAM frame.
An example of a complete CCM Ethernet frame is pictured in gure 5. The multicast destination MAC-address of the CCM PDUs is 0180.C200.003y, where y is the MEG Level (MEL). Apart from the CCM frames there exists one other type of PDU which uses multicast but which is not used within AMS - IX: Link Trace Messages (LTMs). All other messages and replies to those messages including the delay measurement messages 17
Research
Chapter 5
are unicast. The source address is always the devices own MAC-address and the EtherType is dened as the hexadecimal-value 0x8902. The PDU also includes the MEG -level, Ethernet OAM version number and an OpCode which denes the type of PDU , which for CCM is 0 (zero). The ags include the Remote Defect Indicator-bit and the specication of the interval. The sequence number remains empty for this specication, as is the reserved eld at the the end of the frame. In between these elds the congured MEP and MEG ID s are included, as well as the counters for dual-ended frame loss.[7] The differences in multicast and unicast destination addresses causes some unexpected behaviour on the AMS - IX MPLS platform. Because the platform learns all destination MAC addresses and bases its hardware forwarding on that information, it can quickly switch any unicast Delay Measurement Message (DMM) from one MEP to the port on which the other resides, provided that the destination MAC has already been learned. With multicast addresses this is different, these frames will have to pass through the CPU for rate-limiting purposes and as such have a much higher delay. For measuring frame loss this shouldnt be much of a problem, until the CPU becomes loaded with other tasks or other broad/multicast frames. When the CPU on the module is too busy it might occur that a CCM PDU gets dropped. This wont affect the OAM service in general but would count for loss of a frame when this shouldnt be true for standard unicast packets. To solve this problem the VPLS instance in which the devices reside can be congured with the vpls-cpu-protection option. This enables the switches to ood all broadcast and multicast frames in hardware without sending it up in to the CPU.[11] Although this enables the switches to forward the multicast frames directly it also prevents the CPU from rate-limiting the trafc. This could potentially allow for high levels of broadcast trafc and therefore rate-limiting layer 2 Access Control Lists (ACLs) must be congured on the physical ports which connect to the NIDs. 5.2.2 Label Switched Paths Another problem arose while talking to fellow NOC engineers. They noted that the DMMs which were being exchanged by the MEP s would always take the same path over the platform as long as this measurement would persist. This of course would not matter if this path would be the same for all trafc but within the AMS - IX the paths are different for all kinds of trafc streams. Because the OAM stream sends a frame every 100 milliseconds this stream never changes and its path wouldnt either (as seen in gure 6a). This means that trafc travelling another path could encounter higher delays which would not be measured by the NIDs. These paths are the four LSPs which run from PE-router to PE-router over every one of the four P-routers. In gure 6 there are two 10 GigE customers presented on the top with paths over the core in the middle, to a 1 GigE customer on the bottom. Different trafcstreams are load-balanced over four of those so different streams may take different 18
Research
Chapter 5
paths. Besides these LSPs there are other paths that wont change if the streams wont: the physical links in a Link Aggregation Group (LAG). A backbone connection between an AMS - IX PE and a P-router consists of multiple ber links which aggregate into one large logical link. The OAM-stream will start sending data over one of these physical links and wont change, unless interrupted.
PXC
PXC
PE-1-blue MLX-32
PE-1-red MLX-16
PE-1-blue MLX-32
PE-1-red MLX-16
core-loctation-1 MLX-32
core-location-1 MLX-32
core-loctation-1 MLX-32
PE-3 MLX-8
PE-3 MLX-8
(a) OAM measurement using only one a backup path.
LSP
with
(b) OAM measurements using all LSPs
Figure 6: The LSPs over the AMS - IX platform To mitigate these circumstances Virtual Leased Lines (VLLs) can be congured which can be set to use only one of the LSPs. Every NID should then be able to give a different VLAN -tag to each frame so every one of them could be sent over the different VLL s. This would mean all LSPs would be used, as seen in gure 6b. Unfortunate the Accedian devices at the moment do not support conguring more than one VLAN on the OAM measurements. This has been requested as a feature with the vendor and has been promised to be included in the rmware by Quarter 2 of 2011. The second problem regarding the physical links will be a lot harder to solve. This has to do with the way the trafc is load-balanced over the links in a LAG. The algorithm 19
Research
Chapter 5
used for this is Brocade proprietary and so it can not be known. Reverse-engineering the algorithm would be a possibility, were it not that it is most likely changed with every rmware revision, according to NOC engineers. Because of this obstacle it was decided not to go to deep in to this hardware limitation and leave the KPIs as they were intended: as indicators of the network performance. 5.2.3 Redundancy
One of the design requirements is also to be able to put the devices directly in the connection between the customer and the platform. AMS - IX provides a very redundant network which uses the PXCs to switch a customer from one of the PE-routers to another in case of a failure (see also section 2.1.1). It also uses MPLS to provide backup-paths over each core switch. This very redundant and resilient network is as weak as its weakest link. If a device like a EtherNID would be placed between a customer and the platform a new single point of failure would be created. For this reason this requirement has been let go for the time being. Because the NIDs are all connected in the same way that a customer would and the trafc they send is completely separate from their trafc, there will not be a signicant difference in the measurements for a new single point of failure to be created.
5.3
Timing
While running test measurements on the equipment provided by Accedian there were some unusual patterns to be seen in the graphs between the MetroNIDs and the MetroNODE. The MetroNODE one way delay graphs were shaped like a sawtooth from one of the NIDs with an inverse of that graph to that NID. This could indicate some kind of unstable path, but the two way delay measurements were unaffected by this problem. After consulting with the vendor it became clear that there was a clock drift issue between the two models; a typo in a constant used by the time syncing algorithm. Solving this problem helped to understand the importance of synchronised clocks between the different NIDs. Because of the high speed network used by AMS - IX variations of a couple of microseconds are noticeable in the graphs. 5.3.1 Hardware Timestamping in NTP
The probes use NTP to synchronise to a common time source which is a TTM server. These distributed servers all use GPS antennas on the roof of their datacenters to receive their proper timing. To reduce the variations Accedian recommended to look for a time source which could time stamp their NTP-packets in hardware, instead of letting the software do it and having to travel all the way through the Operating System (OS) and networking-stack which is the case with the Linux-based RIPE TTM devices. In gure 7 20
Research
Chapter 5
the structure of an NTP packet is shown. The timestamps consist of 64-bits and the last two are used to sync a client to a server. The server stamps the Received eld when it processes the incoming request and stamps the Transmit eld when it sends out a reply.[12] The synchonisation process as well as the formula the client uses to set its own time, can be seen in gure 8.
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
LI
Status
Type Estimated Error Estimated Drift Rate Reference Clock Identier Reference Timestamp Originate Timestamp Receive Timestamp Transmit Timestamp
Precision
Figure 7: Structure of an NTP packet. The RIPE TTM devices would stamp their packets in the Application layer of the OSImodel but with a hardware timestamping device the packet would receive its timestamp just before being put on the wire (as seen in gure 9). Although this is an OSIlayer violation, it would improve the quality of the timing and above all make it more stable. The variations should become much smaller because CPU load wouldnt affect the accuracy of the timestamps. To test this claim, one of the Accedian MetroNIDs was set to be an NTP-server itself. These devices work with hardware-based Field-programmable gate arrays (FPGAs) and always use hardware timestamps. The MetroNODE and other MetroNID were then synchronised to this NID. After letting the synchronisation take place over 24 hours the one way delay graph appeared more smooth and continued with less spikes and dips over the following days.
21
Research
Chapter 5
Client
Server
Originate Timestamp
req ue
st
network latency Received Timestamp Transmit Timestamp

Network/OS stack delay
response time
l rep
Current Time= network latency + TransmitTS = (RecTS - OrigTS) + TransmitTS
Figure 8: Synchronisation process between a client and the NTP-server.
Software Timestamp
NTP Application Presentation Session
Hardware Timestamp
Transport Network Datalink Physical
Figure 9: The path that an NTP-packet travels through the OSI-layers.
22
Implementation
Chapter 6
Implementation
In this chapter the physical implementation of the devices and their conguration will be discussed. The way of collecting data will be looked as well.
6.1
Hardware
In section 5.1.2 it was shown that the only tting solution would be the EtherNID devices from Accedian. After consulting with the supervisor and CTO eight of these devices were ordered. One for every 1 GigE switch on the eight colocations in the Amsterdam area, these switches are represented by the bottom row of devices in gure 1. The 10 GigE MetroNODEs will be ordered and implemented in a later stadium after more testing with the GigE devices. Together with Hardware Implementation Manager Arin Vijn it was cordinated where and how the devices would be placed within the AMS - IX racks. For the most part the devices were connected through copper Gigabit Ethernet links except for the devices on TeleCity 4 and Interxion which connect to the platform using ber links. All the management ports were connected to the different management switches and given an IP address in the AMS - IX space. The ports connecting to the production network remain without IP-addresses because OAM does not need them and can be put to better use when given to customers. The Domain Name System (DNS)-records were already congured for the management ports and the layer 2-ACLs on the switch-ports were also congured with the corresponding MAC-address to allow the NIDs to send trafc. Location Equinix Interxion NIKHEF SARA TeleCity 2 TeleCity 4 GlobalSwitch euNetworks
IP
91.200.16.181 91.200.16.182 91.200.16.183 91.200.16.184 91.200.16.185 91.200.16.186 91.200.16.187 91.200.16.188
Hostname nid-eqx-015 nid-ixn-016 nid-nik-011 nid-sar-010 nid-tel-012 nid-tc4-137 nid-glo-013 nid-eun-014
MAC -address 0015.ad07.171f 0015.ad07.1647 0015.ad06.a2df 0015.ad06.a5cf 0015.ad06.a6df 0015.ad06.a19f 0015.ad07.174f 0015.ad07.176f
Table 2: Device conguration
The hostnames were chosen to reect the location of the NID and the switch to which it connects. Within the AMS - IX platform, this is a standard naming convention. The ports belonging to the NIDs were all put in a separate VPLS-instance on which CPU Protection was congured so that multicast frames could be ooded in hardware (see 23
Implementation
Chapter 6
section 5.2.1). This separate VPLS instance was chosen so that not all ports had to be congured with layer 2-ACLs to rate limit broadcasts. This choice wont change the measurements in any way because the Delay Measurement Messages will still take one of the same paths that the customer-trafc would take. When the vendor has implemented the function to support more VLANs per MEG, separate VLLs will be congured over all four LSPs. 6.1.1 Conguration
NID
The Accedians do not support exporting their conguration in a standard text-le, therefore a list of Command Line Interface (CLI) commands to execute on each of the devices was composed. An example of such a cong is given in Appendix C. The networking conguration would be as shown in table 2 (lines 1 through 11 in Appendix C). All ports except for the Network and Management ports are disabled and only the later gets an IP-address. To poll the values measured by the NIDs SNMP is set to accept connections with specic communities (l. 12-17 ) and also to send traps to a log server (l. 18 ). On top of these traps the devices send their errors and messages to a syslog server on the management network (l. 19-20 ). This double logging is part of the test trial and new logging-method based on the Zabbix software. An important part of the conguration are the NTP time settings (l. 21-24 ). All of the NID s are made to listen to the NID on the NIKHEF location which gets its timing from a RIPE TTM server on the same location. The DNS server contains a record to point to that NID which is time-nid.noc.ams-ix.net so that changing the source would merely be a change to the DNS record. The synchronisation-mode is set to high-resolution which means that each device polls the NTP server every minute to check its time. The conguration of the OAM measurements follows the same structure which has been shown in gure 3. It starts with dening a MD with a Level and a name which follows a DNS-like convention (l. 25 ). It is set on internal index-number 9 below the default MD s in the device. Below this domain, a MA is created that includes the MEPID s of the devices that will participate in the measurements (l. 26 ). It also receives a name, an index-number, a reference to the previously congured MD-index and a value for the CCM -interval. This is the time in milliseconds between each CCM -message between the MEP s. Accedians lowest supported interval is 1000 milliseconds. After these global settings are congured every device needs to get its own MEPID. This is congured together with specifying a port on which OAM needs to be enabled, a reference to its top MA and which errors it should propagate to its peers (l. 27 ). When a MEP is congured it can start measurements to the rest of the MEPs, but some values need to be dened (l. 28-34 ). The interval is set to 100 milliseconds again the lowest value supported by Accedian which means that one and two way delays are measured ten times per second. All these values are aggregated within a specic reference period. This is set to 1 minute: the average value over 600 measurements is 24
Implementation
Chapter 6
taken. To allow alarms to be generated and traps to be sent, three types of thresholds are dened as shown in table 3 Type Max Delay One Way 2 ms Two Way 2 ms Denition This is the absolute maximum delay value which can be measured by one Delay Measurement Message. If the times the maximum delay value is exceeded reaches this value an alarm is sent. If during the reference period this value is exceeded an alarm is immediately sent. This is the absolute maximum jitter value which can be measured by one Delay Measurement Message. If the times the maximum jitter value is exceeded reaches this value an alarm is sent. If during the reference period this value is exceeded an alarm is immediately sent.
Delay Threshold
3 ms
3 ms
Average Delay
1 ms
1 ms
Max Delay Variation
1 ms
1 ms
Delay Variation Threshold
2 ms
2 ms
Average Delay Variation
1 ms
1 ms
Table 3: OAM threshold values
For the SLA measurements the Average Delay values are the most important, while the maximum values are mostly used as indicators when a large error occurs on the platform. The average thresholds are set to their lowest supported values of 1 millisecond while the draft SLA contains much lower values. A feature request has therefore been sent to Accedian to support the values below 1 millisecond. Frame loss is also dened in the settings but here a reference period can not be set. As shown before, the CCMs include the counters for packet loss and so this measurement inherits that value of 1000 milliseconds. A threshold is also set to 1 percent and the reference period is kept the same as the delay measurements (l. 35-41 ).
6.2
Data Collection
A Virtual Machine (VM) running Linux has been set up to collect the data using Multi Router Trafc Grapher (MRTG) which was written to poll routers for their trafc statistics. The program is adaptable to any situation and can also be used to collect the information from the NIDs every minute. To do this special databases called Round Robin 25
Implementation
Chapter 6
Databases (RRDs) are used to store the values over time. These databases store values every interval (eg. one minute) and consolidate these values after a certain period of time. This way not all data has to be kept at a very low resolution, disk space can be saved and computation with those values takes less time. By default MRTG uses its own text-based database but to set the interval to lower than 5 minutes it can call upon RRD tool, the software handling these database-les. An RRD-le can store two KPIs in separate datasets. For the Accedian EtherNID setup the one and two way delays are stored in one le, one and two way jitter in another and frame loss is alone in a third le. Three les are needed for any measurement from on NID to the other. For 8 devices this would correspond to 8 devices * 7 peers * 3 les = 168 RRD-les. Every le needs a conguration within MRTG and thus 168 sections of conguration are needed. Such a section would look like this:
Target[5min_delay_nid-eqx-015_nid-ixn-016]: .1.3.6.1.4.1.22420.2.7.1.1.2.1.10.1&.1.3.6.1.4.1.22420.2.7.1.1.4.1.10.1: COMMUNITY@nid-eqx-015.noc.ams-ix.net Title[5min_delay_nid-eqx-015_nid-ixn-016]: One and Two Way Delay MaxBytes[5min_delay_nid-eqx-015_nid-ixn-016]: 100000000
The rst line contains the target specication between square brackets and after that species the SNMP Object Identiers (OIDs) to be polled with a specic community string at a certain host. This line is the only useful line, the other lines containing Title and MaxBytes are settings required by the MRTG software but are not used by RRDtool. MaxBytes for example is nothing more than the maximum value to be considered realistic but is expressed in bytes because the program was designed to collect exchanged bytes. Another property which shows this specic purpose is that MRTG by default only expects values its measuring to grow over time. Delay measurement vary constantly and so all the datasets are set to the gauge type, to allow the OID values to uctuate. The OIDs to be polled are listed in table 4. OID
.1.3.6.1.4.1.22420.2.7.1.1.2.1.10.index .1.3.6.1.4.1.22420.2.7.1.1.3.1.10.index .1.3.6.1.4.1.22420.2.7.1.1.4.1.10.index .1.3.6.1.4.1.22420.2.7.1.1.5.1.10.index .1.3.6.1.4.1.22420.2.7.1.2.2.1.18.index
Value One Way Average Delay in microseconds One Way Average Jitter in microseconds Two Way Average Delay in microseconds Two Way Average Jitter in microseconds Packet Loss in millions of a percent
Table 4: OIDs for various KPIs
After the whole MRTG conguration is complete and the daemon is started, values start to get collected. The RRD-les which did not exist yet are automatically generated and lled with the average values every minute. By default the values get consolidated to averages over 30 minutes, then 2 hours and nally to 1 day.
26
Implementation
Chapter 6
6.2.1
Scripts
All these collected KPI measurements will eventually need to be presented to the members, the NOC and other parties. RRDtool includes function for that: rrdtool graph. It can visualise the data from the les by graphing it and writing a standard image le. To do this rrdtool graph is given parameters to know which RRD-le it must use, where it should write the image, what kind of values must be written in it, which timeperiod much be shown, etc. Eventually a graph similar to the one in gure 10 would be the result for the one and two way delays from Interxion to NIKHEF
RRDTOOL / TOBI OETIKER
delay from nid-ixn-016 to nid-nik-011

500
microseconds
400 300 200 100 0 Sat 12:00 Sun 00:00 Sun 12:00 Mon 00:00
One Way Two Way Copyright
95th SLA maximum minimum 210.65 sec 500 sec 226.61 sec 174.42 sec 434.51 sec 1000 sec 435.61 sec 428.35 sec 2011 AMS-IX B.V. [updated: 10-Jan-2011 10:58:10 +0100 ]
Figure 10: Graph generated by RRDtool These images can be generated every reference period for all devices which means 168 images would be replaced each minute. This would not be very efcient because they will not be looked at all the time. To make this more efcient a bash-script is written to be called from the web browser which takes arguments for source and destination NID, time period and KPIs. This script is included in appendix D. This script then generates the graph image on-the-y using the raw output from the RRDtool command. This way, there is no image written to disc but directly to the requesting browser. This also adds the advantage that the image is always up-to-date, with every refresh it looks at new data. For one minute devices this would be an improvement in efciency but to report the values to customers each month this method would be too much, considering that these images would not change over the course of one month. A script has been written to run on the rst day of each month to generate a graph and calculate the values of the various KPIs over the previous month. The images are all written to a directory which can be reached from the internal network to include them in the SLA reports to be sent to customers, the values are reported to customers through a 95th percentile. This is a formula in which the highest 5 percent of the data is not included in computing a maximum value over that month. This allows for some uctuations in the graphs. For the NOC engineers an overview page has also been created with a matrix contain27
Implementation
Chapter 6
ing all the measurement graphs that will enlarge when the cursor moves over it. A screenshot of this page is shown in gure 11.
Figure 11: Overview page KPI measurements
6.3
Timing Sources
In the measurement graphs it is obvious that the line representing one way delay is not as consistent as the two way delay. Because this value is the only one that is inuenced by difference in time between the NIDs, this pattern is not seen in any of the other graphs. Two way measurements base their departure-time on the originating clock and because the packet makes a round-trip its arrival-time is also based on the same clock. One way jitter measurements do not base their timing on both devices. One of the NID s receives two packets within a certain period of time and if the period in which it receives the third packet is higher or lower, this would count as delay variation, or jitter. This is shown by the Reserved place for RxTimeStampf in gure 12, here the receiving NID places its timestamp to compare to the previous received values.
28
Implementation
Chapter 6
Using this gure it can also become clear how the one way delay measurements work. The elds in the header of the frame are all discussed in section 5.2.1, apart from the destination MAC-address which is now a unicast address of the receiving NID. The originating device sends this frame with a timestamp of his clock in the TxTimeStampf eld. It travels through the network until the frame reaches its destination address. Here it gets timestamped again immediately after being received in the RxTimeStampf eld. It can then be processed by the OAM system, it subtracts TxTimeStampf from RxTimeStampf which results in the one way delay between the two NIDs.
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Unicast destination MAC-address Source MAC-address EtherType 0x8902 MEL Version Flags TLV Offset (CCM=16) TxTimeStampf Reserved for receiving device (for RxTimeStampf) End TLV
FCS (cont.) FCS
OpCode (CCM=1)
Figure 12: Overview of a One Way Delay Measurement Ethernet OAM frame. One way measurements essentially compare the timestamp of the originating device with the arrival time at the end-device. Subtracting those two values results in a time difference, called delay. Because these delay values are quite low within the AMS - IX platform it becomes quite essential to have the clocks synchronised as precisely as possible. In section 5.3.1 it was already discovered that hardware timestamping would improve these measurements. A new time source would be needed because the RIPE TTM boxes did not support hardware timestamping and were reaching their end-of-life. Near the end of the project a still on-going search was started to nd such a new time source. The requirements for such a device are in the rst place to support hardware timestamping, it should be easily implemented within the current network and it should be as reliable as possible with a small budget. Table 5 lists the options that were looked. Note that only two devices support NTP hardware timestamping. This is because by default NTP is an application layer protocol and thus inserting a hardware timestamp at the data link layer would be an OSI-layer violation. The only possibility to solve this is to implement the whole function in hardware using FPGAs. This is precisely what both the Z-1000 and MetroNID with GPS use to support NTP timestamping. The former is an appliance integrated with the antenna that can be placed directly on the roof. It is powered through the Ethernet cable so there doesnt have to be a power outlet outside and doesnt take up rackspace. The latter is a familiar name and was discovered when talking with Accedian about accurate time-syncing. The vendor produced a small number of devices with a GPS receiver which would enable them to distribute 29
Implementation
Chapter 6
that accurate time. Device EndRun Unison Oscilloquartz 5220 Meinberg M300 Symmetricom S200 Brilliant Telecom Z-1000 Accedian MetroNID GPS GPS yes yes yes yes yes yes GigE no no no yes no yes HW Stamp no no no no yes yes Accuracy <10 microseconds <20 microseconds <50 microseconds 7 microseconds 250 nanoseconds 250 nanoseconds Cost e2,775.00 N/A e2,482.00 e3,995.00 e5,570.00 e1,229.00
Table 5: Comparison of NTP time sources
All devices support GPS as a time source and their accuracy is based on the difference between the current GPS time and the time in the NTP-packet when it is being put on the wire. With hardware timestamping this is as low as possible because the time is set right when the NTP-packet leaves the device. But this is only true when the timestamp has just left the device, while taking its path along the network the packet can face some delay which will make it less accurate. NTP tries to consider this delay in setting the time for a host but only if the delays do not vary a lot over time. A solution that could be implemented to solve the variation in the network is IEEE 1588v2 or Precision Time Protocol (PTP). This relatively new protocol is said to deliver time around a network in sub-second accuracy.[13] The disadvantage of this new standard is that it requires some new hardware within the management network which would also support PTP. Although this is a possibility in the future, at the moment the Accedian NIDs do not support PTP but will do so in the coming year. The four devices that do not support hardware timestamping are for the most part embedded Linux servers with a standard NTP daemon and are subjectable to delays caused by the OS and networking stack. Vendors like Oscilloquartz and Symmetricom do produce hardware timestamping devices but they are targeted at big operators who need synchronisation for their Synchronous Digital Hierarchy (SDH) network, for AMS - IX the performance increase would not be worth the investment. A requirement which was taken for granted is the maximum length of the antenna cable. For roof-access in one of the datacenters this has to be at least 100 meters but preferably longer. The Brilliant Telecom Z-1000 has a maximum of 100 meters which is consistent with the maximum length of Ethernet cabling. The Accedian MetroNIDs with GPS can use any standard antenna and depends on that manufacturer to specify a maximum length of the cable. This can be as short as 30 meters so a reliable vendor must be found. This search for a new reliable time source is still ongoing. In the next chapter the ndings of the research that has been carried out will be summarised and compared to the previously set requirements. 30
Conclusion
Chapter 7
Conclusion
In the introduction of this paper two questions were stated that needed to be answered: 1. How can the performance of an OSI-layer 2 platform, such as provided by AMS - IX to its customers, be measured? 2. How can AMS - IX provide their customers with the measured values to be included in an SLA? The answers to these questions were subject to ve requirements listed in section 3.2 which are discussed in the following paragraphs.
7.1
Requirements
The rst requirement that the solution had to fulll was to measure the KPIs as dened in section 3.3. The possible solutions for this were researched by comparing different standards with each other. The one that could provide the platform with measurements of nearly all KPIs was Y.1731 developed by the ITU - T. Only the uptime measurement could not be performed because this would need some additional checking whether the other provided port of a customer also had the same problems. Showing the data of the other KPIs and making the stats publicly accessible, were the second and third requirements. These were solved by using SNMP polling over the management network to collect the values and graph those on-the-y. The values are one-minute-averages and provide a performance overview of the whole platform. Customers can select between which two devices they want to see the values for delay, jitter and frame loss. The same principle will be true for the public stats, but these will be provided by Synleaf. This company has worked with AMS - IX in a subproject which enables them to show the real-time values measured on the platform. The use of a web portal in which customers can see the data that they need is only part of the fourth requirement. The customer would also be interested in the values over a certain month. This would enable them to see if AMS - IX has met their predened SLA. This is done using the 95th percentile formula, which sorts the values from the past month from high to low, discards the top ve percent and takes the maximum value of the remaining 95 percent. This allows for some sudden changes in the values without affecting the performance indicators over the whole month. The last requirement was actually not taken in to account, as has been discussed in section 5.2.3. It would require to insert a NID in every connection between a customer and the network. Because the measurements are not based on the trafc of the customer and the device is connected in the same manner that any other customer is, this will not improve the results of the measurement signicantly. And because this adds a new 31
Conclusion
Chapter 7
single point of failure in an otherwise completely redundant network, this requirement was dropped.
7.2
Research Questions
In summary, the rst question has been answered by using two standards provided by the IEEE and ITU - T. The measurement functionalities are provided by ITU - T Y.1731 and meet all the measurement demands from section 3.3. The devices meeting these demands as well and also supporting the ITU - T standard are produced by Accedian Networks, where new devices also have been ordered. The values collected by the devices are stored and processed by a server with RRDtool. This answers the second questions as well, because this tool is also being used to graph the data and calculate the monthly values to be provided to the customer with an SLA. All requests by AMS - IX for this project have been met but there are two tasks which have yet to be checked off.
7.3
Recommendations
One of the important observations made while further testing the new devices was that the synchronisation of the clocks would need to be very precise. Because the platform has such a high performance, the smallest differences between the clocks of the devices would be visible in the one way delay graphs. The time sources AMS - IX uses at the moment had their maintenance contracts ended and are not able to timestamp NTP reliable enough, therefore a new set of NTP servers is being searched for. These will be needed in the coming months to provide the platform with a reliable source of time and minimise the variations in the one way delay graphs. Another task that has yet to fullled is the implementation of 10 GigE NIDs, called MetroNODEs. These devices will be needed to supply the customers connecting through a high speed link with reliable values as well. Adding these devices also means that the data stored will increase tremendously. At the moment there are 18 switches with 10 GigE connections, add to them the 8 existing devices and the count of RRD-les will rise to (24 * 23 * 3 =) 1656. This shouldnt pose a problem for the server but the VM-guest might need some speed and memory upgrades. The 10 GigE devices are already ordered with Accedian but only under the condition that three features will be added to the software: Polling of the Instantaneous Values from the OAM DMM measurements. This allows for the generation of real-time graphs, a subproject that was started together with Synleaf[14]. At the moment these values are acquired through the proprietary PAA protocol (section 4.5). 32
Conclusion
Chapter 7
Multiple VLAN-tagging in a MA/MEG of OAM. Using this feature, the measurements of the devices can be switched over each LSP. Without it, the measurement stream will only use one of the paths which generates a less representative result. Threshold for alarms on DMM lower than 1 millisecond. This enables for traps to generated quicker, should a problem arise on the platform. The manufacturer conceded with these requirements, which should all be implemented by the second quarter of 2011. By that time, the new 10 GigE devices would be fully implemented and tested and a reliable time source will be found as well.
33
Performance measurement infrastructure project
Appendix A
Stagiair: Michiel Appelman Begeleider: Attilla de Groot
A.1
Introduction and Purpose of the project
This project aims at building an infrastructure that measures and reports on the performance of the Amsterdam Internet Exchange (AMS - IX) Metro Ethernet Network. Performance of the network can be identied by the following parameters: Availability of the service Packet loss One way or round trip packet delay One way or round trip packet delay variation (jitter) A number of different applications can be distinguished that need or do not need to be implemented. 1. An overall overview of the performance of the network by adding measuring devices to the each of the access routers and measure the performance between these. This will provide an average performance overview as can be experienced by every customer and will set a baseline on AMS - IX service levels. 2. For specic customer groups it might be necessary to have more direct measurements between the participants in these groups. The idea is to put a measuring device in the access connection of the customer and measure the performance between this device and similarly connected devices from customers in the same group. The purpose of this setup is to be able to dene Service Level Agreements (SLAs) and measure adherence to this SLA 3. The denition and specic implementation of Operations, Administration and Maintenance (OAM) on the AMS - IX platform to allow both AMS - IX Network Operations Center (NOC) and possible customers to see performance of the AMS - IX platform the L2 level. For each of the three mentioned applications it is necessary to: investigate how it can be implemented test implementation in the lab 34
Appendix A
evaluate impact in the platform after agreement with the AMS - IX NOC implement the solution An important aspect of the project is presentation of the measurement results and conformance on the dened service levels on the AMS - IX website The project will be split into different stages:
A.2
Stage 1: Measurements and reporting on Key Performance Indicators (KPIs) between a selected subset of the AMS - IX access routers.
Initial devices selected are Accedian MetroNID en MetroNode 10G. For the initial trail we have two metroNIDs connected to the GE access routers at Equinix and Interxion. The single MetroNode10G need can be connected to any suitable 10GE access router on any of the other sites. The following need to be done: 1. Setup all three devices to measure the critical KPIs (Layer 3) Availability of the service Packet loss One way or round trip packet delay One way or round trip packet delay variation (jitter)
2. Describe how the parameters are derived and give an estimate of the reliability of the measured parameters. Specically this is the case for the one way delay parameter as this is dependent on a synchronized clock or a vendor specic algorithm. 3. Create a web page where for the different KPIs performance matrices are displayed. 4. Based on the experience with the devices decide if these are the right devices to build the service on. (a) If yes continue with Stage 2 (b) If no select and repeat with different device. 5. An additional test and of inuence for the nal decision to go on with the Accedian boxes is the performance of the box in a customer connection. To be tested in the lab: (a) Simulate customer connection to the platform terminated on the MetroNid and/or MetroNode10G instead of directly to the platform. (b) Test performance via this indirect connection between two in such a way connected customers. (c) Repeat measurement of the KPIs as above (in this stage no need for website) 35
Appendix A
A.3
Stage 2: Measurements and reporting on KPIs between a selected subset of the AMS - IX access routers
Stage 2 is essentially the generalization of stage 1 point 1 to 4 1. Order devices for all access switches 2. Create web based performance overview between all access routers (see A.2.3) (a) Since there are a lot of devices it is essential to pay specic attention to the user interface to the data. Cooperation with member relations and or marketing might be required here. Other stages to follow later.
36
Plan van Aanpak
Appendix B
B
B.1
Plan van Aanpak

Inleiding
Voor de AMS - IX zal in deze afstudeerperiode een onderzoek worden uitgevoerd met betrekking tot performance measurement op het platform. Dit document zal daarbij als leidraad worden genomen. In de volgende paginas zal een beschrijving worden gegeven van het project. B.1.1
AMS - IX
AMS - IX is n van de grootste Internet Exchanges in de wereld en biedt aan haar klanten een Layer 2 verbinding aan met overige (internationale) netwerken. Het kantoor bevindt zich in de binnenstad van Amsterdam en de apparatuur is verdeeld over meerdere datacentra in en rondom de hoofdstad. Op deze punten bestaat de mogelijkheid voor bijvoorbeeld Internet Service Provider (ISP)s om verbinding te maken met het platform. Dit onderzoek wordt begeleid door Attilla de Groot van het NOC maar heeft betrekking op het gehele bedrijf. In het NOC zitten de network engineers die zich bezig houden met het beheren van het platform van AMS - IX. Ook zorgen zij ervoor dat het netwerk mee gaat met de tijd en kan voldoen aan de steeds groeiende stroom data.
B.1.2
Inter-IPX
Dit onderzoek wordt onderdeel van een groter project voor een nieuwe service bij het bedrijf. De service, Inter-IPX, zorgt voor een verbinding tussen meerdere IP eXchange (IPX)es en is door de Global System for Mobile Communications Association (GSMA) gestandaardiseerd. IPXes zijn netwerken van mobiele operators die data uit willen wisselen buiten het publieke internet om. Deze operators stellen ook eisen met betrekking tot Quality of Service (QoS) aan het platform van AMS - IX. Voorlopig wordt de service als trial aangeboden maar de eisen moeten worden vastgelegd in een nieuwe SLA. Om te weten of aan deze SLA kan worden voldaan zal meetapparatuur nodig zijn.
B.2
Doelstelling
Op dit moment is er nog geen exacte meting mogelijk op het netwerk van AMS - IX. Dit is echter wel nodig als men een SLA heeft waaraan voldaan moet worden. Het doel is om een infrastructuur op te zetten die de performance van het AMS - IX netwerk meet en rapporteert. Onder performance worden de volgende KPIs verstaan:
37
Plan van Aanpak
Appendix B
Availability (up-time) Reliability (packet loss) One Way Delay and Variation (jitter) Two Way Delay and Variation (Round Trip Time (RTT))
B.3
Projectopdracht
Het bedrijf heeft al een drietal apparaten (zgn. probes) geselecteerd van Accedian Networks die als demo worden gebruikt. Deze zijn direct aangesloten op de access switches en zullen in de beginfase van het project worden gebruikt om te zien of ze voldoen aan de gestelde eisen. Mocht dit niet zo zijn dan wordt een lijst van alternatieven opgesteld. Tevens wordt de nauwkeurigheid van de probes met betrekking tot de One Way Delay (OWD) metingen onderzocht. Zodra de geschikte leverancier gevonden is, zal de projectgroep gaan kijken naar de mogelijkheid om de probes direct tussen de klant-aansluiting te plaatsen. Hierdoor komen ze direct in het pad tussen de twee klanten en worden de metingen meer betrouwbaar. Een ander onderdeel houdt in dat de probes uit te lezen moet zijn. De data die opgeslagen is zal overzichtelijk moeten worden gepresenteerd in een website. De data die hiervoor nodig is bestaat uit de verschillende KPIs die al genoemd werden op pagina 37. De gegevens uit de probes zouden tegenover elkaar uitgezet moeten worden per KPI een matrix wordt getoond. Uiteindelijk moet dit leiden tot een implementatie die voldoet aan alle eisen met betrekking tot meting van verschillende KPIs, rapportage aan de klant en stabiliteit in het platform. Van belang is een afstemming met de projectgroep die gaat over de ontwikkeling van de Inter-IPX service. Projectleider hiervan is Luisa Garcia Montoya. Dit onderzoek valt echter direct onder projectlid Attilla de Groot die vanuit het NOC meewerkt. Een gedetailleerde beschrijving van deze opdracht door AMS - IX is te vinden in Bijlage B.
B.4
Activiteiten
De volgende activiteiten worden tijdens de loop van de afstudeerstage uitgevoerd: 1. Informatie verzamelen Opdracht AMS - IX heeft een opdracht geformuleerd (zie ook Bijlage A) die uitgevoerd moet worden. Deze zal allereerst moeten worden vernomen. 38
Plan van Aanpak
Appendix B
Achtergronden Om de achterliggende gedachte van de opdracht te kunnen achterhalen is het van belang om op de hoogte te zijn van de voorgeschiedenis. Dit is te achterhalen door te praten met medewerkers die hier meer ervaring mee hebben. 2. Plan van Aanpak Opstellen Het Plan van Aanpak wordt als leidraad gebruikt voor de rest van het onderzoek. Deze wordt opgesteld aan de hand van informatie die verkregen is over het bedrijf, de afdeling en de opdracht. Bevestigen Met de school- en bedrijfsbegeleider wordt overlegd of het opgestelde document correct is en deze kan worden gebruikt. 3. Onderzoek Opties Is het Plan van Aanpak akkoord dan wordt het onderzoek gestart. Verschillende oplossingen worden bekeken en vergeleken met de eisen van het bedrijf. Oplossing De beste oplossing wordt gekozen met de daarbij behorende toelichting. Deze wordt daarna uitgewerkt. 4. Implementatie Plaatsing Zodra duidelijk is hoe de gekozen oplossing zou behoren te werken wordt deze in het platform toegepast. Dit houdt in dat eventuele hard- en software in gebruik zal worden genomen. 5. Evaluatie Gebruik binnen infrastructuur Verschillende problemen kunnen naar voren komen tijdens het gebruik van de gekozen oplossingen en die zullen opgelost moeten worden. 6. Rapportage Presenteren Alle voorgaande stappen zullen uiteindelijk worden gedocumenteerd in een verslag en gepresenteerd worden aan het bedrijf en de hogeschool.
B.5
Producten
Verschillende producten zullen worden opgeleverd naast het uiteindelijke verslag. B.5.1 Plan van Aanpak
Dit document is het eerste product dat zal worden opgesteld. Hierin worden de planning, andere producten en deelnemers beschreven. Het biedt een houvast gedurende de looptijd van het project. 39
Plan van Aanpak
Appendix B
B.5.2
Planning
Voor AMS - IX is het van belang te weten hoe de komende tijd wordt ingedeeld. Deze planning zal gepresenteerd moeten worden op de project vergadering van maandag 20 september. B.5.3 Onderzoek Apparatuur
Om een beslissing te maken tot aanschaf van apparatuur zal er allereerst een gedegen onderzoek moeten worden gedaan. De uitslag hiervan benvloedt deze beslissing uiteraard. B.5.4 Verslag en Presentatie
Dit zijn twee producten in n die te maken hebben met de afronding van het project en de stageperiode. Afsluitend wordt het verslag gepresenteerd aan de hogeschool.
B.6
Randvoorwaarden
Het project loopt vanaf 1 september 2010 en zal 20 weken in beslag nemen. Het einde van het project zal na oplevering van het eindproduct zijn of na aoop van 100 werkdagen. Als eindproduct wordt een volledig werkend systeem bedoeld waarmee kan worden bepaald of de gestelde SLAs worden gehaald. B.6.1 Randvoorwaarden
Het eindproduct kan als succesvol worden beschouwd als, 1. de gestelde KPIs kunnen worden gemeten over het platform; 2. de data hiervan kan worden weergegeven op een website; 3. geaggregeerde statistieken publiekelijk weergegeven kunnen worden; 4. klanten hun eigen specieke gegevens overzichtelijk kunnen benaderen; 5. het uiteindelijk mogelijk is om metingen tussen specieke klanten in te zien, en 6. deze opzet schaalbaar is naar alle klanten.
40
Plan van Aanpak
Appendix B
B.7
Kwaliteit
De kwaliteit van het project staat of valt door de terugkoppeling en overzichtelijkheid. Voor het Plan van Aanpak en andere documenten geldt dat deze eerst enkel als concept worden opgesteld. Deze wordt dan ter beoordeling gestuurd naar de begeleider van AMS - IX en na zijn goedkeuring terug naar school. De verschillende aanpassingen en versies worden apart bewaard.
B.8
B.8.1
Kosten/baten overzicht
Kosten
Voor AMS - IX worden in de eerste fase kosten gemaakt door de inkoop van testapparatuur en de inzet van manuren in het gehele project. Mocht de apparatuur bevallen dan wordt overgegaan tot een grotere aankoop in een later stadium. B.8.2 Baten
Belangrijk is om een extra service aan te bieden voor klanten. Hierdoor gaat AMS IX mee met de ontwikkelingen in de markt en kan het blijven groeien. Ook stelt dit onderzoek het bedrijf in staat om in een later stadium SLAs aan alle klanten aan te bieden.
B.9
Risico analyse
Door een overzicht te maken met problemen waar tegen aan kan worden gelopen, wordt de kans op een succesvolle afronding vergroot. Op deze manier kan hier op in worden gespeeld. B.9.1 Tijdnood
Het project moet binnen een bepaalde tijd af zijn en dit gaat met de nodige druk gepaard. Om te voorkomen dat hierdoor fouten worden gemaakt is het belangrijk om te communiceren met de overige projectleden, zo kunnen zij rekening houden met een eventuele vertraging of assisteren bij de afronding. B.9.2 Extra BV
Al voordat dit project gestart werd sprak het management team met de leden van de AMS - IX vereniging over het vormen van een extra BV . Dit met als doel om de risicos, 41
Plan van Aanpak
Appendix B
die SLAs en overheidsregulering met zich meebrengen, in te perken. De uitwerking hiervan is echter nog niet volledig en het project wordt nu binnen de bestaande vorm uitgevoerd. Het is onduidelijk in welke bedrijfsstructuur de service uiteindelijk aangeboden gaat worden. Voor het project is dit geen risico aangezien de service in elk geval geleverd moet worden. B.9.3 Expertise
Wat wel een probleem kan vormen is dat de student te weinig kennis heeft over het onderwerp of gebruikte technieken. Dit risico kan worden beperkt door gebruik te maken en te leren van de expertise van de werknemers van AMS - IX.
B.10
Planning
Zie hiervoor pagina 43.
42
Naam
Werk sep
2010, Kw 4 okt nov dec
2011, Kw 1 jan feb Michiel Appelman mrt
2011, apr
Werkzaam bij AMS-IX Plan van Aanpak Goedkeuring PvA IPX VPLS Instance Inhangen Accedian MetroNODE Voorbereiden Presentatie Presenteren Planning Onderzoek mogelijkheden Accedians Beslissing tot aanschaf Uitwerking van mogelijkheden Plaatsen overige Accedians Overzichten in My AMS-IX Welke specicaties in rapportage Rapportage maken Start Trail Evalueren Accedians en Stats Grotere Implementatie Opzetten Rapport schrijven Rapport ter beoordeling Rapport verbeteren Rapport Presentatie voorbereiden Presentatie
100d 11d Michiel Appelman Ger Brouwers 11d 2d 10d Attilla de Groot Attilla de Groot, Michiel Appelman Michiel Appelman Michiel Appelman 7d Michiel Appelman Henk Steemans, Attilla de Groot, Michiel Appelman 11d 2d 5d Michiel Appelman Attilla de Groot, Michiel Appelman Michiel Appelman Attilla de Groot, Michiel Appelman 1d 1d 21d 1d 85d 5d 5d Michiel Appelman Michiel Appelman Attilla de Groot
Michiel Appelman Ger Brouwers, Henk Steemans, Attilla de Groot Michiel Appelman Michiel Appelman
5d
Michiel Appelman Michiel Appelman
WBS Naam 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 Werkzaam bij AMS-IX Plan van Aanpak Goedkeuring PvA IPX VPLS Instance Inhangen Accedian MetroNODE Voorbereiden Presentatie Presenteren Planning Onderzoek mogelijkheden Accedians Beslissing tot aanschaf Uitwerking van mogelijkheden Plaatsen overige Accedians Overzichten in My AMS-IX Welke specicaties in rapportage Rapportage maken Start Trail Evalueren Accedians en Stats Grotere Implementatie Opzetten Rapport schrijven Rapport ter beoordeling Rapport verbeteren Rapport Presentatie voorbereiden Presentatie
Start 1 sep 1 sep 15 sep 1 sep 17 sep 7 sep 20 sep 21 sep 29 sep 30 sep 15 okt 18 okt 22 okt 25 okt 1 nov 2 nov 1 dec 16 sep 13 jan 14 jan 21 jan 21 jan 28 jan
Finish 18 jan 15 sep 15 sep 15 sep 17 sep 20 sep 20 sep 29 sep 29 sep 14 okt 15 okt 22 okt 22 okt 25 okt 1 nov 30 nov 1 dec 12 jan 14 jan 21 jan 21 jan 28 jan 28 jan
Werk 100d 11d n.v.t. 11d 2d 10d n.v.t. 7d n.v.t. 11d 2d 5d n.v.t. 1d 1d 21d 1d 85d 5d 5d n.v.t. 5d n.v.t.
Toegewezen aan Michiel Appelman Michiel Appelman Ger Brouwers Attilla de Groot Attilla de Groot, Michiel Appelman Michiel Appelman Michiel Appelman Michiel Appelman Attilla de Groot, Henk Steemans, Michiel Appelman Michiel Appelman Attilla de Groot, Michiel Appelman Michiel Appelman Attilla de Groot, Michiel Appelman Attilla de Groot
Michiel Appelman Michiel Appelman Michiel Appelman Attilla de Groot, Ger Brouwers, Henk Steemans Michiel Appelman Michiel Appelman Michiel Appelman Michiel Appelman
Conguration Example
Appendix C
Conguration Example
This section gives an example of the conguration used for the Accedian EtherNIDs.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 media-selection select RJ45-A_RJ45-B port edit Monitor-1 state disable port edit Monitor-2 state disable port edit Client state disable interface edit Auto state disable interface delete Management interface add Management address 91.200.16.181 port Management netmask 255.255.255.192 gateway 91.200.16.129 dns edit hostname nid-eqx-015 dns edit domain noc.ams-ix.net dns edit server1 91.200.16.36 dns edit server2 91.200.16.37 snmp edit ro-community *********** snmp edit rw-community *********** snmp edit use-host-name enable snmp edit system-location Equinix snmp edit contact-info noc@ams-ix.net snmp state enable snmp-trap edit v2c 1 host-name zabbix.ams-ix.net host-community ********* host-state enable syslog edit host interipx.stats.ams-ix.net syslog edit priority Notice ntp add time1.noc.ams-ix.net ntp add time-nid.noc.ams-ix.net ntp enable time-nid.noc.ams-ix.net ntp edit sync-mode highres cfm add domain level 1 index 9 format dns-name name noc.ams-ix.net cfm add ma name oam ccm-interval 1000 mepid-list 1,2,3,4,5,6,7,8 md-idx 9 index 1 cfm add mep active yes ma-idx 1 mep-id 1 port Network cci-enable yes lowest-alarm-pri remErrXconAis cfm add dmm enable yes mep-idx 1 index 1 rmep-id 2 reference-period 1 interval 100 owdelay enable ow-max-delay 2 ow-delay-threshold 3 ow-ad-threshold 1 tw-delay enable tw-max-delay 2 tw-delay-threshold 3 tw-ad-threshold 1 ow-dv enable ow-max-dv 1 ow-dv-threshold 2 ow-adv-threshold 1 tw-dv enable tw-max-dv 1 tw-dv-threshold 2 tw-adv-threshold 1 cfm add dmm enable yes mep-idx 1 index 2 rmep-id 3 reference-period 1 interval 100 owdelay enable ow-max-delay 2 ow-delay-threshold 3 ow-ad-threshold 1 tw-delay enable tw-max-delay 2 tw-delay-threshold 3 tw-ad-threshold 1 ow-dv enable ow-max-dv 1 ow-dv-threshold 2 ow-adv-threshold 1 tw-dv enable tw-max-dv 1 tw-dv-threshold 2 tw-adv-threshold 1 cfm add dmm enable yes mep-idx 1 index 3 rmep-id 4 reference-period 1 interval 100 owdelay enable ow-max-delay 2 ow-delay-threshold 3 ow-ad-threshold 1 tw-delay enable tw-max-delay 2 tw-delay-threshold 3 tw-ad-threshold 1 ow-dv enable ow-max-dv 1 ow-dv-threshold 2 ow-adv-threshold 1 tw-dv enable tw-max-dv 1 tw-dv-threshold 2 tw-adv-threshold 1 cfm add dmm enable yes mep-idx 1 index 4 rmep-id 5 reference-period 1 interval 100 owdelay enable ow-max-delay 2 ow-delay-threshold 3 ow-ad-threshold 1 tw-delay enable tw-max-delay 2 tw-delay-threshold 3 tw-ad-threshold 1 ow-dv enable ow-max-dv 1 ow-dv-threshold 2 ow-adv-threshold 1 tw-dv enable tw-max-dv 1 tw-dv-threshold 2 tw-adv-threshold 1 cfm add dmm enable yes mep-idx 1 index 5 rmep-id 6 reference-period 1 interval 100 owdelay enable ow-max-delay 2 ow-delay-threshold 3 ow-ad-threshold 1 tw-delay enable tw-max-delay 2 tw-delay-threshold 3 tw-ad-threshold 1 ow-dv enable ow-max-dv 1 ow-dv-threshold 2 ow-adv-threshold 1 tw-dv enable tw-max-dv 1 tw-dv-threshold 2 tw-adv-threshold 1 cfm add dmm enable yes mep-idx 1 index 6 rmep-id 7 reference-period 1 interval 100 owdelay enable ow-max-delay 2 ow-delay-threshold 3 ow-ad-threshold 1 tw-delay enable
29
30
31
32
33
45
Conguration Example
Appendix C
34
35 36 37 38 39 40 41 42
tw-max-delay 2 tw-delay-threshold 3 tw-ad-threshold 1 ow-dv enable ow-max-dv 1 ow-dv-threshold 2 ow-adv-threshold 1 tw-dv enable tw-max-dv 1 tw-dv-threshold 2 tw-adv-threshold 1 cfm add dmm enable yes mep-idx 1 index 7 rmep-id 8 reference-period 1 interval 100 owdelay enable ow-max-delay 2 ow-delay-threshold 3 ow-ad-threshold 1 tw-delay enable tw-max-delay 2 tw-delay-threshold 3 tw-ad-threshold 1 ow-dv enable ow-max-dv 1 ow-dv-threshold 2 ow-adv-threshold 1 tw-dv enable tw-max-dv 1 tw-dv-threshold 2 tw-adv-threshold 1 cfm add pkt-loss enable yes mep-idx 1 index 1 rmep-id 2 reference-period 1 thresholdratio 1 cfm add pkt-loss enable yes mep-idx 1 index 2 rmep-id 3 reference-period 1 thresholdratio 1 cfm add pkt-loss enable yes mep-idx 1 index 3 rmep-id 4 reference-period 1 thresholdratio 1 cfm add pkt-loss enable yes mep-idx 1 index 4 rmep-id 5 reference-period 1 thresholdratio 1 cfm add pkt-loss enable yes mep-idx 1 index 5 rmep-id 6 reference-period 1 thresholdratio 1 cfm add pkt-loss enable yes mep-idx 1 index 6 rmep-id 7 reference-period 1 thresholdratio 1 cfm add pkt-loss enable yes mep-idx 1 index 7 rmep-id 8 reference-period 1 thresholdratio 1 user edit admin password
46
Graph Generating Bash Script
Appendix D
D
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58
#!/bin/bash # Written by michiel.appelman@ams-ix.net # Working Directory for the RRD files DIR="/opt/mrtg/nids" WWWDIR="/var/www/stats" ## Strip HTML GET String into local variables ## hour=echo "$QUERY_STRING" | sed -n s/^.*hour=$[^&]*$.*$/\1/p | sed "s/%20/ /g" end=echo "$QUERY_STRING" | sed -n s/^.*end=$[^&]*$.*$/\1/p | sed "s/%20/ /g" kpi=echo "$QUERY_STRING" | sed -n s/^.*kpi=$[^&]*$.*$/\1/p | sed "s/%20/ /g" nid1=echo "$QUERY_STRING" | sed -n s/^.*srcnid=$[^&]*$.*$/\1/p | sed "s/%20/ /g" nid2=echo "$QUERY_STRING" | sed -n s/^.*dstnid=$[^&]*$.*$/\1/p | sed "s/%20/ /g" # If hour variable is empty set it to 48 if [ -z $hour ] ; then hour="48" fi # If end time is empty set it to now or else to starting time + end time if [ -z $end ] ; then end="now" else end="start+${end}h" fi # Check to see if two different NIDs are selected if [ $nid1 == $nid2 ] ; then exit 1 fi # Delay or jitter function if [ "$kpi" == "delay" ] || [ "$kpi" == "jitter" ] then # Change colors depending on KPI if [ "$kpi" == "delay" ] ; then color1="#00ff00" color2="#0000ff" sla1="500" sla2="1000" elif [ "$kpi" == "jitter" ] ; then color1="#00ffff" color2="#ff00ff" sla1="50" sla2="100" fi ## Run RRDtool an send output to web page ## echo "Content-type: image/png " echo rrdtool graph -X0 -l0 -Y --imginfo --title "$kpi from $nid1 to $nid2" --start now-${ hour}h --end ${end} --vertical-label microseconds - \ DEF:oadv_$nid1\_$nid2=$DIR/5min_$kpi\_$nid1\_$nid2.rrd:ds0:AVERAGE \ DEF:tadv_$nid1\_$nid2=$DIR/5min_$kpi\_$nid1\_$nid2.rrd:ds1:AVERAGE \ VDEF:oadvmax_$nid1\_$nid2=oadv_$nid1\_$nid2,MAXIMUM \ VDEF:oadvavg_$nid1\_$nid2=oadv_$nid1\_$nid2,AVERAGE \ VDEF:oadvmin_$nid1\_$nid2=oadv_$nid1\_$nid2,MINIMUM \ VDEF:oadv95t_$nid1\_$nid2=oadv_$nid1\_$nid2,95,PERCENT \ COMMENT:" 95th SLA maximum minimum\l" \ LINE1:oadv_$nid1\_${nid2}${color1}:"One Way " \ GPRINT:oadv95t_$nid1\_$nid2:"%6.2lf sec " \ t COMMENT:"$sla1 sec " \ t
47
Appendix D
59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100
GPRINT:oadvmax_$nid1\_$nid2:"%6.2lf sec " \ t GPRINT:oadvmin_$nid1\_$nid2:"%6.2lf sec\l" \ t VDEF:tadvmax_$nid1\_$nid2=tadv_$nid1\_$nid2,MAXIMUM \ VDEF:tadvavg_$nid1\_$nid2=tadv_$nid1\_$nid2,AVERAGE \ VDEF:tadvmin_$nid1\_$nid2=tadv_$nid1\_$nid2,MINIMUM \ VDEF:tadv95t_$nid1\_$nid2=tadv_$nid1\_$nid2,95,PERCENT \ LINE1:tadv_$nid1\_${nid2}${color2}:"Two Way " \ GPRINT:tadv95t_$nid1\_$nid2:"%6.2lf sec " \ t COMMENT:"$sla2 sec " \ t GPRINT:tadvmax_$nid1\_$nid2:"%6.2lf sec " \ t GPRINT:tadvmin_$nid1\_$nid2:"%6.2lf sec\l" \ t COMMENT:"Copyright l date +%Y AMS-IX B.V. [updated\: date "+%d-%b-%Y %H\:%M\:%S %z " ]\c" 2>/dev/null # Exit without errors exit 0 # Frame Loss function elif [ "$kpi" == "frameloss" ] then ## Run RRDtool and send output to web page ## echo "Content-type: image/png " echo rrdtool graph -X0 -l0 -Y --imginfo --title "$kpi from $nid1 to $nid2" --start now-${ hour}h --end ${end} --vertical-label percentage - \ DEF:fl0_$nid1\_$nid2=$DIR/5min_frameloss_$nid1\_$nid2.rrd:ds0:AVERAGE \ CDEF:fl_$nid1\_$nid2=fl0_$nid1\_$nid2,1000000,/ \ VDEF:flmax_$nid1\_$nid2=fl_$nid1\_$nid2,MAXIMUM \ VDEF:flavg_$nid1\_$nid2=fl_$nid1\_$nid2,AVERAGE \ VDEF:flmin_$nid1\_$nid2=fl_$nid1\_$nid2,MINIMUM \ VDEF:fl95t_$nid1\_$nid2=fl_$nid1\_$nid2,95,PERCENT \ COMMENT:" 95th SLA maximum minimum\l" \ LINE1:fl_$nid1\_$nid2#000000:"Frame Loss " \ GPRINT:fl95t_$nid1\_$nid2:"%3.6lf %% " \ COMMENT:"0.005000 % " \ GPRINT:flmax_$nid1\_$nid2:"%3.6lf %% " \ GPRINT:flmin_$nid1\_$nid2:"%3.6lf %%\l" \ COMMENT:"Copyright l date +%Y AMS-IX B.V. [updated\: date "+%d-%b-%Y %H\:%M\:%S %z " ]\c" 2> /dev/null # Exit without errors exit 0 # Invalid $kpi - exit with error else exit 1 fi
48
Acronyms
Appendix E
IEEE
E
1 DM
ACL
Acronyms
One Way Delay Measurement Access Control List Amsterdam Internet Exchange Asynchronous Transfer Mode Border Gateway Protocol Continuity Check Message Collission Detection
Institute of Electrical and Electronics Engineers Internet Protocol

IP
IP IPX ISP IT ITU - T
eXchange
AMS - IX ATM BGP CCM CD CFM CLI CPU CSMA CTO DMM DNS DWDM
Internet Service Provider Information Technology International Telecommunication Union - Telecommunication Standardization Sector Key Performance Indicator Link Aggregation Group Local Area Network Leap Indicator Label Switched Path Link Trace Message Maintenance Association Media Access Control Maintenance Domain Maintenance Entity Maintenance Entity Group
MEG
KPI
Connection Fault Management

LAG
Command Line Interface

LAN
Central Processing Unit

LI
Carrier Sense Multiple Access

LSP
Chief Technology Ofcer

LTM
Delay Measurement Message

MA
Domain Name System

MAC
Dense wavelength division multiplexing Ethernet in the First Mile Frame Check Sequence Field-programmable gate array Gigabit Ethernet Global Position System General Packet Radio Service Global System for Mobile Communications Association
GPRS
MD ME MEG MEL MEP MEP MEPID MHF MIP MPLS MRTG
EFM FCS FPGA
Level
GigE
GPS GPRS GSMA
Maintenance End Point

MEG MEP MIP
End Point IDentier
Half Function
Maintenance Intermediate Point Multiprotocol Label Switching Multi Router Trafc Grapher
GRX ICMP
Roaming eXchange
Internet Control Message Protocol
49
Acronyms
Appendix E
RRD RTT SDH SLA SLD SFP SNMP
NGN NID NIST
Next Generation Network Network Interface Device National Institute of Standards and Technology Network Operations Center Network Time Protocol Operations, Administration and Maintenance One Way Average Delay One Way Average Delay Variation Object Identier Operating System Open System Interconnection One Way Delay Provider Performance Assurance Agent Protocol Data Unit Provider Edge Performance Measurement Passive Optical Network Point of Presence Precision Time Protocol Photonic Cross Connect Quality of Service Remote Defect Indicator Request for Comment Rseaux IP Europens
Round Robin Database Round Trip Time Synchronous Digital Hierarchy Service Level Agreement Service Level Description Small form-factor pluggable Simple Network Management Protocol Service Provider
NOC NTP OAM
OAD OADV OID OS OSI OWD P PAA PDU PE PM PON POP PTP PXC QoS RDI RFC RIPE
SP
SyncE Synchronous Ethernet

TAD TADV TCP TLV TTM VLAN VLL VM VPLS WAN
Two Way Average Delay Two Way Average Delay Variation Transmission Control Protocol Type, Length, Value Test Trafc Measurement Virtual LAN Virtual Leased Line Virtual Machine Virtual Private LAN Service Wide Area Network
50
Appendix F
Bibliography
[1] Global System for Mobile Communications Association, Evolution to ip, December 2010. http://www.gsmworld.com/our-work/ programmes-and-initiatives/evolution_to_ip.htm. [2] E. Rosen, A. Viswanathan, and R. Callon, Request for Comment 3031: Multiprotocol Label Switching, January 2001. http://www.ietf.org/rfc/rfc3031.txt. [3] M. Lasserre and V. Kompella, Request for Comment 4762: Virtual Private LAN Service using Label Distribution Protocol (LDP) signaling, January 2007. http: //www.faqs.org/rfcs/rfc4762.html. [4] Institute of Electrical and Electronics Engineers (IEEE), IEEE 802.3 Working Group Approves New Study Group for Ethernet in the First Mile (EFM), November 2000. http://web.archive.org/web/20070217122603/http: //standards.ieee.org/announcements/8023nsg.html. [5] Institute of Electrical and Electronics Engineers, IEEE Standard 802.3: Carrier Sense Multiple Access (CSMA) with Collission Detection (CD) (CSMA/CD) access method and Physical Layer specications Section Five, 2008. http://standards. ieee.org/getieee802/802.3.html. [6] Institute of Electrical and Electronics Engineers, Virtual Bridged Local Area Networks - Amendment 5: Connectivity Fault Management, 2007. http://www. ieee802.org/1/pages/802.1ag.html. [7] International Telecommunication Union - Telecommunication Standardization Sector, OAM functions and mechanisms for Ethernet based networks, 2008. http: //www.itu.int/rec/T-REC-Y.1731/en. [8] Institute of Electrical and Electronics Engineers, EtherType Field Public Assignments, 2004. http://standards.ieee.org/regauth/ethertype/eth. txt. [9] Brocade, NetIron Conguration Guide Supporting Multi-Service IronWare Unied NetIron R05.0.00. April 2010. page 2239. [10] MRV, OptiSwitch 940 - 10GE Carrier-Ethernet Access Data Sheet, October 2010. http://www.mrv.com/datasheets/OS/PDF300/MRV-OS-OS940_ A4_HI.pdf. [11] Brocade, NetIron Conguration Guide Supporting Multi-Service IronWare Unied NetIron R05.0.00. April 2010. page 1557. 51
Appendix F [12] D. L. Mills, Request for Comment 958: Network Time Protocol, September 1985. http://www.faqs.org/rfcs/rfc958.html. [13] National Institute of Standards and Technology, IEEE 1588 - Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems, May 2010. http://ieee1588.nist.gov/. [14] A. Ramkisoen, Synleaf, December 2010. http://www.synleaf.com.
52
Acknowledgements
Thanks to Attilla de Groot, Henk Steenman, Martin Pels, Arin Vijn, Niels Bakker, Steven Bakker and all the other staff at AMS - IX, for providing me with the opportunity, knowledge and skills to successfully complete this research project.

Ethernet Performance Measurement

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Ethernet Performance Measurement

Uploaded by

Copyright:

Available Formats

Ethernet Performance Measurement

Michiel Appelman 2011

Ethernet Performance Measurement

By: Michiel A PPELMAN

Supervisor: Attilla DE G ROOT

January 14, 2011

Conclusion 7.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

MLX8 10/100/1000 10/100/1000 10/100/1000

Figure 1: Topology of the AMS - IX platform

Operations, Administration and Maintenance

Operations, Administration and Maintenance

Operations, Administration and Maintenance

Operations, Administration and Maintenance

Figure 4: Inter-Domain structure in OAM

Operations, Administration and Maintenance

4.5 Performance Assurance Agent

With some exception on the AMS - IX platform, see 5.2.2.

yes yes no yes yes yes

Table 1: Comparison of different vendors

MEG ID (48 octets)

TxFCf RxFCb TxFCb Reserved (0) End TLV

Figure 5: Overview of a Continuity Check Message Ethernet OAM frame.

(a) OAM measurement using only one a backup path.

(b) OAM measurements using all LSPs

network latency Received Timestamp Transmit Timestamp

Current Time= network latency + TransmitTS = (RecTS - OrigTS) + TransmitTS

Figure 8: Synchronisation process between a client and the NTP-server.

Transport Network Datalink Physical

Figure 9: The path that an NTP-packet travels through the OSI-layers.

91.200.16.181 91.200.16.182 91.200.16.183 91.200.16.184 91.200.16.185 91.200.16.186 91.200.16.187 91.200.16.188

Hostname nid-eqx-015 nid-ixn-016 nid-nik-011 nid-sar-010 nid-tel-012 nid-tc4-137 nid-glo-013 nid-eun-014

Table 2: Device conguration

Max Delay Variation

Delay Variation Threshold

Average Delay Variation

Table 3: OAM threshold values

Table 4: OIDs for various KPIs

delay from nid-ixn-016 to nid-nik-011

One Way Two Way Copyright

Figure 11: Overview page KPI measurements

Table 5: Comparison of NTP time sources

Performance measurement infrastructure project

Performance measurement infrastructure project

Stagiair: Michiel Appelman Begeleider: Attilla de Groot

Introduction and Purpose of the project

Performance measurement infrastructure project

Performance measurement infrastructure project

Plan van Aanpak

Plan van Aanpak

Plan van Aanpak

Plan van Aanpak

Plan van Aanpak

Plan van Aanpak

Plan van Aanpak

Zie hiervoor pagina 43.

2010, Kw 4 okt nov dec

2011, Kw 1 jan feb Michiel Appelman mrt

Michiel Appelman Michiel Appelman

Graph Generating Bash Script

Graph Generating Bash Script

Graph Generating Bash Script

Institute of Electrical and Electronics Engineers Internet Protocol

IP IPX ISP IT ITU - T

Connection Fault Management