P. 1
Ethernet Performance Measurement

Ethernet Performance Measurement

|Views: 390|Likes:
Published by Michiel Appelman
Final research paper for the Network Infrastructure Design education at HS Zuyd. Done internally at AMS-IX.
Final research paper for the Network Infrastructure Design education at HS Zuyd. Done internally at AMS-IX.

More info:

Published by: Michiel Appelman on May 30, 2012
Copyright:Attribution Non-commercial

Availability:

Read on Scribd mobile: iPhone, iPad and Android.
download as PDF or read online from Scribd
See more
See less

12/09/2013

pdf

Sections

Ethernet Performance Measurement

on an MPLS platform

Michiel Appelman 2011

Ethernet Performance Measurement
on an MPLS platform

By: Michiel A PPELMAN

Supervisor: Attilla DE G ROOT

January 14, 2011

Summary
The Amsterdam Internet Exchange (AMS - IX) is an organization to facilitate the exchange of IP traffic between its participants and is set up as a not for profit membership based association. The members consist of large ISPs, mobile operators, content providers and other communication companies. AMS - IX has been searching for a way to measure values like delay and jitter over this platform, after a need for a new service called Inter-IP eXchange (IPX) arose. Mobile operators are planning to use this new exchange structure to separate their communication-specific data from the ‘regular’ internet. The IPX standard requires Service Level Agreements ( SLA s) to be met by all parties involved in providing end-to-end connections, including AMS - IX as an traffic exchange platform. To provide its customers with a reliable network and prove that the SLA will be met, AMS - IX must be able to collect the statistics of the network. Because the network uses a combination of Ethernet, MPLS and VPLS in exchanging traffic, traditional higher level Operations, Administration and Maintenance (OAM) utilities could not be used to measure its performance reliably. At the moment there is one new standard defining measurement of the various Key Performance Indicators (KPIs) over Ethernet and is developed by the ITU - T in coordination with the IEEE. It’s referenced by Y.1731 and is commonly known as Ethernet OAM Performance Measurement. With this knowledge, a fitting hardware solution supporting Performance Measurement was searched for among different vendors. Accedian Networks already provided AMS IX with some test Network Interface Devices ( NID s) to set a baseline. Other devices were looked at as well and a list of requirements was compiled. Eventually the test devices by Accedian were the only models which could pass all criteria and had a performance that would be of use on the AMS - IX platform. Accedian was the choice as a manufacturer and the first devices for 1 Gigabit Ethernet (GigE) connections were ordered and installed. While testing these new devices it became clear that an important external part of the measurements is the synchronisation of the clocks between them. The slightest difference between clocks will result in a significant variation in the one way delay values, the only KPI influenced by this. Because the performance of the platform is of such a high level, differences in milliseconds are not acceptable. Using the Accedian NIDs the clocks can be set to within a couple of microseconds but a new external time source will be needed to provide them with a valid time. A search for this new NTP-server is still on-going. The data collected by the devices using ITU - T Y.1731 over the platform is graphed to provide the customers and the company itself with easy to interpret images. This is done by polling all the devices every minute through SNMP, retrieving their average values and graphing them. The images are then displayed in a web portal where users can select source and destination of the measurements. The stored values are consolidated over time and at the end of each month the data is processed and the values are i

then reported to customers through a 95th percentile. At the moment the measurements on the platform are all operational and the values are being collected. The Inter-IPX VLAN has yet to have customers connect to it but the data is enough to give AMS - IX itself a valuable insight in their platform. The Inter-IPX service will be provided in the coming months and SLAs will also be made available to other customers who wish to have such a contract. All this is enabled by the relatively new ITU - T Y.1731 standard and the NIDs manufactured by Accedian. The remaining tasks to make the whole measurement system complete, are the implementation of new and reliable time sources and the expansion of the devices to also include 10 GigE connections to the network. Doing so will provide a full picture of the network, from and to every switch. Both recommendations will be implemented in the first quarter of this year.

ii

Contents
Summary 1 2 Introduction Background 2.1 AMS - IX . . . . . . . . 2.1.1 The platform 2.1.2 Organisation 2.2 Inter-IPX . . . . . . . i 1 3 3 3 4 4 6 6 6 7 8 8 8 9 9 9 10 10 11 11 12 12 12 16 16 16 18 20 20 20 23 23 24 25 27 28

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

3

Project Details 3.1 Service Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . Operations, Administration and Maintenance 4.1 Ethernet OAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 IEEE 802.3ah . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 IEEE 802.1ag . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.1 Maintenance Domains and Levels . . . . . . . . . . . . . . . . . . 4.3.2 Maintenance Association and Maintenance End Point . . . . . . . . 4.3.3 Inter-Domain Operations, Administration and Maintenance (OAM) 4.4 ITU - T Y.1731 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Performance Assurance Agent . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Research 5.1 Probe Hardware . . . . . . . . . . . . . . 5.1.1 Vendors . . . . . . . . . . . . . . 5.1.2 Comparison . . . . . . . . . . . . 5.2 Platform Behavior . . . . . . . . . . . . . 5.2.1 Multicast Frames . . . . . . . . . 5.2.2 Label Switched Paths . . . . . . . . 5.2.3 Redundancy . . . . . . . . . . . . 5.3 Timing . . . . . . . . . . . . . . . . . . . 5.3.1 Hardware Timestamping in NTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4

5

6

Implementation 6.1 Hardware . . . . . . . . . . . . . . . . . . . . . . . . 6.1.1 Network Interface Device (NID) Configuration 6.2 Data Collection . . . . . . . . . . . . . . . . . . . . . 6.2.1 Scripts . . . . . . . . . . . . . . . . . . . . . . 6.3 Timing Sources . . . . . . . . . . . . . . . . . . . . . iii

7

Conclusion 7.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

31 31 32 32

Appendices A Performance measurement infrastructure project A.1 Introduction and Purpose of the project . . . . . . . . . . . . . . . . . . . A.2 Stage 1: Measurements and reporting on Key Performance Indicators (KPIs) between a selected subset of the AMS - IX access routers. . . . . . . . . . . A.3 Stage 2: Measurements and reporting on KPIs between a selected subset of the AMS - IX access routers . . . . . . . . . . . . . . . . . . . . . . . . . . B Plan van Aanpak B.1 Inleiding . . . . . . . . . . . . . B.1.1 AMS - IX . . . . . . . . . . B.1.2 Inter-IPX . . . . . . . . . B.2 Doelstelling . . . . . . . . . . . B.3 Projectopdracht . . . . . . . . . B.4 Activiteiten . . . . . . . . . . . . B.5 Producten . . . . . . . . . . . . B.5.1 Plan van Aanpak . . . . B.5.2 Planning . . . . . . . . . B.5.3 Onderzoek Apparatuur B.5.4 Verslag en Presentatie . B.6 Randvoorwaarden . . . . . . . B.6.1 Randvoorwaarden . . . B.7 Kwaliteit . . . . . . . . . . . . . B.8 Kosten/baten overzicht . . . . B.8.1 Kosten . . . . . . . . . . B.8.2 Baten . . . . . . . . . . . B.9 Risico analyse . . . . . . . . . . B.9.1 Tijdnood . . . . . . . . . B.9.2 Extra BV . . . . . . . . . B.9.3 Expertise . . . . . . . . . B.10 Planning . . . . . . . . . . . . . C Configuration Example D Graph Generating Bash Script E Acronyms F Bibliography iv . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 34 35 36 37 37 37 37 37 38 38 39 39 40 40 40 40 40 41 41 41 41 41 41 41 42 42 45 47 49 51

List of Figures
1 2 3 4 5 6 7 8 9 10 11 12 Topology of the Amsterdam Internet Exchange (AMS - IX) platform The GPRS Roaming eXchange (GRX)-structure . . . . . . . . . . . . Structure of Ethernet OAM . . . . . . . . . . . . . . . . . . . . . . Inter-Domain structure in OAM . . . . . . . . . . . . . . . . . . . Overview of a Continuity Check Message Ethernet OAM frame. . . The Label Switched Paths (LSPs) over the AMS - IX platform . . . . Structure of an NTP packet. . . . . . . . . . . . . . . . . . . . . . Synchronisation process between a client and the NTP-server. . The path that an NTP-packet travels through the OSI-layers. . . . Graph generated by Round Robin Database (RRD)tool . . . . . . . Overview page KPI measurements . . . . . . . . . . . . . . . . . Overview of a One Way Delay Measurement Ethernet OAM frame. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 5 9 10 17 19 21 22 22 27 28 29

List of Tables
1 2 3 4 5 Comparison of different vendors . . . . . . . . . . . . . . Device configuration . . . . . . . . . . . . . . . . . . . . . OAM threshold values . . . . . . . . . . . . . . . . . . . . Object Identifiers (OIDs) for various KPIs . . . . . . . . . . Comparison of Network Time Protocol (NTP) time sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 23 25 26 30

v

Introduction

Chapter 1

1

Introduction

This report is a product of a research project that was commissioned by the Amsterdam Internet Exchange (AMS - IX), a provider of an Internet Protocol (IP) traffic exchange platform. A significant number of companies connect to this platform such as Internet Service Providers (ISPs), content providers, mobile operators, etc. They connect to one of the eight Point of Presences (POPs) in the Amsterdam area which enables them to establish peering sessions with other members and exchange traffic with them. Reason Mobile operators connect to each other through GPRS Roaming eXchange (GRX) providers to provide authentication to their customers when they are roaming on another network. A trend within the telecommunications industry is to consolidate all traffic into one transport mode, be it voice, data, video or any other format.[1] This is called the Next Generation Network (NGN) and operators are now looking at connecting with each other over a network which supports IP-traffic, called IP eXchange (IPX). AMS - IX role in this network would be to provide IPX-providers with a similar exchange platform as is now in place for GRX providers. This project was started while developing this new service that would require Service Level Agreements (SLAs) to guarantee delivery of this vulnerable traffic. The values of these SLAs need to be measured to provide the customers and AMS - IX itself with a clear overview of the performance of the network. Research Questions This research paper will need to provide answers to the following questions: 1. How can the performance of an OSI-layer 2 platform, such as provided by AMS - IX to its customers, be measured? 2. How can AMS - IX provide their customers with the measured values and include this data in an SLA? Structure Further project details including other requirements set by AMS - IX will be looked at in chapter 3. In the following chapter a more detailed view of the structure and operation of AMS - IX will be given. Chapter 4 discusses the techniques which will be used to measure the SLA values. This includes standards by the IEEE and ITU - T. The Y.1731 standard by the ITU - T will be discussed in detail in chapter 5 along with the devices that were considered in finding a 1

Introduction

Chapter 1

fitting solution. The difficulties with regards to the Multiprotocol Label Switching (MPLS) platform and timing between the devices are also brought up and solved where possible. The findings of this research are then implemented in chapter 6. The devices which have been purchased are installed and configured to support the measurements of the defined Key Performance Indicators (KPIs). Also shown is how through various tools, the data collected by the devices is stored and then visualised in graphs, as well as a more deeper study in timing sources for the network. The conclusion summarises the results of the research and answers the afore mentioned questions but also provides some recommendations for tasks still to be finished.

2

Background

Chapter 2

2
2.1

Background
AMS - IX

The Amsterdam Internet Exchange (AMS - IX) provides an Multiprotocol Label Switching (MPLS)-platform to which a customer can connect their network and exchange data with the networks of others. AMS - IX has several Point of Presences (POPs) distributed around the Amsterdam-area where customers have the possibility to connect their routers to the platform. There are two datacenters between which the core of the network is divided, it consists of four switches with connections to every other access switch in the network (see figure 1).
Interxion AMS5 SARA NIKHEF euNetworks GlobalSwitch Telecity 2 Telecity 4 Equinix

10GE

10GE

10GE

10GE

10GE

10GE

10GE

pxc-eqx-118 PXC

PXC

PXC

PXC

PXC

PXC

PXC

PXC

MLX16

MLX16

MLX32

MLX32

MLX32 MLX32

MLX32 MLX32

MLX16

MLX16

MLX16

MLX16

MLX32 MLX32

MLX32 MLX32

MLX16

MLX16

MLX32

MLX32

MLX32

MLX32

MLX8 GE

MLX8 10/100/1000

MLX8 10/100/1000

MLX8 10/100/1000

MLX8

MLX8 10/100/1000 10/100/1000 10/100/1000

MLX8 GE

Figure 1: Topology of the AMS - IX platform

2.1.1

The platform

was founded by several Internet Service Providers (ISPs) who could save a lot of money if they exchanged traffic directly instead of paying their upstream-providers to reach each other. What they did was buy a shared infrastructure device on which they could all connect to each other. This concept of a shared platform holds true today although the scale has changed significantly. Because of this, a separate, neutral entity was needed to maintain this piece of shared infrastructure. AMS - IX became this entity and it’s neutrality is ensured through an organisational structure which is described in section 2.1.2. The platform uses MPLS[2] for transporting data from one customer to the other and using Virtual Private LAN Service (VPLS), it appears as a single large broadcast domain[3] from a customer’s point of view, so they can setup their peering sessions.
AMS - IX

To peer is for two customers to exchange information about their networks through BGP . Although the customers can reach each other over the platform, this does not mean that they immediately can start exchanging traffic. They will have to talk to the

3

Background

Chapter 2

other customers to peer with. When they have established a session they can start sending and receiving traffic to each other. The optical link of a customer with a 10 Gigabit Ethernet (GigE) connection is being terminated on a so called Photonic Cross Connect (PXC). This device receives the light from the customer router and is able to direct that light to one of two Provider Edge (PE)router. This is all done in hardware without converting the light to an electrical signal. To do this the device essentially uses micromirrors to switch it from one fiber output to the other. This system is able to provide a redundant connection to the network which can recover from a broken switch within 20 milliseconds and without any downtime for the customer. 2.1.2 Organisation

AMS - IX is split up in two legal entities: a corporation and an association. The association is the one and only stockholder of the corporation and consists of the members which are the parties connecting to the AMS - IX platform. Each member has an equal vote in the decision-making of the corporation. Twice a year the members get together for a general meeting to discuss the future of the company. Being a member – and thus stockholder – also means that if the corporation makes any profit this could be beneficial because prices may go down.

2.2

Inter-IPX

Over time AMS - IX also started offering other services than just the ISP peering, one of which are Inter-GRX connections. A GPRS Roaming eXchange (GRX) is an exchange point for mobile operators to exchange roaming traffic and authenticate subscribers of network A when they are using network B, eg. when they are in a different country. This exchange is maintained by a GRX-provider, which connects several operators, mostly from the same continent. Not all mobile operators are connected to the same GRXprovider though, and so the need for a connection between those exchanges arises. A company providing that service is called an Inter-GRX provider, of which three exist in the world: Singapore, Ashburn (VS) and the one at AMS - IX, which is the largest. The GRX providers connect to each other using a separate VPLS -instance on the platform – similar to a Virtual LAN (VLAN). Figure 2 illustrates the structure of the various GRXelements. A new standard that has been developed by the Global System for Mobile Communications Association (GSMA) is IP eXchange (IPX). The GSMA is the developer of standards like General Packet Radio Service (GPRS) and GRX and has been working on IPX since the trend of Next Generation Networks (NGNs) started, networks which use the same mode of transport regardless the payload.[1] More and more traffic between operators is being

4

Background

Chapter 2

Figure 2: The GRX-structure sent over Internet Protocol (IP), be it data, voice or video. The goal of IPX is to supersede and include GRX and provide the following advantages: 1. connections between every kind of service provider instead of mobile operators only; 2. end-to-end Quality of Service (QoS) agreements for IP traffic; 3. any service based on IP can be used between peers; 4. connections can be sold as upstream to other providers. To make the transition to IPX without much impact to the traffic the current GRX structure is essentially upgraded. The diagram from figure 2 would not change drastically after the change. AMS - IX started a trial period of the Inter-IPX service in the end of 2010 and eventually wants to provide this service to all potential customers.

5

Project Details

Chapter 3

3

Project Details

In chapter 1 two research questions were proposed and will need to be answered but there are some requirements that the answers of the questions will also need to meet.

3.1

Service Levels

As was shown in the previous chapter, there are some advantages to the new IPX standard. One of these is the fact that there will be QoS-agreements that have to be met. These agreements are commonly referred to as Service Level Agreements (SLAs) and form an important aspect to this project. Traditionally AMS - IX has provided their members with an Service Level Description (SLD), a document that contains some basic information about connecting to the platform. It defines terms like the demarcation point, availability, performance, and maintenance periods. Although it includes some performance targets, there are no penalties when failing to meet them and therefore is called a Description instead of an Agreement. The Inter-IPX service will require a penalty scheme because of the end-to-end QoS agreements that will need to be met at all times. This means that the targeted values that are present in the SLD will become maximum values in the new SLA that should not be exceeded. To ensure this doesn’t happen AMS - IX will need to start measuring these various indicators. Before the beginning of the trial AMS - IX has been using three demo probes capable of doing so. The probes were tested on three different locations and produced by Accedian Networks. They were directly connected to the access switches, ran some standard measurements and helped baseline the platform to get some initial values for the SLA.

3.2

Assignment

This project will need to make sure that the Key Performance Indicators (KPIs) included in an SLA can be measured over time. If hardware is to be purchased all options will need to be looked at and compared. The data that results from the measurements needs to be stored and visualised in a clear matrix with the data from each probe opposite of each other. This way a clear image of the performance of the network is formed for both AMS - IX and it’s members in the earlier stage of testing. When expanding to more than twenty devices this will become less useful. Eventually a solution has to be provided that gives a clear insight in the measurement of the KPIs, can provide monthly reports to customers with an SLA and also performs reliably in the platform. The Inter-IPX working group within AMS - IX starting the trial will also be followed closely to map their requirements for this assignment. The projectleader is Luisa Garcia Montoya although this research project is supervised by Attilla 6

Project Details

Chapter 3

de Groot, a member of the Network Operations Center (NOC). A detailed description of the assignment provided by Chief Technology Officer (CTO) Henk Steenman has been included in Appendix A. The implementation and visualisation within the web-portal will need to be coördinated with the web developers, monthly reports will be drafted with the Member Relations department and the technical details will be discussed with the NOC. This project will be marked as complete when it’s possible to: 1. measure the defined KPIs over the platform; 2. show this data in the web portal; 3. present aggregated stats publicly; 4. show each customer data that is of interest to them, and 5. scale the solution up to a device per customer. The time schedule and agreements made are also included in the so called ‘Plan van Aanpak’ which is included in appendix B (in Dutch).

3.3

Hardware Requirements

A potential hardware solution has to meet the following requirements: 1. It can measure: (a) One Way Average Delay; (b) Two Way Average Delay; (c) One Way Average Delay Variation also known as jitter; (d) Two Way Average Delay Variation also known as jitter; (e) Frame Loss, and (f) Uptime. 2. Values from these measurements can be stored locally or remotely; 3. There is a possibility to place the device between the customer router and the AMS - IX PE -router.

7

Operations, Administration and Maintenance

Chapter 4

4

Operations, Administration and Maintenance

Operations, Administration and Maintenance (OAM) is a technique used by network administrators to manage their long-distance networks, which can spread over multiple cities, countries or even continents. It enables them to be notified when one of the links in the network shows errors, high delays or any other issues. Doing this manually would be a huge and tiresome task. Based on the events detected by the OAM process other actions can be triggered – eg. taking another routing path – although this is not an internal function of OAM. These kind of functionalities are included in Asynchronous Transfer Mode (ATM) for example, and MPLS includes some tools to troubleshoot Label Switched Paths (LSPs). Higher up the stack, IP also includes some of these manual tools which are included with Internet Control Message Protocol (ICMP).

4.1

Ethernet OAM

Ethernet wasn’t developed with long distances in mind, it is traditionally used as a Local Area Network (LAN) protocol. The links in this case are not of great length and easily manageable by a single person. When Ethernet-networks started to expand, more personnel was needed to look after these links, a company could have multiple locations with different engineers but the network was still in their control. The demand for OAM functionalities remained quite low. These days Ethernet is also being used to connect networks from different companies to each other. Instead of a LAN protocol it is more and more used for Wide Area Networks (WANs), replacing traditional protocols like ATM while users still expect the same highquality connections. Around the begin of the century the demand for the same OAM functionalities grew and were starting to be added to the Ethernet-protocol. In this chapter, four different standards will be discussed that try to fulfill this demand.

4.2

IEEE

802.3ah

In November 2000 a new study group was founded after 300 representatives of different companies got together on a “Call for Interest.” The goal of this group was to add functionalities to Ethernet to use it as Ethernet in the First Mile (EFM). The First Mile essentially being the local loop to the customer. [4] It added various new features that were of importance to Service Providers (SPs) one of which were new types of media that could be used, eg. Passive Optical Networks (PONs) and legacy voice-grade copper wiring. It also provided discovery and monitoring OAM tools on link level to the Institute of Electrical and Electronics Engineers (IEEE) 802.3 standard for Carrier Sense Multiple Access (CSMA)/Collission Detection (CD) networks in 2008. [5]

8

Operations, Administration and Maintenance

Chapter 4

4.3

IEEE

802.1ag

Although IEEE 802.3ah adds some OAM features they are still limited to one link, ie. it can’t provide monitoring for a whole path. This is why in 2007 the IEEE approved a new draft to be included in the 802.1Q standard. This section is known as Connection Fault Management (CFM) and provides end-to-end OAM functionality. [6] 4.3.1 Maintenance Domains and Levels 802.1ag defines a new hierarchy within the different networks that it covers. The top most level consists of a Maintenance Domain (MD) with a specified MD-level. There a 8 levels that can be chosen of which the lowest levels are for the companies on the lowest level of the network, eg. SPs and carriers. The top most levels are for end users and customers. This means that higher level data may be carried over lower levels but lower level data may not propagate to higher level domains. 4.3.2 Maintenance Association and Maintenance End Point Below MDs there exist Maintenance Associations (MAs): groups that consist of two or more Maintenance End Points (MEPs). A MEP is a device that resides at the edge of the domain, has it’s own unique MEP IDentifier (MEPID) and knows all of its peers through a list defined in the MA. It maintains connections called Maintenance Entitys (MEs), with all the other MEPs provided that they share the same MA and MD-level. Figure 3 tries to illustrate this structure.

Figure 3: Structure of Ethernet OAM Over the ME-paths the MEPs keep each other informed about their status and the links in between. These paths are represented by the dotted lines in figure 4. Errors can be propagated and paths can be traced to check for any faulty links. Intermediate devices 9

Operations, Administration and Maintenance

Chapter 4

can also participate in these checks. These would be called Maintenance Intermediate Points (MIPs) and are passive in the OAM domain, ie. they only respond to incoming messages and never originate messages. 4.3.3
OAM

Inter-Domain OAM

measurements can be setup between any two MEPs, even if they are located on two different continents. But it is possible that this connection traverses several other networks which also use OAM. To distinguish between frames from both networks the previously mentioned MD-levels are used. Any lower level MD transparently passes on the frames from the higher levels until the frame reaches a MEP of the same level and is then processed. But it’s also the other way around: lower level frames are not allowed to enter the higher level MDs. For this to work a device which connects a higher level network to a lower level one would need to appear as a MIP to the higher level. This is a partial function of this OAM entity and is therefore also called a MIP Half Function (MHF). See also figure 4.

Figure 4: Inter-Domain structure in OAM

4.4

ITU - T

Y.1731

At the same time the IEEE was developing 802.1ag, the International Telecommunication Union - Telecommunication Standardization Sector (ITU - T) was also looking into Ethernet OAM functionalities. The two bodies made some agreements as not to overlap each other, while both standards would be using the same EtherType (0x8902) and frame format. They did not agree on the naming of the different parts of the OAM hierarchy however, and thus the ITU - T has a different naming convention. A MA is called a Maintenance Entity Group (MEG) by their standard and although the acronym MEP remains the same, in this case it’s short for MEG End Point. The Y.1731 standard also doesn’t define MDs and only considers the MEG-levels. Figure 3 includes both types of naming. [7] 10

Operations, Administration and Maintenance

Chapter 4

The ITU - T additionally added Performance Measurement (PM) to their standard, an important functionality for the mobile operators. These companies require to know how long it would take for a frame to get from one side of the network to the other. Measurements for delay, jitter and frame loss were being implemented and eventually this standard was made official even before the IEEE approved 802.1ag in 2007.

4.5 Performance Assurance Agent
While testing the demo devices from Accedian that AMS - IX had been using, there also was a proprietary protocol that was looked at capable of measuring these indicators, developed by Accedian and called Performance Assurance Agent (PAA). According to the IEEE assignment database [8] EtherType 0x88fc contains the following: The protocol will use a common header multiplexing various subprotocols into a single ethertype. The format is as follow: Version Subtype Flag Offset 8 8 8 8 bits bits bits bits Version of the common header Subprotocol identifier Dependent subprotocol flag bits Offset to the subprotocol PDU

The actual subprotocol PDU location is computed by adding the Offset value to the address of the Offset field itself. For example an Offset of 0 point to the byte following the Offset field. Further details about the contents of the subprotocol Protocol Data Units (PDUs) are unfortunately not available.

4.6

Solution

The protocol of choice on the AMS - IX platform would be the two supporting the measurements of delays from end-to-end: ITU - T Y.1731 or PAA. Because the latter lacks documentation and therefore the way the measurements work can not be checked, it is deprecated. With Y.1731 AMS - IX will be able to measure their required KPIs and the standard is both open and more widely adopted.

11

Research

Chapter 5

5

Research

After knowing the basics of the theory and a good understanding of the demands set by AMS - IX it is possible to search for a fitting solution. Possible vendors for measurement devices will be compared and further details about the OAM standards will be provided.

5.1

Probe Hardware

To know what kind of hardware to use it is important to understand which requirements it must fill. The more important ones were already discussed in paragraph 3.3 and in summary consisted of: • One Way measurements; • Two Way measurements; • Frame Loss; • Uptime; • Storage of values and • using it as an intermediate device. At the moment, only 1 GigE customers are taken in to consideration but eventually there might be an upgrade to include 10 GigE ports as well. This has to be taken in to account when searching for possible solutions. It is also important for these devices to sync to each other so that they all share the same time, this is an essential requirement for the One Way delay measurements. On the other hand, uptime is a value which can not be calculated using one device. This is a result of the requirement that has been set in a draft Inter-IPX SLA, that every customer has to have two connections on two different locations to the AMS - IX platform. This means that the results of both devices need to be compared and only when both of them are down outside of a scheduled maintenance window, it counts as downtime. For this reason, this indicator is not taken into account but will be calculated after the data has been collected and is to be presented to the customer. 5.1.1 Vendors

Knowing the requirements which have to be fulfilled by a certain device makes searching for one a more manageable task. In this section the devices that were looked at will be discussed.

12

Research

Chapter 5

Accedian Accedian is a vendor of service level assurance devices which are primarily built for Ethernet backhauls of large mobile carrier networks. For these operators measuring delays between their sites is essential to ensure high quality voice connections. They have models which support 1 GigE and others that also support 10 GigE, an interesting option when scaling is necessary. These devices had the advantage to already be in use by AMS - IX before starting this project. The NIDs were performing correctly and purchasing them was already a discussed option. They nevertheless had to be tested further and models from other vendors needed to be considered in case these did not fit the requirements after all. Two 1 GigE MetroNIDs and one 10 GigE MetroNODE were provided to use in the platform and provide it with some baseline-values. This also enabled the engineers to see how the network would react to the data exchanged by the Network Interface Devices (NIDs). The acronym NID is used to describe the actual location of the device in the network: essentially it provides the customer with a network interface to connect to the platform. This is the demarcation point for the equipment of the provider. In section 5.2.3 it will explained why AMS - IX would not use the devices in this way, but the term NID will still be used to refer to these devices. The devices include the standard OAM Performance Measurement but also add the proprietary PAA protocol which was discussed in 4.5. Both provide one and two way measurements and counters for frame loss. The PAA protocol was considered as an option but after testing it became apparent that it would not scale to a large number of devices. Because of the fact that all destination Media Access Control (MAC)-addresses have to be manually configured it would make it a tedious task to configure a large amount NIDs. This is in contrast to the IEEE and ITU - T standards which auto-detect MAC-addresses of their peers. A NID has two active forwarding ports, one facing the network and the other facing a customer. This means that as an intermediate device it can be used to measure the whole path that the customer data takes.1 Models exist with copper or fiber ports, or a combination of both. The model that would fit AMS - IX is the EtherNID GE, it has a combination of two Small form-factor pluggable (SFP) fiber ports and three copper ports of which one is the management port. Using this combination, both customers with fiber as well as with copper connections can be connected to the device. Brocade The devices that are in place right now for switching data are from Brocade. This vendor has minimal support built-in for OAM Performance Measurement. The hardware has
1

With some exception on the AMS - IX platform, see 5.2.2.

13

Research

Chapter 5

the functionality to act as a MIP and as such can participate in any Link Traces originating from MEPs. They can also be configured as MEPs themselves but although this has been implemented, one way delay measurements are not supported because they can not synchronise their clocks properly. [9] Test Traffic Measurement (TTM) servers

RIPE

For some years AMS - IX has been working with Réseaux IP Européens (RIPE) to provide them with statistics regarding the quality of the Internet. They do this using servers hosted by a variety of SPs or other IT-related companies throughout the world. All these servers exchange ICMP ping-messages to measure their delays and frame loss to each other. AMS - IX has been hosting four of these devices in datacenters NIKHEF, SARA, GlobalSwitch and TeleCity. Expanding this setup to all eight datacenters would provide AMS - IX with an overview of their internal network as well. Although this would be very helpful, the administration of the servers and the data collected would be done by another company. This situation would not be desirable and also wouldn’t make it possible to collect the measured values to calculate downtime periods. On top of that the devices have stopped working reliably in the network ever since AMS - IX started using the MPLS platform. This was caused by the different paths available to the device to get out of the network. It could take a different path each time and because it only sent out a message every 30 seconds, the Provider (P)-routers in the path sometimes had to relearn their MAC-address. This would mean that the frame would have to pass over the Central Processing Unit (CPU) for the switch to learn the MAC -address again and this would cause a much higher delay. Besides these problems the devices weren’t very scalable because they all needed a Global Position System (GPS) antenna, they couldn’t be placed as an intermediate devices and also don’t support 10 GigE connections. MRV Another supplier that is familiar within AMS - IX is MRV. Their products are used for Dense wavelength division multiplexing (DWDM) connections between datacenters, these devices multiplex different light-streams onto one fiber strand using different wavelengths. This allows AMS - IX to multiply the capacity of a single fiber strand by a factor eight. MRV recently started producing the OptiSwitch 940 which builds on their Carrier Ethernet portfolio. It is marketed as a demarcation device and so multiple customers can be aggregated on the device to one or more uplinks. The 940 model includes OAM for

14

Research

Chapter 5

the first time to measure one and two way delays, in addition to frame loss. The downside is that the series at this moment lack a stable clock synchronisation method.[10] In a later hardware revision it will be able to use Synchronous Ethernet (SyncE) for timing but that would also require additional hardware in the management network. There also exists an OptiSwitch 930 model which suffers from the same limitation but does support 10 GigE interfaces. RAD Besides the known names in the business there are also two new vendors which provide similar solution. One of them is RAD, a manufacturer of products to support carriers and mobile operators. They provide Ethernet OAM functionality with their ETX devices, one of which is the ETX-202A. This devices allows connections up to 1 GigE and provides them with one and two way delays measurement, frame loss and Simple Network Management Protocol (SNMP) polling for the collected values. As with the other devices, an important factor is the synchronisation between clocks. The supplier of the RAD devices couldn’t give a specific value for the average variation between the 202A devices but did mention the 204A model, which similar to the MRV OS940 could sync through SyncE. This would require the additional networking hardware as well. To support 10 GigE connections the supplier could provide AMS - IX with the RAD 1002 model which is a similar demarcation device. Unfortunately this relatively new product is not yet able to support Ethernet Performance Measurement. Omnitron iConverter Another new vendor capable of providing AMS - IX with OAM devices is Omnitron. Its primary product line is a range of media-type converters called iConverter. An important feature that these devices also support is Ethernet Performance Measurement. The GM3 model supports the same measurements for one and two way delay and frame loss while also making it possible to poll these values through SNMP. The devices are used as a media-type converter and thus always sit in between the customer and the platform. GigE customers are supported but the vendor lacks support for 10 GigE connections. Apart from this limitation the clock synchronisation is not specific: the vendor could not specify a variation value for Network Time Protocol (NTP) and didn’t provide any information about other synchronisation methods.

15

Research

Chapter 5

5.1.2

Comparison

With all specifications considered a matrix can be compiled comparing the features of the various vendors. Table 1 gives an overview of the devices that were looked at.
Accedian EtherNID Brocade internal RIPE TTM boxes MRV OS940 RAD 202A iConverter GM3 10GigE yes* yes no yes no no One Way yes no yes yes yes yes Two Way yes yes yes yes yes yes Loss yes yes yes yes yes yes Intermediate yes yes no yes yes yes
SNMP

yes yes no yes yes yes

Sync Accuracy <20 microseconds none 100 nanoseconds none yet** N/A N/A

Table 1: Comparison of different vendors
* Available on another model called MetroNODE. ** Should be available in a next hardware revision.

From this table it can be concluded that only two options pass all criteria: Accedian and MRV. Both vendors would fulfill the requirements passed by AMS - IX although the MRV OptiSwitch lacks an important detail: clock synchronisation. At this moment this is not optimal and would later be improved by adding SyncE support but this would require more investment in adding hardware to the management network that also support Synchronous Ethernet. Accedian on the other hand claims a synchronisation between clocks using their NTP implementation of less than 20 microseconds. Thanks to the demo models that were in use this claim could and has been verified. These observations allow for the conclusion that the Accedian EtherNIDs and MetroNODEs would be the preferable solution to measure the various KPIs over the AMS - IX platform.

5.2

Platform Behavior

Testing continued with the Accedian NIDs to see how the network would react. Some unexpected behaviour was experienced while running the Performance Measurements considering the paths that the OAM PDUs would take over the platform. To understand this a more detailed view of the OAM frames is need. 5.2.1 Multicast Frames

In chapter 4.1 Ethernet OAM has already been shown to add some important features to the lower levels of the Open System Interconnection (OSI)-model. The protocol has been developed as a network layer independent function and as such does not rely on the TCP / IP -stack. The OAM data gets an Ethernet-header with its own specific EtherType 16

Research

Chapter 5

(0x8902) before being put on the wire. Of course this Ethernet Frame also needs a destination MAC-address to send it to which varies depending on the type of message being sent. The configuration of an OAM Maintenance Domain does not require to manually set the MAC -addresses of all the peers. These can be auto-discovered through sending out a multicast Ethernet frame which announces the MEG End Point (MEP) MAC-address to all the peers. These message also function as a keep-alive message and are thus called Continuity Check Messages (CCMs). They are sent out at a certain interval and are not replied to by the other MEPs in the MEG. If a MEP stops receiving CCMs it marks the peer MEP as ‘failed’ after 3.5 times the interval. Besides these two functions, the CCM PDU s also provide the counters for the frame loss measurements, an essential KPI .
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Multicast destination MAC-address 0180.C200.003y Source MAC-address EtherType 0x8902 OpCode (CCM=1) MEL Version Flags Sequence Number (0) TLV Offset (CCM=70) Sequence Number (cont.) MEPID

MEG ID (48 octets)

TxFCf RxFCb TxFCb Reserved (0) End TLV
FCS (cont.) FCS

Figure 5: Overview of a Continuity Check Message Ethernet OAM frame.

An example of a complete CCM Ethernet frame is pictured in figure 5. The multicast destination MAC-address of the CCM PDUs is 0180.C200.003y, where y is the MEG Level (MEL). Apart from the CCM frames there exists one other type of PDU which uses multicast but which is not used within AMS - IX: Link Trace Messages (LTMs). All other messages and replies to those messages – including the delay measurement messages 17

Research

Chapter 5

– are unicast. The source address is always the devices own MAC-address and the EtherType is defined as the hexadecimal-value 0x8902. The PDU also includes the MEG -level, Ethernet OAM version number and an OpCode which defines the type of PDU , which for CCM is 0 (zero). The flags include the Remote Defect Indicator-bit and the specification of the interval. The sequence number remains empty for this specification, as is the reserved field at the the end of the frame. In between these fields the configured MEP and MEG ID s are included, as well as the counters for dual-ended frame loss.[7] The differences in multicast and unicast destination addresses causes some unexpected behaviour on the AMS - IX MPLS platform. Because the platform learns all destination MAC addresses and bases its hardware forwarding on that information, it can quickly switch any unicast Delay Measurement Message (DMM) from one MEP to the port on which the other resides, provided that the destination MAC has already been learned. With multicast addresses this is different, these frames will have to pass through the CPU for rate-limiting purposes and as such have a much higher delay. For measuring frame loss this shouldn’t be much of a problem, until the CPU becomes loaded with other tasks or other broad/multicast frames. When the CPU on the module is too busy it might occur that a CCM PDU gets dropped. This won’t affect the OAM service in general but would count for loss of a frame when this shouldn’t be true for standard unicast packets. To solve this problem the VPLS instance in which the devices reside can be configured with the vpls-cpu-protection option. This enables the switches to flood all broadcast and multicast frames in hardware without sending it up in to the CPU.[11] Although this enables the switches to forward the multicast frames directly it also prevents the CPU from rate-limiting the traffic. This could potentially allow for high levels of broadcast traffic and therefore rate-limiting layer 2 Access Control Lists (ACLs) must be configured on the physical ports which connect to the NIDs. 5.2.2 Label Switched Paths Another problem arose while talking to fellow NOC engineers. They noted that the DMMs which were being exchanged by the MEP s would always take the same path over the platform as long as this measurement would persist. This of course would not matter if this path would be the same for all traffic but within the AMS - IX the paths are different for all kinds of traffic streams. Because the OAM stream sends a frame every 100 milliseconds this stream never changes and its path wouldn’t either (as seen in figure 6a). This means that traffic travelling another path could encounter higher delays which would not be measured by the NIDs. These paths are the four LSPs which run from PE-router to PE-router over every one of the four P-routers. In figure 6 there are two 10 GigE customers presented on the top with paths over the core in the middle, to a 1 GigE customer on the bottom. Different trafficstreams are load-balanced over four of those so different streams may take different 18

Research

Chapter 5

paths. Besides these LSPs there are other paths that won’t change if the streams won’t: the physical links in a Link Aggregation Group (LAG). A backbone connection between an AMS - IX PE and a P-router consists of multiple fiber links which aggregate into one large logical link. The OAM-stream will start sending data over one of these physical links and won’t change, unless interrupted.

PXC

PXC

PE-1-blue MLX-32

PE-1-red MLX-16

PE-1-blue MLX-32

PE-1-red MLX-16

core-loctation-1 MLX-32

core-location-1 MLX-32

core-location-2 MLX-32

core-location-2 MLX-32

core-loctation-1 MLX-32

core-location-1 MLX-32

core-location-2 MLX-32

core-location-2 MLX-32

PE-3 MLX-8

PE-3 MLX-8

(a) OAM measurement using only one a backup path.

LSP

with

(b) OAM measurements using all LSPs

Figure 6: The LSPs over the AMS - IX platform To mitigate these circumstances Virtual Leased Lines (VLLs) can be configured which can be set to use only one of the LSPs. Every NID should then be able to give a different VLAN -tag to each frame so every one of them could be sent over the different VLL s. This would mean all LSPs would be used, as seen in figure 6b. Unfortunate the Accedian devices at the moment do not support configuring more than one VLAN on the OAM measurements. This has been requested as a feature with the vendor and has been promised to be included in the firmware by Quarter 2 of 2011. The second problem regarding the physical links will be a lot harder to solve. This has to do with the way the traffic is load-balanced over the links in a LAG. The algorithm 19

Research

Chapter 5

used for this is Brocade proprietary and so it can not be known. Reverse-engineering the algorithm would be a possibility, were it not that it is most likely changed with every firmware revision, according to NOC engineers. Because of this obstacle it was decided not to go to deep in to this hardware limitation and leave the KPIs as they were intended: as indicators of the network performance. 5.2.3 Redundancy

One of the design requirements is also to be able to put the devices directly in the connection between the customer and the platform. AMS - IX provides a very redundant network which uses the PXCs to switch a customer from one of the PE-routers to another in case of a failure (see also section 2.1.1). It also uses MPLS to provide backup-paths over each core switch. This very redundant and resilient network is as weak as it’s weakest link. If a device like a EtherNID would be placed between a customer and the platform a new single point of failure would be created. For this reason this requirement has been let go for the time being. Because the NIDs are all connected in the same way that a customer would and the traffic they send is completely separate from their traffic, there will not be a significant difference in the measurements for a new single point of failure to be created.

5.3

Timing

While running test measurements on the equipment provided by Accedian there were some unusual patterns to be seen in the graphs between the MetroNIDs and the MetroNODE. The MetroNODE one way delay graphs were shaped like a sawtooth from one of the NIDs with an inverse of that graph to that NID. This could indicate some kind of unstable path, but the two way delay measurements were unaffected by this problem. After consulting with the vendor it became clear that there was a clock drift issue between the two models; a typo in a constant used by the time syncing algorithm. Solving this problem helped to understand the importance of synchronised clocks between the different NIDs. Because of the high speed network used by AMS - IX variations of a couple of microseconds are noticeable in the graphs. 5.3.1 Hardware Timestamping in NTP

The probes use NTP to synchronise to a common time source which is a TTM server. These distributed servers all use GPS antennas on the roof of their datacenters to receive their proper timing. To reduce the variations Accedian recommended to look for a time source which could time stamp their NTP-packets in hardware, instead of letting the software do it and having to travel all the way through the Operating System (OS) and networking-stack which is the case with the Linux-based RIPE TTM devices. In figure 7 20

Research

Chapter 5

the structure of an NTP packet is shown. The timestamps consist of 64-bits and the last two are used to sync a client to a server. The server stamps the Received field when it processes the incoming request and stamps the Transmit field when it sends out a reply.[12] The synchonisation process as well as the formula the client uses to set its own time, can be seen in figure 8.
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

LI

Status

Type Estimated Error Estimated Drift Rate Reference Clock Identifier Reference Timestamp Originate Timestamp Receive Timestamp Transmit Timestamp

Precision

Figure 7: Structure of an NTP packet. The RIPE TTM devices would stamp their packets in the Application layer of the OSImodel but with a hardware timestamping device the packet would receive its timestamp just before being put on the wire (as seen in figure 9). Although this is an OSIlayer violation, it would improve the quality of the timing and above all make it more stable. The variations should become much smaller because CPU load wouldn’t affect the accuracy of the timestamps. To test this claim, one of the Accedian MetroNIDs was set to be an NTP-server itself. These devices work with hardware-based Field-programmable gate arrays (FPGAs) and always use hardware timestamps. The MetroNODE and other MetroNID were then synchronised to this NID. After letting the synchronisation take place over 24 hours the one way delay graph appeared more smooth and continued with less spikes and dips over the following days.

21

Research

Chapter 5

Client

Server

Originate Timestamp
req ue

st

network latency Received Timestamp Transmit Timestamp
Network/OS stack delay

response time

l rep

y

Current Time= network latency + TransmitTS = (RecTS - OrigTS) + TransmitTS

Figure 8: Synchronisation process between a client and the NTP-server.

Software Timestamp
NTP Application Presentation Session

Hardware Timestamp

Transport Network Datalink Physical

Figure 9: The path that an NTP-packet travels through the OSI-layers.

22

Implementation

Chapter 6

6

Implementation

In this chapter the physical implementation of the devices and their configuration will be discussed. The way of collecting data will be looked as well.

6.1

Hardware

In section 5.1.2 it was shown that the only fitting solution would be the EtherNID devices from Accedian. After consulting with the supervisor and CTO eight of these devices were ordered. One for every 1 GigE switch on the eight colocations in the Amsterdam area, these switches are represented by the bottom row of devices in figure 1. The 10 GigE MetroNODEs will be ordered and implemented in a later stadium after more testing with the GigE devices. Together with Hardware Implementation Manager Ariën Vijn it was coördinated where and how the devices would be placed within the AMS - IX racks. For the most part the devices were connected through copper Gigabit Ethernet links except for the devices on TeleCity 4 and Interxion which connect to the platform using fiber links. All the management ports were connected to the different management switches and given an IP address in the AMS - IX space. The ports connecting to the production network remain without IP-addresses because OAM does not need them and can be put to better use when given to customers. The Domain Name System (DNS)-records were already configured for the management ports and the layer 2-ACLs on the switch-ports were also configured with the corresponding MAC-address to allow the NIDs to send traffic. Location Equinix Interxion NIKHEF SARA TeleCity 2 TeleCity 4 GlobalSwitch euNetworks
IP

91.200.16.181 91.200.16.182 91.200.16.183 91.200.16.184 91.200.16.185 91.200.16.186 91.200.16.187 91.200.16.188

Hostname nid-eqx-015 nid-ixn-016 nid-nik-011 nid-sar-010 nid-tel-012 nid-tc4-137 nid-glo-013 nid-eun-014

MAC -address 0015.ad07.171f 0015.ad07.1647 0015.ad06.a2df 0015.ad06.a5cf 0015.ad06.a6df 0015.ad06.a19f 0015.ad07.174f 0015.ad07.176f

Table 2: Device configuration

The hostnames were chosen to reflect the location of the NID and the switch to which it connects. Within the AMS - IX platform, this is a standard naming convention. The ports belonging to the NIDs were all put in a separate VPLS-instance on which CPU Protection was configured so that multicast frames could be flooded in hardware (see 23

Implementation

Chapter 6

section 5.2.1). This separate VPLS instance was chosen so that not all ports had to be configured with layer 2-ACLs to rate limit broadcasts. This choice won’t change the measurements in any way because the Delay Measurement Messages will still take one of the same paths that the customer-traffic would take. When the vendor has implemented the function to support more VLANs per MEG, separate VLLs will be configured over all four LSPs. 6.1.1 Configuration

NID

The Accedians do not support exporting their configuration in a standard text-file, therefore a list of Command Line Interface (CLI) commands to execute on each of the devices was composed. An example of such a config is given in Appendix C. The networking configuration would be as shown in table 2 (lines 1 through 11 in Appendix C). All ports except for the Network and Management ports are disabled and only the later gets an IP-address. To poll the values measured by the NIDs SNMP is set to accept connections with specific communities (l. 12-17 ) and also to send traps to a log server (l. 18 ). On top of these traps the devices send their errors and messages to a syslog server on the management network (l. 19-20 ). This double logging is part of the test trial and new logging-method based on the Zabbix software. An important part of the configuration are the NTP time settings (l. 21-24 ). All of the NID s are made to listen to the NID on the NIKHEF location which gets its timing from a RIPE TTM server on the same location. The DNS server contains a record to point to that NID which is time-nid.noc.ams-ix.net so that changing the source would merely be a change to the DNS record. The synchronisation-mode is set to high-resolution which means that each device polls the NTP server every minute to check its time. The configuration of the OAM measurements follows the same structure which has been shown in figure 3. It starts with defining a MD with a Level and a name which follows a DNS-like convention (l. 25 ). It is set on internal index-number 9 below the default MD s in the device. Below this domain, a MA is created that includes the MEPID s of the devices that will participate in the measurements (l. 26 ). It also receives a name, an index-number, a reference to the previously configured MD-index and a value for the CCM -interval. This is the time in milliseconds between each CCM -message between the MEP s. Accedian’s lowest supported interval is 1000 milliseconds. After these global settings are configured every device needs to get it’s own MEPID. This is configured together with specifying a port on which OAM needs to be enabled, a reference to it’s top MA and which errors it should propagate to it’s peers (l. 27 ). When a MEP is configured it can start measurements to the rest of the MEPs, but some values need to be defined (l. 28-34 ). The interval is set to 100 milliseconds – again the lowest value supported by Accedian – which means that one and two way delays are measured ten times per second. All these values are aggregated within a specific reference period. This is set to 1 minute: the average value over 600 measurements is 24

Implementation

Chapter 6

taken. To allow alarms to be generated and traps to be sent, three types of thresholds are defined as shown in table 3 Type Max Delay One Way 2 ms Two Way 2 ms Definition This is the absolute maximum delay value which can be measured by one Delay Measurement Message. If the times the maximum delay value is exceeded reaches this value an alarm is sent. If during the reference period this value is exceeded an alarm is immediately sent. This is the absolute maximum jitter value which can be measured by one Delay Measurement Message. If the times the maximum jitter value is exceeded reaches this value an alarm is sent. If during the reference period this value is exceeded an alarm is immediately sent.

Delay Threshold

3 ms

3 ms

Average Delay

1 ms

1 ms

Max Delay Variation

1 ms

1 ms

Delay Variation Threshold

2 ms

2 ms

Average Delay Variation

1 ms

1 ms

Table 3: OAM threshold values

For the SLA measurements the Average Delay values are the most important, while the maximum values are mostly used as indicators when a large error occurs on the platform. The average thresholds are set to their lowest supported values of 1 millisecond while the draft SLA contains much lower values. A feature request has therefore been sent to Accedian to support the values below 1 millisecond. Frame loss is also defined in the settings but here a reference period can not be set. As shown before, the CCMs include the counters for packet loss and so this measurement inherits that value of 1000 milliseconds. A threshold is also set to 1 percent and the reference period is kept the same as the delay measurements (l. 35-41 ).

6.2

Data Collection

A Virtual Machine (VM) running Linux has been set up to collect the data using Multi Router Traffic Grapher (MRTG) which was written to poll routers for their traffic statistics. The program is adaptable to any situation and can also be used to collect the information from the NIDs every minute. To do this special databases called Round Robin 25

Implementation

Chapter 6

Databases (RRDs) are used to store the values over time. These databases store values every interval (eg. one minute) and consolidate these values after a certain period of time. This way not all data has to be kept at a very low resolution, disk space can be saved and computation with those values takes less time. By default MRTG uses it’s own text-based database but to set the interval to lower than 5 minutes it can call upon RRD tool, the software handling these database-files. An RRD-file can store two KPIs in separate datasets. For the Accedian EtherNID setup the one and two way delays are stored in one file, one and two way jitter in another and frame loss is alone in a third file. Three files are needed for any measurement from on NID to the other. For 8 devices this would correspond to 8 devices * 7 peers * 3 files = 168 RRD-files. Every file needs a configuration within MRTG and thus 168 sections of configuration are needed. Such a section would look like this:
Target[5min_delay_nid-eqx-015_nid-ixn-016]: .1.3.6.1.4.1.22420.2.7.1.1.2.1.10.1&.1.3.6.1.4.1.22420.2.7.1.1.4.1.10.1: COMMUNITY@nid-eqx-015.noc.ams-ix.net Title[5min_delay_nid-eqx-015_nid-ixn-016]: One and Two Way Delay MaxBytes[5min_delay_nid-eqx-015_nid-ixn-016]: 100000000

The first line contains the target specification between square brackets and after that specifies the SNMP Object Identifiers (OIDs) to be polled with a specific community string at a certain host. This line is the only useful line, the other lines containing Title and MaxBytes are settings required by the MRTG software but are not used by RRDtool. MaxBytes for example is nothing more than the maximum value to be considered realistic but is expressed in bytes because the program was designed to collect exchanged bytes. Another property which shows this specific purpose is that MRTG by default only expects values it’s measuring to grow over time. Delay measurement vary constantly and so all the datasets are set to the ‘gauge’ type, to allow the OID values to fluctuate. The OIDs to be polled are listed in table 4. OID
.1.3.6.1.4.1.22420.2.7.1.1.2.1.10.index .1.3.6.1.4.1.22420.2.7.1.1.3.1.10.index .1.3.6.1.4.1.22420.2.7.1.1.4.1.10.index .1.3.6.1.4.1.22420.2.7.1.1.5.1.10.index .1.3.6.1.4.1.22420.2.7.1.2.2.1.18.index

Value One Way Average Delay in microseconds One Way Average Jitter in microseconds Two Way Average Delay in microseconds Two Way Average Jitter in microseconds Packet Loss in millions of a percent

Table 4: OIDs for various KPIs

After the whole MRTG configuration is complete and the daemon is started, values start to get collected. The RRD-files which did not exist yet are automatically generated and filled with the average values every minute. By default the values get consolidated to averages over 30 minutes, then 2 hours and finally to 1 day.

26

Implementation

Chapter 6

6.2.1

Scripts

All these collected KPI measurements will eventually need to be presented to the members, the NOC and other parties. RRDtool includes function for that: rrdtool graph. It can visualise the data from the files by graphing it and writing a standard image file. To do this rrdtool graph is given parameters to know which RRD-file it must use, where it should write the image, what kind of values must be written in it, which timeperiod much be shown, etc. Eventually a graph similar to the one in figure 10 would be the result for the one and two way delays from Interxion to NIKHEF
RRDTOOL / TOBI OETIKER

delay from nid-ixn-016 to nid-nik-011
500

microseconds

400 300 200 100 0 Sat 12:00 Sun 00:00 Sun 12:00 Mon 00:00

One Way Two Way Copyright ©

95th SLA maximum minimum 210.65 µsec 500 µsec 226.61 µsec 174.42 µsec 434.51 µsec 1000 µsec 435.61 µsec 428.35 µsec 2011 AMS-IX B.V. [updated: 10-Jan-2011 10:58:10 +0100 ]

Figure 10: Graph generated by RRDtool These images can be generated every reference period for all devices which means 168 images would be replaced each minute. This would not be very efficient because they will not be looked at all the time. To make this more efficient a bash-script is written to be called from the web browser which takes arguments for source and destination NID, time period and KPIs. This script is included in appendix D. This script then generates the graph image on-the-fly using the raw output from the RRDtool command. This way, there is no image written to disc but directly to the requesting browser. This also adds the advantage that the image is always up-to-date, with every refresh it looks at new data. For one minute devices this would be an improvement in efficiency but to report the values to customers each month this method would be too much, considering that these images would not change over the course of one month. A script has been written to run on the first day of each month to generate a graph and calculate the values of the various KPIs over the previous month. The images are all written to a directory which can be reached from the internal network to include them in the SLA reports to be sent to customers, the values are reported to customers through a 95th percentile. This is a formula in which the highest 5 percent of the data is not included in computing a maximum value over that month. This allows for some fluctuations in the graphs. For the NOC engineers an overview page has also been created with a matrix contain27

Implementation

Chapter 6

ing all the measurement graphs that will enlarge when the cursor moves over it. A screenshot of this page is shown in figure 11.

Figure 11: Overview page KPI measurements

6.3

Timing Sources

In the measurement graphs it is obvious that the line representing one way delay is not as consistent as the two way delay. Because this value is the only one that is influenced by difference in time between the NIDs, this pattern is not seen in any of the other graphs. Two way measurements base their departure-time on the originating clock and because the packet makes a round-trip it’s arrival-time is also based on the same clock. One way jitter measurements do not base their timing on both devices. One of the NID s receives two packets within a certain period of time and if the period in which it receives the third packet is higher or lower, this would count as delay variation, or jitter. This is shown by the Reserved place for RxTimeStampf in figure 12, here the receiving NID places it’s timestamp to compare to the previous received values.

28

Implementation

Chapter 6

Using this figure it can also become clear how the one way delay measurements work. The fields in the header of the frame are all discussed in section 5.2.1, apart from the destination MAC-address which is now a unicast address of the receiving NID. The originating device sends this frame with a timestamp of his clock in the TxTimeStampf field. It travels through the network until the frame reaches it’s destination address. Here it gets timestamped again immediately after being received in the RxTimeStampf field. It can then be processed by the OAM system, it subtracts TxTimeStampf from RxTimeStampf which results in the one way delay between the two NIDs.
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Unicast destination MAC-address Source MAC-address EtherType 0x8902 MEL Version Flags TLV Offset (CCM=16) TxTimeStampf Reserved for receiving device (for RxTimeStampf) End TLV
FCS (cont.) FCS

OpCode (CCM=1)

Figure 12: Overview of a One Way Delay Measurement Ethernet OAM frame. One way measurements essentially compare the timestamp of the originating device with the arrival time at the end-device. Subtracting those two values results in a time difference, called delay. Because these delay values are quite low within the AMS - IX platform it becomes quite essential to have the clocks synchronised as precisely as possible. In section 5.3.1 it was already discovered that hardware timestamping would improve these measurements. A new time source would be needed because the RIPE TTM boxes did not support hardware timestamping and were reaching their end-of-life. Near the end of the project a still on-going search was started to find such a new time source. The requirements for such a device are in the first place to support hardware timestamping, it should be easily implemented within the current network and it should be as reliable as possible with a small budget. Table 5 lists the options that were looked. Note that only two devices support NTP hardware timestamping. This is because by default NTP is an application layer protocol and thus inserting a hardware timestamp at the data link layer would be an OSI-layer violation. The only possibility to solve this is to implement the whole function in hardware using FPGAs. This is precisely what both the Z-1000 and MetroNID with GPS use to support NTP timestamping. The former is an appliance integrated with the antenna that can be placed directly on the roof. It is powered through the Ethernet cable so there doesn’t have to be a power outlet outside and doesn’t take up rackspace. The latter is a familiar name and was discovered when talking with Accedian about accurate time-syncing. The vendor produced a small number of devices with a GPS receiver which would enable them to distribute 29

Implementation

Chapter 6

that accurate time. Device EndRun Unison Oscilloquartz 5220 Meinberg M300 Symmetricom S200 Brilliant Telecom Z-1000 Accedian MetroNID GPS GPS yes yes yes yes yes yes GigE no no no yes no yes HW Stamp no no no no yes yes Accuracy <10 microseconds <20 microseconds <50 microseconds 7 microseconds 250 nanoseconds 250 nanoseconds Cost e2,775.00 N/A e2,482.00 e3,995.00 e5,570.00 e1,229.00

Table 5: Comparison of NTP time sources

All devices support GPS as a time source and their accuracy is based on the difference between the current GPS time and the time in the NTP-packet when it is being put on the wire. With hardware timestamping this is as low as possible because the time is set right when the NTP-packet leaves the device. But this is only true when the timestamp has just left the device, while taking it’s path along the network the packet can face some delay which will make it less accurate. NTP tries to consider this delay in setting the time for a host but only if the delays do not vary a lot over time. A solution that could be implemented to solve the variation in the network is IEEE 1588v2 or Precision Time Protocol (PTP). This relatively new protocol is said to deliver time around a network in sub-second accuracy.[13] The disadvantage of this new standard is that it requires some new hardware within the management network which would also support PTP. Although this is a possibility in the future, at the moment the Accedian NIDs do not support PTP but will do so in the coming year. The four devices that do not support hardware timestamping are for the most part embedded Linux servers with a standard NTP daemon and are subjectable to delays caused by the OS and networking stack. Vendors like Oscilloquartz and Symmetricom do produce hardware timestamping devices but they are targeted at big operators who need synchronisation for their Synchronous Digital Hierarchy (SDH) network, for AMS - IX the performance increase would not be worth the investment. A requirement which was taken for granted is the maximum length of the antenna cable. For roof-access in one of the datacenters this has to be at least 100 meters but preferably longer. The Brilliant Telecom Z-1000 has a maximum of 100 meters which is consistent with the maximum length of Ethernet cabling. The Accedian MetroNIDs with GPS can use any standard antenna and depends on that manufacturer to specify a maximum length of the cable. This can be as short as 30 meters so a reliable vendor must be found. This search for a new reliable time source is still ongoing. In the next chapter the findings of the research that has been carried out will be summarised and compared to the previously set requirements. 30

Conclusion

Chapter 7

7

Conclusion

In the introduction of this paper two questions were stated that needed to be answered: 1. How can the performance of an OSI-layer 2 platform, such as provided by AMS - IX to its customers, be measured? 2. How can AMS - IX provide their customers with the measured values to be included in an SLA? The answers to these questions were subject to five requirements listed in section 3.2 which are discussed in the following paragraphs.

7.1

Requirements

The first requirement that the solution had to fulfill was to measure the KPIs as defined in section 3.3. The possible solutions for this were researched by comparing different standards with each other. The one that could provide the platform with measurements of nearly all KPIs was Y.1731 developed by the ITU - T. Only the ‘uptime’ measurement could not be performed because this would need some additional checking whether the other provided port of a customer also had the same problems. Showing the data of the other KPIs and making the stats publicly accessible, were the second and third requirements. These were solved by using SNMP polling over the management network to collect the values and graph those on-the-fly. The values are one-minute-averages and provide a performance overview of the whole platform. Customers can select between which two devices they want to see the values for delay, jitter and frame loss. The same principle will be true for the public stats, but these will be provided by Synleaf. This company has worked with AMS - IX in a subproject which enables them to show the real-time values measured on the platform. The use of a web portal in which customers can see the data that they need is only part of the fourth requirement. The customer would also be interested in the values over a certain month. This would enable them to see if AMS - IX has met their predefined SLA. This is done using the 95th percentile formula, which sorts the values from the past month from high to low, discards the top five percent and takes the maximum value of the remaining 95 percent. This allows for some sudden changes in the values without affecting the performance indicators over the whole month. The last requirement was actually not taken in to account, as has been discussed in section 5.2.3. It would require to insert a NID in every connection between a customer and the network. Because the measurements are not based on the traffic of the customer and the device is connected in the same manner that any other customer is, this will not improve the results of the measurement significantly. And because this adds a new 31

Conclusion

Chapter 7

single point of failure in an otherwise completely redundant network, this requirement was dropped.

7.2

Research Questions

In summary, the first question has been answered by using two standards provided by the IEEE and ITU - T. The measurement functionalities are provided by ITU - T Y.1731 and meet all the measurement demands from section 3.3. The devices meeting these demands as well and also supporting the ITU - T standard are produced by Accedian Networks, where new devices also have been ordered. The values collected by the devices are stored and processed by a server with RRDtool. This answers the second questions as well, because this tool is also being used to graph the data and calculate the monthly values to be provided to the customer with an SLA. All requests by AMS - IX for this project have been met but there are two tasks which have yet to be checked off.

7.3

Recommendations

One of the important observations made while further testing the new devices was that the synchronisation of the clocks would need to be very precise. Because the platform has such a high performance, the smallest differences between the clocks of the devices would be visible in the one way delay graphs. The time sources AMS - IX uses at the moment had their maintenance contracts ended and are not able to timestamp NTP reliable enough, therefore a new set of NTP servers is being searched for. These will be needed in the coming months to provide the platform with a reliable source of time and minimise the variations in the one way delay graphs. Another task that has yet to fulfilled is the implementation of 10 GigE NIDs, called MetroNODEs. These devices will be needed to supply the customers connecting through a high speed link with reliable values as well. Adding these devices also means that the data stored will increase tremendously. At the moment there are 18 switches with 10 GigE connections, add to them the 8 existing devices and the count of RRD-files will rise to (24 * 23 * 3 =) 1656. This shouldn’t pose a problem for the server but the VM-guest might need some speed and memory upgrades. The 10 GigE devices are already ordered with Accedian but only under the condition that three features will be added to the software: Polling of the Instantaneous Values from the OAM DMM measurements. This allows for the generation of real-time graphs, a subproject that was started together with Synleaf[14]. At the moment these values are acquired through the proprietary PAA protocol (section 4.5). 32

Conclusion

Chapter 7

Multiple VLAN-tagging in a MA/MEG of OAM. Using this feature, the measurements of the devices can be switched over each LSP. Without it, the measurement stream will only use one of the paths which generates a less representative result. Threshold for alarms on DMM lower than 1 millisecond. This enables for traps to generated quicker, should a problem arise on the platform. The manufacturer conceded with these requirements, which should all be implemented by the second quarter of 2011. By that time, the new 10 GigE devices would be fully implemented and tested and a reliable time source will be found as well.

33

Performance measurement infrastructure project

Appendix A

A

Performance measurement infrastructure project

Stagiair: Michiel Appelman Begeleider: Attilla de Groot

A.1

Introduction and Purpose of the project

This project aims at building an infrastructure that measures and reports on the performance of the Amsterdam Internet Exchange (AMS - IX) Metro Ethernet Network. Performance of the network can be identified by the following parameters: • Availability of the service • Packet loss • One way or round trip packet delay • One way or round trip packet delay variation (jitter) A number of different applications can be distinguished that need or do not need to be implemented. 1. An overall overview of the performance of the network by adding measuring devices to the each of the access routers and measure the performance between these. This will provide an average performance overview as can be experienced by every customer and will set a baseline on AMS - IX service levels. 2. For specific customer groups it might be necessary to have more direct measurements between the participants in these groups. The idea is to put a measuring device in the access connection of the customer and measure the performance between this device and similarly connected devices from customers in the same group. The purpose of this setup is to be able to define Service Level Agreements (SLAs) and measure adherence to this SLA 3. The definition and specific implementation of Operations, Administration and Maintenance (OAM) on the AMS - IX platform to allow both AMS - IX Network Operations Center (NOC) and possible customers to see performance of the AMS - IX platform the L2 level. For each of the three mentioned applications it is necessary to: • investigate how it can be implemented • test implementation in the lab 34

Performance measurement infrastructure project

Appendix A

• evaluate impact in the platform • after agreement with the AMS - IX NOC implement the solution An important aspect of the project is presentation of the measurement results and conformance on the defined service levels on the AMS - IX website The project will be split into different stages:

A.2

Stage 1: Measurements and reporting on Key Performance Indicators (KPIs) between a selected subset of the AMS - IX access routers.

Initial devices selected are Accedian MetroNID en MetroNode 10G. For the initial trail we have two metroNIDs connected to the GE access routers at Equinix and Interxion. The single MetroNode10G need can be connected to any suitable 10GE access router on any of the other sites. The following need to be done: 1. Setup all three devices to measure the critical KPIs (Layer 3) • • • • Availability of the service Packet loss One way or round trip packet delay One way or round trip packet delay variation (jitter)

2. Describe how the parameters are derived and give an estimate of the reliability of the measured parameters. Specifically this is the case for the one way delay parameter as this is dependent on a synchronized clock or a vendor specific algorithm. 3. Create a web page where for the different KPIs performance matrices are displayed. 4. Based on the experience with the devices decide if these are the right devices to build the service on. (a) If “yes” continue with Stage 2 (b) If “no” select and repeat with different device. 5. An additional test and of influence for the final decision to go on with the Accedian boxes is the performance of the box in a customer connection. To be tested in the lab: (a) Simulate customer connection to the platform terminated on the MetroNid and/or MetroNode10G instead of directly to the platform. (b) Test performance via this indirect connection between two in such a way connected customers. (c) Repeat measurement of the KPIs as above (in this stage no need for website) 35

Performance measurement infrastructure project

Appendix A

A.3

Stage 2: Measurements and reporting on KPIs between a selected subset of the AMS - IX access routers

Stage 2 is essentially the generalization of stage 1 point 1 to 4 1. Order devices for all access switches 2. Create web based performance overview between all access routers (see A.2.3) (a) Since there are a lot of devices it is essential to pay specific attention to the user interface to the data. Cooperation with member relations and or marketing might be required here. Other stages to follow later.

36

Plan van Aanpak

Appendix B

B
B.1

Plan van Aanpak
Inleiding

Voor de AMS - IX zal in deze afstudeerperiode een onderzoek worden uitgevoerd met betrekking tot performance measurement op het platform. Dit document zal daarbij als leidraad worden genomen. In de volgende pagina’s zal een beschrijving worden gegeven van het project. B.1.1
AMS - IX

AMS - IX is één van de grootste Internet Exchanges in de wereld en biedt aan haar klanten een Layer 2 verbinding aan met overige (internationale) netwerken. Het kantoor bevindt zich in de binnenstad van Amsterdam en de apparatuur is verdeeld over meerdere datacentra in en rondom de hoofdstad. Op deze punten bestaat de mogelijkheid voor bijvoorbeeld Internet Service Provider (ISP)’s om verbinding te maken met het platform. Dit onderzoek wordt begeleid door Attilla de Groot van het NOC maar heeft betrekking op het gehele bedrijf. In het NOC zitten de network engineers die zich bezig houden met het beheren van het platform van AMS - IX. Ook zorgen zij ervoor dat het netwerk mee gaat met de tijd en kan voldoen aan de steeds groeiende stroom data.

B.1.2

Inter-IPX

Dit onderzoek wordt onderdeel van een groter project voor een nieuwe service bij het bedrijf. De service, Inter-IPX, zorgt voor een verbinding tussen meerdere IP eXchange (IPX)es en is door de Global System for Mobile Communications Association (GSMA) gestandaardiseerd. IPXes zijn netwerken van mobiele operators die data uit willen wisselen buiten het publieke internet om. Deze operators stellen ook eisen met betrekking tot Quality of Service (QoS) aan het platform van AMS - IX. Voorlopig wordt de service als trial aangeboden maar de eisen moeten worden vastgelegd in een nieuwe SLA. Om te weten of aan deze SLA kan worden voldaan zal meetapparatuur nodig zijn.

B.2

Doelstelling

Op dit moment is er nog geen exacte meting mogelijk op het netwerk van AMS - IX. Dit is echter wel nodig als men een SLA heeft waaraan voldaan moet worden. Het doel is om een infrastructuur op te zetten die de performance van het AMS - IX netwerk meet en rapporteert. Onder performance worden de volgende KPI’s verstaan:

37

Plan van Aanpak

Appendix B

• Availability (up-time) • Reliability (packet loss) • One Way Delay and Variation (jitter) • Two Way Delay and Variation (Round Trip Time (RTT))

B.3

Projectopdracht

Het bedrijf heeft al een drietal apparaten (zgn. probes) geselecteerd van Accedian Networks die als demo worden gebruikt. Deze zijn direct aangesloten op de access switches en zullen in de beginfase van het project worden gebruikt om te zien of ze voldoen aan de gestelde eisen. Mocht dit niet zo zijn dan wordt een lijst van alternatieven opgesteld. Tevens wordt de nauwkeurigheid van de probes met betrekking tot de One Way Delay (OWD) metingen onderzocht. Zodra de geschikte leverancier gevonden is, zal de projectgroep gaan kijken naar de mogelijkheid om de probes direct tussen de klant-aansluiting te plaatsen. Hierdoor komen ze direct in het pad tussen de twee klanten en worden de metingen meer betrouwbaar. Een ander onderdeel houdt in dat de probes uit te lezen moet zijn. De data die opgeslagen is zal overzichtelijk moeten worden gepresenteerd in een website. De data die hiervoor nodig is bestaat uit de verschillende KPI’s die al genoemd werden op pagina 37. De gegevens uit de probes zouden tegenover elkaar uitgezet moeten worden per KPI een matrix wordt getoond. Uiteindelijk moet dit leiden tot een implementatie die voldoet aan alle eisen met betrekking tot meting van verschillende KPI’s, rapportage aan de klant en stabiliteit in het platform. Van belang is een afstemming met de projectgroep die gaat over de ontwikkeling van de Inter-IPX service. Projectleider hiervan is Luisa Garcia Montoya. Dit onderzoek valt echter direct onder projectlid Attilla de Groot die vanuit het NOC meewerkt. Een gedetailleerde beschrijving van deze opdracht door AMS - IX is te vinden in Bijlage B.

B.4

Activiteiten

De volgende activiteiten worden tijdens de loop van de afstudeerstage uitgevoerd: 1. Informatie verzamelen Opdracht AMS - IX heeft een opdracht geformuleerd (zie ook Bijlage A) die uitgevoerd moet worden. Deze zal allereerst moeten worden vernomen. 38

Plan van Aanpak

Appendix B

Achtergronden Om de achterliggende gedachte van de opdracht te kunnen achterhalen is het van belang om op de hoogte te zijn van de voorgeschiedenis. Dit is te achterhalen door te praten met medewerkers die hier meer ervaring mee hebben. 2. Plan van Aanpak Opstellen Het Plan van Aanpak wordt als leidraad gebruikt voor de rest van het onderzoek. Deze wordt opgesteld aan de hand van informatie die verkregen is over het bedrijf, de afdeling en de opdracht. Bevestigen Met de school- en bedrijfsbegeleider wordt overlegd of het opgestelde document correct is en deze kan worden gebruikt. 3. Onderzoek Opties Is het Plan van Aanpak akkoord dan wordt het onderzoek gestart. Verschillende oplossingen worden bekeken en vergeleken met de eisen van het bedrijf. Oplossing De beste oplossing wordt gekozen met de daarbij behorende toelichting. Deze wordt daarna uitgewerkt. 4. Implementatie Plaatsing Zodra duidelijk is hoe de gekozen oplossing zou behoren te werken wordt deze in het platform toegepast. Dit houdt in dat eventuele hard- en software in gebruik zal worden genomen. 5. Evaluatie Gebruik binnen infrastructuur Verschillende problemen kunnen naar voren komen tijdens het gebruik van de gekozen oplossingen en die zullen opgelost moeten worden. 6. Rapportage Presenteren Alle voorgaande stappen zullen uiteindelijk worden gedocumenteerd in een verslag en gepresenteerd worden aan het bedrijf en de hogeschool.

B.5

Producten

Verschillende producten zullen worden opgeleverd naast het uiteindelijke verslag. B.5.1 Plan van Aanpak

Dit document is het eerste product dat zal worden opgesteld. Hierin worden de planning, andere producten en deelnemers beschreven. Het biedt een houvast gedurende de looptijd van het project. 39

Plan van Aanpak

Appendix B

B.5.2

Planning

Voor AMS - IX is het van belang te weten hoe de komende tijd wordt ingedeeld. Deze planning zal gepresenteerd moeten worden op de project vergadering van maandag 20 september. B.5.3 Onderzoek Apparatuur

Om een beslissing te maken tot aanschaf van apparatuur zal er allereerst een gedegen onderzoek moeten worden gedaan. De uitslag hiervan beïnvloedt deze beslissing uiteraard. B.5.4 Verslag en Presentatie

Dit zijn twee producten in één die te maken hebben met de afronding van het project en de stageperiode. Afsluitend wordt het verslag gepresenteerd aan de hogeschool.

B.6

Randvoorwaarden

Het project loopt vanaf 1 september 2010 en zal 20 weken in beslag nemen. Het einde van het project zal na oplevering van het eindproduct zijn of na afloop van 100 werkdagen. Als eindproduct wordt een volledig werkend systeem bedoeld waarmee kan worden bepaald of de gestelde SLA’s worden gehaald. B.6.1 Randvoorwaarden

Het eindproduct kan als succesvol worden beschouwd als, 1. de gestelde KPI’s kunnen worden gemeten over het platform; 2. de data hiervan kan worden weergegeven op een website; 3. geaggregeerde statistieken publiekelijk weergegeven kunnen worden; 4. klanten hun eigen specifieke gegevens overzichtelijk kunnen benaderen; 5. het uiteindelijk mogelijk is om metingen tussen specifieke klanten in te zien, en 6. deze opzet schaalbaar is naar alle klanten.

40

Plan van Aanpak

Appendix B

B.7

Kwaliteit

De kwaliteit van het project staat of valt door de terugkoppeling en overzichtelijkheid. Voor het Plan van Aanpak en andere documenten geldt dat deze eerst enkel als concept worden opgesteld. Deze wordt dan ter beoordeling gestuurd naar de begeleider van AMS - IX en na zijn goedkeuring terug naar school. De verschillende aanpassingen en versies worden apart bewaard.

B.8
B.8.1

Kosten/baten overzicht
Kosten

Voor AMS - IX worden in de eerste fase kosten gemaakt door de inkoop van testapparatuur en de inzet van manuren in het gehele project. Mocht de apparatuur bevallen dan wordt overgegaan tot een grotere aankoop in een later stadium. B.8.2 Baten

Belangrijk is om een extra service aan te bieden voor klanten. Hierdoor gaat AMS IX mee met de ontwikkelingen in de markt en kan het blijven groeien. Ook stelt dit onderzoek het bedrijf in staat om in een later stadium SLA’s aan alle klanten aan te bieden.

B.9

Risico analyse

Door een overzicht te maken met problemen waar tegen aan kan worden gelopen, wordt de kans op een succesvolle afronding vergroot. Op deze manier kan hier op in worden gespeeld. B.9.1 Tijdnood

Het project moet binnen een bepaalde tijd af zijn en dit gaat met de nodige druk gepaard. Om te voorkomen dat hierdoor fouten worden gemaakt is het belangrijk om te communiceren met de overige projectleden, zo kunnen zij rekening houden met een eventuele vertraging of assisteren bij de afronding. B.9.2 Extra BV

Al voordat dit project gestart werd sprak het management team met de leden van de AMS - IX vereniging over het vormen van een extra BV . Dit met als doel om de risico’s, 41

Plan van Aanpak

Appendix B

die SLA’s en overheidsregulering met zich meebrengen, in te perken. De uitwerking hiervan is echter nog niet volledig en het project wordt nu binnen de bestaande vorm uitgevoerd. Het is onduidelijk in welke bedrijfsstructuur de service uiteindelijk aangeboden gaat worden. Voor het project is dit geen risico aangezien de service in elk geval geleverd moet worden. B.9.3 Expertise

Wat wel een probleem kan vormen is dat de student te weinig kennis heeft over het onderwerp of gebruikte technieken. Dit risico kan worden beperkt door gebruik te maken en te leren van de expertise van de werknemers van AMS - IX.

B.10

Planning

Zie hiervoor pagina 43.

42

Naam

Werk sep

2010, Kw 4 okt nov dec

2011, Kw 1 jan feb Michiel Appelman mrt

2011, … apr

Werkzaam bij AMS-IX Plan van Aanpak Goedkeuring PvA IPX VPLS Instance Inhangen Accedian MetroNODE Voorbereiden Presentatie Presenteren Planning Onderzoek mogelijkheden Accedians Beslissing tot aanschaf Uitwerking van mogelijkheden Plaatsen overige Accedians Overzichten in My AMS-IX Welke specificaties in rapportage Rapportage maken Start Trail Evalueren Accedians en Stats Grotere Implementatie Opzetten Rapport schrijven Rapport ter beoordeling Rapport verbeteren Rapport Presentatie voorbereiden Presentatie

100d 11d Michiel Appelman Ger Brouwers 11d 2d 10d Attilla de Groot Attilla de Groot, Michiel Appelman Michiel Appelman Michiel Appelman 7d Michiel Appelman Henk Steemans, Attilla de Groot, Michiel Appelman 11d 2d 5d Michiel Appelman Attilla de Groot, Michiel Appelman Michiel Appelman Attilla de Groot, Michiel Appelman 1d 1d 21d 1d 85d 5d 5d Michiel Appelman Michiel Appelman Attilla de Groot

Michiel Appelman Ger Brouwers, Henk Steemans, Attilla de Groot Michiel Appelman Michiel Appelman

5d

Michiel Appelman Michiel Appelman

WBS Naam 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 Werkzaam bij AMS-IX Plan van Aanpak Goedkeuring PvA IPX VPLS Instance Inhangen Accedian MetroNODE Voorbereiden Presentatie Presenteren Planning Onderzoek mogelijkheden Accedians Beslissing tot aanschaf Uitwerking van mogelijkheden Plaatsen overige Accedians Overzichten in My AMS-IX Welke specificaties in rapportage Rapportage maken Start Trail Evalueren Accedians en Stats Grotere Implementatie Opzetten Rapport schrijven Rapport ter beoordeling Rapport verbeteren Rapport Presentatie voorbereiden Presentatie

Start 1 sep 1 sep 15 sep 1 sep 17 sep 7 sep 20 sep 21 sep 29 sep 30 sep 15 okt 18 okt 22 okt 25 okt 1 nov 2 nov 1 dec 16 sep 13 jan 14 jan 21 jan 21 jan 28 jan

Finish 18 jan 15 sep 15 sep 15 sep 17 sep 20 sep 20 sep 29 sep 29 sep 14 okt 15 okt 22 okt 22 okt 25 okt 1 nov 30 nov 1 dec 12 jan 14 jan 21 jan 21 jan 28 jan 28 jan

Werk 100d 11d n.v.t. 11d 2d 10d n.v.t. 7d n.v.t. 11d 2d 5d n.v.t. 1d 1d 21d 1d 85d 5d 5d n.v.t. 5d n.v.t.

Toegewezen aan Michiel Appelman Michiel Appelman Ger Brouwers Attilla de Groot Attilla de Groot, Michiel Appelman Michiel Appelman Michiel Appelman Michiel Appelman Attilla de Groot, Henk Steemans, Michiel Appelman Michiel Appelman Attilla de Groot, Michiel Appelman Michiel Appelman Attilla de Groot, Michiel Appelman Attilla de Groot

Michiel Appelman Michiel Appelman Michiel Appelman Attilla de Groot, Ger Brouwers, Henk Steemans Michiel Appelman Michiel Appelman Michiel Appelman Michiel Appelman

Configuration Example

Appendix C

C

Configuration Example

This section gives an example of the configuration used for the Accedian EtherNIDs.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 media-selection select RJ45-A_RJ45-B port edit Monitor-1 state disable port edit Monitor-2 state disable port edit Client state disable interface edit Auto state disable interface delete Management interface add Management address 91.200.16.181 port Management netmask 255.255.255.192 gateway 91.200.16.129 dns edit hostname nid-eqx-015 dns edit domain noc.ams-ix.net dns edit server1 91.200.16.36 dns edit server2 91.200.16.37 snmp edit ro-community *********** snmp edit rw-community *********** snmp edit use-host-name enable snmp edit system-location Equinix snmp edit contact-info noc@ams-ix.net snmp state enable snmp-trap edit v2c 1 host-name zabbix.ams-ix.net host-community ********* host-state enable syslog edit host interipx.stats.ams-ix.net syslog edit priority Notice ntp add time1.noc.ams-ix.net ntp add time-nid.noc.ams-ix.net ntp enable time-nid.noc.ams-ix.net ntp edit sync-mode highres cfm add domain level 1 index 9 format dns-name name noc.ams-ix.net cfm add ma name oam ccm-interval 1000 mepid-list 1,2,3,4,5,6,7,8 md-idx 9 index 1 cfm add mep active yes ma-idx 1 mep-id 1 port Network cci-enable yes lowest-alarm-pri remErrXconAis cfm add dmm enable yes mep-idx 1 index 1 rmep-id 2 reference-period 1 interval 100 owdelay enable ow-max-delay 2 ow-delay-threshold 3 ow-ad-threshold 1 tw-delay enable tw-max-delay 2 tw-delay-threshold 3 tw-ad-threshold 1 ow-dv enable ow-max-dv 1 ow-dv-threshold 2 ow-adv-threshold 1 tw-dv enable tw-max-dv 1 tw-dv-threshold 2 tw-adv-threshold 1 cfm add dmm enable yes mep-idx 1 index 2 rmep-id 3 reference-period 1 interval 100 owdelay enable ow-max-delay 2 ow-delay-threshold 3 ow-ad-threshold 1 tw-delay enable tw-max-delay 2 tw-delay-threshold 3 tw-ad-threshold 1 ow-dv enable ow-max-dv 1 ow-dv-threshold 2 ow-adv-threshold 1 tw-dv enable tw-max-dv 1 tw-dv-threshold 2 tw-adv-threshold 1 cfm add dmm enable yes mep-idx 1 index 3 rmep-id 4 reference-period 1 interval 100 owdelay enable ow-max-delay 2 ow-delay-threshold 3 ow-ad-threshold 1 tw-delay enable tw-max-delay 2 tw-delay-threshold 3 tw-ad-threshold 1 ow-dv enable ow-max-dv 1 ow-dv-threshold 2 ow-adv-threshold 1 tw-dv enable tw-max-dv 1 tw-dv-threshold 2 tw-adv-threshold 1 cfm add dmm enable yes mep-idx 1 index 4 rmep-id 5 reference-period 1 interval 100 owdelay enable ow-max-delay 2 ow-delay-threshold 3 ow-ad-threshold 1 tw-delay enable tw-max-delay 2 tw-delay-threshold 3 tw-ad-threshold 1 ow-dv enable ow-max-dv 1 ow-dv-threshold 2 ow-adv-threshold 1 tw-dv enable tw-max-dv 1 tw-dv-threshold 2 tw-adv-threshold 1 cfm add dmm enable yes mep-idx 1 index 5 rmep-id 6 reference-period 1 interval 100 owdelay enable ow-max-delay 2 ow-delay-threshold 3 ow-ad-threshold 1 tw-delay enable tw-max-delay 2 tw-delay-threshold 3 tw-ad-threshold 1 ow-dv enable ow-max-dv 1 ow-dv-threshold 2 ow-adv-threshold 1 tw-dv enable tw-max-dv 1 tw-dv-threshold 2 tw-adv-threshold 1 cfm add dmm enable yes mep-idx 1 index 6 rmep-id 7 reference-period 1 interval 100 owdelay enable ow-max-delay 2 ow-delay-threshold 3 ow-ad-threshold 1 tw-delay enable

29

30

31

32

33

45

Configuration Example

Appendix C

34

35 36 37 38 39 40 41 42

tw-max-delay 2 tw-delay-threshold 3 tw-ad-threshold 1 ow-dv enable ow-max-dv 1 ow-dv-threshold 2 ow-adv-threshold 1 tw-dv enable tw-max-dv 1 tw-dv-threshold 2 tw-adv-threshold 1 cfm add dmm enable yes mep-idx 1 index 7 rmep-id 8 reference-period 1 interval 100 owdelay enable ow-max-delay 2 ow-delay-threshold 3 ow-ad-threshold 1 tw-delay enable tw-max-delay 2 tw-delay-threshold 3 tw-ad-threshold 1 ow-dv enable ow-max-dv 1 ow-dv-threshold 2 ow-adv-threshold 1 tw-dv enable tw-max-dv 1 tw-dv-threshold 2 tw-adv-threshold 1 cfm add pkt-loss enable yes mep-idx 1 index 1 rmep-id 2 reference-period 1 thresholdratio 1 cfm add pkt-loss enable yes mep-idx 1 index 2 rmep-id 3 reference-period 1 thresholdratio 1 cfm add pkt-loss enable yes mep-idx 1 index 3 rmep-id 4 reference-period 1 thresholdratio 1 cfm add pkt-loss enable yes mep-idx 1 index 4 rmep-id 5 reference-period 1 thresholdratio 1 cfm add pkt-loss enable yes mep-idx 1 index 5 rmep-id 6 reference-period 1 thresholdratio 1 cfm add pkt-loss enable yes mep-idx 1 index 6 rmep-id 7 reference-period 1 thresholdratio 1 cfm add pkt-loss enable yes mep-idx 1 index 7 rmep-id 8 reference-period 1 thresholdratio 1 user edit admin password

46

Graph Generating Bash Script

Appendix D

D
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58

Graph Generating Bash Script

#!/bin/bash # Written by michiel.appelman@ams-ix.net # Working Directory for the RRD files DIR="/opt/mrtg/nids" WWWDIR="/var/www/stats" ## Strip HTML GET String into local variables ## hour=‘echo "$QUERY_STRING" | sed -n ’s/^.*hour=\([^&]*\).*$/\1/p’ | sed "s/%20/ /g"‘ end=‘echo "$QUERY_STRING" | sed -n ’s/^.*end=\([^&]*\).*$/\1/p’ | sed "s/%20/ /g"‘ kpi=‘echo "$QUERY_STRING" | sed -n ’s/^.*kpi=\([^&]*\).*$/\1/p’ | sed "s/%20/ /g"‘ nid1=‘echo "$QUERY_STRING" | sed -n ’s/^.*srcnid=\([^&]*\).*$/\1/p’ | sed "s/%20/ /g"‘ nid2=‘echo "$QUERY_STRING" | sed -n ’s/^.*dstnid=\([^&]*\).*$/\1/p’ | sed "s/%20/ /g"‘ # If hour variable is empty set it to 48 if [ -z $hour ] ; then hour="48" fi # If end time is empty set it to now or else to starting time + end time if [ -z $end ] ; then end="now" else end="start+${end}h" fi # Check to see if two different NIDs are selected if [ $nid1 == $nid2 ] ; then exit 1 fi # Delay or jitter function if [ "$kpi" == "delay" ] || [ "$kpi" == "jitter" ] then # Change colors depending on KPI if [ "$kpi" == "delay" ] ; then color1="#00ff00" color2="#0000ff" sla1="500" sla2="1000" elif [ "$kpi" == "jitter" ] ; then color1="#00ffff" color2="#ff00ff" sla1="50" sla2="100" fi ## Run RRDtool an send output to web page ## echo "Content-type: image/png " echo rrdtool graph -X0 -l0 -Y --imginfo ’’ --title "$kpi from $nid1 to $nid2" --start now-${ hour}h --end ${end} --vertical-label microseconds - \ DEF:oadv_$nid1\_$nid2=$DIR/5min_$kpi\_$nid1\_$nid2.rrd:ds0:AVERAGE \ DEF:tadv_$nid1\_$nid2=$DIR/5min_$kpi\_$nid1\_$nid2.rrd:ds1:AVERAGE \ VDEF:oadvmax_$nid1\_$nid2=oadv_$nid1\_$nid2,MAXIMUM \ VDEF:oadvavg_$nid1\_$nid2=oadv_$nid1\_$nid2,AVERAGE \ VDEF:oadvmin_$nid1\_$nid2=oadv_$nid1\_$nid2,MINIMUM \ VDEF:oadv95t_$nid1\_$nid2=oadv_$nid1\_$nid2,95,PERCENT \ COMMENT:" 95th SLA maximum minimum\l" \ LINE1:oadv_$nid1\_${nid2}${color1}:"One Way " \ GPRINT:oadv95t_$nid1\_$nid2:"%6.2lf ¸sec " \ t COMMENT:"$sla1 ¸sec " \ t

47

Graph Generating Bash Script

Appendix D

59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100

GPRINT:oadvmax_$nid1\_$nid2:"%6.2lf ¸sec " \ t GPRINT:oadvmin_$nid1\_$nid2:"%6.2lf ¸sec\l" \ t VDEF:tadvmax_$nid1\_$nid2=tadv_$nid1\_$nid2,MAXIMUM \ VDEF:tadvavg_$nid1\_$nid2=tadv_$nid1\_$nid2,AVERAGE \ VDEF:tadvmin_$nid1\_$nid2=tadv_$nid1\_$nid2,MINIMUM \ VDEF:tadv95t_$nid1\_$nid2=tadv_$nid1\_$nid2,95,PERCENT \ LINE1:tadv_$nid1\_${nid2}${color2}:"Two Way " \ GPRINT:tadv95t_$nid1\_$nid2:"%6.2lf ¸sec " \ t COMMENT:"$sla2 ¸sec " \ t GPRINT:tadvmax_$nid1\_$nid2:"%6.2lf ¸sec " \ t GPRINT:tadvmin_$nid1\_$nid2:"%6.2lf ¸sec\l" \ t COMMENT:"Copyright  l’ ‘date +%Y‘ AMS-IX B.V. [updated\: ‘date "+%d-%b-%Y %H\:%M\:%S %z "‘ ]\c" 2>/dev/null # Exit without errors exit 0 # Frame Loss function elif [ "$kpi" == "frameloss" ] then ## Run RRDtool and send output to web page ## echo "Content-type: image/png " echo rrdtool graph -X0 -l0 -Y --imginfo ’’ --title "$kpi from $nid1 to $nid2" --start now-${ hour}h --end ${end} --vertical-label percentage - \ DEF:fl0_$nid1\_$nid2=$DIR/5min_frameloss_$nid1\_$nid2.rrd:ds0:AVERAGE \ CDEF:fl_$nid1\_$nid2=fl0_$nid1\_$nid2,1000000,/ \ VDEF:flmax_$nid1\_$nid2=fl_$nid1\_$nid2,MAXIMUM \ VDEF:flavg_$nid1\_$nid2=fl_$nid1\_$nid2,AVERAGE \ VDEF:flmin_$nid1\_$nid2=fl_$nid1\_$nid2,MINIMUM \ VDEF:fl95t_$nid1\_$nid2=fl_$nid1\_$nid2,95,PERCENT \ COMMENT:" 95th SLA maximum minimum\l" \ LINE1:fl_$nid1\_$nid2#000000:"Frame Loss " \ GPRINT:fl95t_$nid1\_$nid2:"%3.6lf %% " \ COMMENT:"0.005000 % " \ GPRINT:flmax_$nid1\_$nid2:"%3.6lf %% " \ GPRINT:flmin_$nid1\_$nid2:"%3.6lf %%\l" \ COMMENT:"Copyright  l’ ‘date +%Y‘ AMS-IX B.V. [updated\: ‘date "+%d-%b-%Y %H\:%M\:%S %z "‘ ]\c" 2> /dev/null # Exit without errors exit 0 # Invalid $kpi - exit with error else exit 1 fi

48

Acronyms

Appendix E
IEEE

E
1 DM
ACL

Acronyms
One Way Delay Measurement Access Control List Amsterdam Internet Exchange Asynchronous Transfer Mode Border Gateway Protocol Continuity Check Message Collission Detection

Institute of Electrical and Electronics Engineers Internet Protocol
IP

IP IPX ISP IT ITU - T

eXchange

AMS - IX ATM BGP CCM CD CFM CLI CPU CSMA CTO DMM DNS DWDM

Internet Service Provider Information Technology International Telecommunication Union - Telecommunication Standardization Sector Key Performance Indicator Link Aggregation Group Local Area Network Leap Indicator Label Switched Path Link Trace Message Maintenance Association Media Access Control Maintenance Domain Maintenance Entity Maintenance Entity Group
MEG

KPI

Connection Fault Management
LAG

Command Line Interface
LAN

Central Processing Unit
LI

Carrier Sense Multiple Access
LSP

Chief Technology Officer
LTM

Delay Measurement Message
MA

Domain Name System
MAC

Dense wavelength division multiplexing Ethernet in the First Mile Frame Check Sequence Field-programmable gate array Gigabit Ethernet Global Position System General Packet Radio Service Global System for Mobile Communications Association
GPRS

MD ME MEG MEL MEP MEP MEPID MHF MIP MPLS MRTG

EFM FCS FPGA

Level

GigE
GPS GPRS GSMA

Maintenance End Point
MEG MEP MIP

End Point IDentifier

Half Function

Maintenance Intermediate Point Multiprotocol Label Switching Multi Router Traffic Grapher

GRX ICMP

Roaming eXchange

Internet Control Message Protocol

49

Acronyms

Appendix E
RRD RTT SDH SLA SLD SFP SNMP

NGN NID NIST

Next Generation Network Network Interface Device National Institute of Standards and Technology Network Operations Center Network Time Protocol Operations, Administration and Maintenance One Way Average Delay One Way Average Delay Variation Object Identifier Operating System Open System Interconnection One Way Delay Provider Performance Assurance Agent Protocol Data Unit Provider Edge Performance Measurement Passive Optical Network Point of Presence Precision Time Protocol Photonic Cross Connect Quality of Service Remote Defect Indicator Request for Comment Réseaux IP Européens

Round Robin Database Round Trip Time Synchronous Digital Hierarchy Service Level Agreement Service Level Description Small form-factor pluggable Simple Network Management Protocol Service Provider

NOC NTP OAM

OAD OADV OID OS OSI OWD P PAA PDU PE PM PON POP PTP PXC QoS RDI RFC RIPE

SP

SyncE Synchronous Ethernet
TAD TADV TCP TLV TTM VLAN VLL VM VPLS WAN

Two Way Average Delay Two Way Average Delay Variation Transmission Control Protocol Type, Length, Value Test Traffic Measurement Virtual LAN Virtual Leased Line Virtual Machine Virtual Private LAN Service Wide Area Network

50

Appendix F

F

Bibliography

[1] Global System for Mobile Communications Association, “Evolution to ip,” December 2010. http://www.gsmworld.com/our-work/ programmes-and-initiatives/evolution_to_ip.htm. [2] E. Rosen, A. Viswanathan, and R. Callon, “Request for Comment 3031: Multiprotocol Label Switching,” January 2001. http://www.ietf.org/rfc/rfc3031.txt. [3] M. Lasserre and V. Kompella, “Request for Comment 4762: Virtual Private LAN Service using Label Distribution Protocol (LDP) signaling,” January 2007. http: //www.faqs.org/rfcs/rfc4762.html. [4] Institute of Electrical and Electronics Engineers (IEEE), “IEEE 802.3 Working Group Approves New Study Group for Ethernet in the First Mile (EFM),” November 2000. http://web.archive.org/web/20070217122603/http: //standards.ieee.org/announcements/8023nsg.html. [5] Institute of Electrical and Electronics Engineers, “IEEE Standard 802.3: Carrier Sense Multiple Access (CSMA) with Collission Detection (CD) (CSMA/CD) access method and Physical Layer specifications – Section Five,” 2008. http://standards. ieee.org/getieee802/802.3.html. [6] Institute of Electrical and Electronics Engineers, “Virtual Bridged Local Area Networks - Amendment 5: Connectivity Fault Management,” 2007. http://www. ieee802.org/1/pages/802.1ag.html. [7] International Telecommunication Union - Telecommunication Standardization Sector, “OAM functions and mechanisms for Ethernet based networks,” 2008. http: //www.itu.int/rec/T-REC-Y.1731/en. [8] Institute of Electrical and Electronics Engineers, “EtherType Field Public Assignments,” 2004. http://standards.ieee.org/regauth/ethertype/eth. txt. [9] Brocade, NetIron Configuration Guide – Supporting Multi-Service IronWare Unified NetIron R05.0.00. April 2010. page 2239. [10] MRV, “OptiSwitch 940 - 10GE Carrier-Ethernet Access Data Sheet,” October 2010. http://www.mrv.com/datasheets/OS/PDF300/MRV-OS-OS940_ A4_HI.pdf. [11] Brocade, NetIron Configuration Guide – Supporting Multi-Service IronWare Unified NetIron R05.0.00. April 2010. page 1557. 51

Appendix F [12] D. L. Mills, “Request for Comment 958: Network Time Protocol,” September 1985. http://www.faqs.org/rfcs/rfc958.html. [13] National Institute of Standards and Technology, “IEEE 1588 - Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems,” May 2010. http://ieee1588.nist.gov/. [14] A. Ramkisoen, “Synleaf,” December 2010. http://www.synleaf.com.

52

Acknowledgements
Thanks to Attilla de Groot, Henk Steenman, Martin Pels, Ariën Vijn, Niels Bakker, Steven Bakker and all the other staff at AMS - IX, for providing me with the opportunity, knowledge and skills to successfully complete this research project.

You're Reading a Free Preview

Download
scribd
/*********** DO NOT ALTER ANYTHING BELOW THIS LINE ! ************/ var s_code=s.t();if(s_code)document.write(s_code)//-->