Quick Start Guide

INSTALLING OR UPGRADING THE MNA CACHE

WITH ARISTA 7060

OCT 2021

Meta Platforms, Inc.

Copyright & Trademarks
© 2021 Meta Platforms, Inc. All rights reserved.

Meta Platforms, Inc. Quick Start Guide: Installing or Upgrading an MNA Cache | 2
Table of Contents
Introduction ........................................................................................................................................ 4

Hardware Inventory .......................................................................................................................... 5

Pre-Installation Checklist..................................................................................................................7

Physical Installation .......................................................................................................................... 8

1. INSTALL THE SERVERS AND THE SWITCH. ............................................................................ 8

2. CONNECT THE NETWORK CABLES FROM THE SERVERS TO THE SWITCH ........................ 10

3. INSTALL OPTICS INTO PORTS AND CONNECT THE MNA SWITCH TO YOUR ROUTER......... 12

4. INSTALL QSFP-SFP ADAPTERS AND SFP+ OPTICAL TRANSCEIVERS ............................. 12

5. POWER ON THE MNA .......................................................................................................... 13

6. VERIFY SERVER OPERATION ...............................................................................................14

7. CHECK OPTICAL LIGHT LEVELS ..........................................................................................14

Network Configuration ...................................................................................................................15

8. CONFIGURE POINT-TO-POINT LINK........................................................................................15

9. CONFIGURE BORDER GATEWAY PROTOCOL (BGP) ..............................................................16

10. STEP-BY-STEP REPRESENTATION OF IP/BGP CONNECTIVITY AND TRAFFIC FLOW.............18

Appendix 1: HPE Proliant DL380 Generation 10 Overview .................................................... 20

Appendix 2: Server <-> Switch Full Port Map (For Future Upgrades) .................................. 22

Meta Platforms, Inc. Quick Start Guide: Installing or Upgrading an MNA Cache | 3
Introduction
Thank you for choosing to install the Meta Network Appliance (MNA)!

Meta Network Appliance is Meta’s content caching program. MNA provides Internet
Service Providers (ISPs) with an efficient means of delivering static Meta content from
within their network. Upon deployment, an ISP will offload a significant amount of Meta
content from its backbone network and vastly improve the Meta user experience.

The MNA cluster consists of a Top-of-Rack (ToR) switch and from two to twenty servers.
The illustrations in this guide will show a typical 4-server cache deployment, so please
consider these illustrations as an example and follow the instructions taking into account
the number of servers actually deployed.

This document provides hardware specifications and instructions on installing your MNA
and configuring it, whether it is a new cluster installation or an upgrade kit for an existing
cluster. We have a separate document titled “Refresh Guide” with the instructions for
The MNA hardware refresh.

Meta Platforms, Inc. Quick Start Guide: Installing or Upgrading an MNA Cache | 4
Hardware Inventory
Ensure that you have received all equipment in accordance with the checklist below. The
table below is a reference for a 4-server deployment of a new cache. The number of items
will vary depending on deployment size.

An upgrade kit will be almost identical, but I will not include a switch. This is because an
upgrade kit adds servers and uplinks to an existing cluster that already has a switch.

If you are missing any items, please contact Meta via the Network Partner Portal (NPP).

MNA Provided Hardware – New Cluster Kit

Dimensions
QTY Item
per Unit

(1) Arista DCS-7060CX-32S-R-DC Switch 1 RU

(4) Hewlett Packard ProLiant Server DL380p G10 Server 2 RU

(4) Post rail kit N/A

(8) AC Power Cord (only if AC power required) 2 Meters

(4) Network Cable (to connect servers to the switch) 3 Meters

(1) HP ProLiant DL380 G10 server user manual [1] N/A

(1) Switch user manual N/A

(4) For 10G uplink, SFP+ optical transceiver, LR or SR2 N/A

(1 per SFP+) For 10G uplink, QSFP to SFP+ adapter[2] N/A

(1-4) For 100G uplink, QSFP transceiver, LR4 or SR42 N/A

1
Model numbers vary per cluster.

2
Uplink capacity is selected for every order, either with 10G or 100G ports. Optical transceivers will be
sent according to the port type selected. Meta provides transceivers only for the MNA, while optics for
the ISP devices must be provided by the ISP. For 10g uplink, by default Meta will send as many SFP+ as
servers in the order. For 100g uplink, Meta will send as many as selected in the order form.

Meta Platforms, Inc. Quick Start Guide: Installing or Upgrading an MNA Cache | 5
 Table 1: MNA Cluster Provided Hardware
The following table lists additional equipment — not provided by Meta — that is required
for the MNA installation.

Equipment Required but Not Included

QTY Item

(1 per uplink port) Fiber patch cables from Arista DCS-7060CX-32S-R-DC Switch to ISP router [1]

As needed QSFP / SFP from router side and rack [2]

1
Fiber connectivity on the MNA side is the Lucent Connector (LC) type. Fiber connectivity on the
router side can be LC-LC, LC-SC, (Lucent Connector – Subscriber Connector), etc.

2
Optics for the ISP devices must be provided by the ISP. The MNA accepts 10G SR and LR, and 100G
SR4 and LR4 optical transceivers. Meta provides the most optimal optical transceiver type,
corresponding to the type you currently use to connect your routers. MNA does not support 40GE
interfaces for the uplink.

Table 2: Equipment Required but Not Included

Meta Platforms, Inc. Quick Start Guide: Installing or Upgrading an MNA Cache | 6
Pre-Installation Checklist

Use the pre-installation checklist below to confirm that your facility has the capacity and
resources required to install the MNA cluster:

Pre-installation Checklist

Item

Sufficient power for each cluster (approx. 1.4kW for a 4-server cluster)

Power connections are divided evenly between two different power sources

n x available input power connections [1]

Sufficient rack space is available [2]

n x 10 Gigabit (10Gb or 100gb) available ports on ISP switch or router [3]

n x available SFP+ (or compatible) optics of same type – Long Range (LR) or Short
Range (SR) [3]

n x available fiber patch cables (multi-mode or single mode) [3]

1
Where n = 2 times the number of devices (e.g., Cluster 1 is (5) devices, so (10) input power
connections are required).

2
For initial deployment (Cluster 1), 9RU (nine rack unit) is required. For additional servers, consider
2RU additional per server.

3
Where n = 1 for each server deployed.

Table 3: Pre-installation Checklist

Meta Platforms, Inc. Quick Start Guide: Installing or Upgrading an MNA Cache | 7
Physical Installation
1. Install the servers and the switch.

a. Install the servers and connect power cables.

i. Install the servers into a compatible rack. Installing servers in a 2-post rack
requires appropriate 4-post to 2-post conversion hardware. Please leave
enough rack space at the top part of the rack to install the switch later.
Also, please install the server labeled as fna001 at the highest position of
the rack. Install the rest of the racks accordingly. Use Figure 1.1 for
guidance. Follow manufacturer’s instructions for the server model included
in your order if further guidance is needed.

ii. Connect AC power cables from each server to the Power Distribution Units
(PDUs). Figure 1.1 illustrates AC power supplies. For DC power installations,
please consult an electrician.

Meta Platforms, Inc. Quick Start Guide: Installing or Upgrading an MNA Cache | 8
Rear of Rack

Fna001

Fna002

Fna003

Fna004

Figure 1.1: Install the servers and connect AC Power

b. Install the Switch.

i. Install the switch on top of the rack with the airflow as por-side exhaust
(switch ports must face the rear of the rack as shown in Figure 1.2). Follow
manufacturer’s instructions for the switch model included in your order if
further guidance is needed.

Rear of Rack

Meta Platforms, Inc. Quick Start Guide: Installing or Upgrading an MNA Cache | 9
 Figure 1.2: Install switch on top of the rack

IMPORTANT: Do not connect the switch to power at this stage, because the
switch will immediately turn on as soon it receives power. That powering-up
needs to be delayed until a later stage.

2. Connect the Network Cables from the Servers to the Switch

a. Each server’s NIC and network cable connector has a green sticker to show the
correct port connection and orientation (as shown in Figure 2.1).

Meta Platforms, Inc. Quick Start Guide: Installing or Upgrading an MNA Cache | 10
 Figure 2.1: Server port connexion and orientation

b. Connect one end of the network cable (green sticker facing up) to the server’s NIC
port 2 (begin with the server at the highest position in the rack labelled fna001)
and connect the other end to the switch’s port 32. Repeat for every server as
illustrated in Figure 2.2. Table 4 provides a port map for a 4-servers cache
deployment, plus an example of a 4 to 6-servers upgrade.

Figure 2.2: Switch <> Server connectivity

Meta Platforms, Inc. Quick Start Guide: Installing or Upgrading an MNA Cache | 11
 Server NIC's port MNA Switch port
Hostname Number number

fna001 2 32
fna002 2 31
4-Servers Cache
fna003 2 30
fna004 2 29

Server NIC's port MNA Switch port

Hostname Number number

fna001 2 32
fna002 2 31
fna003 2 30
6-Servers Cache
fna004 2 29
fna005 2 28
fna006 2 27

Table 4: Server <> Switch connectivity port map (4-servers and 4 to 6-servers cache upgrade)

Note: Go to Appendix 2 at the end of this document to get a complete port map for
future upgrades.

3. Install Optics into ports and connect the MNA Switch to your
Router

4. Install QSFP-SFP adapters and SFP+ optical transceivers

into ports 1, 2, 3 and 4 in MNA switch as shown in Figure 4.1.

Please note: Full switch port assignment is provided in Figure 4.1 for clusters over four

Meta Platforms, Inc. Quick Start Guide: Installing or Upgrading an MNA Cache | 12
servers.

Figure 4.1: Switch port assignment

Ports 1-4

5. Power on the MNA

a. Press the power buttons on all four servers as shown in Figure 5.1.

b. Connect two power cables to the network switch as shown in Figure 5.1 (the
switch powers on when connected to power).

c. The switch’s system status LED displays as blinking green while powering up,
then steady green for Normal. An amber light indicates a fan disconnect or
malfunction. Port LEDs display green for up, yellow for software disabled and
flashing yellow for failed diagnostics (refer to Figure 5.2).

5.b

5.a

Meta Platforms, Inc. Quick Start Guide: Installing or Upgrading an MNA Cache | 13
 Figure 5.1 Figure 5.2
6. Verify Server Operation

a. Verify server operation by checking the status LEDs as illustrated in Figure 6.1.

b. A server is powered on when the status LED is green. If the status LED is amber,
then you need to press the power button until the LED turns green. For Gen10
servers, wait a few seconds until the status LED turns green.

Figure 6.1

7. Check Optical Light Levels

a. Check the optical light levels to ensure that the cluster has adequate signal
stretch. Light levels need to be between -2 dB and -7 dB.

Meta Platforms, Inc. Quick Start Guide: Installing or Upgrading an MNA Cache | 14
Network Configuration

8. Configure Point-to-Point Link

a. The point-to-point connection is configured on the Link-Aggregation Control
Protocol (LACP) interface. Verify the following parameter configurations:

i. Set a static route to the allocated subnets via the link-aggregation

interface.

ii. Set the appropriate maximum transmission unit (MTU) for your router
software:

• For CISCO IOS XR router software, set the MTU to 1514. CISCO IOS
SR software includes both the L2 and L3 overhead in the interface
MTU command.

• For all other router software, set the MTU to 1500.

The MNA cluster must have internet connectivity before BGP peering can be
established. The BGP peering session does not need to be routed before the MNA
has internet connectivity.

The MNA kit is preconfigured with the IP addresses specified in the order.

If IP addresses need to be changed at any time, please let us know by posting in

the kit-request ticket in the Network Partner Portal.

Our team monitors the updates in the ticket portal, where you can also request
assistance. You may also contact the MNA Operations team for support by
emailing fna-deployment@meta.com

Meta Platforms, Inc. Quick Start Guide: Installing or Upgrading an MNA Cache | 15
9. Configure Border Gateway Protocol (BGP)

iii. Use the IPv4 and IPv6 addresses originally indicated in the MNA order
form.

iv. The IP prefixes assigned to the MNA should be announced to all your Meta
peering connections and to your transit provider(s) to ensure they are
reachable through the internet.

v. (Optional) Enable Graceful Restart (GR). Enabling GR is not required, but it

is recommended.

vi. Enable External BGP (EBGP) Multihop.

vii. Use the peering Autonomous System Number (ASN): 63293

viii. Configure your router’s BGP settings.

ix. After the BGP settings are saved, the MNA cluster will begin downloading
application data. Within two business days, BGP peering will be
established:

1. The connection will be coming from the 16th IP address in each

allocated subnet (for example, 192.168.1.16 for IPv4, and 2001:db8::10
for IPv6).

2. MNA will not advertise any routes/prefixes to the ISP.

x. BGP advertisements to the MNA:

1. All your customer prefixes and every servable public IP prefix should
be announced to the MNA for network traffic optimization.

2. Do not advertise private IP addresses to the MNA. This is not

supported.

3. The smallest IP prefix sizes accepted are /30 for IPv4 and /64 for
IPv6.

4. We do not support MD5 on MNA BGP sessions.

Meta Platforms, Inc. Quick Start Guide: Installing or Upgrading an MNA Cache | 16
 xi. We hold a limit on the number of IPv4 / IPv6 advertisements allowed to
your MNA:

• For IPv4 we allow a maximum of 50k

• For IPv6 we allow a maximum of 8k

xii. Because this may indicate prefix leakage, we will inform you via a Direct
Support ticket in the Network Partner Portal when you have reached 90%
of these thresholds. Should you cross either of these thresholds, traffic will
be drained from your cluster. To prevent this, please ensure that your BGP
advertisements are below the defined thresholds. Should you have a reason
to exceed this, please contact noc@meta.com.

xiii. When installation is complete, please share an update in the kit-request

ticket in the Network Partner Portal. Our team monitors the updates in the
portal ticket, where you can also request assistance. You may also contact
the MNA Operations team for support by emailing fna-
deployment@meta.com

Meta Platforms, Inc. Quick Start Guide: Installing or Upgrading an MNA Cache | 17
10. Step-by-step depiction of IP/BGP connectivity and traffic flow

STEP 1

Figure 10.1: Physical connection and LACP uplink

STEP 2

Meta Platforms, Inc. Quick Start Guide: Installing or Upgrading an MNA Cache | 18
 Figure 10.2: Configuring BGP
STEP 3

Figure 10.3: Adding prefixes, MNA starts serving traffic

STEP 4

Figure 10.4: Full diagram with example IP addressing

Meta Platforms, Inc. Quick Start Guide: Installing or Upgrading an MNA Cache | 19
Appendix 1:
HPE ProLiant DL380 Generation 10 Overview

Figure A1.1

Figure A1.2

You can find the technical specifications for the MNA devices in the Technical Overview
document available in the Network Partner Portal documentation.

Meta Platforms, Inc. Quick Start Guide: Installing or Upgrading an MNA Cache | 20
Meta Platforms, Inc. Quick Start Guide: Installing or Upgrading an MNA Cache | 21
Appendix 2: Server <-> Switch full port map

Server Hostname NIC's port Number MNA Switch port number

fna001 2 32

fna002 2 31

fna003 2 30

fna004 2 29

fna005 2 28

fna006 2 27

fna007 2 26

fna008 2 25

fna009 2 24

fna010 2 23

fna011 2 22

fna012 2 21

fna013 2 20

fna014 2 19

fna015 2 18

fna016 2 17

fna017 2 16

fna018 2 15

fna019 2 14

fna020 2 13

Table 5: Server <-> Switch full port map

Meta Platforms, Inc. Quick Start Guide: Installing or Upgrading an MNA Cache | 22
Meta Platforms, Inc. Quick Start Guide: Installing or Upgrading an MNA Cache | 23