QUICK START GUIDE

Installing an FNA cache

May 2020
Copyright & Trademarks
© 2020 Facebook, Inc. All rights reserved.

Quick Start Guide: Installing an FNA Cache | 2

Table of Contents

Physical installation .................................................................................................................................................................... 4

1. RACK THE SERVERS ................................................................................................................................................................ 4

2. RACK THE SWITCH ................................................................................................................................................................. 4

3. CONNECT THE NETWORK CABLES FROM THE SERVERS TO THE SWITCH .................................................................... 5

4. INSTALL OPTICS INTO PORTS ............................................................................................................................................... 5

5. CONNECT THE SWITCH TO THE ROUTER........................................................................................................................... 5

6. POWER ON THE FNA.............................................................................................................................................................. 5

7. VERIFY SERVER OPERATION .................................................................................................................................................. 6

8. CHECK OPTICAL LIGHT LEVELS ............................................................................................................................................ 7

Network Configuration ............................................................................................................................................................ 7

9. CONFIGURE POINT-TO-POINT LINK .................................................................................................................................... 7

10. CONFIGURE BORDER GATEWAY PROTOCOL (BGP) ....................................................................................................... 9

11. STEP-BY-STEP REPRESENTATION OF IP/BGP CONNECTIVITY AND TRAFFIC FLOW ......................................... 10

Appendix 1: HPE ProLiant DL360 Generation 10 Overview ............................................................................... 11

Quick Start Guide: Installing an FNA Cache | 3

The illustrations in this guide will show a typical 4-server cache deployment. Caches can be deployed with
any number of servers from 2 to 20, so please consider these illustrations as an example and follow the
instructions taking into account the number of servers actually deployed.

Physical Installation
1. Rack the Servers

a. Install the servers into a compatible rack. Installing servers in a two-post rack requires
appropriate conversion hardware.
b. Connect the power cables from each server to the Power Distribution Units (PDUs). AC power
supplies are illustrated. For DC power installations, please consult an electrician.

2. Rack the Switch

a. Install the switch into the rack with the airflow as port side exhaust (switch ports must face the
rear of the rack as shown in Figure 2-1).

Figure 2-1 Figure 2-2

Quick Start Guide: Installing an FNA Cache | 4

 Figure 2-3

3. Connect the Network Cables from the Servers to the Switch

a. Connect the network cable (green sticker facing up) to the Network Interface Controller (NIC)
port 2 as seen in Figure 2-1, 2-2 and 2-3.
b. Connect a server at the highest position in the rack (fna001) to QSFP+ port 32 on the switch.
Utilize the illustration as reference for port connectivity for the remaining servers (four are
illustrated in Figure 2-1).

4. Install Optics into Ports

a. Install QSFP-SFP adapters, optical transceivers and QSFP optics into ports 1, 2, 3 and 4 as
shown in Figure 2-1.
b. For Clusters over four servers, follow the port assignments illustrated in Figure 4-1.

Figure 4-1

Quick Start Guide: Installing an FNA Cache | 5

5. Connect the Switch to the Router

a. Install QSFP-SFP adapters, optical transceivers and QSFP optics into ports 1, 2, 3 and 4 as
shown in Figure 2-1.

6. Power on the FNA

a. Press the power buttons on all four servers as shown in Figure 6-1.
b. Connect two power cables to the network switch as shown in Figure 6-1 (the switch powers on
when connected to power).
c. The switch’s system status LED displays as blinking green while powering up, then steady green
for normal and amber indicates a fan disconnect or malfunction. Port LEDs display green for
up, yellow for software disabled and flashing yellow for failed diagnostics (refer to Figure 6-2).
d. Only if instructed by Facebook, refer to the USB-Based Software Installation document on
the Network Partner Portal (NPP), “Docs” section. Note that this is rarely necessary and should
be done only if asked by Facebook’s FNA Operations team.

Figure 6-1 Figure 6-2

7. Verify Server Operation

a. Verify server operation by checking the behavior of the LEDs as illustrated on each server.

Quick Start Guide: Installing an FNA Cache | 6

 Figure 8-1

8. Check Optical Light Levels

a. Check the optical light levels to ensure the cluster has adequate signal stretch. Light levels
need to be between -2 dB and -7 dB.

Network Configuration
9. Configure Point-to-Point Link
The point-to-point connection is configured on the Link-Aggregation Control Protocol (LACP)
interface. Make sure the following Parameters are configured as follows:

i. Set a static route to the allocated subnets via the link-aggregation interface.
ii. Set the appropriate maximum transmission unit (MTU) for your router software:
1. For CISCO IOS XR router software, set the MTU to 1514. CISCO IOS SR
software includes both the L2 and L3 overhead in the interface MTU
command.
2. For all other router software, set the MTU to 1500.
The FNA cluster must have internet connectivity BEFORE BGP peering can be established.
The BGP peering session does not need to be routed before the FNA has internet
connectivity.

The FNA kit is preconfigured with the IP addresses that were specified in the order.

If IP addresses need to be changed at any time, please let us know by posting in the kit

Quick Start Guide: Installing an FNA Cache | 7

request ticket in the Network Partner Portal.

Our team monitors the updates in the portal ticket, where you can also request assistance.
You may also contact the FNA Operations team and ask for support by emailing
fna-ops@fb.com

Quick Start Guide: Installing an FNA Cache | 8

10. Configure Border Gateway Protocol (BGP)

iii. Use the IPv4 and IPv6 addresses originally indicated in the FNA order form.
iv. The IP prefixes assigned to the FNA should be announced to all your Facebook
peering connections and to your transit provider(s) to ensure they are reachable
through the internet.
v. (Optional) Enable Graceful Restart (GR). Enabling GR is not required, but it is
recommended.
vi. Enable External BGP (EBGP) Multihop.
vii. Use the peering Autonomous System Number (ASN): 63293
viii. Configure your router’s BGP settings.
ix. After the BGP settings are saved, the FNA cluster will begin downloading
application data. Within two business days, BGP peering will be established:
1. The connection will be coming from the 16th IP address in each allocated
subnet (for example, 192.168.1.16 for IPv4, and 2001:db8::10 for IPv6).
2. FNA will not advertise any routes/prefixes to the ISP.
x. BGP advertisements to the FNA:
1. All your customer prefixes and every servable public IP prefix should be
announced to the FNA for network traffic optimization
2. Do now advertise private IP addresses to the FNA, as this is not supported.
3. The smallest IP prefix sizes accepted are /30 for IPv4 and /64 for IPv6
4. We hold a limit on the number of IPv4 / IPv6 advertisements allowed to
your FNA:
a. For IPv4 we allow a maximum of 90k
b. For IPv6 we allow a maximum of 25k
c. As this may be indicative of prefix leakage, we will inform you via a
Direct Support ticket in the Network Partner Portal when you have
reached 90% of these thresholds. Should you cross either of these
thresholds, traffic will be drained from your cluster. To prevent this,
please ensure your BGP advertisements are below the defined
thresholds. Should you have a reason to cross this, please contact
noc@fb.com
xi. When installation is complete, please share an update in the kit request ticket in the
Network Partner Portal. Our team monitors the updates in the portal ticket, where
you can also request assistance. You may also contact the FNA Operations team and
ask for support by emailing fna-ops@fb.com

Quick Start Guide: Installing an FNA Cache | 9

11. Step-by-step representation of IP/BGP connectivity and traffic flow

STEP 1

Figure 11-1: Physical connection and LACP uplink

STEP 2

Figure 11-2: Configuring BGP

Quick Start Guide: Installing an FNA Cache | 10

 STEP 3

Figure 11-3: Adding prefixes, FNA starts serving traffic

STEP 4

Figure 11-4: Full diagram with example IP addressing

Quick Start Guide: Installing an FNA Cache | 11

Appendix 1: HPE ProLiant DL380 Generation 10 Overview

Figure A1-1

Figure A1-2

You can find the technical specifications for the FNA devices in the White Paper document available in the
Network Partner Portal documentation.

Quick Start Guide: Installing an FNA Cache | 12

Quick Start Guide: Installing an FNA Cache | 13