Facebook Network Appliance Deployment | 1
© 2015-2016 Facebook. All rights reserved. Cisco®, Cisco Nexus®, and the Cisco Logo are registered trademarks of Cisco Systems, Inc. FreeBSD® FreeBSD® and the FreeBSD Logo are registered trademarks of The FreeBSD Foundation. IBM® is a trademarks or registered trademarks of International Business Machines Corporation in the United States and/or other countries. Juniper Networks® Networks® and Junos® Junos® are [registered] trademarks of Juniper Networks, Inc. in the United States and other countries. Linux® Linux® is a registered trademark o f Linus Torvalds in the United States. Mac® Mac ® and Mac OS® OS® are registered trademarks of Apple Inc. Hewlett-Packard® and HP® are registered trademarks of Hewlett-Packard Company. Windows® is either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.
Facebook Network Appliance Deployment | 1
© 2015-2016 Facebook. All rights reserved. Cisco®, Cisco Nexus®, and the Cisco Logo are registered trademarks of Cisco Systems, Inc. FreeBSD® FreeBSD® and the FreeBSD Logo are registered trademarks of The FreeBSD Foundation. IBM® is a trademarks or registered trademarks of International Business Machines Corporation in the United States and/or other countries. Juniper Networks® Networks® and Junos® Junos® are [registered] trademarks of Juniper Networks, Inc. in the United States and other countries. Linux® Linux® is a registered trademark o f Linus Torvalds in the United States. Mac® Mac ® and Mac OS® OS® are registered trademarks of Apple Inc. Hewlett-Packard® and HP® are registered trademarks of Hewlett-Packard Company. Windows® is either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.
Facebook Network Appliance Deployment | 2
Facebook Network Appliance Deployment | 3
Facebook Network Appliance Deployment | 4
Thank you for choosing to install the Facebook Network Appliance (FNA)! Facebook Network Appliance is Facebook’s content caching program. FNA provides Internet Service Providers (ISPs) with an efficient means of delivering static Facebook content from within their existing network. Upon deployment, an ISP will offload a significant amount of Facebook content from its backbone network and vastly improve the Facebook user experience. The FNA kit consists of a Cisco Nexus Top-of-Rack (ToR) switch, four Hewlett Packard ProLiant DL380 servers, and their required interconnects. The hardware is suitable for deployment in data centers, colocation facilities, and outside plant environments (the industry standard 19-inch form factor allows integration into most network environments). This document describes the procedures necessary to install, configure, and operate the FNA.
Facebook Network Appliance Deployment | 5
This section lists equipment required for FNA installation. Ensure that you have received all of the equipment required for either your SKU1 or SKU2 order (Tables 1 and 2, respectively).
The following table lists equipment that is included with a SKU1 order. If equipment is missing, refer to the Support section for contact information and seek assistance. Table 1: FNA SKU1 Provided Equipment
(1)
Cisco Nexus 3132Q-X switch 2
1RU
(4)
HP ProLiant DL380 Generation 9 (Gen9) server 3
2RU
(4)
4-Post rail kit
N/A
(8)
C13-C14 AC Power Cord
2 Meters
(4)
Network Cable
3 Meters
(1)
HP ProLiant DL380 Generation 9 (Gen9) server user manual
N/A
(1)
Cisco Nexus 3132Q-X switch user manual
N/A
(2)
SFP+ optical transceiver, LR or SR, (2) per SKU 4
N/A
1
See Appendix I: Hardware Dimensions and Weight for more dimension details.
2Hereafter
referred to as Nexus switch.
3Hereafter
referred to as ProLiant DL380 server.
4Caution:
SFP+ optical transceivers are sensitive to static electricity.
Facebook Network Appliance Deployment | 6
The following table shows equipment that is included a SKU2 order. If equipment is missing, refer to the Support section for contact information and seek assistance. Table 2: FNA SKU2 Provided Equipment
(4)
HP ProLiant DL380 Generation 8 (Gen9) server 2
2RU1
(4)
4-Post rail kit
(8)
C13-C14 power cable
2 Meters
(4)
Network cable
3 Meters
(4)
HP ProLiant DL380p Generation 8 (Gen8) server user manual
N/A
(1)
Text configuration file
N/A
(2)
SFP+ optical transceiver, LR or SR, (2) per SKU 3
N/A
N/A
1
See Appendix I: Hardware Dimensions and Weight for more dimension details.
3Hereafter
referred to as ProLiant DL380 server.
3
Caution: SFP+ optical transceivers are sensitive to static electricity.
NOTE: SKU2 does not include a Cisco Nexus switch. However, the order includes SFP+ optical transceivers for additional uplink capacity. Use the existing switch at your FNA deployment. NOTE: The quantity of servers you receive depends on your current and forecasted Facebook traffic volume. Servers are shipped in groups of four. The equipment will arrive at your facility in one shipment. IBM (FNA’s logistics partner) will contact you with shipping information.
Facebook Network Appliance Deployment | 7
This section shows additional equipment required for the FNA installation that is not included with a SKU order. Table 3: FNA SKU1 and SKU2 Equipment Required but Not Included
(2)
Fiber Patch Cables from Cisco Nexus 3132Q-X switch to router 1,2
1
The FNA accepts both SR and LR optical transceivers. Facebook provides the most optimal optical transceiver type corresponding to the type you currently use to connect your routers. 2
Fiber connectivity on the FNA side is the Lucent Connector (LC) type. Fiber connectivity on the router side can be LC-LC, LC-SC, (Lucent Connector – Subscriber Connector), etc.
Facebook Network Appliance Deployment | 8
This section describes how to install equipment to build an FNA cluster. After installing the equipment, you will need to configure the equipment to work in your facility. This information is provided in section 4 Network Configuration.
Use the pre-installation checklist to confirm that your environment has the resources required to install the FNA into your existing network: Table 4: FNA Pre-Install Checklist
n x 10 Gigabit (10Gb) available ports on ISP switch or router 1 n available SFP+ (or compatible) optics of same type - Long Range (LR) or Short Range (SR) 1 n available fiber patch cables (multi-mode or single-mode) 1 n available input power connections 2 Power connections are divided evenly between two different power sources Sufficient power (8kW) for each Stock Keeping Unit (SKU) Sufficient rack space is available 3 1
Where n = 2 for every FNA SKU deployed.
2
Where n = 2 times the number of devices (e.g., SKU1 is (5) devices, so (10) input power connections are required). 3
For initial deployment (SKU1), 9RU (nine rack unit) is required. For additional capacity (SKU2), 8RU is required.
Facebook Network Appliance Deployment | 9
Install the ProLiant DL380 servers into a compatible server rack. Utilize the manufacturer instructions for the model included in your order: DL380P Gen8: Server User Guide (Edition: 5) DL380 Gen9: Server User Guide (Edition: 3) Connect C13-C14 AC power cables from each server to your power distribution units (PDUs). NOTE: If you install the servers in a 2-post rack, you must provide the proper 4-post to 2-post conversion hardware.
Install the Cisco Nexus switch in the rack. Utilize the instructions in the Cisco Nexus 3000 Series Hardware Installation Guide (April 22,2015). Connect power cables from the switch to your PDUs.
Using the supplied network cables, connect each ProLiant DL380 server from network interface controller (NIC) port 2 to their respective QSFP+ ports on the Cisco Nexus switch (see and 3). Begin with the server at the highest location in the rack (labeled fna001) and work your way to the bottom-most server. Connect fna001 to port 32 on the switch, fna002 to port 31, and so on. See Figure 30 in Appendix F: Additional Installation Images for a reference on server connectivity order. NOTE: Each ProLiant DL380 server and QSFP+ connector have a round green sticker to show the correct orientation. Ensure both green dots on each QSFP+ connector and server chassis are facing upward. NOTE: The Cisco Nexus switch QSFP+ ports 9 ~ 12 are not used and should not be utilized for any FNA or non-FNA equipment. Each Cisco Nexus switch supports up to twenty servers.
Facebook Network Appliance Deployment | 10
Connect the Cisco Nexus switch to your router. Use SFP+ ports 1 and 2 for the first (4) servers (see Figure 4). These ports are pre-configured for the Link Aggregation Control Protocol (LACP). See Appendix C: Link Aggregation/Link Aggregation Control Protocol (LACP) Configuration for more information. NOTE: The Cisco Nexus’ (4) SFP+ ports and QSFP+ port 1 cannot be utilized at the same time. These ports are internally multiplexed. See Figure 30 for more information
Facebook Network Appliance Deployment | 11
Power on all FNA hardware. Press the power buttons on the ProLiant DL380 servers and power on any interfacing systems (i.e., PDUs, existing router, etc.).
NOTE: The Cisco Nexus switch does not have power buttons. They receive power when connected to a PDU with adequate power.
Facebook Network Appliance Deployment | 12
Check and record the LED states for each PSU, NIC, and Solid State Drive (SSD) for each ProLiant DL380 server in the cluster. For LED definitions, refer to the LED Definitions section in Appendix B: Troubleshooting.
Check the light levels (for deployments utilizing a Cisco Nexus switch, refer to Measuring the Optical Level through IOS, Document ID: 16150 ). Light levels need to be between -2dB and -7dB for proper operation. Light levels outside of this range may be indicative of a connectivity issue. If you find the levels out of range, see Appendix B: Troubleshooting. This completes the FNA hardware installation procedure. Equipment installation should look similar to Figure 7.
Facebook Network Appliance Deployment | 13
NOTE: If you are adding additional equipment to your initial FNA installation, see section 6 System Upgrades.
Facebook Network Appliance Deployment | 14
This section describes the procedure to configure the FNA to peer with Facebook’ s network.
Connect a VGA monitor and USB keyboard to one of the FNA servers. Upon server boot-up, you will see the FNA Main Screen in Figure 8:
NOTE: The servers should be preconfigured with IP addresses. If this is not the case, you will need to create and use a custom USB boot disk. See Section 9.1 Creating a Using a USB Boot Disk.
Configure the point-to-point connection. This connection will be configured on the link-aggregation control interface (LACP). See Appendix C: Link Aggregation/Link Aggregation Control Protocol (LACP) Configuration for additional configuration details . Untag the Virtual LAN (VLAN) link-aggregation interface. Set a static route to the allocated subnets via the link-aggregation interface. NOTE: Facebook requires that the FNA have internet connectivity before BGP peering can be established. The BGP peering session does not need to be routing before the FNA has internet connectivity.
Configure BGP. Use the IPv4 and IPv6 addresses originally supplied in the FNA order form. Enable Graceful Restart (GR). See Appendix D: BGP Configuration Details for configuration information. Enable External BGP (EBGP) Multi-hop. The Autonomous System Number (ASN) that you will be peering with is 63293. Configure your router’s BGP settings. See Appendix D: BGP Configuration Details for configuration information.
Facebook Network Appliance Deployment | 15 NOTE: Once the router’s BGP settings have been properly configured, a latency based routing test will be initiated to optimize routing paths. It is required that all FNA servable netblocks be announced for the purpose of this test. This process takes about twenty-four hours to complete. BGP establishment is not required during this time. See section 4.4 Latency Based Routing for more information. NOTE: The BGP peering session is intended to be a router-to-server configuration and not a switch-to-switch configuration. NOTE: Multi-hop sessions are supported.
Facebook will begin initializing the FNA system for operation. The FNA cluster will begin to download application data. The time to finish the download depends on your connection speed. Within about two business days, BGP peering will be established: The connection will be initiated by one of the FNA servers. The connection will be coming from the 16 th IP address in each allocated subnet (e.g. 192.168.1.16, fe80::10). FNA will not advertise any routes/prefixes. NOTE: The remote software delivery session will begin only after all of the FNA hardware has been powered on, the point-to-point connection has been established, and routing to the internet is enabled.
This completes the network configuration of the FNA. Contact
[email protected] to inform them that you have completed the installation.
FNA applies inbound filtering to reject the following address spaces on BGP peering sessions to ensure successful BGP establishment:
Prefixes smaller than /24 (255 IPs)
Default route
Bogon prefixes
RFC 1918
Facebook’s IP prefixes
Facebook Network Appliance Deployment | 16
The FNA utilizes a latency based routing test to optimize traffic routing. For the purpose of this test, it is critical that all netblocks that can be served be announced to the FNA. Note that prefixes announced to the FNA only implies those prefixes are eligible to serve Facebook traffic. The latency based routing test evaluates the round-trip latency of your network. This is accomplished by sending a small sample of static content from the netblock population you announced. Initially, only a small sample set is sent to the FNA cluster. The sample volume will ramp up during this test. This makes it possible for Facebook to observe and construct the most optimal routing paths for your FNA deployment. The accuracy of the latency based routing service is dependent on the percentage of FNA servable netblocks announced. Providing a large percentage of your FNA servable netblock population allows the service to construct more optimal (low-latency) routing paths. A small percentage of netblocks announced (i.e., a test netblock) will not allow the test to function as intended and traffic may be routed in a seemingly random course. It is critical that every FNA servable netblock is provided to Facebook to optimize routing paths for your FNA deployment.
Facebook Network Appliance Deployment | 17
Please complete this checklist after you have installed the FNA to verify proper installation. Table 5: FNA Installation Completion Checklist
All ProLiant DL380 servers in the cluster are powered on. Each power supply of each ProLiant DL380 server indicates an ON, Steady Green state. The Life (L) and Status (S) LEDs for each SSD in each ProLiant DL380 server indicate an On, Steady Green state. All accompanying LEDs for the connected fiber uplink SFP+ ports are green. All network cables are seated properly to each ProLiant DL380 server. The port 2 (P2) Link LED (LNK) on each NIC in the cluster indicates an On, Steady Green state. Light level readings for all uplinks are between -2dB and -7dB
Facebook Network Appliance Deployment | 18
This section describes how to add additional capacity your original FNA deployment.
WARNING: This procedure requires updating the Link Aggregation (LAG) configuration. Updating the LAG may cause service interruption. If this is the case, it is recommended to drain network traffic first. See section 7 Maintenance and Monitoring for instructions on how to drain network traffic.
Refer to section 2.2 SKU2 Provided Equipment for a list of provided hardware. Make sure all of the hardware is available. Rack the additional ProLiant DL380 servers, connect the peripheral cables (power and network), and power on the servers. Utilize section 3 Hardware Installation for reference. Connect (2) additional 10G fiber uplink cables from port 3 and port 4 on the Cisco Nexus switch to the same router (or switch) where the initial two 10G fiber uplinks are connected (see Figure 9).
Update the Link Aggregation (LAG) configuration on the router (or switch) to accommodate the additional interfaces. NOTE: All cluster augments are implemented with an increment of four ProLiant DL380 servers.
Facebook Network Appliance Deployment | 19
WARNING: This procedure requires updating the Link Aggregation (LAG) configuration. Updating the LAG may cause service interruption. If this is the case, it is recommended to drain network traffic first. See the Maintenance section for instructions on how to drain network traffic.
Refer to section 2.2 SKU2 Provided Equipment for a list of provided hardware. Make sure all of the hardware is available. Rack the ProLiant DL380 servers, connect the peripheral cables (power and network), and power on the additional servers. Utilize the Hardware Installation section for reference. Connect the provided QSFP+ to 4x SFP+ breakout cable assembly to port 2 on the Cisco Nexus switch. See Figure 10 for breakout cable interface location. 1 If you are upgrading from a (16) server to a (20) server cluster, connect the 2 nd provided QSFP to 4x SFP+ breakout cable to port 3 on the Cisco Nexus switch.
Connect (2) additional 10G fiber uplink cables for each group of (4) servers from the Cisco Nexus switch to the same router (or switch) where the initial 2x 10G fiber uplinks are connected. Update the LAG configuration on the router (or switch) to accommodate the additional interfaces. 1
NOTE: The provided QSFP to 4x SFP+ breakout cable assembly only supports LR single-mode connections.
NOTE: Refer to Figure 32: System Architecture Diagram for a diagram of a (20) server FNA cluster.
Facebook Network Appliance Deployment | 20
This section provides a reference for best practices regarding augmentations. This growth path optimizes for deployment failover scenarios by reducing the quantity of Single Points Of Failure (SPOF).
NOTE: When building an FNA cluster beyond eight servers, it is best practice to install the additional servers in a new rack and maintain equal size clusters (as depicted in growth path 3 and 4).
Facebook Network Appliance Deployment | 21
When performing either singular or recurring maintenance jobs, email
[email protected] to report the planned maintenance.
Before powering down the FNA cluster for maintenance, all network traffic must be properly drained. This procedure describes how to properly drain traffic from the cluster: IMPORTANT: This procedure requires withdrawing prefixes. It is important that the BGP peering session is maintained while withdrawing prefixes.
Withdraw prefixes. While maintaining the BGP peering session, begin withdrawing BGP prefixes. Traffic will begin to drain. Traffic will fully drain in less than one hour.
You should shut down the FNA only if it is absolutely required and traffic has been properly drained from the system (see 7.2 Draining Network Traffic). This should be a rare occurrence. To shut down the FNA, follow the procedure outlined below or contact
[email protected] for assistance: Connect a monitor and keyboard to the server you want to power off. Use the arrow keys navigate to the Shut Down text. Press the [Enter] key. The FNA will power off.
Facebook regularly monitors internet reachability from various simulated endpoints. When reachability issues are detected on these Virtual IPs (VIPs), Facebook may drain traffic from an FNA cluster. This measure ensures that Facebook maintains a high quality of service for the Facebook platform.
Facebook Network Appliance Deployment | 22
The Terms of Service agreements that you signed when you installed the FNA in your facility included specific language pertaining to the disposal of storage media. That language is replicated here for your convenience. Your ISP has agreed to:
Only deal with and process Edge user data in compliance with, and subject to, the instructions received from Edge and will not use or process the Edge User Data for any other purpose whatsoever.
Take custody of and secure, and support the destruction of, all server storage media (example: hard disk, solid state disks, flash memory cards, etc.) that are uninstalled from the servers, and provide secure handling, storage, packing, and shipping of the storage media per IBM provided instructions and at IBM's expense to an IBM designated asset recovery center for destruction.
Facebook Network Appliance Deployment | 23
FNA equipment comes preconfigured with the base software installation and IP configuration that was specified in the FNA kit order. In some instances, the IP configuration may need to be changed (such as configuration changes made between processing an order and receiving the kit). If this is the case, you will need to reconfigure the equipment by creating and using a USB Boot Disk.
To create a USB boot disk with to change the system configuration, follow these steps: Log into the FNA portal. Click the Caching tab and Manage FNA. Locate the name of the FNA kit that needs to be reconfigured under the Installed Kits pane. Verify that the FNA IP Prefix and every P2P address are correct. If any information is incorrect, stop and inform the FNA team of the changes that need to be made (see section 11 Support). Only continue to the next step after the information has been corrected.
Locate the USB Boot Disk item under the Installed Kits pane. Click the Disk Image and Kit Config links to begin downloading (see Figure 13).
Facebook Network Appliance Deployment | 24
Connect a blank USB flash drive into a computer running either Windows, MacOS, or Linux. Instructions for creating a USB boot disk for each of these operating systems is provided in this section. Create the USB boot disk. Utilize the instructions for the appropriate operating system: WARNING: This drive will be completely overwritten.
Instructions for Windows Operating Systems Download and install Win32 Disk Imager and 7-Zip. Use 7-Zip to extract the disk image (see Figure 14).
Facebook Network Appliance Deployment | 25
Use Win32 Disk Imager to write the Disk Image to the appropriate USB drive.
Instructions for Mac OS Run diskutil list to find the mount point of the USB drive.
Diskutil list /dev/disk0 (internal, physical): /dev/disk1 (internal, virtual): /dev/disk2 (external, physical): #: TYPE NAME SIZE 0: FDisk_partition_scheme *15.5 GB 1: DOS_FAT_32 UNTITLED 15.5 GB Unlocked Encrypted
IDENTIFIER disk2 disk2s1
Confirm you have selected the correct mounting point by running diskutil info disk# and verifying the drive name, size, and protocol. Run diskutil unmountDisk
to unmount the USB drive.
Diskutil unmountDisk /dev/disk2 Unmount of all volumes on disk2 was successful Run gunzip to unzip the disk image. WARNING: dd will overwrite any disk you specify. Ensure you have selected the correct USB disk before continuing.
gunzip –d ~/Downloads/caelus.img.gz Run dd to write the disk image to the USB drive.
Sudo dd if=~/Downloads/caelus.img of=/dev/disk2 bs=4000000 && sync 67+1 records in 67+1 records out 268435456 bytes transferred in 183.506575 secs (1462811 bytes/sec)
Facebook Network Appliance Deployment | 26
Instructions for Linux Operating Systems Run mount or blkid to find the mount point of the USB drive. Run umount to unmount the USB drive. Run gunzip to unzip the disk image:
gunzip –d ~/Downloads/caelus.img.gz Run dd to write the disk image to the USB drive. WARNING: dd will overwrite any disk you specify. Ensure you have selected the correct USB disk before continuing.
Sudo dd if=~/Downloads/caelus.img of=/dev/disk2 bs=4000000 && sync 67+1 records in 67+1 records out 268435456 bytes transferred in 183.506575 secs (1462811 bytes/sec) Configure the USB boot disk by dragging the FNA kit config JSON file (_config.json ) to the CONFIG partition on the USB drive.
Connect the USB drive to the USB port on the server labeled ‘fna001’ (see Figure 17). NOTE: The USB boot disk will configure the entire FNA system. You do not need to connect the USB drive into the remaining servers.
Facebook Network Appliance Deployment | 27
Reboot ‘fna001’ while the USB drive is connected. The server will automatically boot from the USB drive. If it does not automatically boot from the USB drive, access the BIOS settings and set the USB drive as the first boot option. The remainder of the configuration process is automated. You do not need to interact with the FNA. You can connect a VGA monitor into the VGA port of ‘fna001’ to watch the configuration process. After a few minutes, the host will boot from the USB boot disk and you will see the FreeBSD boot screen (Figure 18):
The USB boot disk will attempt to access the internet and download software necessary for the FNA to operate.
Facebook Network Appliance Deployment | 28
NOTE: This process will take a significant amount of time. If any errors appear on the monitor, take a picture or note the errors seen. Contact the FNA team (see the Support section) and submit these errors.
Facebook Network Appliance Deployment | 29
The network partner portal is the central hub for requesting additional FNA kits, reviewing deployment information and metrics, and troubleshooting issues you may encounter with your deployment. URL: https://partners.facebook.com/network/
Logging into the NPP is as easy as logging into Facebook. If you have an existing Facebook account associated with your work email address, the portal will use that account to log in. If you have a Facebook account associated with your personal email address, you must associate your work email address with your Facebook account. If you don't have a Facebook account, you need to create one. Go to http://www.facebook.com and follow the instructions for creating an account. To associate your work email address with your Facebook account, perform the following steps: Click the Menu icon (
) located at the top right of a Facebook page. Click Settings.
Click Email. Click Add another email or mobile number . Enter your work email address in the New Email field. Click Add. Re-enter your Facebook password when prompted. A confirmation code will be sent to verify the email address. Click Confirm and enter the code. Click OK. Click Save Changes.
Facebook Network Appliance Deployment | 30
The portal allows you to authorize others as administrators and set their permissions.
To add a new administrator, perform the following steps: Click the Settings icon ( ) located at the top right of the portal page. Enter the name and email of the user you wish to make an administrator. Click Add. An invite will be sent to the user's email address. Their permission status will remain "pending" until they accept the invite.
NOTE: Admin access grants access and control to all settings and data within the portal. NOTE: You can also set the user's permissions to "Read-Only Access". This gives the user complete access to all portal data, but does not allow access to any settings or controls. To set a user's permission to read-only access, follow the steps in the Modifying Permissions section and select Read-Only Access.
To modify an existing user's permissions, perform the following steps: Click the Settings icon ( ). Click the Edit link on the user's card. Click the chevron icon ( Access.
) below the user's email address. Select either Admin Access or Read-Only
Facebook Network Appliance Deployment | 31
Click Done.
To remove a user from the portal, perform the following steps: Locate the user you would like to remove. Click the Remove link on the user's card. Enter your Facebook password if prompted.
Facebook Network Appliance Deployment | 32
FNA deployments need to connect to your network. To enable this connection, you need to provide your ISP's Autonomous System Number (ASN). Facebook requires this data to estimate the number of servers your FNA deployment will require. Facebook also uses this information during the provisioning process to properly route traffic to the FNA. To add your ASN, follow these steps: Click the Settings icon ( ). Click Autonomous Systems located in the left menu. Enter your ASN and AS Name. Enter your ISP's ASN and AS Name. Click Add.
To remove an ASN, follow these steps: Click the Settings icon ( ). Click Autonomous Systems located in the left menu. Click the Remove link next to the ASN entry you wish to remove. NOTE: You can enter multiple ASNs. All ASNs should be kept up-to-date. NOTE: If the ASN entry has a yellow clock icon (
), it is currently being reviewed by Facebook.
Requesting an FNA kit is a multi-step process that takes several weeks to complete. Facebook evaluates
Facebook Network Appliance Deployment | 33
submissions for qualified ISPs that can utilize an FNA. If your application is approved to receive an FNA, Facebook will send an FNA kit to the designated ISP address. To request an FNA kit, follow these steps: Click the Support link at the top of the page. The Support page will appear. Click the Request a New Kit hyperlink. Enter the deployment country where the FNA will be installed. Click Next. Select the Type of Kit. Review the information and click the Begin Order Form button. An online order form will appear in a new window or tab. Use table 6 as a guideline for order form field entry:
Shipping Contact
Installation Contact
Engineering Contact
Provide contact information for the primary contact that will receive FNA shipments.
This can be the engineering contact.
Provide contact information for the primary contact that will install or lead installation of the FNA equipment.
This can be the engineering contact.
Provide contact information for the primary engineering contact that can answer technical questions about the network.
Shipping Address
Provide the location where FNA equipment will be sent.
Site Installation Address
Provide the location where FNA equipment will be installed.
Facebook Network Appliance Deployment | 34
Special Shipping Instructions
Provide any notable unique shipping instructions.
This is an optional field.
Rack Name
Enter a name for the FNA rack.
This is an optional field.
Facility / Deployment Technical Information
Provide power source requirements for the FNA deployment (AC or DC).
Provide the type of optical transceiver your deployment would require – either long range (LR) or short range (SR)
Provide the power outlet type your deployment would require.
Networking and IP Information
IPv4 Settings
IPv6 Settings
Provide the primary peer ASN. Provide any additional ASNs or communities to be served.
Provide IPv4 allocation for caches, gateway address and network, cache address and network, and Border Gateway Protocol (BGP) peer address.
Provide IPv6 allocation for caches, gateway address and network, cache address and network, and BGP peer address.
Gateway and Cache Peerto-Peer (P2P) IP addresses must be within the same subnet.
BGP address / network should not overlap with IP allocation for caches.
Gateway and Cache Peerto-Peer (P2P) IP addresses must be within the same subnet.
BGP address / network should not overlap with IP allocation for caches.
Fill out and submit any required order forms located in the Required Forms section near the top of the order form. Forms that require submission are denoted with the "X" icon ( ). Forms that are currently pending review are denoted with the "exclamation" icon ( ). Click Submit if you have completed the order form or Save Draft if you plan to complete the order form at a later time.
Facebook Network Appliance Deployment | 35
If Facebook selects you to participate in the FNA program, International Business Machines (IBM) will contact the shipping contact you indicated with further details and will make arrangements to deliver FNA equipment. IBM is Facebook's logistics partner for FNA equipment.
Facebook Network Appliance Deployment | 36
Once you have installed your FNA equipment and it is brought online, access the Manage FNA to:
Review Installed Kit Information
View Traffic Information
Review Contact, Exchange, and Facility Information
View Peering Sessions
The installed kit information page allows you to view the number of servers per cluster, IPv4 and IPv6 network addresses, advertised prefixes, and disk image. To view this page, follow these steps: Click the Caching tab located at the top of the portal. Click Manage FNA. This page displays FNA information sorted by cluster name. You can also request a new kit by clicking Request a New Kit in the top-right corner.
Facebook Network Appliance Deployment | 37
FNA traffic metrics are located in the Insights page. This page allow you to observe cache-fill and retransmit information sorted by FNA cluster name. To view traffic information, follow these steps: Click the Caching tab located at the top of the portal. Click Insights. The FNA Traffic information will appear. Each plot represents either "To Subscribers" or cache fill for each FNA cluster installed. To view traffic retransmission information, click the button at the top left of the graph. Click FNA Retransmits. This page will show traffic retransmission as a percentage of the total FNA traffic.
The contact information you provided on the FNA order form can be found in the Peering page. To review this information, follow these steps: Click the Peering tab. Click Peering Summary (this should load by default). This page allows you to review your contacts, exchanges, and facilities information.
Facebook Network Appliance Deployment | 38
The contact information you provided on the FNA order form can be found in the Peering page. To review this information, follow these steps: Click the Peering tab. Click Peering Sessions. This page allows you to view peering session details, including ASN, location, peer type, link speed, router address, peer port, and status.
Facebook Network Appliance Deployment | 39
If you have issues receiving equipment or installation and configuration issues, use Table 7 to contact the most appropriate support team for assistance. Table 7: Support and Contact Information
Email: [email protected] Facebook Network Appliance
Network Support is Required
BGP Peering Issues
Email: [email protected]
IBM
Assistance with Draining Network Traffic
Hardware Support is Required
Hardware is missing from order. Provide the following: o Missing Item(s) o ISP Name o ISP Address o Equipment Received Date o Your Contact Information
Notification of Hardware Installation Completion
NOTE: In some cases, it may be necessary to return the equipment. It is recommended to keep the original packaging material for two months.
Facebook Network Appliance Deployment | 40
What type of data is being stored on FNA? An FNA cluster will cache the following types of data:
Static Facebook content
Static Instagram content
FNA does not cache WhatsApp or Oculus content.
How much traffic can the FNA cluster serve? The amount of traffic that the FNA cluster may serve varies based on:
The size of the cluster and the number of available machines
The amount of prefixes being advertised / number of users
The RTT latency to advertised prefixes
Content type variance: profile pictures, large images, and video
Each server in the cluster is capable of delivering ~4Gbps of traffic. However, depending on the deployment region, service type (e.g., mobile vs. fixed line), and variance in content type (e.g., images and video), the FNA cluster may deliver less overall traffic (~3Gbps is expected). Generally, a high variance in content type will yield lower traffic throughput and low content type variance will yield higher traffic throughput.
The installation procedure indicates installing the supplied switch directly to our existing router. Our facility aggregates to our existing switches before connecting to a router. How does this modify the installation process? FNA provides aggregate connections from the cache servers directly to the router. However, if you connect the supplied FNA switch to an intermediary aggregation switch, this configuration is fine and the logical topology will remain the same.
Is FNA Carrier-Grade NAT (CGNAT) compatible? Yes. FNA is compatible with CGNAT for the RFC 1918 and RFC 6598 address spaces.
Facebook Network Appliance Deployment | 41
I’ve advertised a certain prefix but it doesn’t seem to be served by the FNA. Why is this happening? Advertising a prefix marks means it as eligible to be served from the FNA, but does not guarantee it will be served by FNA. There are a few reasons why the FNA may not be serving a prefix which has been advertised to it:
If it has been less than 24 hours since the FNA has been online or since you have started advertising the prefix (see Latency Based Routing). Facebook may not have collected enough latency samples in order to decide whether the FNA cluster will serve the prefix.
If Facebook is observing less latency to the prefix from a nearby Point of Presence (PoP), then the PoP will be preferred for serving the prefix.
If the system resources (e.g. CPU utilization) have been exhausted.
Which routes should I advertise towards Facebook and how does failover occur between FNA’s ? In most cases, the best option is to announce all customer prefixes. In cases where there may be congestion from region to region, the best practice is to announce local routes only. Why should I advertise all of my prefixes? Facebook’s content routing platform measures Round Trip Time (RTT) latency between each Point of Presence (PoP) and FNA deployment to all prefixes being advertised by a peer ISP. Based on this measurement and a number of other factors (such as available system resources and link capacity), a client is routed to the most optimal location for making content requests.
Facebook Network Appliance Deployment | 42
This section provides troubleshooting resources for events that may be encountered during installation or operation.
Each ProLiant DL380 server and Nexus switch has LEDs that can be used to diagnose the FNA for issues. These LEDs also useful for verifying whether the FNA was installed correctly and for receiving quick technical support.
Activity (A)
Life (L)
Status (S)
Off
____________________
The server is powered off or the power supply has failed.
On, Steady Green
____________________
The power supply is on.
On, Blinking Green
____________________
There is data activity on the card.
On, Steady Green
____________________
The drive has sufficient life for programming and erasing the memory.
____________________
The drive has approximately 10% or less life remaining for programming and erasing the memory.
On, Steady Red
____________________
The drive no longer has program or erase cycles. Data can be read, but writing data is at risk.
On, Steady Green
____________________
The drive is operating normally.
On, Blinking Green
____________________
A network administrator is toggling the LED to locate the server.
On, Steady Yellow
Drive warning. One of the following has occurred: On, Steady Yellow
____________________
On, Blinking Red
____________________
The maximum temperature threshold has been reached. Another issue has occurred.
A firmware fault code has been issued.
Facebook Network Appliance Deployment | 43
Fatal drive warning. One of the following has occurred:
Status (S)
On, Steady Red
____________________
Drive failure.
Drive has exceeded its maximum temperature threshold.
The RAID volume has failed.
The RAID volume is not configured.
The drive’s internal backup power supply has failed.
Another issue has occurred.
Activity (ACT)
On, Blinking Green
____________________
There is network activity.
Link (LNK)
On, Steady Green
____________________
A connection exists between the server and the network.
Power Supply
On, Steady Green
____________________
The power supply is on.
____________________
The fan is powered on.
On, Steady Green
____________________
The system is operating normally.
On, Steady Orange
____________________
The system has posted an environmental issue code.
On, Steady Red
____________________
The system has posted an environmental issue code.
Fan
System Status
On, Steady Green
Facebook Network Appliance Deployment | 44
In cases where light levels are outside the normal range (between -2dB and -7dB), utilize Table 9 to resolve the issue. Table 9: Signal Level Troubleshooting
Above -2
The signal is too strong. This could be caused by many reasons.
Fix with your normal operating procedures for strong signals.
None (Normal Operating Range)
None (Normal Operating Range)
-2 to -7
-8 to -20
A degraded fiber optic cable is installed
A degraded SFP+ optical transceiver is installed
A fiber optic cable has a dirty connection
A fiber optic cable is bent
-21 to -40
If this does not resolve the issue, either replace TX/RX hardware or solve with your normal fiber replacement procedures.
Inspect the fiber optic cable(s) along its length. Gently roll the fiber if you observe bends or kinks.
Check that the port connection is enabled
Ensure the cables are connected to the proper port interface. See Figure 10 for port interfaces.
Network connectivity has not been established All cables are connected to their proper ports, but the port(s) is disabled All ports are enabled, but a cable is connected to the incorrect port
Disconnect, clean, and reconnect the fiber optic cable(s) and/or SFP+ optical transceiver(s).
Facebook Network Appliance Deployment | 45
The following are examples of LACP configurations for the most common Cisco IOS and Juniper Junos routers. Apply the lines of code (below) that are appropriate to your specific router model.
The following configuration settings should be applied to a Juniper Junos router for FNA LACP compatibility:
ae1 { aggregated-ether-options { lacp { active; periodic fast; } } unit 0 { description FACEBOOK-FNA; family inet { address ; } family inet6 { address ; } } } xe-0/0/0 { description FACEBOOk-FNA; gigether-options { 802.3ad ae1; } } xe-1/0/0 { description FACEBOOk-FNA; gigether-options { 802.3ad ae1; } }
To confirm whether the LACP settings were properly applied, run the following commands:
show lacp interfaces xe-4/3/3 show interfaces ae3 | grep Physical If the following output appears, the LACP settings were properly configured for the Juniper Junos router:
Facebook Network Appliance Deployment | 46
show lacp interfaces xe-4/3/3 Aggregated interface: ae4 LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity xe-4/3/3 Actor No No Yes Yes Yes Yes Fast Active xe-4/3/3 Partner No No Yes Yes Yes Yes Fast Active LACP protocol: Receive State Transmit State Mux State xe-4/3/3 Current Fast periodic Collecting distributing show interfaces ae4 | grep Physical Physical interface: ae4, Enabled, Physical link is Up
The following configuration settings should be applied to a Cisco IOS-XR router for FNA LACP compatibility:
! interface Bundle-Ether1 description FACEBOOK-FNA ipv4 address ipv6 address ! interface TenGigE0/0/0/0 description FACEBOOK-FNA bundle id 1 mode active lacp period short ! interface TenGigE0/1/0/0 description FACEBOOK-FNA bundle id 1 mode active lacp period short ! To confirm whether the Cisco IOS-XR LACP was properly configured, run the following command:
show port-channel sum interface port-channel 911 If the following output appears, the LACP settings were properly configured on the Cisco IOS-XR router:
Facebook Network Appliance Deployment | 47
show port-channel sum interface port-channel 911 Flags: D - Down P - Up in port-channel (members) I - Individual H - Hot-standby (LACP only) s - Suspended r - Module-removed S - Switched R - Routed U - Up (port-channel) M - Not in use. Min-links not met -------------------------------------------------------------------------------Group PortType Protocol Member Ports Channel -------------------------------------------------------------------------------911 Po911(RU) Eth LACP Eth18/9(P) Eth18/10(P) Eth18/11(P) Eth18/12(P)
Facebook Network Appliance Deployment | 48
The following are examples of BGP peering code for the most common Cisco OOS and Juniper Junos routers. These examples allow you to configure BGP peering sessions to send Facebook traffic to the FNA cluster. Contact Facebook ([email protected] ) if you encounter problems when setting up BGP peering. NOTE: After the FNA cluster has network access, it will begin downloading application data that enables the FNA to operate. All application data must be installed before the BPG session is established. The application data may take some time to download.
The following router BGP configuration should be used for FNA deployments that utilize a Juniper Junos router:
protocols { bgp { group FACEBOOK-FNA-IPv4 { neighbor { multihop; family inet { unicast; } export FACEBOOK-FNA-EXPORT; peer-as 63293; } } group FACEBOOK-FNA-IPv6 { neighbor { multihop; family inet6 { unicast; } export FACEBOOK-FNA-EXPORT; peer-as 63293; } } } } policy-options { policy-statement FACEBOOK-FNA-EXPORT { term export { from { prefix-list FACEBOOK-FNA; } then accept; } term reject { then reject; }
Facebook Network Appliance Deployment | 49
To enable BGP Graceful Restart on a Juniper Junos router, perform the following procedure:
Utilize Juniper’s Configuring Graceful Restart Options for BGP reference document. Navigate to the appropriate hierarchy level and change the disable statement to enable. Evaluate whether GR was successfully negotiated by running the following command:
show bgp neighbor | grep Options Locate the applicable Junos device and evaluate whether GracefulResart is within the Options output:
show bgp neighbor | grep Options Options:
PrefixLimit Refresh> Options:
If the Options output string contains GracefulRestart, GR was successfully negotiated. REMINDER: GR is disabled on Junos devices by default. Enabling GR is required for the FNA deployment. NOTE: Enabling Graceful Restart on the Junos device does not imply that it is in use. Both sides of the session must support and negotiate Graceful Restart.
The following router BGP configuration should be used for FNA deployments that utilize a Cisco IOX-XR routers:
! router bgp x ! address-family ipv4 unicast ! address-family ipv6 unicast ! neighbor remote-as 63293 address-family ipv4 unicast route-policy FACEBOOK-FNA-EXPORT out ! neighbor remote-as 63293 address-family ipv6 unicast route-policy FACEBOOK-FNA-EXPORT out !
Facebook Network Appliance Deployment | 50
To enable BGP Graceful Restart on a Cisco NX-OS router, perform the following procedure:
Utilize Cisco’s Configuring a Graceful Restart procedure.
show ip bgp neighbors | grep "Graceful Restart" If the output string ‘Graceful Restart capability: advertised received ’ appears, GR was successfully negotiated.
Facebook Network Appliance Deployment | 51
The following diagram demonstrates the route of a Facebook request to your ISP both before and after the FNA deployment:
Facebook Network Appliance Deployment | 52
REMINDER: QSFP+ port 1 is internally multiplexed with the (4) SFP+ uplink ports. The (4) SFP+ ports and the QSFP+ breakout port cannot be utilized at the same time.
Facebook Network Appliance Deployment | 53
The following diagram shows the topology of a (20) server FNA installation:
NOTE: This diagram depicts an FNA cluster with the maximum number of supported servers per switch. However, when building an FNA cluster beyond eight servers, it is best practice to install the additional servers in a new rack and maintain equal size clusters (as demonstrated in Section 6.2 Growth Path).
Facebook Network Appliance Deployment | 54
Table 10: FNA System Benchmarks
Expected Bandwidth Offload
65%
Maximum Sustained Throughput
4Gbps per cluster
Facebook Network Appliance Deployment | 55
Table 11: AC Power and Temperature Requirements
Input Current (by PSU)
8.9A @100V 4.3A @ 200V
Steady State Power
750W @ 100 V to 120V 750W @ 200V to 240V
Frequency
50-60Hz
Dual Path Power
Required
Dual PDUs
Required
Operating Temperature Range
10°C to 35°C
Typical Operating Power
210W with Twinax @ 100% load; (2) PSUs 240W with SR4 optics @ 100% load; (2) PSUs
Maximum Power
340W
Input Voltage by PSU
100V to 240V
Frequency
50-60Hz
Efficiency
89% to 91% @ 220V
Dual Path Power
Required
Dual PDUs
Required
Operating Temperature Range
0°C to 40°C
Typical Heat Dissipation
717 BTU/hr (with Twinax at 100% load) 819 BTU/hr (with SR4 optics at 100% load)
Maximum Heat Dissipation
1160 BTU/hr
Cordset Rating
10A @ 250V
SKU1
~ 800W (per SKU)
Facebook Network Appliance Deployment | 56
Table 12: DC Power and Temperature Requirements
Input Voltage Range (V DC)
36-72V DC
Nominal Input Voltage (V DC)
36/48/72V DC
Maximum Rated Output Wattage Rating (W)=
750W @ 36V DC / 48V DC / 72V DC
Nominal Input Current (A)
23A @ 36V DC 17A @ 48V DC 11A @ 72V DC
Maximum Rated Input Wattage Rating (Watts)
815W @ 36V DC 805W @ 48V DC 795W @ 72V DC
Maximum British Thermal Unit Rating (BTU-Hr)
2780 BTU-Hr @ 36 V DC 2740 BTU-Hr @ 48 V DC 2720 BTU-Hr @ 72 V DC
Operating Temperature Range
10°C to 35°C
Typical Operating Power
210W with Twinax @ 100% load; (2) PSUs 240W with SR4 optical transceiver @ 100% load; (2) PSUs
Maximum Power
340W
Input Voltage by PSU
–40V DC to –72V DC
Maximum Current
33A
Efficiency
85% to 88%
Power Supply Efficiency
89% to 91% @ 220V
Dual Path Power
Required
Dual PDUs
Required
Operating Temperature Range
0°C to 40°C
Typical Heat Dissipation
717 BTU/hr (with Twinax @ 100% load) 819 BTU/hr (with SR4 optical transceiver @ 100% load)
Maximum Heat Dissipation
1160 BTU/hr