VELUX_Access_Layer_Low_Level_Design_v0.3.pdf

Advanced Services

VELUX VELUX Access Layer Low Level Design Version 0.3

Corporate Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 526-4100 CISCO CONFIDENTIAL

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS. THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY. The following information is for FCC compliance of Class A devices: This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to part 15 of the FCC rules. rules. These limits are are designed to provide provide reasonable protection protection against harmful harmful interference interference when the equipment is is operated in a commercial commercial environment. environment. This equipment generates, uses, and can radiate radio-frequency energy and, if not installed and used in accordance with the instruction manual, may cause harmful interference to radio communications. communications. Operation of this equipment in a residential area is likely to cause harmful interference, in which case users will be required to correct the interference at their own expense. The following information is for FCC compliance of Class B devices: The equipment described in this manual generates and may radiate radio-frequency energy. If it is not installed in accordance with Cisco’s installation instructions, it may cause interference with radio and television reception. This equipment has been tested and found to comply with the limits for a Class B digital device in accordance with the specifications in part 15 of the FCC rules. These specifications are designed to provide reasonable protection against such interference in a residential installation. However, there is no guarantee that interference will not occur in a particular installation. You can determine whether your equipment is causing interference by turning it off. If the interference stops, it was probably caused by the Cisco equipment or one of its peripheral devices. If the equipment causes interference to radio or television reception, try to correct the interference by using one or more of the following measures: measures: Turn the television or radio antenna until the interference stops. Move the equipment to one side or the other of the television or radio. Move the equipment farther away from the television or radio. Plug the equipment into an outlet that is on a different circuit from the television or radio. (That is, make certain the equipment and the television or radio are on circuits controlled by different circuit circuit breakers or fuses.) fuses.) Modifications to this product not authorized by Cisco Systems, Inc. could void the FCC approval and negate your authority to operate the product. The following third-party software may be included with your product and will be subject to the software license agreement: CiscoWorks software and documentation are based in part on HP OpenView under license from the Hewlett-Packard Company. Company. HP OpenView is a trademark of the Hewlett-Packard Company. Copyright ! 1992, 1993 Hewlett-Packard Company. The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright ! 1981, Regents of the University of California. Network Time Protocol Protocol (NTP). Copyright Copyright ! 1992, David L. Mills. The University of Delaware makes no representations representations about the suitability of this software for any purpose. Point-to-Point Protocol. Copyright ! 1989, Carnegie-Mellon University. All rights reserved. The name of the University may not be used to endorse or promote products derived from this software without specific prior written permission. The Cisco implementation of TN3270 is an adaptation of the TN3270, curses, and termcap programs developed by the University of California, Berkeley (UCB) as part of the UCB’s public domain version version of the UNIX operating system. system. All rights reserved. reserved. Copyright ! 1981-1988, Regents of the University of California.

Cisco incorporates Fastmac and TrueView software and the RingRunner chip in some Token Ring products. Fastmac software is licensed to Cisco by Madge Networks Limited, and the RingRunner chip is licensed to Cisco by Madge NV. Fastmac, RingRunner, and TrueView are trademarks and in some jurisdictions registered trademarks of Madge Networks Limited. Copyright ! 1995, Madge Networks Limited. All rights reserved. Xremote is a trademark of Network Computing Devices, Inc. Copyright ! 1989, Network Computing Devices, Inc., Mountain View, California. NCD makes no representations about the suitability of this software for any purpose. The X Window System is a trademark of the X Consortium, Cambridge, Massachusetts. All rights reserved. NOTWITHSTANDING NOTWITHSTANDING ANY OTHER OTHER WARRANTY WARRANTY HEREIN, ALL ALL DOCUMENT FILES FILES AND SOFTWARE SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PRACTICAL PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. AccessPath, AtmDirector, AtmDirector, Browse with Me, CCDE, CCIP, CCSI, CD-PAC, CiscoLink , the Cisco NetWorks NetWorks logo, logo, the Cisco Powered Cisco Powered Network Network logo, Cisco Systems Networking Academy, Fast Step, Follow Me Browsing, FormShare, FrameShare, FrameShare, GigaStack, IGX, Internet Quotient, IP/VC, iQ Breakthrough, iQ Expertise, iQ FastTrack, the iQ logo, iQ Net Readiness Scorecard, MGX, the Networkers logo, Packet logo, Packet , RateMUX, ScriptBuilder, ScriptShare, SlideCast, SMARTnet, TransPath, Unity, Voice LAN, Wavelength Router, and WebViewer are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn, Discover All That’s Possible, and Empowering the Internet Generation, are service marks of Cisco Systems, Inc.; and Aironet, ASIST, BPX, Catalyst, CCDA, CCDP, CCIE, CCNA, CCNP, Cisco, the Cisco Certified Internetwork Expert Logo, Cisco IOS, the Cisco IOS logo, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Enterprise/Solver, EtherChannel, EtherChannel, EtherSwitch, FastHub, FastSwitch, IOS, IP/TV, LightStream, MICA, Network Registrar, PIX, Post-Routing, Pre-Routing, Registrar, StrataView Plus, Stratm, SwitchProbe, TeleRouter, and VCO are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and certain other countries. All other trademarks mentioned in this document or Web site are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. company. (0105R) INTELLECTUAL PROPERTY RIGHTS: THIS DOCUMENT CONTAINS VALUABLE TRADE SECRETS AND CONFIDENTIAL INFORMATION OF CISCO SYSTEMS, INC. AND IT’S SUPPLIERS, AND SHALL NOT BE DISCLOSED DISCLOSED TO ANY PERSON, PERSON, ORGANIZATION, ORGANIZATION, OR ENTITY ENTITY UNLESS SUCH SUCH DISCLOSURE DISCLOSURE IS SUBJECT TO THE THE PROVISIONS PROVISIONS OF A WRITTEN NONNONDISCLOSURE AND PROPRIETARY RIGHTS AGREEMENT OR INTELLECTUAL PROPERTY LICENSE AGREEMENT APPROVED BY CISCO SYSTEMS, INC. THE DISTRIBUTION OF THIS DOCUMENT DOES NOT GRANT ANY LICENSE IN OR RIGHTS, IN WHOLE OR IN PART, TO THE CONTENT, THE PRODUCT(S), TECHNOLOGY OF INTELLECTUAL PROPERTY DESCRIBED HEREIN.

VELUX Access Layer Low Level Design Design Copyright ! 2009, Cisco Systems, Inc. All rights reserved.

1.

Contents 1.

Contents

3

2.

Figures

5

3.

Tables

6

4.

Configuration Examples

8

5.

Document Information

9

5.1.

Review and Distribution Distribution

6.

Document Acceptance Certificate

10

7.

Introduction

12

8.

7.1.

Preface

12

7.2.

Audience

12

7.3.

Scope

12

7.4.

Related Documents

12

High Level Design Overview 8.1.

9.

14

High Level Design Overview

14

Access Layer Physical Design

16

9.1.

Hardware Components

9.1.1. 9.1.2.

Nexus5596UP Switch Nexus2248TP-E Fabric Extender

16 16 16

9.2.

Naming convention

17

9.3.

FEX Connectivity

18

9.3.3. 9.3.4. 9.3.5.

9.4.

9.5.

Nexus 2232PP to Nexus 5596UP Nexus 2248TP to Nexus 5596UP Nexus 5596UP to Catalyst 6500 (Existing Distribution Layer)

FEX 2000 Connectivity

9.4.6. 9.4.7. 9.4.8. 9.4.9.

Management of FEX 2000 Etherchannel Fabric Interface Connection FEX in Straight-Through Configuration FEX Port Numbering Convention

Host Connectivity

9.5.10.

Connectivity to HP Servers

18 18 18

19 19 20 20 21

22 23

9.5.10.1.

HP NIC Teaming Options

23

9.5.10.2.

HP Blade Servers with 3Com Blade Switches

23

9.5.10.3.

HP Virtual Connect

24

9.5.10.4.

Nexus B22HP

24

9.6.

Port Mapping Details

25

9.7.

Power Connectivity

26

VELUX Access Layer Low Level Design 3

9

CISCO CONFIDENTIAL 19 March 2013

Contents 10.

Access Layer Logical Design

10.1.

Layer 2 Features

10.1.1. 10.1.2. 10.1.3. 10.1.4. 10.1.5.

10.1.6. 10.1.7. 10.1.8. 10.1.9.

10.2.

27 28 28 29 29

10.1.5.1.

Portfast and BPDU Guard

29

10.1.5.2.

Loop Guard

30

10.1.5.3. Bridge Assurance UDLD Error Disable Recovery Interface Speed/Duplex Port Channels

31 31 33 34 34

VPC Design

35

Virtual Port Channel (vPC) Overview

10.2.10.4.

35

Dual Control Plane but Single Layer 2 Node Behavior

10.2.10.5. vPC and LACP Interaction 10.2.11. vPC and Spanning Tree Interaction 10.2.12. vPC Design Considerations

36 37 37 38

10.2.12.6.

vPC Role and Priority

38

10.2.12.7.

vPC Domain ID

38

10.2.12.8.

vPC Peer Keep-alive Link

38

10.2.12.9.

vPC Peer Link

40

10.2.12.10. vPC Ports 10.2.13. vPC Connectivity to the L2/3 Distribution Layer 10.2.14. Velux DC Interconnect Implementation 10.2.15. Proposed vPC Numbering Scheme

Network Management

40 41 43 46

47

11.1.

Host Name

48

11.2.

Banner

48

11.3.

Network Time Protocol (NTP)

49

11.4.

Time Zone

49

11.5.

Logging

49

11.6.

DNS

50

11.7.

SNMP

50

Security

52

12.

12.1.

General Device Security

12.1.1. 12.1.2. 12.1.3.

Enable Protection Services Disable DHCP service TCP Device Access Security

52 52 52 52

13.

Cisco NX-OS Software Recommendation

54

14.

Configuration Templates

55


27

VLAN Trunks VLAN Pruning VLAN Trunking Protocol (VTP) Spanning-Tree Protocol (STP) Layer 2 Feature Placement

10.2.10.

11.

27


2.

Figures Figure 1: Access Layer Design Figure 2: Nexus5596UP switch with three expansion modules Figure 3 Nexus2248TP-E Fabric Extender Figure 4: N2232PP Fabric Extender Figure 5: FEX in Straight-Through Configuration Figure 6: Server Connectivity Model Figure 7 Connectivity to HP Virtual Connect Figure 8: B32HP Connectivity to the Nexus5000 Figure 9: N5596UP port allocation scheme Figure 10: vPC Overview Figure 11: Connectivity to aggregation Figure 12 Velux Inter-DC Implementation Figure 13 DCI BPDU Filter – Backdoor issue Figure 14: Proposed vPC numbering scheme


15 16 16 17 19 22 24 25 26 35 42 43 45 46


3.

Tables Table 1: Related Documents Table 2: FEX Chassis ID scheme Table 3: Cisco NX-OS Code Versions


12 21 54


Tables



Configuration Examples

4.

Configuration Examples Example 3: Interface Configuration for single-attached Host Ports on FEX 22 Example 4: Interface Configuration for Dual Attached Active-Active Host Ports on FEX 23 Example 5: Vlan Pruning Configuration 28 Example 6: VTP Configuration 29 Example 7: Loop Guard Configuration 30 Example 8: UDLD Configuration 33 Example 9: Error Disable Recovery 33 Example 10: Enabling vPC feature configuration on Nexus 5000 38 Example 11: vPC role and priority configuration on Nexus 5000 38 Example 12: vPC peer keepalive link configuration using the mgmt0 39 Example 13: vPC peer keepalive link configuration using SVI interface39 Example 14: vPC peer link configuration on Nexus 5500 40 Example 15: vPC port configuration on Nexus 5000 40 Example 16: vPC connectivity to the aggregation switches 42 Example 17: Interface Configuration for UCS FI connection on Nexus5500 45 Example 18: Configuring IP Address on Mgmt Interface on Nexus 500047 Example 19: Configuring Hostname 48 Example 20: Configuring Banner 48 Example 21: Configuring NTP 49 Example 22: Configuring Timezone 49 Example 23: Configuring Logging 49 Example 24: Configuring DNS 50 Example 25: SNMP Configuration 50 Example 26: Disable DHCP Service on Nexus 52 Example 27: SSH Configuration on Nexus 53



5.

Document Information Author: Barbara Ledda – Cisco Advanced Services Change Authority: Advanced Services Change Forecast: Low Document Version 0.3

5.1.

Review and Distribution Organization

Name

Title



A printed copy of this document is considered uncontrolled

6.

Document Acceptance Certificate Title: Version:

VELUX Access Layer Low Level Design 0.3

Name

Name

Title

Title

Company

Company

Signature

Signature

Date

Date

Name

Name

Title

Title

Company

Company

Signature

Signature

Date

Date

Name

Name

Title

Title

Company

Company




Document Acceptance Certificate

Signature

Signature

Date

Date




7.

Introduction

7.1.

Preface The recommendations in this document are based on input from VELUX, Cisco AS and industry wide leading practices. The intent of this document is to provide design recommendation and configuration templates for the VELUX Network located in Gelsted. This document will be used by the Customer as a guideline for physical implementation and for deriving the actual configurations for the different network components.

7.2.

Audience This document is intended for members of the VELUX and Accenture architecture and engineering teams who have direct responsibility for the network infrastructure inside the Data Centers. In addition, it is advisable that those IT persons responsible for managing server and application environments hosted inside the Data Centers also review this document and provide comments and feedback where applicable .

7.3.

Scope The scope of this document is limited to providing a design recommendations and configuration templates for the VELUX data centers initial deployment of Nexus 5000 and Nexus 2000 in the Access Layer. The scope of the document is limited to the following network blocks or zones: •

7.4.

Access Layer Network at DNKDE and DNKDW Datacenters

Related Documents Table 1: Related Documents

Document

Author

VELUX Network Assessment Document

Cisco




Introduction

Document

Author AS

VELUX DC Network Conceptual Design

Cisco AS

VELUX High Level Design

Cisco AS

[1]




8.

High Level Design Overview The following design principles will be followed for implementing the new Access Layer at VELUX: The access layer and L2 Aggregation will consist two pairs of Cisco Nexus 5596UP, a pair on each Datacenter location •

•

•

• •

8.1.

L2 connectivity between Datacenter will be extended via L2 Etherchannel between the Nexus5596 pairs. A double-sided vPC configuration will be used to ensure high availability and full link utilization. Fabric Extender technology (i.e., Cisco Nexus 2248TP and Nexus 2232PP) will be used to form a “distributed access fabric”. The Fabric Extended will be deployed in Straight-Through configuration vPC can be used as an option for connecting servers to ensure high availability and full link utilization

High Level Design Overview In the initial phase, VELUX will deploy the following number of devices: - 4 x Nexus 5596UP Switches, 2 on each DC location - 4 x Nexus 2248TP-E Fabric Extenders in Straight-Through configuration, 2 on each DC location. - 8 x Nexus 2232PP Fabric Extenders in Straight-Through configuration, 2 on each DC location At the L2/L3 Distribution Layer, the Nexus5500 will connect via vPC to a pre-existing Cat6509 on each Datacenter.

The proposed network design can be seen in Figure 1




High Level Design Overview Figure 1: Access Layer Design

.





9.


9.1.

Hardware Components The following section describes the access layer network elements present in the Velux Datacenters network which shall be covered in depth throughout this document.

9.1.1.

Nexus5596UP Switch The Cisco Nexus 5596UP Switch is a 2RU 10 Gigabit Ethernet, Fibre Channel, and FCoE switch offering up to 1920 Gbps of throughput and up to 96 ports. The switch has 48 unified ports and three expansion slots. At Velux, the Nexus 5596UP will be used, in conjunction with the Nexus 2000 Fabric Extender (referred as FEX hereafter) for 1/10GE copper or 100Mb Connectivity. At the initial deployment, each pair of Nexus 5000 will manage 6 FEX switches in a Straght-Through configuration. Figure 2: Nexus5596UP switch with three expansion modules

9.1.2.

Nexus2248TP-E Fabric Extender The Cisco Nexus 2248TP-E Fabric Extender has four 10 GE fabric interfaces and 48 100/1000M Ethernet host interfaces. With this system, the oversubscription models will vary depeding on the number of uplinks as follows: No oversubscription — Use only 40 of 48 host interfaces, with four fabric interfaces 1.2 to 1 oversubscription — 48 host interfaces for four fabric interfaces 4.8 to 1 oversubscription — 48 host interfaces for one fabric interface • •

Figure 3 Nexus2248TP-E Fabric Extender




Access Layer Physical Design The Cisco Nexus 2232PP Fabric Extender has eight 10 GE fabric interfaces and 32 10G Ethernet host interfaces. With this system, the oversubscription models will vary depeding on the number of uplinks as follows: 4 to 1 oversubscription — 32 host interfaces for eight fabric interfaces 32 to 1 oversubscription — 32 host interfaces for one fabric interface • •

Figure 4: N2232PP Fabric Extender

9.2.

Naming convention The following naming convention will be applied at VELUX

--

Where CI Types can be: !

SW: Switch

!

RT: Router

!

AP: Access Point

!

WLC: Wireless LAN Controller

!

FW: Firewall

!

WO: WAN Optimization

!

IPS: Intrusion Prevention System

Table 2: Switch naming Convention

!"#$%&'()* !"#$%&$'(()*$+, !"#$%&$'(()*$+# 78$&9$'(()*$+, 78$&9$'(()*$+#

+*,%-#.$#/' '-./0(()*12 '-./0(()*12 '-./0(()*12 '-./0(()*12

0/%($#/' 3&45!"#6 3&45!"#6 3&:5786 3&:5786





9.3.

FEX Connectivity

9.3.3.

Nexus 2232PP to Nexus 5596UP During the initial phase of the implementation, the connectivity between the N2232PP and N5596UP will be provided via 4 x 10GE uplink. This will result in an oversubscription ratio of 8:1 between N2232PP and the parent N5596UP. Accenture has determined that this is an acceptable oversubscription ratio for the initial deployment.

9.3.4.

Nexus 2248TP to Nexus 5596UP The connectivity between the N2248TP and the N5596UP will be provided by 4 x 10GE uplinks, resuting ia a 1.2:1 oversubscription ratio. Accenture has determined that this is an acceptable oversubscription ratio for this environment.

9.3.5.

Nexus 5596UP to Catalyst 6500 (Existing Distribution Layer) The following devices are planned for the initial deployment of Nexus 5000 and 2000: 2 x Nexus 5596UP 4 x Nexus 2232PP 2 x Nexus 2248TP Each Nexus 5596UP will have two links to each Nexus 2232PP and two links to each Nexus 2248TP. In summary each Nexus 5596UP will have (4 x 2 + 2 x 2) x 10 G = 12 x 10 G = 120 G towards Nexus 2000s. Combining both Nexus 5596UP, we have 2 x 10G links connected to existing distribution layer of Catalyst 6509, hence 10 G from each Nexus 5596UP. Therefore a 12:1 oversubscription ratio exists between Access and Distribution layer for the traffic coming from inside the same DC site. Accenture determined that this is an acceptable oversubscription ratio in this environment for the initial deployment. The capacity to the Distribution/Core layer will be increased as part of the expected Nexus 7000 deployment project.

Note: - Cisco AS recommends reviewing the current design once the expected Nexus7000 deployment project will start. The following aspects should be considered: Increased Oversubscription on the ISLs between Nexus5000 and the existing Cat6500 Distribution switches due to the traffic coming from the remote DC site that needs to be routed or sent to the Firewall appliances. This could lead to congestion on the ISLs between the Nexus5500 pair and the Cat6500. The Nexus5596 switches will be replacing a pair of existing Cat6500 at the L2 Aggregation layer. Since the Nexus5596 have shallower port buffers compared to the •

•




Access Layer Physical Design Cat6500 switches, it is recommended to monitor for packet drops that could occur in case of bursty traffic.

9.4.

FEX 2000 Connectivity In the initial implementation, each pair of FEX will be connected in StraightThrough configuration to the Nexus5596UP and port-channel pinning will be used. Figure 5 shows FEXes connectivity in a straight-through configuration. Figure 5: FEX in Straight-Through Configuration

Nexus5596UP-1

Nexus5596UP-2

port-channel

port-channel

FEX 101

FEX 100 NEXUS2000

NEXUS2000

FEX Straight-Through

9.4.6.

Management of FEX 2000 The Cisco Nexus 2000 Series Fabric Extender is managed by its parent switch, over the fabric interfaces, through a zero-touch configuration model. The Fabric Extender is discovered by the switch by detecting the fabric interfaces of the Fabric Extender, which are connected to the 10GE ports of the Nexus 5000. After discovery, if the Fabric Extender has been correctly associated with the parent switch, the following steps are performed: The switch checks the software image compatibility and upgrades the Fabric Extender, if necessary. The switch and Fabric Extender establish in-band IP connectivity with each other. The switch assigns the Fabric Extender an IP address in the range of loopback addresses (127.0.0.0/8), to avoid conflicts with IP addresses that may be in use on the network. The switch pushes the configuration data to the Fabric Extender. The Fabric Extender does not store any configuration locally. •

•

•





•

The Fabric Extender updates the switch with its operational status. All Fabric Extender information is displayed using the switch commands for monitoring and troubleshooting.

To provide load balancing between the host interfaces and the parent switch, the Fabric Extender will use EtherChannel fabric interface connection. This connection will bundle 2 x 10GE fabric interfaces into a single logical channel.

9.4.7.

Etherchannel Fabric Interface Connection When configured to use an EtherChannel fabric interface connection to its parent switch, the switch will load balance the traffic from the hosts that are connected to the host interface ports by using the following load-balancing criteria to select the link, For a Layer 2 frame, the switch uses the source and destination MAC addresses. For a Layer 3 frame, the switch uses the source and destination MAC addresses and the source and destination IP addresses. •

•

A fabric interface that fails in the EtherChannel will not trigger a change to the host interfaces. Traffic is automatically redistributed across the remaining links in the EtherChannel fabric interface.

! Note

9.4.8.

A The Fabric Extender provides end-host connectivity into the network fabric. As a result, Bridge Protocol Data Unit (BPDU) Guard is enabled on all its host interfaces. If a bridge or switch is connected to a host interface, that interface is placed in an error-disabled state when a BPDU is received. BPDU Guard cannot be disabled on host interfaces of the Fabric Extender.

FEX in Straight-Through Configuration The following configuration illustrates how to connect a FEX to a Nexus 5000 switch in a straight-through configuration

Example 1: Nexus 2000 Association to Nexus 5000 in Straight-Through configuration

fex 100 pinning max-links 1 description "Nx2248-TP-1" type N2248T interface Ethernet1/6 description ** Fabric To Fex100 ** switchport mode fex-fabric fex associate 100 channel-group 100 interface Ethernet1/7 description ** Fabric To Fex100 **




Access Layer Physical Design switchport mode fex-fabric fex associate 100 channel-group 100 interface port-channel1100 description ** PortChannel To Fex100 ** switchport mode fex-fabric fex associate 100

9.4.9.

FEX Port Numbering Convention The port numbering convention used for the Fabric Extender is: Interface ethernet chassis/slot/port , where: Chassis is configured by the administrator when assigning the fex. Slot identifies the slot number on the Fabric Extender. Port identifies the port number on a specific slot and chassis ID. • • •

A Fabric Extender must be directly connected to its parent switch through the individual fabric interfaces or an EtherChannel fabric interface. Configure a chassis ID on a physical Ethernet interface or EtherChannel on the switch to identify the Fabric Extender discovered through those interfaces. Valid Fabric Extender Chassis ID ranges are from 100 to 199. This chassis ID is required only to access a host interface on the Fabric Extender. A value of less than 100 indicates a slot on the parent switch. The table below shows the proposed chassis ID scheme for the initial implementation phase Table 3: FEX Chassis ID scheme

!"# %&'(()( *+ !"" !"! !"$ !"/ !"% !"1 !/" !/! !/$ !// !/% !/1

+,(-.)/0)12 #$$%&'()'*'! #$$%&'()'*'$ #$$/$'))'! #$$/$'))'$ #$$/$'))'/ #$$/$'))'% #$$%&'()'*'/ #$$%&'()'*'% #$$/$'))'1 #$$/$'))'4 #$$/$'))'5 #$$/$'))'&

31-'0)12 +,*-./$0 +,*-./$0 +,*-./$0 +,*-./$0 +,*-./$0 +,*-./$0 +,2-3&0 +,2-3&0 +,2-3&0 +,2-3&0 +,2-3&0 +,2-3&0





9.5.

Host Connectivity VELUX has a requirement to providing connectivity to dual attached host is activeactive configuration. In order to provide high availability, the server will be connected with vPC to a pair of Nexus2000 Figure 6 shows the proposed server connectivity model Figure 6: Server Connectivity Model

Nexus 5500

Nexus2000 eth100/1/1

Nexus 5500

Straight-Through Fabric Extenders

Nexus2000

/ 1 / 1 1 0 1 h t e

vPC

Dual-Attached Host Active-Active

The following examples provides sample configurations required to configure a host connecting to a FEX, Example 2: Interface Configuration for single-attached Host Ports on FEX

! interface Ethernet101/1/3 Description switchport mode access switchport access vlan speed 1000 duplex full no shutdwon





Example 3: Interface Configuration for Dual Attached Active-Active Host Ports on FEX

nexus5500-1 configuration:

interface Ethernet100/1/1 switchport mode trunk switchport trunk allowed vlan channel-group 12 interface port-channel12 switchport mode trunk switchport trunk allowed vlan spanning-tree port type edge trunk spanning-tree bpdufilter enable nexus5500-2 configuration :

interface Ethernet101/1/1 switchport mode trunk switchport trunk allowed vlan channel-group 12 interface port-channel12 switchport mode trunk switchport trunk allowed vlan spanning-tree port type edge trunk spanning-tree bpdufilter enable

9.5.10.

Connectivity to HP Servers

VELUX is planning to connect a wide variety of HP servers to the new access layer infrastructure. The following sections provide recommendations for connectivity of HP blade centers.

9.5.10.1. HP NIC Teaming Options

The following options are normally available on HP Servers for NIC Teaming Automatic: this option detects what is configured upstream and configures the teaming option accordingly 803.3ad dynamic: this option requires LACP support and vPC since the NICs are normally split across FEXes Switch Assisted Load Balancing : All members of the SLB Team transmit and receive frames with the same MAC address, in a static port-channel fashion. This option requires vPC only Transmit Load Balancing . In this configuration, 1 port is RX while all ports in the team are TX. TLB provides increased bandwidth in the TX direction and does not requires specific configuration on the switch side Network Fault Tolerant: Active/Standby configuration. •

•

•

•

•

9.5.10.2. HP Blade Servers with 3Com Blade Switches

Since only end-host port are supported on the Nexus2000, it is recommended to connect the blade switches directly on the Nexus5596UP




Access Layer Physical Design 9.5.10.3. HP Virtual Connect

In the case of HP c7000 with the Virtual Connect modules, it is recommended to connect the 10G uplinks for the VC directly to the Nexus5596UP in order to avoid additional oversubscription introduced by the Nexus2000. Figure 7 Connectivity to HP Virtual Connect

N5596UP-1

vPC

N5596UP-2

vPC

HP c7000 with Flex10 Modules

9.5.10.4. Nexus B22HP

The Cisco Nexus B22 Blade Fabric Extender for HP (Cisco Nexus B22HP) provides an extension of the Cisco Nexus switch fabric to the HP server edge. Logically, it behaves like a remote line card to a parent Cisco Nexus 5000 Series Switch. The fabric extender and the parent Cisco Nexus 5000 Series Switch together form a distributed modular system. The Cisco Nexus B22HP forwards all traffic to the parent Cisco Nexus 5000 Series Switch over eight 10 Gigabit Ethernet uplinks. Low-cost uplink connections of up to 10 meters can be made with copper Twinax cable, and longer connections of up to 100 meters can use the Cisco Fabric Extender Transceiver (FET-10G). Standard 10-Gbps optics, such as short reach (SR) and long reach (LR), are also supported. Downlinks to each server are auto negotiating for 1 and 10 Gigabit Ethernet and work with all HP Ethernet and converged network adapter (CNA) mezzanines, allowing customers a choice of Ethernet, Fibre Channel over Ethernet (FCoE), or Small Computer System Interface over IP (iSCSI) connections. Because the Cisco Nexus B22 is a transparent extension of a Cisco Nexus 5000 Series Switch, traffic can be switched according to policies established by the Cisco Nexus 5000 Series Switch with a single point of management. The diagram below shows the proposed connectivity model for the B32HP to the Nexus5000:




Access Layer Physical Design Figure 8: B32HP Connectivity to the Nexus5000

N5596UP-1

vPC

N5596UP-2

vPC

HP c7000 with B32HP Modules

9.6.

Port Mapping Details In order to achieve a level of standardization during the access layer deployment, the following port allocation scheme is proposed on the Nexus5596UP Port Eth1/1: ISL connectivity towards the Distribution Layer Port Eth1/2-3: vPC peer Link Port Eth1/4-5: DCI links towards the remote DC site Port Eth1/6-48: connectivity towards the FEX and blade switches Figure 9 shows the proposed port allocation scheme for the initial deployment • • • •




Access Layer Physical Design Figure 9: N5596UP port allocation scheme

Expansion Module

Expansion Module

Expansion Module

Eth1/1: ISL to Aggregation Eth1/2-3: vPC Peer-Link Eth1/4-5 DCI Links to Remote DC Eth1/6-48 : FEX Fabric Links and Blade Switches

9.7.

Power Connectivity The Nexus5596UP use two 660 Watts power supplies and takes 100-240 volt input. The Power Supplies should be connected to at least two different power distribution units (Grids) that utilize two different power sources for redundancy.





10.


10.1. Layer 2 Features VELUX Data Center will leverage VPC feature in Nexus platform for Layer 2 forwarding in the Access-Distribution Layers. The Spanning-Tree protocol (STP) is still required and highly recommended to be configured in conjunction with vPC in order to prevent L2 loops.

10.1.1.

VLAN Trunks Trunks extend VLANs between devices by temporarily identifying and marking the original Ethernet frames. This enables the frames to be multiplexed over a single link. This also ensures that separate VLAN broadcast and security domains are maintained between switches. Content Addressable Memory (CAM) tables maintain the frame to VLAN mapping inside the switches. In VELUX Data Center network, VLAN’s will be trunked using 802.1Q on all the inter switch links.

! Note

There is a potential security consideration with 802.1Q caused by the implicit tagging of the Native VLAN, as it may be possible to send frames from one VLAN to another without a router. The workaround is to use a VLAN ID for the trunk’s native VLAN that is not used for end-user access . VELUX will use VLAN 888 as the standard Native VLAN. The following VLANs will be allowed on all trunks : VLAN 5-6 VLAN 10 VLAN 20, 22,23 VLAN 30, 35,36 VLAN 41-43, 47 VLAN 115 VLAN 120 VLAN 131,133-135, 137,140 VLAN 208-211 VLAN 250,251,253,254 VLAN 320 VLAN 350,351 VLAN 1137 VLAN 1804, 1808-1810 • • • • • • • • • • • • • •





10.1.2.

VLAN Pruning Unnecessary VLANs should be manually pruned from all trunk switches where not required. As mentioned before, the native VLAN on the trunks should be an obscure VLAN ID that is not used elsewhere in the network, Example 4: Vlan Pruning Configuration

! interface type switchport trunk allowed vlan < vlan_list> !

10.1.3.

VLAN Trunking Protocol (VTP) VTP is designed to allow users to make VLAN configuration changes centrally on one or more switches, and have those changes automatically be communicated to all the other switches in the (network) VTP domain. A VTP domain, also called a VLAN management domain, is made up of one or more interconnected switches via trunk that share the same VTP domain name. A switch can be configured to be in only one VTP domain. An IOS switch can be configured to operate in any one of three VTP modes: •

•

•

Server - In VTP server mode, you can create, modify, delete VLANs and specify other configuration parameters (such as VTP version and VTP pruning) for the entire VTP domain. VTP servers advertise their VLAN configuration to other switches in the same VTP domain and synchronize their VLAN configuration with other switches based on advertisements received over trunk links. VTP server is the default mode for Catalyst switches. Client - VTP clients behave the same way as VTP servers, but you cannot create, change, or delete VLANs on a VTP client. Moreover, the client does not remember the VLAN after a reboot (no VLAN information is written in NVRAM). Transparent - VTP transparent switches do not participate in VTP. A VTP transparent switch does not advertise its VLAN configuration and does not synchronize its VLAN configuration based on received advertisements. However, in VTP version 2, transparent switches do forward VTP advertisements that they receive out their trunk interfaces.

The benefits of using transparent mode over Client/Server mode include, •

It encourages good change control practice i.e. as a requirement to change a VLAN on a switch or trunk port has to be considered one switch at a time.





•

•

•

•

It limits the risk of administrator error such as deleting a VLAN accidentally and impacting the entire domain. There is no risk of a new switch being introduced to the network with a higher VTP revision number and overwriting the entire domain’s VLAN configuration. The extended VLAN range in numbers 1025-4094, can only be configured in this way Unnecessary STP instances are removed, which reduces CPU utilization and thereby improving overall scalability

At the time of this writing , VELUX/Accenture has taken the decision not to use VTP and to manually prune VLANs. Example 5: VTP Configuration

! Switch(config)#VTP domain Switch(config)#VTP mode transparent Switch(config)#VTP password !

10.1.4.

Spanning-Tree Protocol (STP) The Spanning-Tree Protocol, as originally specified in the IEEE 802.1D standard, typically recovers a link failure within 50 seconds [convergence time = (2 x Forward_Delay) + Max_Age]. While such outage may be adequate for data applications, currently many mission-critical applications require faster network convergence. To speed up network convergence and address scalability limitations related to Spanning Tree and Vlan interaction, the IEEE committee developed two new standards, Rapid Spanning-Tree Protocol (RSTP) defined in IEEE 802.1w and Multiple Spanning-Tree Protocol (MST) defined in IEEE 802.1s. As discussed above Spanning Tree Protocol will be configured on the Nexus5596UP switches as a protective mechanism for preventing Layer 2 loops. Rapid PVST+ is selected as the STP protocol in VELUX Data Center.

! Note

10.1.5.

Rapid PVST+ is enabled by default on Nexus switches.

Layer 2 Feature Placement This section explains the Cisco AS recommendations and best practices on optimizing a traditional Data Centre design from a Layer 2 perspective.

10.1.5.1. Portfast and BPDU Guard

Because spanning-tree categorizes ports into edge and non-edge, which is based on the duplex information as well as the assignment of PortFast to a port, it is




Access Layer Logical Design important to configure the PortFast feature on all eligible ports. PortFast transitions the port directly into forwarding after linkup rather than going through the spanning tree transition states, which delays link bring up by forward delay time (15 seconds, by default) at each transition. When using 802.1w it is extremely important to assign PortFast to all the eligible ports. There are two main advantages in doing this. First, if flapping occurs on edge ports, 802.1w does not generate a topology change notification. Secondly, an edge port does not change its forwarding state when there is a topology recalculation thus making the network more stable. Sometimes topology changes happen as the consequence of the attachment of an incorrectly configured access switch to a port that should be used by a server. You can use BPDU Guard with spanning tree PortFast to limit the chance for such an occurrence. The PortFast BPDU-Guard feature provides a method for preventing loops by moving a nontrunking port into an ErrDisable state when a BPDU is received on the port. A BPDU packet should never be received on an access-port configured for PortFast, as designed host ports should not be attached to switches. Finally, if a port is placed in ErrDisable state, by default it will remain down. Ports that are error-disabled due to PortFast BPDU Guard symptoms must be manually enabled again. It is worth noting that there is a command that will re-enable ports after a time-out interval (300s by default) if desired. " Recommendation

PortFast and BPDU Guard should be enabled on all the edge ports in the Access Layer switches.

10.1.5.2. Loop Guard

Loop-guard protects Layer-2 networks from loops caused (for example) by malfunctioning NICs /ASICs/Drivers, busy CPU’s, unidirectional links, or any other reason that may prevent the normal forwarding of BPDUs. Loop-guard operates with Spanning-Tree to prevent an Alternate Port or a Root Port from assuming a Designated Role if this is because of the absence of BPDUs. When Loop-guard doesn’t receive BPDUs on a Root Port or a Blocking Port it will put / keep the port into a Blocking State and mark the port as Loopinconsistent. Loop-guard should never be enabled on shared links: Loop-guard could isolate local hosts directly attached to the shared medium. An Alternate Port is a blocked port that provides an alternate path to the root. A backup port is a blocked port because connected in a loopback fashion. Example 6: Loop Guard Configuration

! Switch(config)#spanning-tree guard loop !





Loop-guard shall be configured on: Access-Switch Uplink ports STP Primary Root -> Secondary Root inter-Distribution Switch Link ports. • •

10.1.5.3. Bridge Assurance

With bridge assurance, a port can be now be classified in two categories: Edge ports—ports connected to end stations (existing portfast) Network ports—ports connected point to point to another bridge • •

This feature is applicable on NX-OS. Cisco NX-OS improves the stability of the network by protecting STP from bridging loops. Bridge assurance works in conjunction with Rapid-PVST BPDUs, and is enabled globally by default in NXOS. When a Bridge Assurance-enabled network port does not receive any BPDUs for a specified period that interface moves into the blocking state. After the network port receives a BPDU again, the port begins its normal spanning tree transitions. Bridge assurance works in conjunction with the spanning-tree port type command. The default port type for all ports in the switch is “normal” for backward compatibility with devices that do not yet support bridge assurance; therefore, even though bridge assurance is enabled globally, it is not active by default on these ports. The port must be configured to a spanning tree port type of “network” for bridge assurance to function on that port. Both ends of a pointto-point Rapid-PVST connection must have the switches enabled for bridge assurance, and have the connecting ports set to type “network” for bridge assurance to function properly. Bridge assurance is enabled by default on the vPC peer-link

! Note

Bridge assurance should also be enabled on the vPC member ports and on the L2 port-channel of the neighbour switch (access-layer or services chassis). But due to a current incompatibility (defect) between the Bridge Assurance detection mechanism and the vPC port management infrastructure, it is recommended not to put the vPC member port and the L2 port-channel on the neighbour switch into the “network” type.

10.1.6.

UDLD

The Unidirectional Link Detection (UDLD) protocol is a light-weight L2 protocol that detects and disables one-way connections before they create undesired




Access Layer Logical Design situation such as Spanning-Tree loops. It runs on physical interfaces only, but not on logical interface like portchannels. A one-way connection occurs whenever traffic transmitted by the local device over a link is received by the neighbor but traffic transmitted from the neighbor is not received by the local device or vice versa. This partial communication between peer’s is usually caused by misconnected fiber strands or hardware malfunctions and can create Spanning-Tree loops. UDLD is complementary feature to SpanningTree protocol and has to be enabled and supported on both end of the link. Once UDLD feature is enabled, it will be running on all enabled Ethernet interfaces on the Nexus 5K Series switches, including interface populated with Twinax transceivers. UDLD is running as a conditional feature, it needs to be enabled manually. UDLD has 2 modes of operations: •

•

Normal (default): In normal detection mode, UDLD detects the link error by examining the incoming UDLD packets from the peer port. UDLD will then errdisable the port. These error conditions are: Empty Echo packet, Uni-direction, TX/RX loop, and Neighbor Mismatch. If there is no packet coming in from the peer port, UDLD will NOT take any action. Aggressive mode: Same error detection as in normal mode with the additional cases: (a) One side of a link has a port stuck (both TX and RX) and (b) one side of a link remains up while the other side of the link has gone down. With UDLD aggressive mode enabled, after all the neighbors of a port have aged out, either in the advertisement or in the detection phase, UDLD aggressive mode restarts the linkup sequence in an effort to resynchronize with any potentially out-of-sync neighbors. If after a fast train of messages (eight failed retries) the link is still deemed undetermined, the port is put into the errdisable state. (With normal mode UDLD, the port is NOT shut down in this case).

! Note

Aggressive UDLD may lead to false positive events. The risk of aggressive UDLD mode is when the remote end stops responding for some reason that is NOT a unidirectional link, such as high CPU conditions. The Nexus 5K Series switch within the Velux 10Gbit/s Nexus Access Project acts as a pure L2 aggregation switch. All links are point to point L2 links between switches. These links will benefits from the UDLD protection in case of physical media failures. Hence we will enable the feature UDLD. Additionally we will enable UDLD explicitly on each Ethernet interface, even this task is not




Access Layer Logical Design necessary. The reason behind this task is to easier identify the interfaces which have UDLD enabled and to distinguish between the FEX fabric links. The diagram below illustrates the UDLD deployment.

! Note

The UDLD is not supported and not required for the FEX fabric link. There is a bidirectional keepalive message mechanism running between FEX and Nexus 5K Series switches that provide similar function as UDLD already.

Example 7: UDLD Configuration

! feature udld ! interface ethernet slot#/port# udld enable !

10.1.7.

Error Disable Recovery A cause (bpduguard, dtp-flap, gbic-invalid, l2ptguard, link-flap, pagp-flap, psecure-violation, udld) is defined as the reason why the error-disabled state occurred. When a cause is detected on an interface, the interface is placed in error-disabled state, an operational state similar to link-down state. If you do not enable errdisable recovery for the cause, the interface stays in error-disabled state until you enter a shutdown and no shutdown interface configuration command. If you enable the recovery for a cause, the interface is brought out of the error-disabled state and allowed to retry the operation again when all the causes have timed out. Example 8: Error Disable Recovery

! Switch(config)# errdisable recovery cause udld Switch(config)# errdisable recovery cause bpduguard Switch(config)# errdisable recovery interval 900 !

" Recommendation

Cisco does not recommend errdisable recovery at the core (DC Backbone) of the network because there are typically multiple entry points into a core, and automatic recovery in the core can lead to recurring problems. Therefore, you must manually re-enable a port at the core if UDLD disables the port.





10.1.8.

Interface Speed/Duplex The default value on Nexus Gigabit interfaces is auto-negotiation, which can be left unchanged. End users, mobile workers, and transient hosts need autonegotiation to minimize management of these hosts. Auto-negotiation is supported on Nexus switches, although the latest NIC drivers are often required on the end-hosts. In general, speed and duplex for server ports are left un-configured (autonegotiation). For certain server ports, speed and duplex can be hard coded, if there are issues with auto-negotiation.

10.1.9.

Port Channels This section describes the design considerations for non-vPC port channels if desired and configured inside VELUX Data Center networks. Ether-Channel technologies allow the inverse multiplexing of up to eight links into a single logical channel. There are two protocols to control the channel. Port Aggregation Protocol (PagP) is Cisco proprietary and Link Aggregation Control Protocol (LACP), which is an IEEE standard (802.3ad) that will provide vendor interoperability. Load-Balancing can be performed by one of the following criteria: Destination IP address and L4 port •

•

Destination IP address, L4 port and VLAN

•

Destination IP address and VLAN

•

Destination MAC address

•

Destination L4 port

•

Source & Destination IP address and L4 port

•

Source & Destination IP address, L4 port and VLAN

•

Source & Destination IP address and VLAN

•

Source & Destination MAC address

•

Source & Destination L4 port

•

Source IP address and L4 port

•

Source IP address, L4 port and VLAN

•

Source IP address and VLAN

•

Source MAC address

•

Source L4 port




Access Layer Logical Design By default layer-2 interfaces use Source & Destination MAC Address information for load-balancing. If a segment within an Ether-Channel fails, traffic previously carried over the failed link switches to the remaining segments within the Ether-Channel.

" ecommendation

LACP mode active is considered the recommended configuration for both ends of the link. It is also also recommended to configure the command “lacp suspendindividual” respective “port-channel standalone-disable” under the portchannel interface on network point to point links

10.2. VPC Design 10.2.10. Virtual Port Channel (vPC) Overview Virtual Port-Channel (vPC) technology on the Nexus platforms allows a single device to use a port channel across two upstream switches, similar to MEC (Multi Chassis Etherchannel) in Cat6K VSS environment. While a pair of switches acting as a vPC peer endpoint look like a single logical entity to portchannel attached devices, the two devices that act as the logical port-channel endpoint are two separate devices. This environment has the benefits of hardware redundancy combined with the benefits of port-channel loop management. Figure below shows a network topology with and without vPC. While vPC does potentially remove the need to run Spanning Tree Protocol within a directly vPC attached network, STP is still a requirement, to deal with loop detection outside of the direct attached environment of the vPC peers, and in order to deal with potential loops in the case of a client or peer misconfiguration or mis-cabling. Figure 10: vPC Overview





OOB Mgmt Switch

vPC Peer Keepalive carried over the OOB management network int mgmt 0

Primary N5K

Secondary N5K vPC Peer Link vPC Peer-Switch

vPC Domain

vPC

vPC Member Port

The components of a vPC system are as follows, vPC peer – a vPC switch (primary or secondary) •

•

•

•

•

•

•

vPC Domain – This is the common domain configured across two vPC peer devices. vPC member port – one of a set of ports (port channels) that form a vPC vPC – the combined port channel between the vPC peers and the downstream device vPC peer-link – Link used to synchronize state between vPC peer devices and carry traffic from one system to the other when needed vPC peer-keepalive link – the peer-keepalive link between vPC peer devices CFS – Cisco Fabric Services protocol, used for state synchronization and configuration validation between vPC peer devices

vPC has the following advantages, It eliminates STP blocked ports •

•

•

•

It leverages all available uplink bandwidth It enables dual-homed servers operate in active-active mode, if servers are directly uplinked to Nexus switches It provides fast convergence upon link/device failure

10.2.10.4. Dual Control Plane but Single Layer 2 Node Behavior

While still operating with two separate control planes, vPC ensures that the neighboring devices perceive the vPC peers as a single spanning-tree and LACP entity. For this to happen the system has to perform IEEE 802.3ad control plane operations in a slightly modified way (which is not noticeable to the neighbor switch).




Access Layer Logical Design 10.2.10.5. vPC and LACP Interaction

IEEE 802.3ad specifies the standard implementation of port channeling. Port channeling specifications provide a standard protocol, known as the Link Aggregation Control Protocol (LACP), which makes it possible to negotiate port bundling. In order to allow Link Aggregation Control to determine whether a set of links connect to the same System, and to determine whether those links are compatible from the point of view of aggregation, it is necessary to be able to establish: A globally unique identifier for each System that participates in Link Aggregation (i.e. the switch itself needs to be “unique”) A means of identifying a Link Aggregation Group The way that a switch decides which ports can bundle together is based on the Link Aggregation Group Identifier (LAG-ID). This number includes the system identifier (ID for the physical switch), a key that identifies the aggregation group itself (i.e. the equivalent of the channel-group number). vPC makes the system-id look the same so that the downstream switch believes it is connected to a single upstream device. •

•

10.2.11. vPC and Spanning Tree Interaction vPC modifies in two ways how spanning tree works on the switch: It makes sure that the peer link is always forwarding, in fact even if the switch has a direct path to the root, the secondary vPC peer always sees the peer-link as the Root Port towards the primary vPC device. It ensures that only the primary switch forwards BPDUs on vPCs so that the other switches connected to the vPC system perceive the two peers as a single entity from a spanning tree perspective. This modification is strictly limited to vPC-member ports. As a result the BPDUs that may be received by the secondary vPC peer on a vPC port are forwarded to the primary vPC peer via the peer link for processing. •

•

! Note

Non-vPC ports operate like regular ports. The special behavior of the primary vPC member applies uniquely for ports that are part of a vPC. The best practice for vPC is to configure the network as one would a standard STP network, and then enable vPC on the particular member ports as necessary. This includes common configurations such as edge port configuration. One area where vPC currently requires a non-standard STP configuration is specifically relative to Bridge Assurance on vPC member ports. Due to a current incompatibility between the Bridge Assurance detection mechanism and the vPC port management infrastructure, it is recommended that you not put the vPC member port into “network” type, disabling Bridge Assurance on those ports.





10.2.12. vPC Design Considerations To begin with the following configuration is needed to enable the vPC feature within the VDC, Example 9: Enabling vPC feature configuration on Nexus 5000

(config)# feature vpc

10.2.12.6. vPC Role and Priority

A domain needs to be defined and priorities are configured to define primary and secondary roles in the vPC configuration. The lower number has higher priority. It should also be noted that the role is non-preemptive, so a device may be operationally primary but secondary from a configuration perspective. Because spanning tree is preemptive, this may result in a mismatch between the spanning tree root and the vpc operational primary but that has no consequences for traffic forwarding. Example 10: vPC role and priority configuration on Nexus 5000

Vpc domain 1 Role priority 100

10.2.12.7. vPC Domain ID

When configuring the vpc domain-id you should make sure it is different from the one used by a neighboring vpc-capable device with which you plan to configure double-sided vPC. This is because the LAGID is derived by the system-id, which in turn is derived from the MAC address identifier of the switch, and for vPC this MAC address is in turn derived from the domain-id. As a result, in a back-to-back vPC configuration, if the neighboring switches use the same domain-id, there’s a risk of conflicting LAGID in the LACP negotiation that could lead to an unsuccessful LACP negotiation. 10.2.12.8. vPC Peer Keep-alive Link

vPC peer-keepalive link is another key in the vPC loop management chain. This link provides a The peer keepalive is an out-of-band monitoring mechanism that is used for vPC peers to arbitrate roles and to resolve peer-link failures. Peer-keepalive connectivity can be configured either through the mgmt0 interface or a switch virtual interface (SVI) and a separate port.




Access Layer Logical Design For example, each Cisco Nexus 5500 the mgmt0 interface may already be connected to the management network, in which case the vPC configuration can simply use the existing connectivity for peer-keepalive purposes. Routing the peer keepalive over mgmt0 is the preferred option since it has the advantage that the mgmt0 virtual route forwarding (VRF) instance is routed independently of the default VRF instance, and that this configuration is compatible with ISSU. Routing the peer keepalive from an SVI over a regular front-panel port provides the advantage that the Nexus 5000 Switches can be connected back to back ; it also provides additional verification of the health of the ASICs through which the peer keepalive flows. It provides the disadvantage that the peer keepalive is routed according to the information contained in the default VRF instance (which may cause the traffic to use the peer link instead of a different path); in addition, ISSU is not compatible with this configuration. The following configuration illustrates the use of the mgmt0 interface Example 11: vPC peer keepalive link configuration using the mgmt0

vpc domain 2 peer-keepalive destination 10.128.50.102 source 10.128.50.101 vrf-management

The following configuration illustrates the use an SVI interface Example 12: vPC peer keepalive link configuration using SVI interface

! Port connecting Nexus5k01 to Nexus 5k02 directly interface Ethernet1/48 description direct-peer-keepalive switchport access spanning-tree port type edge ! Clearing the peer-link from the VLAN that is used for the peerkeepalive interface Po30 switchport trunk allowed vlan remove 10 ! Defining the SVI that is used by the peer keepalive, in this case this SVI ! is defined within the VRF default interface Vlan10 no ip redirects ip address 10.128.50.200/24 no shutdown ! configuration of the peer-keepalive information with the associated vrf vpc domain 2 peer-keepalive destination 10.128.50.201 source 10.128.50.200 vrf default

At the time of this writing, Accenture has confirmed that the preferred option is to configure the vpc keep-alive link via the mgmt0 interface. The mgmt0 interfaces on the two vPC peers in the same location will be connected back-toback.




Access Layer Logical Design Both configuration have been kept in this document for reference. 10.2.12.9. vPC Peer Link

In order to provide loop free direct attached topologies, the virtual port-channel capability of the Nexus platform requires specific hardware and Layer 2 connectivity in order to provide the required loop management and detection capabilities. The peer-link is the key communications link between the two vPC peers. This link should be as redundant and robust as possible, in order to limit the likelihood of a single failure creating a dual active system scenario. The characteristics of the peer link are Standard 802.1Q Trunk Carries all vPC and non vPC VLANs Carries Cisco Fabric Services messages (with default CoS 6) Carries flooded traffic from first hop peer Carries STP BPDUs, HSRP Hellos, IGMP updates, etc. • • • • •

The recommendations for designing the peer-link is to deploy a minimum of 2 x10GE ports The vPC peer-link carries not only control plane traffic (which while of low volume and size is still critical to the proper functioning of the vPC pair), but also traffic to/from vPC member ports in the case where one vPC peer looses it’s access to a particular vPC. Example 13: vPC peer link configuration on Nexus 5500

nexus5500(config)# interface port-channel21 nexus5500(config-if)# vpc peer-link nexus5500(config-if)# switchport trunk allowed vlan 5,6,10,20,22,23,30,35,36,41-43,47,115,120,131,133-135,137-140,208211,250,251,253,254,320,350,351,1137,1804,1808-1810

10.2.12.10.

vPC Ports

Virtual Port channels are configured by bundling Layer 2 ports (switchports) on each Nexus switch via the command vpc as shown below. The system issues an error message if the port channel wasn’t previously configured as a switchport. Example 14: vPC port configuration on Nexus 5000

! vPC Primary interface ethernet2/9 channel-group 51 mode active interface Port-channel 51 switchport vpc 51





!vPC Secondary interface ethernet2/9 channel-group 51 mode active interface Port-channel 51 switchport vpc 51 !

After a port is defined as part of a vpc, any further configurations, such as activation or disablement of Bridge Assurance, trunking mode, etc, are performed under the interface port channel configuration mode, and trying to configure spanning tree properties for the physical interface instead of the port channel will result in an error message. If the Consistency check doesn’t show Success, it is recommended to verify the Consistency Parameters using the vPC consistency show commands mentioned below, • •

show vpc consistency-parameters global show vpc consistency-parameters int port-channel X

Typical reason for the vpc not to form include (i) the vLAN that is defined in the trunk doesn’t exist or it is not defined on the peer link, (ii) one member port is configured as access and the other as trunk, (iii) mismatches in the vlans that are carried on the trunk etc.

10.2.13. vPC Connectivity to the L2/3 Distribution Layer At the L2/L3 Distribution layer, each pair of Nexus5500 will connect with vPC with an existing Cat6509 as shown in the diagram below:




Access Layer Logical Design Figure 11: Connectivity to aggregation

L32-Y31-6509-1 Catalyst 6509 Existing L2/L3 Distribution

vPC21

L32-OC-N5596-01

L32-OC-N5596-01

Nexus 2000 Nexus 2000

The example below shows a sample configuration for the connectivity to the Cat6509 Example 15: vPC connectivity to the aggregation switches

------ nexus5500-1 configuration----interface Ethernet1/1 description *** to Catalyst 6509-1 *** switchport mode trunk switchport trunk native vlan 888 switchport trunk allowed vlan 5,6,10,20,22,23,30,35,36,41-43,47,115,120,131,133135,137-140,208-211,250,251,253,254,320,350,351,1137,1804,1808-1810

logging event port link-status udld enable channel-group 20 mode active ! no shutdown interface port-channel20 description * TO Cat6509-1* switchport mode trunk lacp suspend-individual switchport trunk native vlan 888 switchport trunk allowed vlan 5,6,10,20,22,23,30,35,36,4143,47,115,120,131,133-135,137-140,208211,250,251,253,254,320,350,351,1137,1804,1808-1810

spanning-tree port type normal vpc 20 ------ Cat6509-1 configuration----!interface Port-channel20 switchport switchport trunk encapsulation dot1q switchport trunk native vlan 888 switchport trunk allowed vlan 5,6,10,20,22,23,30,35,36,41-43,47,115,120,131,133135,137-140,208-211,250,251,253,254,320,350,351,1137,1804,1808-1810

switchport mode trunk switchport nonegotiate port-channel standalone-disable




Access Layer Logical Design interface TenGigabitEthernet3/3 switchport switchport trunk encapsulation dot1q switchport trunk native vlan 888 switchport trunk allowed vlan 5,6,10,20,22,23,30,35,36,41-43,47,115,120,131,133135,137-140,208-211,250,251,253,254,320,350,351,1137,1804,1808-1810

switchport mode trunk switchport nonegotiate udld port channel-protocol lacp !channel-group 20 mode active interface TenGigabitEthernet3/4 switchport switchport trunk encapsulation dot1q switchport trunk native vlan 888 switchport trunk allowed vlan 5,6,10,20,22,23,30,35,36,41-43,47,115,120,131,133135,137-140,208-211,250,251,253,254,320,350,351,1137,1804,1808-1810

switchport mode trunk switchport nonegotiate udld port channel-protocol lacp !channel-group 20 mode active

10.2.14. Velux DC Interconnect Implementation Velux has the need to extend the L2 connectivity between the DCE and DCW Datacenters. The two DC sites will be interconnected via vPC over the existing DWDM infrastructure, as shown in Figure 12. Figure 12

Velux Inter-DC Implementation

DCE(L32)

DCW(K8)

DWDM

vPC40

L32-OC-N5596-01

NX2000

L32-S6-N5596-01

NX2000

vPC40

K8-C4-N5596-01

NX2000


K8-C5-N5596-01

NX2000



Access Layer Logical Design This solution leverages the vPC functionality and it gets the following benefits: BPDU Filtering Removing the need of spanning tree being each couple of Nexus 5596UP virtualized a single switch (even if with doubled control plane) Avoiding any single point of failure: no single link or device failure will cause data center isolation Load balancing packet over the etherchannel links between the two DC: removing the spanning tree, no ports will be in blocked state Avoiding the possibility to have topology loop, behaving the two Nexus in each DC as a single virtualized device • •

•

•

•

Filtering BPDUs on the interconnect One of the main problems with bridging is that L2 addresses don’t carry any notion of location. Unlike IP for example, there is no way of determining the destination of a mac address from its “value”. As a result, it’s up to the network to make sure that frames are always forwarded along a tree (i.e. a topology with no loop). Bridges running STP build a tree topology this by exchanging BPDUs with their peers. As soon as BPDUs are filtered on a link, the link is hidden from STP and it will not be able to prevent a loop that would go through this link. Filtering STP off the interconnect could introduce a loop should a wrong cabling open a backdoor between the data centers. The following figure illustrates the scenario before and after the backdoor insertion.




Access Layer Logical Design Figure 13

DCI BPDU Filter – Backdoor issue

Considering that introducing a backdoor connecting remote data centers together is rather unlikely, the benefit of filtering the BPDUs across the DCI generally outweighs its drawback: STP scalability •

•

STP failure domain containment

•

BPDU filtering is native with most of multipoints solutions

Anyway it is suggested to implement the following hints in order to protect from that particular situations: disable any not used interface on the aggregation switches •

•

when adding any new switch in cascade to the access switch, implement the Root Guard STP extension towards the new switch

Example 16: Enable BPDU FIlterring

nexus5500(config)# interface ethernet 1/4 nexus5500(config-if)# spanning-tree bpdufilter enable





10.2.15. Proposed vPC Numbering Scheme The table below show the proposed vPC Numbering scheme. Figure 14: Proposed vPC numbering scheme

! "# $ & '( )* +

!"#$%&$'(()*$+, !"#$1*$'(()*$+, 20$&.$'(()*$+, 20$&($'(()*$+,

, #" )- .+ /0 01 25# &' () *+

, #" )-*+/001 2 3 4 6,. 3 4

#+ L32-Y31-6509-1 #+ L32-Y31-6509-1 "+ K8-C-5-6509-1 "+ K8-C-5-6509-1

7 22 #'18 9:7 ;

#+ #+ "+ "+

#+ #+ #, #,

(-*-,+-#+-##-#"-"+-"(-"*-.,$."-./-,,(-,#+-,",-,""$,"(-,"/$,.+-#+0$#,,-#(+-#(,-#("-#(.-"#+-"(+-"(,-,,"/-,0+.-,0+0$,0,+ (-*-,+-#+-##-#"-"+-"(-"*-.,$."-./-,,(-,#+-,",-,""$,"(-,"/$,.+-#+0$#,,-#(+-#(,-#("-#(.-"#+-"(+-"(,-,,"/-,0+.-,0+0$,0,+ (-*-,+-#+-##-#"-"+-"(-"*-.,$."-./-,,(-,#+-,",-,""$,"(-,"/$,.+-#+0$#,,-#(+-#(,-#("-#(.-"#+-"(+-"(,-,,"/-,0+.-,0+0$,0,+ (-*-,+-#+-##-#"-"+-"(-"*-.,$."-./-,,(-,#+-,",-,""$,"(-,"/$,.+-#+0$#,,-#(+-#(,-#("-#(.-"#+-"(+-"(,-,,"/-,0+.-,0+0$,0,+

!"#$%&$'(()*$+, !"#$1*$'(()*$+, !"#$%&$'(()*$+, !"#$1*$'(()*$+, !"#$%&$'(()*$+, !"#$1*$'(()*$+,

,++ ,+, ,+# ,+" ,+. ,+(

'##.0$34$5$, '##.0$34$5$# '##"#$44$, '##"#$44$# '##"#$44$" '##"#$44$.

,++ ,+, ,+# ,+" ,+. ,+(

,++ ,+, ,+# ,+" ,+. ,+(

(-*-,+-#+-##-#"-"+-"(-"*-.,$."-./-,,(-,#+-,",-,""$,"(-,"/$,.+-#+0$#,,-#(+-#(,-#("-#(.-"#+-"(+-"(,-,,"/-,0+.-,0+0$,0,+ (-*-,+-#+-##-#"-"+-"(-"*-.,$."-./-,,(-,#+-,",-,""$,"(-,"/$,.+-#+0$#,,-#(+-#(,-#("-#(.-"#+-"(+-"(,-,,"/-,0+.-,0+0$,0,+ (-*-,+-#+-##-#"-"+-"(-"*-.,$."-./-,,(-,#+-,",-,""$,"(-,"/$,.+-#+0$#,,-#(+-#(,-#("-#(.-"#+-"(+-"(,-,,"/-,0+.-,0+0$,0,+ (-*-,+-#+-##-#"-"+-"(-"*-.,$."-./-,,(-,#+-,",-,""$,"(-,"/$,.+-#+0$#,,-#(+-#(,-#("-#(.-"#+-"(+-"(,-,,"/-,0+.-,0+0$,0,+ (-*-,+-#+-##-#"-"+-"(-"*-.,$."-./-,,(-,#+-,",-,""$,"(-,"/$,.+-#+0$#,,-#(+-#(,-#("-#(.-"#+-"(+-"(,-,,"/-,0+.-,0+0$,0,+ (-*-,+-#+-##-#"-"+-"(-"*-.,$."-./-,,(-,#+-,",-,""$,"(-,"/$,.+-#+0$#,,-#(+-#(,-#("-#(.-"#+-"(+-"(,-,,"/-,0+.-,0+0$,0,+

20$&.$'(()*$+, 20$&($'(()*$+, 20$&.$'(()*$+, 20$&($'(()*$+, 20$&.$'(()*$+, 20$&($'(()*$+,

,"+ ,", ,"# ,"" ,". ,"(

'##.0$34$5$" '##.0$34$5$. '##"#$44$( '##"#$44$* '##"#$44$/ '##"#$44$0

,"+ ,", ,"# ,"" ,". ,"(

678 678 678 678 678 678

(-*-,+-#+-##-#"-"+-"(-"*-.,$."-./-,,(-,#+-,",-,""$,"(-,"/$,.+-#+0$#,,-#(+-#(,-#("-#(.-"#+-"(+-"(,-,,"/-,0+.-,0+0$,0,+ (-*-,+-#+-##-#"-"+-"(-"*-.,$."-./-,,(-,#+-,",-,""$,"(-,"/$,.+-#+0$#,,-#(+-#(,-#("-#(.-"#+-"(+-"(,-,,"/-,0+.-,0+0$,0,+ (-*-,+-#+-##-#"-"+-"(-"*-.,$."-./-,,(-,#+-,",-,""$,"(-,"/$,.+-#+0$#,,-#(+-#(,-#("-#(.-"#+-"(+-"(,-,,"/-,0+.-,0+0$,0,+ (-*-,+-#+-##-#"-"+-"(-"*-.,$."-./-,,(-,#+-,",-,""$,"(-,"/$,.+-#+0$#,,-#(+-#(,-#("-#(.-"#+-"(+-"(,-,,"/-,0+.-,0+0$,0,+ (-*-,+-#+-##-#"-"+-"(-"*-.,$."-./-,,(-,#+-,",-,""$,"(-,"/$,.+-#+0$#,,-#(+-#(,-#("-#(.-"#+-"(+-"(,-,,"/-,0+.-,0+0$,0,+ (-*-,+-#+-##-#"-"+-"(-"*-.,$."-./-,,(-,#+-,",-,""$,"(-,"/$,.+-#+0$#,,-#(+-#(,-#("-#(.-"#+-"(+-"(,-,,"/-,0+.-,0+0$,0,+

!"#$%&$'(()*$+, 20$&.$'(()*$+,

#, !"#$1*$'(()*$+, ", 20$&.$'(()*$+#

#, ",

678 (-*-,+-#+-##-#"-"+-"(-"*-.,$."-./-,,(-,#+-,",-,""$,"(-,"/$,.+-#+0$#,,-#(+-#(,-#("-#(.-"#+-"(+-"(,-,,"/-,0+.-,0+0$,0,+ 678 (-*-,+-#+-##-#"-"+-"(-"*-.,$."-./-,,(-,#+-,",-,""$,"(-,"/$,.+-#+0$#,,-#(+-#(,-#("-#(.-"#+-"(+-"(,-,,"/-,0+.-,0+0$,0,+

!"#$%&$'(()*$+, !"#$1*$'(()*$+,

.+ 20$&.$'(()*$+, .+ 20$&.$'(()*$+#

.+ .+

.+ (-*-,+-#+-##-#"-"+-"(-"*-.,$."-./-,,(-,#+-,",-,""$,"(-,"/$,.+-#+0$#,,-#(+-#(,-#("-#(.-"#+-"(+-"(,-,,"/-,0+.-,0+0$,0,+ .+ (-*-,+-#+-##-#"-"+-"(-"*-.,$."-./-,,(-,#+-,",-,""$,"(-,"/$,.+-#+0$#,,-#(+-#(,-#("-#(.-"#+-"(+-"(,-,,"/-,0+.-,0+0$,0,+




Network Management

11.

Network Management It is very important to have the right means in place in order to properly monitor the status and health of the network and be notified when a potential harm or event is happening. The management infrastructure should be in place to allow access to the managed devices even during network instability. To deliver all the objectives described above the following options can be implemented: •

•

•

SNMP/Syslog: Syslog messages from network devices are invaluable when troubleshooting network problems or security threats. In-band Management: Provides network management applications access to the production network for monitoring and managing the individual devices. This option is provided by the In-band management. Out-of-band Management (OOB): Provides the means to connect to the managed devices through the console/management ports.

VELUX will use the mgmt0 interface only for the vPC peer-keepalive, the management of the Nexus5000 will be In-band. Example 17: Configuring IP Address on Mgmt Interface on Nexus 5000

interface mgmt 0 ip address 10.128.50.101/24

VELUX will use VLAN 10 for In-band management. The VLAN network interface feature must be enabled before VLAN interface can be configured. ! feature interface-vlan Interface vlan 10 management Ip address 10.128.50.103/24 vrf context default ip route 0.0.0.0/0 10.128.50.1 !

Table 4 shows the IP addresses used for in-band and out-of-band management.




Network Management Table 4: Switch Management IP addresses

!"#$%&'()* 2;<=>?=4@@AB=65 2;<=!B=4@@AB=65 GD=?F=4@@AB=65 GD=?@=4@@AB=65

+,+- /0 1234 56 /0 56C5
0&78#%(9 2:%($#:' 2;<=E4*">?E 2;<=!D GD=?F GD=?@

11.1. Host Name Every device on the network needs to be identified using a hostname. Hostnames should follow the current VELUX device naming standard. The hostname will be used as default for the device’s prompt when logged into it and is used by most NMS systems as the device name (SNMP sysname). Example 18: Configuring Hostname

hostname

Table 5 shows the switch hostnames that will be used at VELUX. Table 5: Hostnames

!"#$%&'()* !"#$%&$'(()*$+, !"#$7*$'(()*$+, 89$&:$'(()*$+, 89$&($'(()*$+,

+*,%-#.$#/' '-./0(()*12 '-./0(()*12 '-./0(()*12 '-./0(()*12

0/%($#/' 3&45!"#6 3&45!"#6 3&;5896 3&;5896

11.2. Banner The configuration of the login banner provides a way to display a message to anyone who accesses the switch. Login banners can be useful to re-emphasize the potential law infringement represented by an unauthorized access to the device in question. It can also be used to provide information about the location of the device, the contact details of the administrator, or a message of the month. Example 19: Configuring Banner

banner motd ^C

”************************************ * %%HOSTNAME%% * * IP address: %%INLINE-MGMT-IP-ADDRESS%% * %%MODEL-NUMBER%%

*

*

****************** (c) Accenture *** ^C




Network Management

11.3. Network Time Protocol (NTP) Accurate and reliable time can be very useful for logging purposes, such as for forensic investigations in case of an outage or root cause analysis. The NTP configuration is typically using NTP authentication and NTP sourceinterface, additional to this, for redundancy purposes, multiple NTP servers is a best practice. Within NX-OS is possible to configure NTP logging for NTP events (disabled by default). Example 20: Configuring NTP

ntp server 10.128.50.105 prefer use-vrf default key [key-number] ntp source-interface vlan 10 ntp server 10.128.50.108 use-vrf default key [key-number] ntp source-interface vlan 10 ntp authenticate ntp authentication-key [key-number] md5 [pwd] ntp trusted-key [key-number] ntp logging logging level ntp 5

11.4. Time Zone All the devices within the Data Center will be configured with UTC timezone. Example 21: Configuring Timezone

clock timezone utc 0

11.5. Logging Network Components should be configured to send logging messages on the terminal to an appropriate log server using syslog. All messages should be sent to the log server regardless of severity, only critical messages should be displayed on a console. Example 22: Configuring Logging

! logging logging logging logging

source-interface server 10.45.3.39 7 use-vrf default server 10.45.133.139 7 use-vrf default server 10.45.133.140 7 use-vrf default

!




Network Management

11.6. DNS Network Devices will be configured to use accepted name servers in line with the current VELUX Domain Name Service (DNS). This protocol uses DNS servers to resolve assigned names to IP addresses. Example 23: Configuring DNS

! ip name-server 10.45.7.2 use-vrf default ip name-server 10.45.7.1 use-vrf default ip domain-lookup !

11.7. SNMP SNMP is an application-layer protocol that provides a message format for communication between managers and agents. The SNMP system consists of an SNMP manager, an SNMP agent, and a MIB. The SNMP manager can be part of a network management system (NMS) such as CiscoWorks/HP OpenView etc. The agent and MIB reside on the switch. To configure SNMP on the switch, you define the relationship between the manager and the agent. Trap operations allow an SNMP agent to send unsolicited notifications that an event has occurred. Traps, often termed “fire and forget”, are sent on a best effort basis using the User Datagram Protocol (“UDP”). All Cisco devices are capable of generating SNMP traps to notify NMS applications of network events. Cisco routers and switches implement the Cisco syslog MIB as a means of generating SNMP traps in place of, or in addition to, syslog messages. SNMPv3 provides secure access to devices by a combination of authenticating and encrypting frames over the network. The security features provided in SNMPv3 are as follows: Message integrity—Ensures that a packet has not been tampered with intransit. Authentication—Determines the message is from a valid source. Encryption—Scrambles the packet contents to prevent it from being seen by unauthorized sources. •

• •

Example 24: SNMP Configuration

! snmp-server contact DK Network Team snmp-server location DNKDE snmp-server community ro snmp-server community rw




Network Management snmp-server user Admin auth sha priv snmp-server host 10.45.133.139 informs version 3 auth snmp-server host 10.45.133.139 use-vrf default snmp-server source-interface traps vlan 10 snmp-server host 10.45.133.140 informs version 3 auth snmp-server host 10.45.133.140 use-vrf default snmp-server source-interface traps vlan 10

snmp enable traps snmp enable traps stpx snmp enable traps bridge !




12.

Security

12.1. General Device Security 12.1.1.

Enable Protection Services Cisco routers and switches support a number of services that can be enabled on a device to improve the overall security of a network. Before any of these services are enabled, they should be reviewed in the context of the network to ensure that they will not effect the current operation of the network.

12.1.2.

Disable DHCP service TCP Disabling the DHCP server or relay agent on devices will mitigate a potential for denial-of-service attacks such as the ‘blocked input-queue’ attack described in the recent Cisco Security Advisory: Cisco IOS DHCP Blocked Interface Denial-of-Service. Of course this service is needed on certain devices and cannot be disabled therefore in such cases the appropriate security measures should be taken to protect against this type of attack. The command to disable DHCP on a device is: Example 25: Disable DHCP Service on Nexus

no service dhcp

12.1.3.

Device Access Security Cisco recommends SSH to be used as the primary device access method. You can use the SSH server to enable an SSH client to make a secure, encrypted connection to a Cisco NX-OS device. SSH uses strong encryption for authentication. The SSH server in the Cisco NX-OS software can interoperate with publicly and commercially available SSH clients. By default, the Cisco NX-OS software generates an RSA key using 1024 bits. SSH supports the following public key formats: • • •

OpenSSH IETF Secure Shell (SECSH) Public Key Certificate in Privacy-Enhanced Mail (PEM)




! Note

SSH configuration and operation are local to the virtual device context (VDC). Example 26: SSH Configuration on Nexus

NX-OS(config)#no ssh server enable NX-OS(config)#ssh key {dsa [force] | rsa [bits [force]]} NX-OS(config)#ssh server enable NX-OS#show ssh key ************************************** rsa Keys generated:Fri Apr 10 20:13:21 2009




13.

Cisco NX-OS Software Recommendation Table 6: Cisco NX-OS Code Versions

Platform Nexus 5000

Code 5.1(3)N2(1c)

14.

Configuration Templates Nexus 5596UP Configuration templates ! feature fex feature tacacs feature lacp feature udld feauture vpc feature vlan-interface

!//-------Spanning-Tree Configuration----//! spanning-tree spanning-tree spanning-tree spanning-tree spanning-tree

bridge assurance pathcost method long mode rapid-pvst port type edge default port type edge bpduguard default

!//------- VLAN Configuration ---------//! vlan 5 name vlan 6 name vlan 10 name vlan 888 name NATIVE

!//----------- FEX Configuration -----------//! fex 100 pinning max-links 1 description "Nx2248-TP-1" type N2248T fex 102 pinning max-links 1 description "Nx2232PP" type N2232P fex 104 pinning max-links 1 description "Nx2232PP" type N2232P interface Ethernet1/6 description ** Fabric To Fex100 ** switchport mode fex-fabric fex associate 100 channel-group 100 interface Ethernet1/7 description ** Fabric To Fex100 ** VELUX Access Layer Low Level Design 55



switchport mode fex-fabric fex associate 100 channel-group 100 interface port-channel1100 description ** PortChannel To Fex100 ** switchport mode fex-fabric fex associate 100

interface Ethernet1/8 description ** Fabric To Fex102 ** switchport mode fex-fabric fex associate 102 channel-group 102 interface Ethernet1/9 description ** Fabric To Fex102 ** switchport mode fex-fabric fex associate 102 channel-group 102 interface port-channel1102 description ** PortChannel To Fex102 ** switchport mode fee-fabric fex associate 102 interface Ethernet1/10 description ** Fabric To Fex104 ** switchport mode fex-fabric fex associate 104 channel-group 104 interface Ethernet1/11 description ** Fabric To Fex104 ** switchport mode fen-fabric fex associate 104 channel-group 104 interface port-channel1104 description ** PortChannel To Fex104 ** switchport mode fex-fabric fex associate 104

!//-------------vPC Configuration ----------------------//! vpc domain 2 role priority 100 peer-keepalive destination 10.128.50.102 source 10.128.50.101 vrf-management auto-recovery

interface Ethernet1/2 description vPC peer-link member switchport mode trunk switchport trunk native vlan 888 switchport trunk allowed vlan 5,6,10,20,22,23,30,35,36,4143,47,115,120,131,133-135,137-140,208211,250,251,253,254,320,350,351,1137,1804,1808-1810 logging event port link-status udld enable channel-group 21 mode active VELUX Access Layer Low Level Design 56



no shutdown interface Ethernet1/3 description vPC peer-link member switchport mode trunk switchport trunk native vlan 888 switchport trunk allowed vlan 5,6,10,20,22,23,30,35,36,4143,47,115,120,131,133-135,137-140,208211,250,251,253,254,320,350,351,1137,1804,1808-1810 logging event port link-status udld enable channel-group 21 mode active no shutdown interface port-channel21 description description switchport mode trunk lacp suspend-individual vpc peer-link switchport trunk native vlan 888 switchport trunk allowed vlan 5,6,10,20,22,23,30,35,36,4143,47,115,120,131,133-135,137-140,208211,250,251,253,254,320,350,351,1137,1804,1808-1810 spanning-tree port type network logging event port link-status ! logging event port trunk-status

!//----UPLINKS to Cat6509-1 ---------//! interface Ethernet1/1 description *** to Catalyst 6509-1 *** switchport mode trunk switchport trunk native vlan 888 switchport trunk allowed vlan 5,6,10,20,22,23,30,35,36,4143,47,115,120,131,133-135,137-140,208211,250,251,253,254,320,350,351,1137,1804,1808-1810 logging event port link-status udld enable channel-group 20 mode active ! no shutdown interface port-channel20 description * TO Cat6509-1* switchport mode trunk lacp suspend-individual switchport trunk native vlan 888 switchport trunk allowed vlan 5,6,10,20,22,23,30,35,36,4143,47,115,120,131,133-135,137-140,208211,250,251,253,254,320,350,351,1137,1804,1808-1810 spanning-tree port type normal vpc 20

!//----DCI Links to remote site ---------//! interface Ethernet1/4 description *** to remote nexus5596UP *** switchport mode trunk switchport trunk native vlan 888 VELUX Access Layer Low Level Design 57



switchport trunk allowed vlan 5,6,10,20,22,23,30,35,36,4143,47,115,120,131,133-135,137-140,208211,250,251,253,254,320,350,351,1137,1804,1808-1810 logging event port link-status udld enable spanning-tree bpdufilter enable channel-group 40 mode active ! no shutdown interface Ethernet1/5 description *** to remote nexus5596UP *** switchport mode trunk switchport trunk native vlan 888 switchport trunk allowed vlan 5,6,10,20,22,23,30,35,36,4143,47,115,120,131,133-135,137-140,208211,250,251,253,254,320,350,351,1137,1804,1808-1810 logging event port link-status udld enable spanning-tree bpdufilter enable channel-group 40 mode active ! no shutdown interface port-channel40 description * TO Remote Site* switchport mode trunk lacp suspend-individual switchport trunk native vlan 888 switchport trunk allowed vlan 5,6,10,20,22,23,30,35,36,4143,47,115,120,131,133-135,137-140,208211,250,251,253,254,320,350,351,1137,1804,1808-1810 spanning-tree port type normal vpc 40

!//-----TACACS Configuration ---//! tacacs-server host key 7 xxxxxxxx tacacs-server host key 7 xxxxxxxx aaa group server tacacs+ < name > server server use-vrf default aaa aaa aaa aaa aaa aaa

authentication login default group authentication login console group authorization config-commands default group authorization commands default group local accounting default group authentication login error-enable

local

!---snip---

!//-------

VRF Management -------- //!

interface mgmt0 description ip address 10.128.50.101/24 VELUX Access Layer Low Level Design 58



VELUX_Access_Layer_Low_Level_Design_v0.3.pdf

Recommend Documents