Cisco Nexus 5000 Series Switches license
Multicore computing and virtualization are rapidly changing the data center landscape, furthering the need for high-bandwidth, low-latency switching. These technologies increase efficiency by increasing server utilization, but they also promote an ever-increasing demand for bandwidth. Most data centers grappling with the bandwidth challenge are migrating to 10 Gigabit Ethernet to alleviate their IP network bottlenecks. In addition, most data centers support dual Fibre Channel links per server to access their storage networks, and some data centers supporting high-performance computing (HPC) environments also support multiple interprocess communication (IPC) networks per server.
Cisco offers a better solution to these challenges in the form of the Cisco Nexus 5000 Series Switches. Designed as access-layer switches for in-rack deployment, the Cisco Nexus 5000 Series helps simplify data center infrastructure and reduce total cost of ownership (TCO). It supports I/O consolidation at the rack level, reducing the number of adapters, cables, switches, and transceivers that each server must support, all while protecting investment in existing storage assets.
The Cisco Nexus 5000 Series delivers these benefits to data centers through the following product features:
- High performance 10 Gigabit Ethernet: The Cisco Nexus 5000 Series is a family of line-rate, low-latency, cost-effective 10 Gigabit switches designed for access-layer applications.
- Fibre Channel over Ethernet (FCoE): The Cisco Nexus 5000 Series is the first open-standards-based access-layer switch to support I/O consolidation at the rack level through FCoE.
- IEEE Data Center Bridging (DCB): The switch family incorporates a series of Ethernet enhancements designed for the data center, including flow control and network congestion management.
- VM Optimized Services: The switch family supports end-port virtualization and virtual machine optimized services, helping increase the scalability of virtual Layer 2 networks and enhancing application performance and security.
This document describes how Cisco has designed the Cisco Nexus 5000 Series Switches as both high-bandwidth, low-latency, access-layer switches for rack deployment and as the basis for a unified network fabric that can help simplify data center infrastructure while reducing capital and operational costs. This document provides a brief overview of the switch features and benefits and then details the series’ 10 Gigabit Ethernet, I/O consolidation, and virtualization capabilities. Internally, the switches are based on only two custom application-specific integrated circuits (ASICs): a unified port controller that handles all packet-processing operations on ingress and egress, and a unified crossbar fabric that schedules and switches packets. Every design decision made in these two devices is precisely targeted to support I/O consolidation and virtualization features with the most efficient use of transistor logic, helping minimize power consumption and maximize performance.
Introducing the Cisco Nexus 5000 Series
The Cisco Nexus 5000 Series is designed to be deployed in server racks, and the series is designed much like the servers it supports. All ports and power entry connections are at the rear of the switches, simplifying cabling and minimizing cable length .Cooling is front-to-back, supporting hot- and cold-aisle configurations that help increase cooling efficiency. The front panel includes status indicators and hot-swappable, N+1 redundant power supplies and cooling modules. All serviceable components are accessible from the front panel, allowing the switch to be serviced while in operation and without disturbing network cabling. The switch family’s port density is such that, depending on the switch model and server rack configuration, switches can support top-of-rack, adjacent-rack, and end-of-row configurations.
Cisco Nexus 5000 Series Feature Highlights
Features and Benefits
The switch family’s rich feature set makes the series ideal for rack-level, access-layer applications. It protects investments in data center racks with standards based Ethernet and FCoE features that allow IT departments to consolidate networks based on their own requirements and timing.
- The combination of high port density, wire-speed performance, and extremely low latency makes the switch an ideal product to meet the growing demand for 10 Gigabit Ethernet at the rack level. The switch family has sufficient port density to support single or multiple racks fully populated with blade and rack-mount servers.
- Built for today’s data centers, the switches are designed just like the servers they support. Ports and power connections are at the rear, closer to server ports, helping keep cable lengths as short and efficient as possible. Hot-swappable power and cooling modules can be accessed from the front panel, where status lights offer an at-a-glance view of switch operation. Front-to-back cooling is consistent with server designs, supporting efficient data center hot- and cold-aisle designs. Serviceability is enhanced with all customer-replaceable units accessible from the front panel. The use of SFP+ ports offers increased flexibility to use a range of interconnect solutions, including copper for short runs and fiber for long runs.
- Fibre Channel over Ethernet and IEEE Data Center Bridging features supports I/O consolidation, eases management of multiple traffic flows, and optimizes performance. Although implementing SAN consolidation requires only the lossless fabric provided by the Ethernet pause mechanism, the Cisco Nexus 5000 Series provides additional features that create an even more easily managed, high-performance, unified network fabric.
10 Gigabit Ethernet and Unified Fabric Features
The Cisco Nexus 5000 Series is first and foremost a family of outstanding access switches for 10 Gigabit Ethernet connectivity. Most of the features on the switches are designed for high performance with 10 Gigabit Ethernet. The Cisco Nexus 5000 Series also supports FCoE on each 10 Gigabit Ethernet port that can be used to implement a unified data center fabric, consolidating LAN, SAN, and server clustering traffic.
Nonblocking Line-Rate Performance
All the 10 Gigabit Ethernet ports on the Cisco Nexus 5000 Series Switches can handle packet flows at wire speed. The absence of resource sharing helps ensure the best performance of each port regardless of the traffic patterns on other ports. The Cisco Nexus 5020 can have 52 Ethernet ports at 10 Gbps sending packets simultaneously without any effect on performance, offering true 1.04-Tbps bidirectional bandwidth.
The crossbar fabric on the Cisco Nexus 5000 Series Switches is implemented as a single-stage fabric, thus eliminating any bottleneck within the switch. Single-stage fabric means that a single crossbar fabric scheduler has full visibility of the entire system and can therefore make optimal scheduling decisions without building congestion within the switch. With a single-stage fabric, the bandwidth you see is the bandwidth you get, and congestion becomes exclusively a function of your network design; the switch does not contribute to it.
The cut-through switching technology used in the Cisco Nexus 5000 Series ASICs enables the product to offer a low latency of 3.2 microseconds, which remains constant regardless of the size of the packet being switched. This latency was measured on fully configured interfaces, with access control lists (ACLs), quality of service (QoS), and all other data path features turned on. The low latency on the Cisco Nexus 5000 Series enables application-to-application latency on the order of 10 microseconds (depending on the network interface card [NIC]). These numbers, together with the congestion management features described next, make the Cisco Nexus 5000 Series a great choice for latency-sensitive environments.
Keeping latency low is not the only critical element for a high-performance network solution. Servers tend to generate traffic in bursts, and when too many bursts occur at the same time, a short period of congestion occurs. Depending on how the burst of congestion is smoothed out, the overall network performance can be affected. The Cisco Nexus 5000 Series offers a full portfolio of congestion management features to minimize congestion. These features, described next, address congestion at different stages and offer maximum granularity of control over the performance of the network.
Virtual Output Queues
The Cisco Nexus 5000 Series implements virtual output queues (VOQs) on all ingress interfaces, so that a congested egress port does not affect traffic directed to other egress ports. But virtual output queuing does not stop there: every IEEE 802.1p class of service (CoS) uses a separate VOQ in the Cisco Nexus 5000 Series architecture, resulting in a total of 8 VOQs per egress on each ingress interface, or a total of 416 VOQs on each ingress interface. The extensive use of VOQs in the system helps ensure maximum throughput on a per-egress, per-CoS basis. Congestion on one egress port in one CoS does not affect traffic destined for other CoSs or other egress interfaces, thus avoiding head-of-line (HOL) blocking, which would otherwise cause congestion to spread.
Lossless Ethernet (Priority Flow Control)
By default, Ethernet is designed to drop packets when a switching node cannot sustain the pace of the incoming traffic. Packet drops make Ethernet very flexible in managing random traffic patterns injected into the network, but they effectively make Ethernet unreliable and push the burden of flow control and congestion management up at a higher level in the network stack.
IEEE 802.1Qbb Priority Flow Control (PFC) offers point-to-point flow control of Ethernet traffic based on IEEE 802.1p CoS. With a flow control mechanism in place, congestion does not result in drops, transforming Ethernet into a reliable medium. The CoS granularity then allows some CoSs to gain a no-drop, reliable, behavior while allowing other classes to retain traditional best-effort Ethernet behavior. A networking device implementing PFC makes an implicit agreement with the other end of the wire: any accepted packet will be delivered to the next hop and never be locally dropped. To keep this promise, the device must signal the peer when no more packets can reliably be accepted, and that, essentially, is the flow control function performed by PFC. The benefits are significant for any protocol that assumes reliability at the media level, such as FCoE.
Traditional Ethernet is unreliable, and the only way to postpone packet drops in case of congestion is to increase the buffering capabilities of the interfaces. With more buffers, short-lived congestion can be handled without causing any packet drops, and the regular drop behavior takes over if the congestion lasts longer. Tuning the amount of buffer space available effectively means tuning the definition of “short-lived congestion.”
PFC changes the equation by pushing back the buffering requirements to the source. PFC works very well for protocols like FCoE that require a reliable medium, but it makes short-lived congestion and persistent congestion undistinguishable.
Delayed drop mediates between traditional Ethernet behavior and PFC behavior. With delayed drop, a CoS can be flow controlled and the duration of the congestion monitored, so that the traditional drop behavior follows if the congestion is not resolved. Delayed drop offers the capability to tune the definition of “short-lived congestion” with PCF, hence removing the need to increase physical buffers on the interfaces.
Fibre Channel over Ethernet
FCoE is a standard-based encapsulation of Fibre Channel frames into Ethernet packets. By implementing FCoE and enabling a broad range of partners to terminate FCoE on the host side, the Cisco Nexus 5000 Series introduces storage I/O consolidation on top of Ethernet.
Hardware-Level I/O Consolidation
The Cisco Nexus 5000 Series ASICs can transparently forward Ethernet, Fibre Channel, and FCoE, providing true I/O consolidation at the hardware level. The solution adopted by the Cisco Nexus 5000 Series minimizes the costs of consolidation through a high level of integration in the ASICs. The result is a full-featured Ethernet switch and a full-featured Fibre Channel switch combined in one product.
- Ethernet: Ethernet host virtualizer (EHV): In most network designs, access switches are attached to multiple distribution switches for high-availability purposes. The physically redundant paths are not all active in the loop-free logical topology created by the Spanning Tree Protocol, and that affects the amount of active bandwidth available to the LAN core. Using EHV, the default switching behavior can be changed in the Cisco Nexus 5000 Series and replaced by a different loop-prevention scheme at the access layer. EHV allows the switch to behave like a giant end-host for the network, representing all the hosts (servers) directly attached to it. Because of this behavior, EHV is completely transparent to the rest of the network and shrinks the Spanning Tree domain one level up to the distribution layer, giving full access to all the bandwidth physically available between the access and distribution layers.
- Fibre Channel: N_port virtualization (NPV): Because of the use of hierarchically structured addresses (Fibre Channel IDs [FC_IDs]), Fibre Channel switches can offer L2MP, thus resolving the forwarding limitations of the Spanning Tree Protocol in Ethernet. However, the fixed address structure limits the scalability of a Fibre Channel fabric to a maximum of 239 switches, constraining the network design choices available to SAN architects. The Cisco Nexus 5000 Series frees the SAN of these constraints by offering the option to run the switch in NPV mode. When NPV mode is enabled on the Cisco Nexus 5000 Series, the switch becomes a transparent proxy that does not participate in the SAN fabric services, and it can aggregate all the directly attached initiators and targets directed toward the SAN fabric core as if it were a simple multipoint link. Used in conjunction with NPIV on the perimeter of the SAN fabric, NPV is a powerful tool for scaling the SAN beyond the port density of traditional Fibre Channel switches.
Cisco Nexus 5000 Series Internal Architecture
On the control plane side, the Cisco Nexus 5000 Series runs Cisco NX-OS on a single-core 1.66-GHz Intel LV Xeon CPU with 2 GB of DRAM. The supervisor complex is connected to the data plane in-band through 2 internal ports running 1-Gbps Ethernet, and the system is managed in-band, or through the out-of-band 10/100/1000-Mbps management port.