The need for optimal path selection

When embarking on a journey from one location to another one, we always try to determine the best route to follow. This happens according to some preference criteria. Having a single route is the simplest situation but that would lead to long delays in case of any problem occurring along the way. Availability of multiple paths to destination is good in terms of reliability. Of course, we would need an optimal path selection tool and clear street signs, or a navigation system with GPS, to avoid loops or getting lost.

In a similar way, data networks are designed with multiple paths to destination for higher availability. Specific protocols enable optimal path selection and loop avoidance. Ethernet networks have used the Spanning Tree Protocol or more recent standards like TRILL. IP networks rely on routing protocols like BGP, OSPF and RIP to determine the best end-to-end path. Fibre Channel fabrics have their own standard routing protocol, called Fabric Shortest Path First (FSPF), defined by INCITS T11 FC-SW-8.

FSPF on Cisco MDS switches

The FSPF protocol is enabled by default on all Cisco Fibre Channel switches. Normally you do not need to configure any FSPF parameters. FSPF automatically calculates the best path between any two switches in a fabric. It can also select an alternative path in the event of the failure of a given link. FSPF regulates traffic routing no matter how complex the fabric might be, including dual datacenter core-edge designs.

FSPF supports multipath routing and bases path status on a link state protocol called Shortest Path First. It runs on E ports or TE ports, providing a loop free topology. Routing happens hop by hop, based only on the destination domain ID. FSPF uses a topology database to keep track of the state of the links on all switches in the fabric and associates a cost with each link. It makes use of Dijkstra algorithm and guarantees a fast reconvergence time in case of a topology change. Every VSAN runs its own FSPF instance. By combining VSAN and FSPF technologies, traffic engineering can be achieved on a fabric. One use case would be to force traffic for a VSAN on a specific ISL. Also, the use of PortChannels instead of individual ISLs makes the implementation very efficient as fewer FSPF calculations are required.

FSPF link cost calculation

FSPF protocol uses link costs to determine the shortest path in a fabric between a source switch and a destination switch. The protocol tracks the state of links on all switches in the fabric and associates a cost with each link in its database. Also, FSPF determines path cost by adding the costs of all the ISLs in each path. Finally, FSPF compares the cost of various paths and chooses the path with minimum cost. If multiple paths exist with the same minimum cost, FSPF distributes the load among them.

You can administratively set the cost associated with an ISL link as an integer value from 1 to 30000. However, this operation is not necessary and typically FSPF will use its own default mechanism for associating a cost to all links. This is specified within INCITS T11 FC-SW-8 standard. Essentially, the link cost is calculated based on the speed of the link times an administrative multiplier factor. By default, the value of this multiplier is S=1. Practically the link cost is inversely proportional to its bandwidth. Hence the default cost for 1 Gbps links is 1000, for 2 Gbps is 500, for 4 Gbps is 250, for 32 Gbps is 31 and so on.

FSPF link cost calculation challenges

It is easy to realize that  high-speed links introduce some challenges because the link cost computes smaller and smaller. This becomes a significant issue when the total link bandwidth is over 128 Gbps. For these high-speed links, the default link costs become too similar to one another and so leading to inefficiencies.

The situation gets even worse for logical links. FSPF treats PortChannels as a single logical link between two switches. On Cisco MDS 9000 series, a PortChannel can have a maximum of 16 member links. With multiple physical links combined into a PortChannel, the aggregate bandwidth scales upward and the logical link cost reduces accordingly. Consequently, different paths may appear to have the same cost although they have a different member count and different bandwidths. Path inefficiencies may occur when PortChannels with as low as 9 x 16 Gbps members are present. This leads to poor path selection by FSPF. For example, imagine two alternative paths to same destination, one traversing a 9x16G PortChannel and one traversing a 10x16G PortChannel. Despite the two PortChannels have a different aggregate bandwidth, their link cost would compute to the same value.

FSPF link cost multiplier feature

To address the challenge, for now and the future, Cisco MDS NX-OS 9.3(1) release introduced the FSPF link cost multiplier feature. This new feature should be configured when parallel paths above the 128 Gbps threshold exist in a fabric. By doing so, FSPF can properly distinguish higher bandwidth links from one another and is able to select the best path.

All switches in a fabric must use the same FSPF link cost multiplier value. This way they all use the same basis for path cost calculations. This feature automatically distributes the configured FSPF link cost multiplier to all Cisco MDS switches in the fabric with Cisco NX-OS versions that support the feature. If any switches are present in the fabric that do not support the feature, then the configuration fails and is not applied to any switches. After all switches accept the new FSPF link cost multiplier value, a delay of 20 seconds occurs before being applied. This ensures that all switches apply the update simultaneously.

The new FSPF link cost multiplier value is S=20, as opposed to 1 in the traditional implementation. With a simple change to one parameter, Cisco implementation keeps using the same standard based formula as before. With the new value for this parameter, the FSPF link cost computation will stay optimal even for PortChannels with 16 members of up to 128 Gbps speed.

Conclusion

Cisco is leading the way in supporting the design and implementation of high-speed Fibre Channel fabrics. The ongoing development of 128 Gbps serial interfaces by standard bodies, combined with the Cisco-only capability to create logical links with up to 16 members, demanded for a change to the traditional FSPF link cost calculation. That's why the MDS NX-OS 9.3(1) release introduced support for the new FSPF link cost multiplier feature. Thanks to this, your SAN built with MDS 9000 devices can optimally operate today and tomorrow, while maintaining FC-SW-8 compliance.

To learn more, review these resources:

Cisco MDS 9000 NX-OS 9.3(1) release notes

Cisco MDS 9000 NX-OS 9.x fabric configuration guide

Unlock the full value of your Storage Area Network

Store data securely and access it faster with Cisco Storage Area Networking solutions. Protect your investment with 64G-capable speed and high performance for NVMe/FC and all-flash array storage environments. Work smarter with automation, gain deep visibility, and safeguard your data with Cisco SAN solutions.