Before the pandemic, industries were turned upside down as a digital transformation wave forced IT departments to think of new ways to implement services and address this new business challenge. When business travel starts up again, each of us will see examples: taxi's replaced by Uber and Lyft; newspapers replaced by a smartphone; radio replaced by Spotify. Each industry struggles to remain relevant. The impact on IT?  The huge growth in applications that draw data from more sources, and the speed to implement required today. Oracle databases and the server infrastructures that support have to support larger workloads without sacrificing performance. The challenge is how to architect these systems to meet uncertain growth requirements yet keep their finance department happy.

Foresight

Cisco foresaw this requirement a couple of years ago and invested in a set of Cisco Validated Designs demonstrating the benefits of NVMe (Non-Volatile Memory Express) over Fabrics partnering with Pure Storage initially and more recently with NetApp.

Customers generally fall into two categories:

• Those running I/O over ethernet and would more naturally move to RDMA
• SAN based customers who desire low latency but within a SAN infrastructure

Cisco has developed a proven solution for each of these two scenarios, see details below.

In 2019, Cisco and Pure Storage tested and validated a FlashStack solution highlighting the benefits of RoCE V2 -Oracle RAC 19c Databases running on Cisco UCS with Pure Storage FlashArray //X90R2 using NVMe-oF RoCE V2  (RoCE  -RDMA over Converged Ethernet version 2). Here the standard FlashStack Converged Infrastructure (depicted below) was set up with NVMe located in the servers and used RoCE to move the data traffic between the servers and the All-Flash storage subsystem.  SLOB (Silly Little Oracle Benchmark) was used to replicate users and the system was scaled to 512 users demonstrating the following benefits:

• Lower latency compared to other traditional protocols
• Higher IOPS (I/O per second) and scaled linearly
• Higher bandwidth to address higher data traffic requirements
• Improved protocol efficiency by reducing the "I/O stack"
• Lower host CPU utilization, documented at 30% less
• Indirectly, as CPU utilization was lowered, more processor cycles are available to process work, therefore fewer Intel processor cores need to be licensed to achieve performance.

This was a welcome design incorporated by many companies from commercial to large enterprise as it addressed a pressing need -how to stretch the IT budget to complete more work on the current system.  The NVMe interface is defined to enable host software to communicate with nonvolatile memory over PCI Express (PCIe). It was designed from the ground up for low-latency solid state media, eliminating many of the bottlenecks seen in the legacy protocols for running enterprise applications. NVMe devices are connected to the PCIe bus inside a server. NVMe-oF extends the high-performance and low-latency benefits of NVMe across network fabrics that connect servers and storage. NVMe-oF takes the lightweight and streamlined NVMe command set, and the more efficient queueing model, and replaces the PCIe transport with alternate transports, like Fibre Channel, RDMA over Converged Ethernet (RoCE v2), TCP.

In 2020, the Pandemic hit.

Pandemic Driven Shifts

COVID-19 caused many IT organizations to shift focus from database to remote worker implementations initially conceived as short-term solutions, now moving to longer term designs. Businesses are returning to a focus on stretching their database infrastructure solutions, and Cisco has partnered with NetApp on a new solution to meet this goal.

In April 2021, Cisco and NetApp published a new Cisco Validated Design called FlexPod Datacenter with Oracle 19c RAC Databases on Cisco UCS and NetApp AFF with NVMe/FC. The proven design using NVMe is now proven work with a Fibric Channel twist.

NVMe over Fibre Channel (NVMe/FC) is implemented through the Fibre Channel NVMe (FC-NVMe) standard which is designed to enable NVMe based message commands to transfer data and status information between a host computer and a target storage subsystem over a Fibre Channel network fabric. FC-NVMe simplifies the NVMe command sets into basic FCP instructions. Because Fibre Channel is designed for storage traffic, functionality such as discovery, management and end-to-end qualification of equipment is built into the system.

Almost all high-performance latency sensitive applications and workloads are running on the same underlying transport protocol (FCP) today. Because NVMe/FC and Fibre Channel networks use the same FCP, they can use common hardware components. It's even possible to use the same switches, cables, and NetApp ONTAP target port to communicate with both protocols at the same time. The ability to use either protocol by itself or both at the same time on the same hardware makes transitioning from FCP to NVMe/FC both simple and seamless.

Large-scale block flash-based storage environments that use Fibre Channel are the most likely to adopt NVMe over FC. FC-NVMe offers the same structure, predictability and reliability characteristics for NVMe-oF that Fibre Channel does for SCSI. Plus, NVMe-oF traffic and traditional SCSI-based traffic can run simultaneously on the same FC fabric.

The design for new FlexPod is depicted below and follows the proven design that has led FlexPod to become a most popular Converged Infrastructure in the market for several years.

The same low latency, high performance benefits of the previous CVD are proven once again in this NVM/FC design.  As such, customers now have a choice as to how to implement a modern SAN to run the heart of their IT shop -the Oracle Database.

Business will continue to challenge their IT departments, some are planned challenges while others are completely unforecasted. Picking a design that can grow to meet these future requirements, where each element in the design can be upgraded independently as circumstances warrant, while meeting performance requirements with an eye toward Oracle licensing costs is the challenge that Cisco's low latency solutions have met. These are the solutions your organization should take a closer look at for your future Oracle deployments.

Resources:

Learn more about low latency