This Blog is a joint collaboration between Ravi Balakrishnan (Cisco) and Eric Ji  (F5)

Kubernetes is all about abstracting away complexity. As Kubernetes continues to evolve, it becomes more intelligent and will become even more powerful when it comes to helping enterprises manage their data center, not just at the cloud. While enterprises have had to deal with the challenges associated with managing different types of modern applications (AI/ML, Big data, and analytics) to process that data, they are faced with the challenge to maintain top-level network and security policies and gaining better control of the workload, to ensure operational and functional consistency. This is where Cisco ACI and F5 Container Ingress Services come into the picture.

F5 Container Ingress Services (CIS) and Cisco ACI

Cisco ACI offers these customers an integrated network fabric for Kubernetes. Recently, F5 and Cisco joined forces by integrating F5 CIS with Cisco ACI to bring L4-7 services into the Kubernetes environment, to further simplify the user experience in deploying, scaling, and managing containerized applications. This integration specifically enables:

• Unified networking: Containers, VMs, and bare metal
• Secure multi-tenancy and seamless integration of Kubernetes network policies and ACI policies
• A single point of automation with enhanced visibility for ACI and BIG-IP.
• F5 Application Services natively integrated into Container and Platform as a Service (PaaS)Environments

One of the key benefits of such implementation is the ACI encapsulation normalization. The ACI fabric, as the normalizer for the encapsulation, allows you to merge different network technologies or encapsulations be it VLAN or VXLAN into a single policy model. BIG-IP through a simple VLAN connection to ACI, with no need for an additional gateway, can communicate with any service anywhere.

Solution Deployment

To integrate F5 CIS with the Cisco ACI for the Kubernetes environment, you perform a series of tasks. Some you perform in the network to set up the Cisco Application Policy Infrastructure Controller (APIC); others you perform on the Kubernetes server(s). Rather than getting down to the nitty-gritty, I will just highlight the steps to deploy the joint solution.

Pre-requisites

The BIG-IP CIS and Cisco ACI joint solution deployment assumes that you have the following in place:

• A working Cisco ACI installation
• ACI must be integrated with vCenter VDS
• Fabric tenant pre-provisioned with the required VRFs/EPGs/L3OUTs.
• BIG-IP already running for non-container workload

Deploying Kubernetes Clusters to ACI Fabrics

The following steps will provide you a complete cluster configuration: 

Step 1. Run ACI provisioning tool to prepare Cisco ACI to work with Kubernetes

Cisco provides anacc_provisiontool, to provision the fabric for the Kubernetes VMM domain and generate a .yamlfile that Kubernetes uses to deploy the required Cisco Application Centric Infrastructure (ACI) container components. If needed, download the provisioning tool.

Next, you can use this provision tool to generate a sample configuration file that you can edit.

$acc-provision--sample > aci-containers-config.yaml

We can now edit the sample configuration file to provide information from your network. With such a configuration file, now you can run the following command to provision the Cisco ACI fabric:

acc-provision -c aci-containers-config.yaml -o aci-containers.yaml -f kubernetes-<version> -a -u [apic username] -p [apic password]

Step 2. Prepare the ACI CNI Plugin configuration File

The above command also generates the file aci-containers.yaml that you use after installing Kubernetes.

Step 3. Preparing the Kubernetes Nodes - Set up networking for the node to support Kubernetes installation.

With provisioned ACI, you start to prepare networking for the Kubernetes nodes. This includes steps such as Configuring the VMs interface toward the ACI fabric, configuring a static route for the multicast subnet, configuring the DHCP Client to work with ACI, etc.

Step 4. Installing Kubernetes cluster

After you provision Cisco ACI and prepare the Kubernetes nodes, you can install Kubernetes and ACI containers. You can use any installation method you choose appropriate to your environment.

Step 5. Deploy Cisco ACI CNI plugin

When the Kubernetes cluster is up and running, you can copy the preciously generated CNI configuration to the master node, and install the CNI plug-in using the following command:

kubectl apply -f aci-containers.yaml

The command installs the following (PODs):

• ACI Containers Host Agent and OpFlex agent in a DaemonSet called aci-containers-host
• Open vSwitch in a DaemonSet called aci-containers-openvswitch
• ACI Containers Controller in a deployment called aci-containers-controller.
• Other required configurations, including service accounts, roles, and security context

For 'the authoritative word on this specific implementation', you can click here the workflow for integrating k8s into Cisco ACI for the latest and greatest.

After you have performed the previous steps, you can verify the integration in the Cisco APIC GUI. The integration creates a tenant, three EPGs, and a VMM domain. Each tenant will have the visibility of all the Kubernetes POD's.

Install the BIG-IP Controller

The F5 BIG-IP Controller (k8s-bigip-ctlr) or Container Ingress Services, if you aren't familiar, is a Kubernetes native service that provides the glue between container services and BIG-IP. It watches for changes and communicates those to BIG-IP delivered application services. These, in turn, keep up with the changes in container environments and enable the enforcement of security policies.

Once you have a running Kubernetes cluster deployed to ACI Fabric, you can follow these instructions to install BIG-IP Controller.

Use the kubectl get command to verify that the k8s-bigip-ctlr Pod launched successfully.

BIG-IP as a north-south load balancer for External Services

For Kubernetes services that are exposed externally and need to be load balanced, Kubernetes does not handle the provisioning of the load balancing. It is expected that the load balancing network function is implemented separately. For these services, Cisco ACI takes advantage of the symmetric policy-based redirect (PBR) feature available in the Cisco Nexus 9300-EX or FX leaf switches in ACI mode.

This is where BIG-IP Container Ingress Services (or CIS) comes into the picture, as the north-south load balancer. On ingress, incoming traffic to an externally exposed service is redirected by PBR to BIG-IP for that particular service.

If a Kubernetes cluster contains more than one IP pod for a particular service, BIG-IP will load balance the traffic across all the pods for that service. Besides, each new POD is added to BIG-IP pool dynamically.

Conclusion

F5 CIS and Cisco ACI together offer unified control, visibility, security, and application services, for both container and non-container workload. The solution offers a seamless developer experience intended to maintain the simplicity of Kubernetes while still enabling advanced capabilities within the Cisco ACI fabric, and maintaining application availability, security, and visibility across the infrastructure.

Further Resources

F5 Container Ingress Services

Cisco ACI and Kubernetes Integration