At the OpenStack Summit in Barcelona one of our trail-blazing customers – BBVA bank – showcased how they are leveraging the Nuage Networks Virtualized Cloud Services (VCS) solution to deliver next-generation infrastructure provisioning services using OpenStack on top of Kubernetes.
The video recording of this talk – called “Next Generation Overlay for Kubernetes and OpenStack” is available here.
Using their extensive experience with OpenStack, BBVA showed how, by combining Linux Containers and VM provisioning services, they were able to achieve several milestones. In this blog post we will highlight some of those achievements.
OpenStack on top of Kubernetes – the way forward
After several years of experience of running OpenStack in production using the Nuage Networks plug-in, BBVA has concluded that the existing tools for installation, scaling and upgrading OpenStack fall short of their promises.
With the growing maturity of Kubernetes as the leading container orchestration platform, BBVA has adopted Kubernetes for delivering OpenStack services. This was performed by packaging OpenStack services as Kubernetes “applications”. In doing so, BBVA achieved several goals:
- Separate packaging of OpenStack services – All core OpenStack Services – Keystone, Nova, Cinder, Neutron, etc. – are packaged as separate Docker images, built from upstream source code repositories. This allows easier upgrade, migration, bug fixes, etc. This includes any third-party platform integrations – in this case, the Nuage Networks plugin for OpenStack Neutron.
- Easy migration between OpenStack versions – This has long been the Achilles heel of running OpenStack in production. By leveraging native Kubernetes mechanisms – like rolling-updates using Kubernetes Replication Controllers (see this blog post for details) – BBVA can migrate seamlessly between OpenStack versions. By simply updating the the individual Docker images for OpenStack services, BBVA is able to upgrade OpenStack services independently, all by using Kubernetes native tools.
- Reliability (HA) and Scaling – This also has traditionally been a problem in running OpenStack services at scale. Several approaches were tried and then abandoned in the OpenStack community. Instead of trying yet another purpose-built tool, BBVA showed how relying on Kubernetes mechanisms for “pod health-checks” (see here for details and example) achieves the desired reliability without any other tools. Additionally, by simply increasing the number of Kubernetes pods running a given core OpenStack service – and independently for all the other services — the underlying Kubernetes platform enables easy scaling and federation between different hardware infrastructure resource pools.
- Policy-driven networking for both Kubernetes and OpenStack – Lastly but most importantly, BBVA is leveraging the policy engine provided by the Nuage Networks VCS for both Kubernetes networking and OpenStack networking. This provides several advantages:
– Very fine granularity in controlling and monitoring the OpenStack control plane: By isolating each core OpenStack service in its own Kubernetes namespace, the Nuage Network VCS solution gives the ability to control the OpenStack control plane communication.
– Very fine granularity in controlling and monitoring the OpenStack tenant network topology. By using the Neutron plugin for Nuage Networks VCS, BBVA has visibility into the end-user level entities (routers, networks, subnets, etc) through the same policy engine as above.
– Single pane of glass for domain administrators to manage both Kubernetes and OpenStack networking policies.
Networking policies for Kubernetes and OpenStack
As a concrete example, please consider first the screenshot for the Nuage Networks Virtualized Services Directory (VSD) policy repository without OpenStack service segregation:
In the first picture we see that the Kubernetes networking plane – denoted as “K8S-Enterprise” in the left pane – has a single networking domain, consisting of two zones, “default” and “kube-system”. These correspond to the two built-in Kubernetes namespaces (“default” and “kube-system”). In the “default” policy zone / Kubernetes namespace we see the different entities, each corresponding to a core OpenStack service. Out of those, we see the “nova-compute” details highlighted on the right: Its “host” IP address (i.e. the corresponding Kubernetes pod IP address), etc.
For further refined control, OpenStack services were segregated in their own, distinct Kubernetes namespace. The result is as follows:
In this second screenshot we see the same OpenStack services but now clearly separated in their own policy zone, i.e. Kubernetes namespace. In addition to the traditional OpenStack core services we note a “backend” zone (which, as the name implies, implements OpenStack MySQL backend service) and “default” zone, containing other Kubernetes applications.
These constructs are also visible in the Kubernetes dashboard, where we see the OpenStack constructs and Nuage policy engine appearing as workloads and Replication Controllers, all running in the respective Kubernetes namespace:
Open source joint project – a first for both BBVA and Nuage Networks
With this solution BBVA, with the help of Nuage Networks, has developed a new way of deploying OpenStack which is much more flexible and scalable than before. BBVA and Nuage Networks are providing the results of this collaboration as an open source project. We encourage others to adapt and improve on this solution for easy and scalable deployment of OpenStack services using Kubernetes.
The open source repository for this project can be found on GitHub. This includes the additions for deploying the Nuage Networks environment as well.
If you are interested in more details on both the deployment of OpenStack using Kubernetes or Nuage Networks, please contact your local Nuage Networks team or use the contact form on our website, or even the comments section of this blog post.