When considering your cluster architecture you will, of course, be considering dependencies. Kubernetes cannot exist in isolation, so it is important to ensure that your supporting infrastructure is both resilient and highly available. After all, your cluster is only as resilient as its weakest component. This article addresses a few primary concerns regarding some of these services.
Service Discovery
DNS
Depending on how you want to segment your fleet, you will need to ensure that your DNS hierarchy is highly available, resilient, and scalable. This is where Cloud Service Providers demonstrate added value. It is extremely easy to integrate their DNS services with your Kubernetes Cluster, and you can be sure that their DNS infrastructure will scale accordingly with demand.
You might want to consider two schemas. The first will ensure that your infrastructure is addressable. The second will ensure that your services are addressable and have no bearing on the physical location of the service, e.g:
-
node01.cluster01.uk.mydomain.com is the DNS name and IP address for a physical cluster node.
e.g. [node-name].[cluster-name].[datacenter].[internal-domain-name].[domain-suffix]
-
my-first-service.mydomain.com is the DNS name and IP address for a hosted service.
e.g. [service-friendly-name].[internal-domain-name].[domain-suffix]
It seems obvious but, having said that, I’ve seen Service addressing being complicated by confusing the fully-qualified domain name (FQDN) with physical references - i.e., service1.node1.cluster1.uk.mydomain.com
This is NOT a good idea as it tethers the service to a fixed location - cluster1 in this example - and does not give you the flexibility that you would need to transparently shift the service from one node to another, say for instance if you have an outage, or planned maintenance where a service needs to be published elsewhere. Ultimately, your service name should be fully portable and not contain fixed references to any location.
Container Registry
A Private Container Registry is a MUST. You wouldn't trust installing just any old software on your personal laptop, so why would you trust an external container image? A container image pipeline with controls and checks can help to ensure the provenance of the images you want to publish for use in your Private Container Registry.
Continual scanning of containerised images will ensure that they do not become stale and insecure over time, and should a vulnerability be highlighted, automated controls could be engaged to pull down the latest image for ‘on-boarding’. An automated container image pipeline could build a temporary disposable environment, where the container image can be subjected to automated tests that enable you to approve and promote the image. If testing is robust and comprehensive, over time you might have the confidence to promote into production and silently roll it out across your estate. In my experience, this is typically a stage of maturity that most organisations are working towards but haven’t always had the confidence to execute.
Load Balancing
Kubernetes supports 3 types of service endpoints:
- A ClusterIP is a cluster-assigned resource which allows communication between resources INSIDE the cluster.
- A NodeIP is a cluster-assigned resource which allows communication from resources OUTSIDE the cluster.
- A LoadBalancer only becomes available natively if Kubernetes is installed on supporting infrastructure. This is another key area where Cloud Service Providers demonstrate value as LoadBalancers can be provisioned automatically by Kubernetes.
IP
Primary Subnets and Secondary Subnets should be segregated, as a routable subnet for cluster ingress and egress, and non-routable subnet for local network traffic. This is one of many small design choices that help to mitigate the threat of container breakout, whereby any attempts to access anything at all directly from a node is thwarted by the absence of any network routing or internet access. In case you are wondering, the use of private links or endpoints is recommended in lieu of network routes.
IP exhaustion is typically one of the first thresholds encountered when running a cluster. This is often because initial capacity planning is conservative and has not accounted for the fast growth of innovation that a Kubernetes cluster unlocks. I would always advocate using at least a 16-bit subnet, although recently I have worked with a customer who has exceeded this in their first 24 months (a cluster migration fixed this).
Top Tip: Plan ahead and be bold in your estimates during capacity planning. Be mindful of growth and do not build-in any cap to this, as it may save time and effort in the longer term.
API (Application Programming Interface)
API limits are a very real concern. It is quite easy to underestimate just how busy your cluster will be once it is in full flight. I often see the out-of-the-box threshold limits for major cloud providers being exceeded and subsequently throttled. Monitoring these limits closely and requesting an increase in soft limits ahead of time, should be enough for most organisations. However, large enterprise should also consider service hard limits and whether or not you are likely to approach these thresholds. It is easy to mitigate this problem by making the necessary design decisions during capacity planning and before building your cluster.
Infrastructure-as-Code (IaC)
Building anything manually in a Production environment should not be allowed. Ever. Not even in the event of a service outage. Rare exceptions can be made for developers, and only then in non-Production environments.
The best practice of using Infrastructure-as-Code should be adopted and your entire cluster build (including ALL dependencies) should be written as code and committed to a version control repository. This version control repository should be controlled by RBAC (Role-Based Access Control) and a git flow that includes manual code peer reviews (at least for each PULL request), plus automated controls to identify errors. Controls can and should be applied that include (but are not limited to) code linting, code structure validating (e.g. for Helm Charts, or basic YAML), and security scans for anything that shouldn’t be hardcoded (e.g. passwords). Ideally, you would include synthetic transaction testing where, for example, a dry-run flag is used to execute code without building anything at all as a result.
Terraform is our preferred IaC tool because it is platform agnostic, comprehensively battle-tested and supported universally. Using a Terraform plan is also a useful way to dry-run your code to identify any errors, and a useful way to predict results and implement change budgets. Change budgeting involves setting limits that can be used as guard rails e.g. If you manually run a Terraform plan and it wants to build 20 resources, change 0 resources and delete 0 resources, then a change budget in your pipeline that is configured with these thresholds will prevent anything untoward from happening, i.e. deleting or changing anything that you are not expecting.
Using Infrastructure-as-Code also enables granular release control and a standardized approach to building, configuring, and restoring your infrastructure.
Policy Engine
Policies are guardrails that can be established to protect your cluster from unauthorised configuration changes. It is a good idea to establish policies upfront as doing this in retrospect can be time consuming. Establishing guidelines at an early stage is critical for the success of any infrastructure or process, and Kubernetes is no exception.
Binding a Policy engine to your Kubernetes Admission Controller is a straightforward way to protect your cluster from being fed garbage, although it should be your last line of defense in that poorly-written code should be weeded out already by the code pipeline. In fact, every test should be "shifted left", with server-side controls being relied on as the last line of defense.
Some basic examples of policies are:
- All images must be from approved repositories
- All ingress hostnames must be globally unique
- All pods must have resource limits
- All namespaces must have a label that lists a point-of-contact
By building your own policies you can ensure that cluster resources are built to your own specific requirements, and that non-compliant resources cannot be created. Policies are where things begin to get interesting as there are many engines (and languages) you can use to establish compliance control.
As with Infrastructure, I would advocate policy-as-code, given that using a structured code repository gives us full accountability, auditability, granular control and roll-back capability. With this particular requirement, however, your options become much narrower, although this can be a good thing as it highlights those engines that are truly cloud-native and lend themselves easily to a GitOps approach to continuous delivery and deployment. I will talk about my approach to GitOps in detail in a future article.
Dedicated DB Services
While not an external dependency, I think that it is worth mentioning that Kubernetes is underpinned by etcd. As a key-value store of all Kubernetes cluster resources, it facilitates relationships between resources inside the cluster, and beyond via ingress and egress gateways.
While not an external dependency (it will be installed by default) it is vitally important to ensure that file storage for the etcd database is itself resilient and highly available. Cloud Service Providers make this setup relatively easy. It is also important to ensure that the etcd database features in your backup plans, and that you are comfortable with the procedure for restoring etcd to a cluster in case of failure.
Other databases should be completely de-coupled by design and should not be referenced using hardcoded physical FQDN or IP address. An architectural review before any design is approved should capture the use of hardcoded values and eliminate them by ensuring that references to Databases are published as services, i.e. [service-friendly-name-db01].[internal-domain-name].[domain-suffix]
Note: the addition of ‘-db01’ has been added to the end of the Service name. This is to distinguish this service as a database; perhaps one of many that are used in support of your service.
Summary
Ensuring that your dependencies do not become bottlenecks is paramount for managing a stable cluster. While no restrictions are insurmountable, some will require considerable time and effort to remediate. Following the guidance above, you can build on my lessons learnt and decide for yourself if there are any risks that you would like to minimise upfront when planning your own Kubernetes platform. Capacity planning for years 1-2 (and forward planning for years 2-5) can pay dividends at a later date.
Other articles
Choosing a Kubernetes Platform is not an easy decision. To offer you a bit of background and highlight a few popular choices, we have authored other articles that you might like to read before taking your next step:
Airwalk Reply has designed and deployed Kubernetes infrastructure, controls and pipelines, for some of the world’s largest organisations. Contact us for more information about our Kubernetes consulting services, advice or assistance.
