How to Spot Gaps in Your Public Cloud Kubernetes Security Posture

28 April 2022 by Graeme Colman

Breaking down the core areas that you should be aware of when considering security around public cloud Kubernetes

The Control Plane

API Server

The API server provides the entry point specifically for the management of a Kubernetes cluster. The API server endpoint is secured through public cloud IAM and also Kubernetes RBAC - however, it is open to the internet if not configured correctly and hence presents an attack vector.

  • Protect API Server endpoint through the use of private clusters or IP whitelist and blacklisting. 
  • Protect the access to the API server through effective IAM controls.
  • Define effective segregation of duties within the cluster RBAC controls.

Microsoft AKS

Google GKE

Amazon EKS

ETCD

ETCD is a distributed data store that contains cluster data including Kubernetes secrets, a mechanism by which sensitive data is made available to containers such as user certificates, passwords, or API keys.

AKS, EKS and GKE all provide encryption at rest for ETCD to protect stored information, however, Kubernetes secrets by default are only base64 encoded (plaintext).

  • Kubernetes secrets are available to ALL pods within the same namespace. 
  • By default, the secret will be mounted into a pod in plaintext (base 64 encoded).
  • Protect sensitive information by encrypting secrets and rotating certificates
    • Use public cloud encryption services (e.g. AWS KMS, Azure Key Vault).
    • Use a 3rd party secrets management tool (e.g.  Hashicorp Vault).

Google GKE

Microsoft AKS

Amazon EKS

  • Secrets encrypted at rest (depending on the ETCD storage, needs configuration).
  • Secret audit configurable (when secrets were used pushed to cloud watch).
  • Data encryption and secrets management

The Network

Control plane network/worker network

Typically the control plane in public cloud Kubernetes runs in a separate network to the worker nodes (where applications are running). As a consumer of the service, you have no access to the control plane network, which is managed and secured by the cloud provider.

The access point between the two networks must be considered when securing your clusters. Providers have different mechanisms for mitigating security gaps:

Microsoft AKS

  • AKS provides the capability for public and private endpoints for worker nodes to communicate with the control plane. AKS also provides IP and CIDR address whitelisting and blacklisting when using public endpoints or private interfaces for keeping communications within a VPN. 
  • Use Network Security Groups.
  • Security concepts for applications and clusters

Amazon EKS

Google GKE

Network policy for pod network

By default, all pods within a Kubernetes cluster can communicate freely with other pods. This presents a considerable threat of internal attacks from applications running within the cluster.

These threats should be mitigated through network security constructs that exist within the cluster. Network policies are Kubernetes resources that control the traffic between pods and/or network endpoints and control data ingress and egress between pods and namespaces. 

Microsoft AKS

  • Recommends using Calico network provider to enable Network Policy.
  • Network Security

Amazon EKS

  • Recommends using Calico network provider to enable Network Policy.
  • Network security

Google GKE

The VM (node), OS and container runtime

Applications in a Kubernetes cluster run in containers within a virtual server called a node. That node has, amongst other things, a host operating system and container runtime (which runs the container). Both of these components require attention within your Kubernetes security posture as a breach in a host will compromise everything you are running there, including Kubernetes and the workloads inside.

The public cloud providers do a lot of work in mitigating security breaches at this level including running CIS benchmarks against the host OS and regularly patching and updating. However, as the end user, you still have responsibilities to make sure that you are adopting the cloud patches and upgrades.

Microsoft AKS

  • Uses optimized Ubuntu for Linux or Windows 2019 (managed nodes).
  • Containerd by default, Docker as option.
  • Node Security

Amazon EKS

  • Managed or self-managed nodes have different security considerations - check docs.
  • Containerd default from EKS 1.23.
  • EKS Nodes

Google GKE

Operating system

The public cloud Kubernetes services allow a varying range of node options when creating clusters, from Windows to Linux hosts and also container runtime optimized operating systems. Choosing the right OS for your nodes includes understanding the ongoing security patching strategies.

“Be aware of differences in public cloud patching strategies”

Patching and updating

Different cloud providers have different methods of patching and updating the OS and container runtimes in their offerings.

In general, if you are using “self-managed” VMs as hosts then you are responsible for patching and upgrading. If you are using provider-managed VMs then some or all of the patching and upgrading process is automatically applied.

There are several things you can do to harden self-provided node images, like removing package managers, having read only filesystems, etc. The best guidance will be with the OS provider on hardening images.

Microsoft AKS

  • Microsoft security policies automatically apply OS patches to all nodes on a nightly basis, however, patches requiring a node reboot will not force the nodes to be rebooted and must be handled manually. 
  • Patched images are available on a weekly schedule, there is no automated alerting of new patches but Azure Service Health can be used to identify new security patch releases: Microsoft Service Health overview
  • AWS bottlerocket (created specifically to run containers with a minimal OS and hence smaller attack vector) is an option for more secure nodes.
  • Apply security and kernel updates to Linux nodes

Amazon EKS

Google GKE

  • Provides updated node images used to apply patches or upgrades. Customers can subscribe to the update channels to automatically upgrade nodes to the latest version, or apply manually.
  • GKE publishes upgrade notifications to a pub/sub channel: Auto-upgrading nodes
  • Security patching

Security is a joint responsibility

Joint responsibility is never more true than when working with node security and patching. You must be aware of the methodologies that your cloud provider uses for security patches and updates, and ensure that appropriate processes are in place to review and apply the right process to keep node security up to date.

The Workload

Securing the container & deployment

Security around containers and container workloads is a large subject with many implications that is not in the scope of this article. However, you should be aware of the areas that need to be addressed with regard to protecting systems and data.

One important area to think about is the safety of the supply chain.

Container specific security

Securing container workloads starts with how the container interfaces with the container runtime and hence the underlying worker node and networks. Containers pose a number of security concerns that must be addressed through the application of policy-based protection.

Least privilege

Application containers must be created to only expose the services and components needed to do the role of the container. Moreover, policies must be in place to deny certain aspects of container runtime interactions such as:

  • Disallow “privileged containers” 
    • privileged containers can break into the root of the worker node and hence any part of the Kubernetes cluster.
  • Restrict userid   
    • Restrict the users (userid) of processes that are executing within the container, e.g. do not allow root users to execute processes within containers.
  • Restrict network, file system and namespace access
    • Containers should not have the capability to access host filesystems or network.

The list above is just a selection to give a view of the importance of container security and where policy needs to be created to protect and secure.

Pod Security Policy was deprecated from Kubernetes in 1.21 and will be removed completely in 1.25 (April 2022). Pod Security Policy was a core feature used to define pod and OS level security restriction policies described above. Be aware if your existing solutions make use of this. Take a look at these blogs for more information on the rationale behind this and alternative tools:
PodSecurityPolicy is dead. Long live...?

Policy as Versioned Code

How to use the security profiles operator

Container supply chain

An area that has been spawned from the ubiquity of containers is in Software Supply Chain security. A feature of container packaging format enables software to be decomposed into “layers” which may come from other sources such as 3rd party software providers. The layered software approach presents challenges in identifying what is contained in the software “layers” and what security controls mitigations have been applied.

Moreover, base container images can be sourced from public repositories where the content of the software layers is unknown with a high potential for malicious software embedded.

Care must be taken in your software delivery process to ensure that container images used in development have come from trusted sources and have been image scanned to ensure that versions of software used within the container have the latest patches and vulnerable image layers identified.

One important area to consider is the internal container build identification. Version numbers as labels are good for identification but not enough to ensure that what you have in the container is what you are expecting. Take a look at using the SHA checksum to ensure that the image is the same as expected. Also, take a look at image signing which gives even greater confidence that the image is exactly the same as when it was signed.

Supply chain issues are discussed further in this blog:

Tutorial: Keyless sign and verify your container images with cosign

Application access into the cluster

Public cloud Kubernetes applications that need to expose public endpoints have a few options. Typically these fall into three areas:

  • Ingress 
  • Node Port 
  • Load Balancer

Kubernetes Ingress

Kubernetes ingress is configured with two objects: the ingress controller and the ingress object.

An ingress controller is a piece of software that provides reverse proxy, configurable traffic routing, and TLS termination for Kubernetes services. Kubernetes ingress resources are used to configure the ingress rules and routes for individual Kubernetes services. Using an ingress controller and ingress rules, a single IP address can be used to route traffic to multiple services in a Kubernetes cluster.

Node Port

Creates a port mapping on the underlying node that allows the application to be accessed directly with the node IP address and port.

Load Balancer

Uses a load balancer resource that has an external IP address that connects to the requested pods. To allow customers' traffic to reach the application, load balancing rules are created on the desired ports.

Security is a joint responsibility

The public cloud Kubernetes providers generally hand the security concerns with ingress back to the user. It is up to you to configure ingress controllers, certificates and policy to secure access into your services from the outside. Sufficient policy and process should be in place to identify misconfigured ingress that could pose a security threat.

Take a look at this blog post which talks about ingress in EKS with TLS:

Tutorial: How to expose Kubernetes services on EKS with DNS and TLS

Container Registry

By default, Kubernetes can pull container images defined in your Pod specifications from any destination, as long as it's reachable from the Node. This introduces the risk that a user (authorized or not) could accidentally or maliciously specify an image to run from an untrusted source, which may cause direct harm to the platform and neighboring services, resulting in data exfiltration, and increased hosting and auditing costs. 

Employing cluster-level image policy controls significantly reduces this risk and can be achieved via the use of custom admission controllers (i.e. OpenPolicyAgent Gatekeeper).

What to do:

  • Only allow image pulls from trusted repositories. Always reference image tags and digests (for example, by using OPA Gatekeeper).

Take a look at the following blog that talks about verifying the container supply chain. Appvia Wayfinder also implements a custom admission controller which provides policy enforcement of trusted repositories:

5 things to improve your Kubernetes security posture

Summary

Hopefully this has given a reasonable high-level account of most of the security areas that you will need to be aware of when thinking about public cloud Kubernetes as a tool to deliver applications. Each of the areas I have mentioned has much more detailed levels of what and how to secure, but you should now have an appreciation of the scope of where to invest your time.

Kubernetes and containers are a great set of technologies to make building scalable applications in a simpler, more standardized way. Yes, it is still complex, but a lot less complex than building snowflake infrastructure and architectures of the past. Devising a well-thought-out security posture requires you to understand the moving parts as misconfiguration of any of these may lead to vulnerabilities to your organization’s data.

Here at Appvia we have products and services that focus on providing security guardrails and policy-based compliance for public cloud Kubernetes. Appvia Wayfinder brings you peace of mind and compliance in delivering on your security concerns.

Share this article

About the author

Picture of Graeme Colman

Graeme Colman

Head of PreSales, EMEA

Related articles