Extending Kubernetes Clusters into a New AWS Availability Zone

Table of Contents

Kops, self-described as ‘kubectl for clusters‘, is an official Kubernetes tool that allows you to build and manage your Kubernetes infrastructure in cloud. Namely, it’s a strong tool for deploying clusters to Amazon Web Services (AWS). In aligning with the industry and contributing to tools that are a part of the Kubernetes ecosystem, we’ve been running kops. This article is intended to help others in a similar position in leveraging an additional Availability Zone once they have a running cluster.

In June 2018, AWS announced the launch of a 3rd Availability Zone in the London Region. Below is the process taken to leverage this with links to PRs we’ve created to help make the process easier.

To utilize this guide, you’ll need to have:

  1. A working Kubernetes Cluster built with kops 
  2. VPC, Route Tables & Subnets managed by kops
  3. A subnet range available within your VPC (you would not be able to do this if the VPC CIDR is fully allocated)
  4. Five Master instances distributed across two Availability Zones
  5. ETCDv3 with TLS
  6. Using this version of kops (or more recent)

Let’s run a validate on the cluster to make sure everything is healthy before we start:

➜ kops validate cluster 

Using cluster from kubectl context: test.k8s.appvia.io Validating cluster test.k8s.appvia.io 

INSTANCE GROUPS 
NAME ROLE MACHINETYPE MIN MAX SUBNETS 
eu-west-2a-master1 Master t2.medium 1 1 eu-west-2a eu-west-2a-master2 Master t2.medium 1 1 eu-west-2a eu-west-2a-master3 Master t2.medium 1 1 eu-west-2a eu-west-2b-master1 Master t2.medium 1 1 eu-west-2b eu-west-2b-master2 Master t2.medium 1 1 eu-west-2b nodes Node t2.medium 2 2 eu-west-2a,eu-west-2b 

NODE STATUS 
NAME ROLE READY 
ip-10–100–0–13.eu-west-2.compute.internal master True ip-10–100–0–170.eu-west-2.compute.internal node True ip-10–100–0–21.eu-west-2.compute.internal master True ip-10–100–0–56.eu-west-2.compute.internal master True ip-10–100–1–170.eu-west-2.compute.internal master True ip-10–100–1–23.eu-west-2.compute.internal node True ip-10–100–1–238.eu-west-2.compute.internal master True 

Your cluster test.k8s.appvia.io is ready

COPY

Now, we need to edit the kops ClusterSpec, defining a new subnet that would be located within the 3rd Availability Zone:  kops edit cluster test.k8s.appvia.io

Jump down to the ‘subnets’ section, adding the new subnet into the list.

At the same time, you can define new Master instances to be provisioned on the AZ-C Subnet that will be created. To maintain a quorum for etcd, you need to provision and persist an odd number of Master instances (3, 5, 7 etc) at all times. The kops utility will prevent you from creating an odd number of Master instances.

For this example, I created an additional two instances to reside within AZ-C, bringing the total to seven. We’ve used a CoreOS AMI to provision all the nodes, but feel free to change this to whatever is more suitable for your environment.

➜ cat euw2c-master1.yaml
apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
 labels:
 kops.k8s.io/cluster: test.k8s.appvia.io
 name: eu-west-2c-master1
spec:
 image: coreos.com/CoreOS-stable-1576.5.0-hvm
 machineType: t2.medium
 maxSize: 1
 minSize: 1
 role: Master
 subnets:
 — eu-west-2c

➜ kops create -f euw2c-master1.yaml
Created instancegroup/eu-west-2c-master1
To deploy these resources, run: kops update cluster test.k8s.appvia.io --yes

➜ cat euw2c-master2.yaml
apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
 labels:
 kops.k8s.io/cluster: test.k8s.appvia.io
 name: eu-west-2c-master2
spec:
 image: coreos.com/CoreOS-stable-1576.5.0-hvm
 machineType: t2.medium
 maxSize: 1
 minSize: 1
 role: Master
 subnets:
 — eu-west-2c

➜ kops create -f euw2c-master2.yaml
Created instancegroup/eu-west-2c-master2
To deploy these resources, run: kops update cluster test.k8s.appvia.io --yes

COPY

Before deploying these new Instance Groups (IGs), one more update needs to take place within the ClusterSpec to define new associated etcd members:kops edit cluster test.k8s.appvia.io

Jump down to the etcdClusters section, adding etcd-main and etcd-events EBS volumes for the two new IGs. 

Finally, you can run a kops update to interact with the AWS API, creating the new AWS resources that have been defined in the Cluster Spec:

➜ kops update cluster --yes Using cluster from kubectl context: test.k8s.appvia.io I0123 15:27:15.413444 11637 executor.go:91] Tasks: 0 done / 118 total; 46 can run I0123 15:27:15.863181 11637 executor.go:91] Tasks: 46 done / 118 total; 29 can run I0123 15:27:16.374682 11637 executor.go:91] Tasks: 75 done / 118 total; 27 can run I0123 15:27:17.905691 11637 executor.go:91] Tasks: 102 done / 118 total; 9 can run I0123 15:27:18.035842 11637 dnsname.go:111] AliasTarget for "api.test.k8s.appvia.io." is "api-test-k8s-fvii3v-1272530458.eu-west-2.elb.amazonaws.com." I0123 15:27:18.153972 11637 executor.go:91] Tasks: 111 done / 118 total; 7 can run I0123 15:27:18.856931 11637 executor.go:91] Tasks: 118 done / 118 total; 0 can run I0123 15:27:18.856995 11637 dns.go:153] Pre-creating DNS records I0123 15:27:19.232648 11637 update_cluster.go:253] Exporting kubecfg for cluster kops has set your kubectl context to test.k8s.appvia.io Cluster changes have been applied to the cloud. Changes may require instances to restart: kops rolling-update cluster

COPY

At this point, you will have:

  • Created one new Subnet within your VPC
  • Created two new Master Instance Groups, residing within the subnet named ‘eu-west-2c’
  • Created four new EBS volumes, 2x ‘etcd-main’ && ‘etcd-events’, for the new Instance Groups

Unfortunately the new Master instances will not automatically join the etcd cluster without some manual intervention. The etcd member list will need updating to include hosts for the two new instances, and then these new instances will require some reconfiguration to join the existing cluster.

Firstly, SSH into an existing Master instance. Once in the instance, run these commands

If etcdctl is not already present on the Host OS, you can download it from here (grab the same version as what you’re running): 

➜ ${ETCD_MAIN} member add etcd-c1 --peer-urls="https://etcd-c1.internal.test.k8s.appvia.io:2380" Member 356cb0ee3ba9d1f6 added to cluster 5024708754869ab3 ➜ ${ETCD_MAIN} member add etcd-c2 --peer-urls="https://etcd-c2.internal.test.k8s.appvia.io:2380" Member 4e65c79f2332a2ae added to cluster 5024708754869ab3 ➜ ${ETCD_EVENTS} member add etcd-events-c1 --peer-urls="https://etcd-events-c1.internal.test.k8s.appvia.io:2381" Member fa29d33b3e164931 added to cluster df1655d3f512ef29 ➜ ${ETCD_EVENTS} member add etcd-events-c2 --peer-urls="https://etcd-events-c2.internal.test.k8s.appvia.io:2381" Member de67c45c59dc7476 added to cluster df1655d3f512ef29

COPY

Now that the new members have been added, they should initially be listed as unstarted

At this point, the etcd Clusters have been configured to register the new clients once they become available on their respective addresses. The newly provisioned Master instances now require amendments to their etcd configuration to be able to join the existing Clusters.

Next, log-in via SSH to the new Master instances and perform these commands

Validate that the members have started successfully and joined the cluster by running a ‘member list’ command on any of the master nodes that you are connected to. 

Once the etcd clusters have recovered as above, you can start the protokube service back up on the Master instances within the new Availability Zone: systemctl start protokube

Your cluster should eventually validate successfully via kops after a few minutes once the new Master instances have caught up:

➜ kops validate cluster Using cluster from kubectl context: test.k8s.appvia.io Validating cluster test.k8s.appvia.io 

INSTANCE GROUPS 
NAME ROLE MACHINETYPE MIN MAX SUBNETS 
eu-west-2a-master1 Master t2.medium 1 1 eu-west-2a eu-west-2a-master2 Master t2.medium 1 1 eu-west-2a eu-west-2a-master3 Master t2.medium 1 1 eu-west-2a eu-west-2b-master1 Master t2.medium 1 1 eu-west-2b eu-west-2b-master2 Master t2.medium 1 1 eu-west-2b eu-west-2c-master1 Master t2.medium 1 1 eu-west-2c eu-west-2c-master2 Master t2.medium 1 1 eu-west-2c nodes Node t2.medium 2 2 eu-west-2a,eu-west-2b 

NODE STATUS 
NAME ROLE READY 
ip-10-100-0-13.eu-west-2.compute.internal master True ip-10-100-0-170.eu-west-2.compute.internal node True ip-10-100-0-21.eu-west-2.compute.internal master True ip-10-100-0-56.eu-west-2.compute.internal master True ip-10-100-1-170.eu-west-2.compute.internal master True ip-10-100-1-23.eu-west-2.compute.internal node True ip-10-100-1-238.eu-west-2.compute.internal master True ip-10-100-2-118.eu-west-2.compute.internal master True ip-10-100-2-140.eu-west-2.compute.internal master True 

Your cluster test.k8s.appvia.io is ready

COPY

Finally, you can now configure your node pools to make use of the new subnet. Run kops edit ig <nodes-ig> and add the new Availability Zone

One more kops update cluster test.k8s.appvia.io --yes and you’re good to go.

About Appvia

Appvia enables businesses to solve complex cloud challenges with products and services that make Kubernetes secure, cost-effective and scalable.

Our founders have worked with Kubernetes in highly regulated, highly secure environments since 2016, contributing heavily to innovative projects such as Kops and fully utilizing Kubernetes ahead of the curve. We’ve mastered Kubernetes, and experienced its complexities, so our customers don’t have to. 

Share this article
Twitter
LinkedIn
Facebook
profile-112x112-crop-1
Kashif Saadat
SRE LEAD
Throughout my career I’ve moved across a few roles covering customer support, test automation, software development and platform engineering. In my spare time I chase around my fearless baby as she climbs up walls, kitchen cabinets, furniture and windows trying to give me a heart attack.

The podcast that takes a lighthearted look at the who, what, when, where, why, how and OMGs of cloud computing

Related insights

Managing Kubernetes Secrets with HashiCorp Vault vs. Azure Key Vault Keeping secrets secure...
Namespaces are a vital feature of Kubernetes. They allow you to separate uniquely named...
DevOps teams have rapidly adopted Kubernetes as the standard way to deploy and...
Once you start working with Kubernetes, it’s natural to think about how you...
Self-service of cloud resources Kubernetes has been brilliant at delivering an ecosystem for...
Pods, deployments, and services are just some of the concepts that you need to understand in...
Last week I published a blog, “How to spot gaps in your Public Cloud...
Breaking down the core areas that you should be aware of when considering...
5 tips to help you manage more with less Not every manager of...
Public cloud has provided huge benefits in getting infrastructure and services to people...
This is the story of how three Appvia Engineers contributed so much to...
Overview The UK Home Office is a large government organisation, whose projects and...