BLOGKubernetes

The Kubernetes Scaling Problem: People vs. Tools

Category
Kubernetes
Time to read
Published
February 20, 2024
Author

Key Takeaways

Understanding the roles of Workload Identities, Cluster Service Accounts, IAM Policies, and IAM Roles in managing access controls within AWS environments.

Exploring real-world use cases to illustrate the importance of effective IAM policy management in securing multi-tenant environments and aligning access controls with business requirements.

Comparing manual IAM policy management with streamlined approaches, such as Wayfinder's Package Workload Identities, to highlight the benefits of automation and centralised policy management.

Kubernetes is a rapidly growing, emerging technology within most industries, with significant levels of adoption already. According to the CNCF the use of containers in production has increased to 92%, which is up from 84% last year, and up 300% from their first survey in 2016.

Similarly, Kubernetes use in production has increased to 83%, which is up from 78% last year. It provides an operating model that is far more scalable than traditional infrastructure-based approaches and can provide a very elastic runtime for business-critical applications.

During the COVID19 pandemic, 68% of IT professionals have increased their use of Kubernetes.

Source: 2021 Kubernetes Adoption Report

The promise of scalability is a massive contributor to the rapid adoption of Kubernetes. There are still challenges to realising this value though. Historically, organisations have scaled their use of technologies through a combination of adding additional tools to support automation, but primarily through expanding the headcount of the teams tasked with the systems management and usage.

This is the fundamental problem when put in the context of Kubernetes.

Kubernetes is complex and relatively new

Because of that, the number of highly-qualified engineers available in the market is small.  Engineers are often motivated to work on ‘prestige' projects and when they possess an advanced skill that is in high demand, they're unlikely to join organisations that don't provide projects that match their expectations. This presents a challenge for the majority of organisations in hiring and retaining these people.

Kubernetes experts command LARGE salaries and are attracted to cutting edge organisations with disruptive products. This scarcity of resources has created a large problem for the average, established enterprise: Even if these resources can be attracted and hired, they are an inherent flight risk. If they aren't presented with sufficiently "interesting" problems and a commensurate salary, they will simply leave when a better offer is made.

Over 40% of organisations that have begun the process say that the inherent complexity and shortage of skills is preventing them from advancing.

Kubernetes adoption trajectory

The Kubernetes experience follows a common trajectory across most organisations, which is littered with pitfalls for inexperienced engineers.

The organisation starts off with no containers and no orchestration. Developers will try a POV/POC application and discover that it works relatively well. Then they will try containerising existing applications, where they’ll quickly find that major refactoring is needed and organisational behaviours will have to change to accommodate it.

Around this time, engineers may grow disillusioned with Kubernetes and ask if the conversion is really worth the effort.

So if scaling Kubernetes isn't practical by scaling the team, what can organisations that desire to use it do?

Seeing Kubernetes success with tooling

The answer is tooling and automation. When implemented effectively, containers and Kubernetes hold the key to unlocking the value of cloud, but the inherent complexity of the technologies (and the required changes) present barriers to operationalisation that are not easily overcome.

The Kubernetes technical capability comes ahead of usability, so users have a hard time fully leveraging the true potential of Kubernetes. This is true even for managed versions like AKS, EKS and GKE.

Appvia Wayfinder was designed to address these challenges so organisations can fully utilise Kubernetes and scale their business - like a Kubernetes team in a box. Our experience with Kubernetes in the context of high-growth, high-scale and highly regulated environments has been captured in the construction of the products, which come out-of-the-box with highly automated functions that encapsulate the skills and expertise that highly qualified engineers would possess.

Wayfinder is an abstraction layer that sits on top of the cloud providers, which simplifies the creation and maintenance of your Kubernetes clusters.

Over 40% of organisations that have started the process of Kubernetes adoption say that the inherent complexity and shortage of skills is preventing them from advancing. We’re here to make sure that doesn’t happen, helping organisations reach operationalised Kubernetes and make the promise of cloud a reality.  

Related Posts

Related Resources