Managing Kubernetes Secrets with HashiCorp Vault vs. Azure Key Vault

Table of Contents

Managing Kubernetes Secrets with HashiCorp Vault vs. Azure Key Vault

Keeping secrets secure on Kubernetes is crucial for every organization. However, it’s easier said than done.

Although Kubernetes has its own mechanism to manage secrets, it’s not up to current security challenges due to the way Kubernetes handles secrets:

  • By design, Kubernetes saves secrets in the cluster where they are created. This complicates secrets management when working with multiple clusters.
  • Kubernetes encodes secrets in base64, which is far from the level of security offered by current encryption algorithms.
  • All secrets are stored in the etcd backend. The problem with this approach is that etcd does not account for auditing who uses each secret, when each secret is used, or what each secret is used for. Also, etcd doesn’t have a mechanism that allows secrets to be rotated periodically, nor does it have a security mechanism to prevent anyone with root access to the node from accessing the data.

While these are not the only drawbacks of the native Kubernetes mechanism for handling secrets, they are the most critical, as they defy best security practices. There are four basic aspects that should be implemented to handle secrets securely:

  • A centralized location where secrets are encrypted before being stored. This guarantees more convenient and secure management of multiple clusters and environments.
  • The ability to implement access control to said secrets, which is essential to protect sensitive resources.
  • Audit mechanisms that allow for monitoring the lifecycle of each secret, which would enable the detection and resolution of any possible intrusion.
  • Mechanisms that allow secrets to be rotated periodically, thus limiting the usefulness of a secret that is compromised.

The simplest way to fulfill these requirements is through the use of third-party secret managers such as HashiCorp Vault and Azure Key Vault. Both solutions exceed the minimum security features listed above, but they use very different approaches to do so. In this article, you’ll take a look at the pros and cons of each tool so that you can decide which one best suits your needs and requirements.

How Will the Solutions Be Compared?

With any comparison, it’s important to set clear parameters for how each tool will be judged. In this article, each solution will be evaluated by taking into account the following:

  • How it works: You’ll look at things such as what architecture is used, how flexible the tool’s approach is, and how each solution actually injects secrets into Kubernetes.
  • Security: Security is the primary reason to use a secrets broker. It’s important to know how the lifecycle of secrets is managed, as well as possible security issues with these solutions.
  • Ease of use: The last thing developers need is a service that’s difficult to use or that adds a lot of administrative overhead. To address this point, you’ll explore how convenient each tool is for handling secrets.
  • Kubernetes integration: Last but not least, the level of integration with Kubernetes is obviously another key factor in the process of evaluating each tool.

Azure Key Vault

The following are the main features of Azure Key Vault (AKV):

  • Service type: AKV is a paid cloud-based service. It’s managed by Microsoft and is part of its Azure ecosystem.
  • Secrets management: AKV is designed for securely storing secrets, authentication keys, storage account keys, passwords, tokens, API keys, and more.
  • Data encryption: All data stored in AKV is encrypted at rest.
  • Certificate management: The AKV service can be used to manage the encryption keys for data encryption. It can also provision, manage, and deploy SSL and TLS certificates for use with Azure-integrated services such as Azure Kubernetes Service (AKS).
  • Access policy: The service allows you to set granular access policies for users, applications, and services.
  • Audit trails: Key usage details are logged and can be used for auditing.
  • Key revocation: AKV allows you to revoke or delete any key or secret, and lets you set automated key rotation.

How Does It Work?

Key Vault is a service designed to seamlessly integrate with the rest of the Azure ecosystem. In that sense, it’s not surprising that you must create an Azure account to access it. Here’s a high-level overview of how to use AKV:

  • Create a key vault and secret, either using the command line or the Azure dashboard UI.
  • Enable the Azure Key Vault Provider for Secrets Store CSI Driver. This will allow AKS to get secrets, keys, and certificates stored in AKV and mount them to pods using a Container Storage Interface (CSI) volume.
  • Grant the appropriate identity permissions to read the key vault and view its contents. You can do this from the command line or the AKV UI.
  • Define and create a secret provider class, in which all the secrets that should be imported from the key vault are included.
  • Create a pod with the secrets. This pod will use the newly created secret provider class to mount the secrets in the pod using a CSI volume.

The above approach allows secrets to be stored at rest in Azure Key Vault in a secure, encrypted manner. Moreover, the secrets are only stored in the AKS cluster when the pod with the secrets is running. Once the pod is removed, the secrets are also removed.

It should be noted that the Secrets Store CSI driver provides several methods to manage the identities of those who can access your Azure Key Vault. These methods include Azure Active Directory (Azure AD) pod-managed identities, user-assigned managed identities, and system-assigned managed identities. For more information on how to provide an identity to access the Azure Key Vault Provider for Secrets Store CSI Driver, you can take a look at the documentation.

Security

Azure Key Vault is a fully managed service, which means that the encryption of secrets is handled by Microsoft Azure. The service offers two tiers of protection:

  • Standard tier: Secrets are encrypted using FIPS 140-2 level 2 compliant software.
  • Premium tier: Secrets are encrypted using FIPS 140-2 level 3 compliant hardware security modules (HSM).

It’s important to note that both tiers offer the security features mentioned previously, such as ACLs, secret handling, automated secret rotation, and audit trails. In short, unless your organization is required to comply with the FIPS 140-2 level 3 standard, the standard tier is sufficient.

That said, there are no known security issues that would make AKV prone to data leaks, which just leaves the number one cause of security flaws: users. Microsoft recommends using separate key vaults for each application, as well as setting strict access controls on secrets. For more information on using Azure Key Vault effectively, you can look at the best practices section of the documentation.

Ease of Use

Setting up Azure Key Vault is straightforward. After creating your Azure account, you’ll be able to access the Azure portal:

kifj0es

From there, type “vault” in the search box and click Key vaults to access the AKV dashboard:

6tu51j6

The Azure Key Vault dashboard allows you to perform tasks such as creating, managing, and deleting vaults. Click Create key vault to create your first vault:

nu1z1os

On the next screen, you’ll be able to assign a name to the vault, select a region and pricing tier, and assign access policies. You can learn more about creating a key vault using the Azure portal in the quickstart guide.

dc22xnj

Once your vault is created, you can call it by name to assign permissions in AKS, create SecretProviderClass, and create pods that expose the secrets. For detailed instructions on how to use Azure Key Vault with AKS, read the documentation. Below are examples of what the code looks like.

Using the Azure CLI, you can assign permissions to new identities using code similar to the following:

# set policy to access keys in your key vault
az keyvault set-policy -n <keyvault-name> --key-permissions get --spn <pod-identity-client-id>
# set policy to access secrets in your key vault
az keyvault set-policy -n <keyvault-name> --secret-permissions get --spn <pod-identity-client-id>
# set policy to access certs in your key vault
az keyvault set-policy -n <keyvault-name> --certificate-permissions get --spn <pod-identity-client-id>

COPY

The following is an example of a SecretProviderClass that uses the name of the previously created vault:

# This is a SecretProviderClass example using aad-pod-identity to access the key vault
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname-podid
spec:
  provider: azure
  parameters:
    usePodIdentity: "true"               # Set to true for using aad-pod-identity to access your key vault
    keyvaultName: <key-vault-name>       # Set to the name of your key vault
    cloudName: ""                        # [OPTIONAL for Azure] if not provided, the Azure environment defaults to AzurePublicCloud
    objects:  |
      array:
        - |
          objectName: secret1
          objectType: secret             # object types: secret, key, or cert
          objectVersion: ""              # [OPTIONAL] object versions, default to latest if empty
        - |
          objectName: key1
          objectType: key
          objectVersion: ""
    tenantId: <tenant-Id>                # The tenant ID of the key vault

COPY

Below is an example of a pod that mounts the secrets to AKS:

# This is a sample pod definition for using SecretProviderClass and aad-pod-identity to access the key vault
kind: Pod
apiVersion: v1
metadata:
  name: busybox-secrets-store-inline-podid
  labels:
    aadpodidbinding: <name>                   # Set the label value to the name of your pod identity
spec:
  containers:
    - name: busybox
      image: k8s.gcr.io/e2e-test-images/busybox:1.29-1
      command:
        - "/bin/sleep"
        - "10000"
      volumeMounts:
      - name: secrets-store01-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
  volumes:
    - name: secrets-store01-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-kvname-podid"

COPY

As you can see, the process can be handled programmatically and doesn’t hinder the workflow of the administrators at all.

Kubernetes Integration

In the previous section, you saw how Azure Key Vault interacts with AKS. As for AKV integration with Kubernetes, although Key Vault is not a Kubernetes-specific solution, it’s designed to offer a high degree of integration with the Azure ecosystem. This allows AKV to work hand in hand with Active Directory and AKS, Microsoft’s Kubernetes distribution.

The above raises the question of whether AKV can be used outside of AKS. The short answer is yes, but while Key Vault offers built-in integration for Azure services, including AKS, it requires a bit more work to use with other Kubernetes distributions. To use it outside of the Azure environment, you’ll probably have to use tools like akv2k8s, which doesn’t use the Azure Key Vault Provider for Secrets Store CSI Driver to inject secrets into Kubernetes. For more about how to install Azure Key Vault to Kubernetes outside of AKS using akv2k8s, you can take a look at the documentation

HashiCorp Vault

The following are the main features of HashiCorp Vault:

  • Service type: HashiCorp Vault is an open source project available as a self-managed binary file or as a fully managed Vault cluster on HashiCorp Cloud Platform (HCP).
  • Secrets management: HashiCorp Vault can be used to securely store all kinds of secrets.
  • Data encryption: All data stored in HashiCorp Vault is encrypted at rest.
  • Certificate management: HashiCorp Vault is intended to dynamically serve credentials, offloading the encryption process to Vault.
  • Access policy: The service allows you to set granular access policies for users, applications, and services.
  • Audit trails: Key usage details are logged and can be used for auditing.
  • Key revocation: HashiCorp Vault allows you to revoke or delete any key or secret, and lets you set automated secret rotation.

How Does It Work?

Unlike Azure Key Vault, HashiCorp Vault uses an open, flexible model that gives you more control over secrets. To achieve this, Vault uses a highly pluggable architecture that allows you to choose which backends to use for authentication, auditing, and secret storage. For example, you could authenticate using LDAPGitHub, or Kubernetes, along with using Splunk to handle audit logs, and MySQLConsul, or Google Spanner to store encrypted secrets at rest. This means that one major advantage of HashiCorp Vault is that it minimizes vendor lock.

HashiCorp Vault’s flexibility can be interpreted as an advantage or a disadvantage depending on your goals, requirements, and use case. While it offers more control, there’s more administrative overhead than AKV. The following is a brief review of how Vault works.

You must set up Vault before you can create secrets. At this point, the differences with AKV begin. In addition to binary packages aimed at local development, Vault can be deployed in multiple ways as listed below. You can:

As you can see, you can even use HashiCorp Vault on AKS if you wish. Moreover, you can go further, injecting secrets into Kubernetes pods via Vault Agent sidecar containers, or mounting Vault secrets through Container Storage Interface (CSI) volumes.

Once Vault is configured, the app, service, or user can be authenticated using almost any agent. Vault can also use a wide variety of secrets engines, and supports different methods for handling audit logs.

HashiCorp Vault uses external components for authentication, audit logs, and secret storage. This is a big difference from AKV, which takes care of everything except authentication.

Security

By design, Vault is set up to handle the entire lifecycle of secrets, assign access control, and keep an audit trail. Moreover, Vault allows you to use dynamic secrets that are served through its API to ensure that encryption is properly implemented. This means that Vault can be used for both injecting secrets into Kubernetes and managing service and user authentication certificates. In either scenario, the dynamic nature of secret generation allows for best security practices, such as short-lived certificates.

Despite the fact that the secrets are encrypted at rest and that Vault is considered a secure platform, there are other risks that need to be considered. These risks have to do with the components used in Vault. It’s been made clear that unlike AKV, Vault is not a standalone service. No matter how secure Vault is, a bad implementation of the secrets engine or authentication agent can compromise your data. If you prefer the convenience and safety of a managed service, Vault on the HashiCorp Cloud Platform is the way to go.

Ease of Use

One aspect that HashiCorp Vault shares with AKV is its ease of use. You can interact with Vault from either the command line or the UI. To illustrate this, you can deploy the Vault dev server for macOS using the following commands:

brew tap hashicorp/tap
brew install hashicorp/tap/vault

COPY

Once installed, you can start the server using the following command:

vault server -dev

COPY

The output should be similar to the following:

lgznarv

Now you just need to copy the token and point your browser to http://127.0.0.1:8200 to connect to the Vault UI:

usmrima

Use the token to enter the Vault dashboard:

q3vrdzo

One of the first steps to using Vault is to set up a secrets engine, which you can do from the dashboard:

mogghyt

In the image above, you can see most of the secrets engines supported by Vault. For more information on how to get started with Vault, take a look at the documentation.

All in all, Vault’s web UI is easy to use, and doesn’t introduce any major complications to the developer’s workflow. That said, you should remember that it can be time consuming to configure the rest of the components that Vault needs.

Kubernetes Integration

HashiCorp Vault isn’t specifically targeted at Kubernetes. However, as you saw earlier, the level of integration with different Kubernetes distributions, including AKS, is exceptional. You can create a Vault cluster using Helm and then inject secrets using sidecars or the Kubernetes Secrets Store CSI driver. For comparison, here is an example definition of the SecretProviderClass:

cat > spc-vault-database.yaml <<EOF
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-database
spec:
  provider: vault
  parameters:
    vaultAddress: "http://vault.default:8200"
    roleName: "database"
    objects: |
      - objectName: "db-password"
        secretPath: "secret/data/db-pass"
        secretKey: "password"

COPY

As you can see, the most notable difference from AKV is that secrets are not accessed by name, but rather through API calls to Vault. But is this approach better? And more importantly, does HashiCorp Vault offer better integration with Kubernetes than AKV?

The answer to both questions is that it depends.

HashiCorp Vault is highly adaptable, allowing it to seamlessly integrate with any Kubernetes deployment. However, depending on your use case, you may not mind tying yourself to the Azure ecosystem and enjoying its integration with AKV.

Conclusion

In this article, you compared two best-in-class solutions that address the problem of handling secrets in Kubernetes. Both Azure Key Vault and HashiCorp Vault overcome all the drawbacks of the default Kubernetes solution for handling secrets. Both tools are effective at encrypting secrets at rest, offering access control, secrets management, audit trails, and more. Here’s a side-by-side comparison:

  • How it works: Each tool uses different approaches. AKV is a service primarily aimed at Azure users, while HashiCorp Vault is an open source solution that allows you to choose components at your convenience. That said, both tools use top-tier authentication agents and are designed following best practices. It’s impossible to say that one is better than the other.
  • Security: It’s difficult to declare a winner in this category, too. In terms of the tools themselves, both are highly secure. HashiCorp’s self-managed Vault involves more moving parts, which introduces complexity and potential security risks. Vault on the HashiCorp Cloud Platform, though, is much more straightforward, and on similar footing as AKV.
  • Ease of use: This category is a tie. Both solutions are easy to use and manage.
  • Kubernetes integration: This category comes down to your use case. HashiCorp Vault is incredibly versatile, as it offers out-of-the-box integrations for major Kubernetes distributions. Azure Key Vault, on the other hand, integrates effortlessly with the Azure ecosystem.

Ultimately, the question of which solution is better comes down to your vision and needs. If you prefer an out-of-the-box solution based on Azure, then Key Vault is a great fit for your needs. If you’d rather not be tied to a specific vendor, HashiCorp’s self-managed Vault is the perfect solution for you.

About Appvia

Appvia enables businesses to solve complex cloud challenges with products and services that make Kubernetes secure, cost-effective and scalable.

Our founders have worked with Kubernetes in highly regulated, highly secure environments since 2016, contributing heavily to innovative projects such as Kops and fully utilizing Kubernetes ahead of the curve. We’ve mastered Kubernetes, and experienced its complexities, so our customers don’t have to. 

Share this article
Twitter
LinkedIn
Facebook
profile-112x112-crop-1 (9)
Damaso Sanoja
Damaso Sanoja is a mechanical engineer with a passion for cars and computers. He’s written for both industries for more than two decades.

The podcast that takes a lighthearted look at the who, what, when, where, why, how and OMGs of cloud computing

Related insights

Managing Kubernetes Secrets with HashiCorp Vault vs. Azure Key Vault Keeping secrets secure...
Namespaces are a vital feature of Kubernetes. They allow you to separate uniquely named...
DevOps teams have rapidly adopted Kubernetes as the standard way to deploy and...
Once you start working with Kubernetes, it’s natural to think about how you...
Self-service of cloud resources Kubernetes has been brilliant at delivering an ecosystem for...
Pods, deployments, and services are just some of the concepts that you need to understand in...
Last week I published a blog, “How to spot gaps in your Public Cloud...
Breaking down the core areas that you should be aware of when considering...
5 tips to help you manage more with less Not every manager of...
Public cloud has provided huge benefits in getting infrastructure and services to people...
This is the story of how three Appvia Engineers contributed so much to...
Overview The UK Home Office is a large government organisation, whose projects and...