Secrets management is a very important part of continuous deployment and infrastructure as code. At Zapier, we started adopting Kubernetes in late 2017. Back then, we used to use Helm and some custom scripts to deploy our applications to Kubernetes from Helm configs in a git repository. We used to use the Helm secrets plugin to manage our secrets in the Helm configs git repository. Learn why we changed from using the Helm secrets plugin to sops-secrets-operator and finally to Vault.
In 2017, we started adopting Kubernetes at Zapier. At that time, there was not much stable tooling around to continuously deploy applications using Helm to Kubernetes. So, we wrote a few Python scripts to deploy applications from helm configs to Kubernetes. These scripts were wired to our CI/CD solution—which was then Jenkins—to run whenever changes got merged to the main branch of the Helm configs repo. To store secrets in our Helm configs in a secure way, we used a Helm plugin called helm-secrets to decrypt, edit and encrypt our secrets in the Helm configs git repository. The
helm-secrets plugin uses SOPS under the hood to encrypt or decrypt secrets using various key providers like PGP, AWS or GCP KMS, etc.
Back in those days, many community Helm charts used to accept secrets in Helm config values. So, we followed the same pattern in our in-house Helm charts. With
helm-secrets plugin, we could dump a part of the helm config values inside a
secrets.yaml file and then encrypt it using helm-secrets before pushing it to git. We also used a tool called helm-wrapper as an alias for Helm CLI and it’d take care of automatically decrypting
secrets.yaml file during application of helm charts from the Helm config files. The developer workflow with helm-secrets was very simple
# Edit secrets file and encrypt afterward $ helm secrets edit path/to/app/cluster-01/secrets.yaml
The CI/CD tool had permission to decrypt the secrets in the git repo and apply the Helm chart for the applications to deploy changes to Kubernetes. This worked pretty well until we started exploring the use of external or managed CI/CD tools. In terms of good security practices, it’s not ideal for tools in the deployment pipeline to be aware of application secrets. With helm-secrets, the CI/CD tool needed to have permission to decrypt the secrets to deploy the application to Kubernetes.
Simple and easy to use. Works out of the box with most upstream Helm charts.
CI/CD tooling needs to have permissions to decrypt secrets
Unencrypted secrets might get committed to repo by mistake
SOPS secrets operators
In 2018, we started migrating off from our custom Helm tooling to ArgoCD to continuously deploy our applications to Kubernetes from our git repositories. Initially, we had tried to customize ArgoCD images to replace the shipped helm binary with helm-wrapper along with
helm-secrets plugin support. However, our initial attempts were not successful and we eventually found a better solution.
That’s when we came across sops-secrets-operator, which manages Kubernetes Secret Resources created from user-defined
SopsSecret custom resource objects using
sops. Here, we’d first create a
SopsSecret custom resource object inside the
templates directory of our application's Helm chart, something like
apiVersion: isindir.github.com/v1alpha3 kind: SopsSecret metadata: name: example-sopssecret spec: secretTemplates: - name: my-secret-name-1 labels: label1: value1 annotations: key1: value1 stringData: data-name0: data-value0 data: data-name1: ZGF0YS12YWx1ZTE=
Then, we’d encrypt the above
SopsSecret YAML file with the help of
$ sops -e --encrypted-suffix=secret_templates path/to/app/templates/sopssecrets.yaml
We’d then reference the secret name generated from the above
SopsSecret in our application’s Helm config, and then commit our changes and push to git. To modify the secrets, we’d do the following
# Decrypt secrets $ sops -d --encrypted-suffix=secret_templates path/to/app/templates/sopssecrets.yaml # Make changes to secrets # Encrypt secrets $ sops -e --encrypted-suffix=secret_templates path/to/app/templates/sopssecrets.yaml # Commit changes and push
In this workflow, ArgoCD does not need to decrypt the SopsSecret object. The
sops-secrets-operator which has the necessary permission to decrypt the secret decrypts the SopsSecret objects to create Kubernetes native
We have been successfully using
sops-secrets-operator at Zapier for a few years now.
Intermediate CI/CD tooling does not need to have access to decrypt secrets
Works on simple and proven technology like
Secrets are encrypted and stored in
Duplicate secrets for each Kubernetes cluster. When secrets are updated, need to update the
SopsSecretartifacts for each cluster
There's a chance that a user might, by mistake, commit and push an unencrypted
No granular access control to secrets. Since we use the same KMS key to encrypt/decrypt all
SopsSecretartifacts in the Argo configs repo, all users with access to the repo can view secrets for other applications they don't own.
In early 2021, we deployed Vault at Zapier, and we started fiddling with options to use Vault for managing secrets in Zapier. Among many other things, Vault provided us with a centralized place for managing secrets, a user-friendly UI and CLI, and granular access control. From our OIDC provider and service catalog, we have automated setting up separate secret engines for teams and giving team members necessary access to the team's secret engines.
To have a quick and simple replacement of
sops-secrets-operator, we used external-secrets to manage secrets in Kubernetes. Similar to our above workflow with
SopsSecret artifacts, we'd add
ExternalSecret artifact(s) in the
templates directory of the application's Helm chart. The
external-secrets operator will reconcile these
ExternalSecret objects and create native Kubernetes
Secret objects by fetching the secrets from Vault. An example
ExternalSecret will look something like the below.
apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: vault-example spec: refreshInterval: "15s" secretStoreRef: name: vault-backend kind: SecretStore target: name: example-sync data: - secretKey: foobar remoteRef: key: secret/foo property: my-value --- # will create a secret with: kind: Secret metadata: name: example-sync data: foobar: czNjcjN0
The major win in
ExternalSecret definition over
SopsSecret is that you only have references to the actual secret in Vault, rather than the actual encrypted secret value. This means that if a secret gets updated, you won't need to update these
ExternalSecret artifacts for all your Kubernetes clusters. You will only need to update these
ExternalSecret artifacts when you add or remove secret keys.
However, the downside to our
external-secret setup is that the
external-secret service account has access to read secrets for all apps in Vault. To work around this shortcoming, we are exploring Vault CSI provider so that we can define access to specific Vault secret engines and paths based on Kubernetes service accounts and namespaces. Similar to Kubernetes secrets, on pod start and restart, the Secrets Store CSI driver communicates with the provider using gRPC to retrieve the secret content from the external Secrets Store specified in the SecretProviderClass custom resource. Then the volume is mounted in the pod as
tmpfs and the secret contents are written to the volume.
We are finding Vault to be a quality secret management solution across various services and applications at Zapier, be it in Kubernetes or not.
Granular access control for developers and services to view and update secrets
Out of the box Kubernetes authentication mapping Kubernetes service accounts to Vault roles
Central place to manage secrets in a secure way
No duplicate secrets for each Kubernetes cluster.
No chance for unencrypted secrets getting committed to repo by mistake. Secret artifacts are readable and contain only references to Vault paths.
We define secrets (in Vault) and use secrets (in ExternalSecret, CSI Secret) in different places. Developers, initially, find it confusing
That's how and why we moved from Helm secrets to SOPS secrets operator to Vault for secrets management in Kubernetes at Zapier. We are thankful to the creators & maintainers of the awesome Open Source tools mentioned above for supporting our journey of secrets management in Kubernetes. We have grown and learned many things during this journey, and we will keep evolving our secret management practices for Kubernetes at Zapier to stay up to date with the best security practices and tools.