--- title: "The journey of secrets in Kubernetes at Zapier" description: "Secrets management is a very important part of continuous deployment and infrastructure as code." --- # The journey of secrets in Kubernetes at Zapier Secrets management is a very important part of continuous deployment and infrastructure as code. Secrets management is a very important part of continuous deployment and infrastructure as code. At Zapier, we started adopting Kubernetes in late 2017. Back then, we used to use Helm and some custom scripts to deploy our applications to Kubernetes from Helm configs in a git repository. We used to use the [Helm secrets plugin](https://github.com/zendesk/helm-secrets) to manage our secrets in the Helm configs git repository. Learn why we changed from using the Helm secrets plugin to [sops-secrets-operator](https://github.com/isindir/sops-secrets-operator) and finally to [Vault](https://www.vaultproject.io/). ## Helm secrets In 2017, we started adopting Kubernetes at Zapier. At that time, there was not much stable tooling around to continuously deploy applications using Helm to Kubernetes. So, we wrote a few [Python scripts](https://zapier.com/blog/python-automation) to deploy applications from helm configs to Kubernetes. These scripts were wired to our CI/CD solution—which was then Jenkins—to run whenever changes got merged to the main branch of the Helm configs repo. To store secrets in our Helm configs in a secure way, we used a Helm plugin called [helm-secrets](https://github.com/zendesk/helm-secrets) to decrypt, edit and encrypt our secrets in the Helm configs git repository. The `helm-secrets` plugin uses [SOPS](https://github.com/mozilla/sops) under the hood to encrypt or decrypt secrets using various key providers like PGP, AWS or GCP KMS, etc. Back in those days, many community Helm charts used to accept secrets in Helm config values. So, we followed the same pattern in our in-house Helm charts. With `helm-secrets` plugin, we could dump a part of the helm config values inside a `secrets.yaml` file and then encrypt it using helm-secrets before pushing it to git. We also used a tool called [helm-wrapper](https://github.com/cradlepoint/helm-wrapper) as an alias for Helm CLI and it’d take care of automatically decrypting `secrets.yaml` file during application of helm charts from the Helm config files. The developer workflow with helm-secrets was very simple The CI/CD tool had permission to decrypt the secrets in the git repo and apply the Helm chart for the applications to deploy changes to Kubernetes. This worked pretty well until we started exploring the use of external or managed CI/CD tools. In terms of good security practices, it’s not ideal for tools in the deployment pipeline to be aware of application secrets. With helm-secrets, the CI/CD tool needed to have permission to decrypt the secrets to deploy the application to Kubernetes. ### Summary #### Pros - **Simple and easy to use.** Works out of the box with most upstream Helm charts. #### Cons - CI/CD tooling needs to have permissions to decrypt secrets - Unencrypted secrets might get committed to repo by mistake ## SOPS secrets operators In 2018, we started migrating off from our custom Helm tooling to [ArgoCD](https://argoproj.github.io/cd/) to continuously deploy our applications to Kubernetes from our git repositories. Initially, we had tried to customize ArgoCD images to replace the shipped helm binary with helm-wrapper along with `helm-secrets` plugin support. However, our initial attempts were not successful and we eventually found a better solution. That’s when we came across [sops-secrets-operator](https://github.com/isindir/sops-secrets-operator), which manages Kubernetes Secret Resources created from user-defined `SopsSecret` custom resource objects using `sops`. Here, we’d first create a `SopsSecret` custom resource object inside the `templates` directory of our application's Helm chart, something like Then, we’d encrypt the above `SopsSecret` YAML file with the help of `sops` using We’d then reference the secret name generated from the above `SopsSecret` in our application’s Helm config, and then commit our changes and push to git. To modify the secrets, we’d do the following In this workflow, ArgoCD does not need to decrypt the SopsSecret object. The `sops-secrets-operator `which has the necessary permission to decrypt the secret decrypts the SopsSecret objects to create Kubernetes native `Secret` objects. ### Summary We have been successfully using` sops-secrets-operator` at Zapier for a few years now. #### Pros - Intermediate CI/CD tooling does not need to have access to decrypt secrets - Works on simple and proven technology like `sops` - Secrets are encrypted and stored in `git` #### Cons - **Duplicate secrets for each Kubernetes cluster.** When secrets are updated, need to update the `SopsSecret` artifacts for each cluster - There's a chance that a user might, by mistake, **commit and push an unencrypted **`**SopsSecret**`** artifact to **`**git**` - **No granular access control to secrets.** Since we use the same KMS key to encrypt/decrypt all `SopsSecret` artifacts in the Argo configs repo, all users with access to the repo can view secrets for other applications they don't own. ## Vault In early 2021, we deployed Vault at Zapier, and we started fiddling with options to use Vault for managing secrets in Zapier. Among many other things, Vault provided us with a centralized place for managing secrets, a user-friendly UI and CLI, and granular access control. From our OIDC provider and service catalog, we have automated setting up separate secret engines for teams and giving team members necessary access to the team's secret engines. To have a quick and simple replacement of `sops-secrets-operator`, we used [external-secrets](https://github.com/external-secrets/external-secrets) to manage secrets in Kubernetes. Similar to our above workflow with `SopsSecret` artifacts, we'd add `ExternalSecret` artifact(s) in the `templates` directory of the application's Helm chart. The `external-secrets` operator will reconcile these `ExternalSecret` objects and create native Kubernetes `Secret` objects by fetching the secrets from Vault. An example `ExternalSecret` will look something like the below. The major win in `ExternalSecret` definition over `SopsSecret` is that you only have references to the actual secret in Vault, rather than the actual encrypted secret value. This means that if a secret gets updated, you won't need to update these `ExternalSecret` artifacts for all your Kubernetes clusters. You will only need to update these `ExternalSecret` artifacts when you add or remove secret keys. However, the downside to our `external-secret` setup is that the `external-secret` service account has access to read secrets for all apps in Vault. To work around this shortcoming, we are exploring [Vault CSI provider](https://www.vaultproject.io/docs/platform/k8s/csi) so that we can define access to specific Vault secret engines and paths based on Kubernetes service accounts and namespaces. Similar to Kubernetes secrets, on pod start and restart, the Secrets Store CSI driver communicates with the provider using gRPC to retrieve the secret content from the external Secrets Store specified in the SecretProviderClass custom resource. Then the volume is mounted in the pod as `tmpfs` and the secret contents are written to the volume. We are finding Vault to be a quality secret management solution across various services and applications at Zapier, be it in Kubernetes or not. ### Summary #### Pros - **Granular access control** for developers and services to view and update secrets - **Out of the box Kubernetes authentication** mapping Kubernetes service accounts to Vault roles - **Central place to manage secrets** in a secure way - **No duplicate secrets** for each Kubernetes cluster. - **No chance for unencrypted secrets getting committed to repo** by mistake. Secret artifacts are readable and contain only references to Vault paths. #### Cons - We define secrets (in Vault) and use secrets (in ExternalSecret, CSI Secret) in different places. Developers, initially, find it confusing ## Conclusion That's how and why we moved from Helm secrets to SOPS secrets operator to Vault for secrets management in Kubernetes at Zapier. We are thankful to the creators & maintainers of the awesome Open Source tools mentioned above for supporting our journey of secrets management in Kubernetes. We have grown and learned many things during this journey, and we will keep evolving our secret management practices for Kubernetes at Zapier to stay up to date with the best security practices and tools.