Secret management problems

In the running process of our application, we usually need some secret, such as database password, third-party API key, private key, etc. To avoid security risks, this data is not submitted directly into the version management system along with the code. There are usually a few simple ways to manage these secrets:

  1. Configure the system environment variables on the machine to be deployed
  2. Put Secret ina CI/CD system, package it at software build, and deploy it to the server with Binary

All the above schemes have the following disadvantages:

  1. Tedious and error-prone, you need to manually SSH to the server or modify in the CI/CD system.
  2. Without versioning, you can’t see who made what changes, and you can’t easily revert. If manual changes go wrong and there is no backup, you may not be able to restore.

Technology selection

There are many secret management solutions on the market. We have investigated SOPS, Vault, AWS Secrets Manager and other products.

Vault was so complex that it was a “cannon on a mosquito” scenario for us. AWS Secrets Manager is pretty easy to use, but it doesn’t work well with Kubernetes and introduces “cloud vendor lock-in” issues.

SOPS takes into account the coordination between simplicity and K8S, which is exactly what we need.

The introduction of SOPS

SOPS is a secret management tool developed by Mozilla. It encrypts YAML, JSON, ENV, INI, and binary files using AWS KMS, GCP KMS, Azure Key Vault, and PGP. Once the secret files are encrypted by SOPS, we can safely store them in version management.

Our company chooses to use AWS KMS service. Yubikey needs to be inserted into users’ computers to pass secondary authentication before using SOPS encryption and decryption, so as to enhance security. AWS KMS can also configure different permissions in IAM according to personnel. For example, user Tony can only decrypt secret of the Development environment of Example application.

After using SOPS, the Secret management workflow within our company became:

  1. Modify Secret locally using SOPS
  2. Commit to version management
  3. CI/CD decrypts Secret and deploys it to the server

Compared to the previous solution, we added Secret directly to the version management and gained the following benefits:

  1. Easy to manage, we can make changes locally with a single command, and then commit.
  2. Easy to backtrack and audit, you can see secret version history, and you can quickly revert errors.
  3. Synchronization of code and Secret (if we made incompatible changes such as secret name modification in the code before, operation and peacekeeping development need to coordinate code deployment and Secret change at the same time).

Of course, there are drawbacks. If your source code and the private key or credential used to encrypt your secret are compromised at the same time, an attacker can decrypt the secret’s plaintext.

However, there is a risk in any scheme, which requires the loss of several pieces of information at the same time to cause a leak. It is certainly better than storing secret in the code repository in clear text, as we can see from the previous code leaks in China, there are many companies that do this.

Deploy Secret to Kubernetes

All the applications of our company are deployed on K8S. It is too risky to package the decrypted Secret into the image when the application is packaged. Once the Docker image is leaked, the consequences will be unimaginable. And developers have permission to pull the image application for local development, but they should not have permission to view the secret.

Fortunately, the community has taken this issue into account. If you are also using Helm to manage K8S applications, you can integrate SOPS with Helm via the Helm-Secrets plugin. Helm value will be directly encrypted during development, helm-secret will automatically help us decrypt the value during Helm Apply.

We just need to declare the Secret resource of K8S in Chart template, and our Secret can be deployed as K8S Secrets. Secret is then mounted to Pod in the form of an environment variable or file that can be read by the application.

conclusion

This article is only our company’s practical experience in using SOPS to manage Secret. Thanks to the simplicity and convenience of SOPS, you can also explore how to use SOPS for other scenarios that need to manage sensitive information. Please remember to share your practical experience with us.

As the official documentation is detailed enough, we did not repeat the creation, please refer to the official documentation for a start.

Welcome to follow our wechat official account “RightCapital”