Managing Secrets through a CI/CD solution in AWS using Terraform and one source of truth
Overview
An automated solution for teams managing sensitive Cloud resources at scale.
Organisations embarking on the exciting journey to the Cloud need to evaluate the effectiveness and resilience of their IT Infrastructure secrets management in this new environment. They also need to understand the risks to business functions if they fail to engage thoroughly in this exercise.
Everyone is aware that sensitive data such as passwords, encryption keys, TLS certificates, must be securely protected, managed and stored. For most organisations, access to sensitive data should be also fully auditable.
In larger firms, the complexity of handling sensitive data in the Cloud will be increased significantly as responsibilities over secrets management will be shared across many teams, e.g. security, infrastructure, application development, operations and the business regulatory environment. In this case scenario, a one size fits all approach might be hard to achieve, but a secrets management strategy should be defined, approved, implemented and for obvious reasons enforced.
Although a wide range of options are now available to teams implementing their IT secrets management strategy, finding the best compromise between security, distribution, usability and resilience will typically bring some interesting challenges.
In this write-up, I will cover one method of managing secrets in AWS using Infrastructure as Code (Terraform) through a Continuous Integration and deployment solution (AWS CodePipeline and CodeBuild).
CodePipeline & CodeBuild secrets management
This solution will leverage native AWS services to run a pipeline with two stages (source & build) and triggered when an approved commit is made to an external Version Control System (VCS) or similar. It’s good practice to have a VCS branch protection rule designed to enforce an approving review and run automatic tests on the commits as part of our workflow.
For example, a Cloud / Application Engineer creates or updates encrypted secrets on a development branch, then raise a Pull request and obtains approval after review and passing status checks which allows her to merge the change to the release branch.
When the code is merged, the pipeline will pull the code from git, and launch a Codebuild container that will run terraform to create/update the secrets in AWS Secrets Manager.
How do we start?
Solid security foundations are a must, and if we are going to build a solution to handle secrets in the Cloud, we will need to do that in a secure environment. We’ll need a dedicated VPC, private subnets, tight security groups and using VPC endpoints where possible. It’s also a good idea to use an Identity Provider for Single Sign-On, and use identity attributes for fine-grained permission access to resources.
On AWS, IAM is the backbone of the access control strategy and using native services integrated closely with each other will facilitate and simplify permissions management at scale.
Pre-requisites
We’ll use Terraform to provision versioned, repeatable resources in AWS from a non-public VCS with access controls, pre-commit hooks, and branch protection rules before the code is merged. Terraform version 0.14 or later allows users to set the Sensitive property on its schema. This will prevent the field’s value from showing up in CLI output.
It’s important to note that the terraform’s Sensitive property will not encrypt or obscure the value in the terraform state files though, so a secured terraform backend for states files (encryption at rest and in transit) as well as a least privilege ACL roles/policies for AWS services and user roles need to be a hard requirement in order to manage secrets!
Two Open-source tools will be used: ejson to facilitate the encryption of secrets at scale in our pipeline, summon for the easy consumption of secrets on instances or containers.
ejson is a utility for managing a collection of secrets in source control. The secrets are encrypted using public key, elliptic curve cryptography (NaCl Box: Curve25519 + Salsa20 + Poly1305-AES). Secrets are collected in a JSON file, in which all the string values are encrypted. Public keys are embedded in the file, and the decrypter looks up the corresponding private key from its local filesystem.
summon is a command-line tool to make working with secrets easier.
It provides an interface for:
Reading a secrets.yml file
Fetching secrets from a trusted store
Exporting secret values to a sub-process environment
Note that teams consuming Open Source software will need to take responsibility to follow closely the open-source projects they use, and watch out for enhancements, bugs and security fixes. They should update their project with updated binaries when they become available.
Why use ejson for encryption?
It’s true that we could use a KMS key and custom scripts to manage the encryption of all our secrets on the git repo, but ejson is more suited to manage secrets at scale for the following reasons:
With ejson, changes to secrets are auditable on a line-by-line basis, as the json file is readable on the repo and only the secret ‘value’ is encrypted.
json is well supported by various tools and terraform which means we are able to structure the secrets across several files.
AWS KMS size limit for data is 4KB for asymmetric operations, whereas we can create ejson keys at no cost, as many as we need for teams / projects /apps.
Let’s get started
Hold on a minute! The plan is to use ejson to encrypt secrets, so to keep true to our methodology, we’ll manage the ejson keys as secrets in Secrets Manager and encrypted in our code repo. The keys will also need to be securely distributed to the Codebuild container and the team managing the secrets.
This means our first step is to provision an AWS KMS key which will be used to encrypt our ejson keys.
Step 1 — Provision a KMS key
Create a new code repo in GitHub and add .gitignore
In the new repo, create a subfolder called ejk-mgmt for the terraform files needed to store our ejson keys in Secrets Manager, a config for a secure backend (stores the terraform state file) and the terraform file to create the KMS key. e.g.
To provision the KMS key, run terraform init && terraform plan, review the output and if the plan looks good, run terraform apply
Let’s not forget to create an IAM policy / roles to restrict access to this key to the team responsible for managing the secrets.
For reference, see the Terraform IAM Policy Document Data Source
Step 2 — create ejson keys in Secrets Manager
Download ejson from https://github.com/Shopify/ejson/releases, add the binary to your $PATH, then write a small shell script to generate encrypted files with aws kms cli in your repo.
#!/bin/bash
kmsalias="alias/kms-secrets"
region="eu-west-1"
team="xpweb"
mkdir ejkeys && chmod 0700 ./ejkeys
ejkfile=$(ejson --keydir ./ejkeys keygen -w)
chmod 0600 "./ejkeys/$ejkfile"
# create file with pub key
echo "$ejkfile" > ./ejkeys/ejson-pub
# encrypt with KMS key
aws --region $region kms encrypt --key-id $kmsalias --plaintext fileb://ejkeys/ejson-pub --query CiphertextBlob --encryption-context team=$team --output text > ejkpub.encrypted
aws --region $region kms encrypt --key-id $kmsalias --plaintext fileb://ejkeys/$ejkfile --query CiphertextBlob --encryption-context team=$team --output text > ejkpriv.encrypted
# remove the key directory, it's no longer needed, so that it doesn't get accidentally committed on git
rm -fr ./ejkeys
encrypt ejson keys
Verify that the unencrypted files have been removed, and that 2 files were created: ejkpub.encrypted & ejkpriv.encrypted.
We are now able to refer to the encrypted files in terraform without exposing the plaintext secret directly.
In this step, we are going to build the ‘automation’ part of managing secrets.
We will aim for simplicity and using off-the-shelf AWS services as they normally come with a pretty high record in availability.
We will of course create all the AWS resources in terraform. 😃
We will configure CodePipeline’s source stage to use a CodeStarSourceConnection (which supports Bitbucket, GitHub and GitHubEnterprise). This will provide the trigger when a new commit is made on the external VCS’ release branch.
The build stage will be handled by CodeBuild, and will run the scripts and terraform secrets provisioning inside a CodeBuild container.
A valid question some might ask is whether anyone could break out of their CodeBuild container and poke around to see other customers’ containers 😅. Here’s a response from AWS:
Though Codebuild makes use of containers, the containers do not constitute a security boundary of the Codebuild service. As part of Codebuild’s design, instances responsible for running customer jobs are isolated from those belonging to other customers. Security Groups are further used to isolate instances at the network level.
In our repo, create a new top folder called ‘pipeline’ alongside the previous terraform folder.
This folder will contain terraform files to provision the following resources:
CodeBuild
CodePipeline
S3 buckets for CodePipeline /CodeBuild
IAM roles and policies for CodePipeline / CodeBuild
A good example of the resources required can be found in this GitHub repo.
Let’s add the aws_codestarconnections_connection resource to the CodePipeline source stage, e.g.
In the above buildspec.yaml, the install phase runs the script repo-dir/scripts/install.sh script to install terraform 0.14, and install the ejson linux binary into repo-dir/ejson/ejson. As a start point, you can copy the shell scripts install.sh and pre-build.sh from the ci-4-terraform-aws-developer-tools-terratest repo, and add the following lines to install.sh
Time to commit all these files on your git repo and provision the pipeline resources with terraform…
Once done, jump on the console/CodePipeline. You should notice that the source stage of the pipeline just created will have run and failed. On the console, Edit the source stage of the pipeline as you will need to manually authenticate to GitHub:
The aws_codestarconnections_host resource is created in the state PENDING. Authentication with the host provider must be completed in the AWS Console.
When complete, you should get a screen like this one:
Next, verify that a commit & merge on the current repo / branch configuration declared in codepipeline.tf will trigger an execution of the pipeline…
And to help diagnose the build stage and see the container logs, click on the Details link in the ‘Build stage’.
We can now write the terraform modules that will manage all the secrets for our project.
Step 4 — managing secrets at scale
You have already encrypted the ejson keys in step 2 using a KMS key, and to be fair, if you only had to manage a handful of secrets in your account, the same method used before would work.
We will now focus on how a project team will manage their app secrets on AWS using the pipeline. These could include database credentials, a Splunk token, api auth tokens, etc.
The project team is aware that the foundations of the secret management solution are in place (pipeline, access to GitHub and trigger on commit) and that the ejson keys they need to encrypt their application secrets with are stored in Secrets Manager’s secret name: team/xpweb
The command-line tool summon is used to retrieve the secrets from Secrets Manager, as it facilitates the retrieval of secrets securely (and you can define how the value of a variable will be processed using YAML tags). summon’s secrets.yml flags:
!file: Resolves the variable value, places it into a tempfile, and returns the path to that file.
!var: Resolves the value as a variable ID from the provider.
!str: Resolves the value as a literal (default).
!default='’: If the value resolution returns an empty string, use this literal value instead for it.
By default summon will look for the file secrets.yml in the directory it is called from and pass any variable matching a valid secret-name#key to the command (or script) that summon wraps.
Within the same repo as before, create a new top folder for the project team’s secrets called webapps-secrets, and add a file to it called secrets.yml where you define 2 variables EJKP and EJKS mapped to the ejson secrets:
secrets.yml will be loaded by the cli summon when run in this directory, and will pass the AWS secrets you have set in step 2 to the environment of the command it wraps. When the wrapped command exits, the secrets stored in that process environment are gone.
Incidentally, the role of the user running the summon command needs to be granted read permission to the ‘team/xpweb’ secrets too, e.g.
resource "aws_iam_role" "xpweb" {
name = "xpweb-role"
assume_role_policy = <
As we’ll need the ejson keys to encrypt our app secrets in the repo, it would be convenient to have the keys installed on our local machine permanently.
However, that’s a poor idea from a security perspective as we only need the keys locally when we want to encrypt a secret in this particular repo.
A possible option is to write a small shell script that retrieves the keys from Secrets Manager, encrypts the secret file, then removes the ejson keys from disk:
Next, create the ejson file containing the app secrets…
the file starts with the public key. (Please be nice and don’t steal my plaintext passwords; these were all made-up for this exercise and are just random values 😃)
and this file can be safely committed to our git repo.
To update a value, you could simply edit the file and replace the string starting with “EJ[“ with a new plain text password, and run summon ./encrypt.sh … again. Running ejson encryptwill encrypt any new plaintext keys in the file, and leave any existing encrypted keys untouched.
Now create the terrafom resources in this same webapps-secrets folder that will store the secrets in Secrets Manager when CodeBuild runs, declaring the secrets variables with sensitive = true
and add a similar secrets module like the one used in step 2!
Provision the resources with terraform init / plan /apply.
On the CodePipeline build stage, use a similar script to encrypt.sh with summon to decrypt the webapp-secrets.tfvars.ejson into a .json file before running terraform.
Let’s call it decrypt.sh
The above command will request the ejson encryption keys from Secrets Manager, and then create the ejson’s keys on the container’s filesystem for the time it takes for ejson to decrypt webapp-secrets.tfvars. At this point, the ejson keys are removed from the container before the sub-process decrypt.sh terminates. Furthermore, when the summon’s process exits, the secret keys that were passed to decrypt.sh are no longer available in any of the container’s environments.
Next, specify the json file when running terraform plan/apply in the build stage of buildspec.yaml.
terraform plan -var-file="webapp-secrets.tfvars.json" -out=terraform.plan -input=false
The CodeBuild’s buildspec.yaml (in the pipeline folder) needs to be adjusted to incorporate the decrypt.sh and terraform commands:
You are now ready to utilise the solution to manage secrets from git in a repeatable and simple way, minimising the exposure of keys needed for encryption.
For example you can update one of the sample secrets, raise a PR, get approval and merge the branch … and the pipeline will take over and run terraform to update the secret:
In the above example, the CodeBuild “build phase” of the “build stage” lasted 12s, and overall the build stage took less than 1 minute including provisioning of the container, installation of binaries and running terraform plan / apply.
In the last step, we described how easily we could update a secret or generate a new ejson file full of secrets. The workflow involved running a script encrypt.sh to encrypt our secrets, before committing the file to our private repo.
The team managing the secrets knows the correct workflow, but we also know that an accidental commit could occur at some stage. It’s therefore best to try to minimise the risks of human error as much as possible by using a preventative tool closely integrated with our git repo.
Git can run custom scripts as part of the development workflow, and local or server-side hooks can be implemented to make decisions on file content while an action is performed.
One very useful implementation is the pre-commit framework with many great contributions and Yelps’ detect-secrets module.
To install run:
Address any issues or false positive found in the last command by configuring detect-secret (see documentation to mark false positives with an inline `pragma: allowlist secret` comment).
Next, modify one file in your repo and commit:
$ git commit -a
Detect secrets...........................................................Passed
hint: Waiting for your editor to close the file...
Brilliant, you have now some detection happening on every commits before submission to code review!
However we need to extend on the detect-secret tool detection to the ejson files we use, as the current entropy detector (v1.1.0) doesn’t pick up all our unencrypted secrets.
pre-commit supports a Repository local hook, which is useful when developing custom scripts that are tightly coupled to the git repo, and in our case, it makes sense to distibute the hook scripts with the Secrets management repo.
We know that all the encrypted values in our ejson files should start with the characters: EJ[1.
Let’s create a quick script in the bin folder at the root of our repo to parse json files containing the _public_key ‘key’ and any values not starting with EJ[1.
#!/bin/bash
set -e
# requires jq
which jq || exit 1
echo Looking for ejson files in repo...
mkdir -p /tmp/check-json
find . -type f ! -regex ".*/.terraform/.*" -exec file {} \; | grep JSON | awk '{print $1}' | sed -e 's/://' > /tmp/check-json/jsonfiles
if [ ! -s "/tmp/check-json/jsonfiles" ]; then
echo "json files not found **" && exit 0
fi
rm -f /tmp/check-json/ejson-key-val
rm -f /tmp/check-json/ejson-err
for file in $(cat /tmp/check-json/jsonfiles); do grep _public_key "$file" && echo "$file" >> /tmp/check-json/ejson-key-val; done
echo ejson files:
cat /tmp/check-json/ejson-key-val
for ejson in $(cat /tmp/check-json/ejson-key-val); do grep -Ev _public_key < "$ejson" | jq '.[]' >> /tmp/check-json/ejson-err ; done
error_count=$( grep -Ecv 'EJ\[1' < /tmp/check-json/ejson-err | awk '{print $1}')
rm -fr /tmp/check-json/
if [ "$error_count" -ne 0 ]; then
echo "-------------" && echo "Error: $error_count unencrypted string(s) found!! fix before committing."
exit 1
fi
echo Check complete
A local hook is added to run the above check-ejson.sh script in our pre-commit configuration:
Let’s check if it all works as expected… by decrypting one of our secrets files and attempt to commit:
Good, it stopped the commit of our plain text ejson secrets.
Now re-encrypt everything… and commit.
Validation passed! You are able to commit with automatic detection in place on the repo, and next raise a Pull Request for code review!
Finally
It might make sense, in most environments, to build one secrets management CodePipeline solution per team, and give each team their own range of secret names in Secrets Manager. It’s quite easy to attach AWS Identity and Access Management (IAM) permission policies to grant or deny access to specific secrets.
Furthermore, application development teams will often need to retrieve the AWS Secrets Manager’s secrets from their applications, and AWS does provide SDKs which are widely used to access the secrets.
However, for companies with several teams supporting hundreds of apps, it might be sensible to de-couple the apps from retrieving the secrets and perhaps rely on a dedicated subprocess through the summon tool that could easily be installed on all instances and / or Docker images.
Conclusion
In this article we have introduced some building blocks for facilitating the creation and management of secrets using native AWS services and Open Source tools. Terraform was used extensively to provision the AWS resources required for this solution, including the pipeline that will run the terraform required to manage the secrets 😃. More importantly all the components of this Secrets management solution are trackable in one source of truth.
Although some effort was required to create the resources we needed in Terraform, there are clear advantages to have a solid, repeatable and resilient solution which simplifies Secrets management.
In complex environments where many teams are involved, it makes sense to adopt and encourage the usage of a secure pattern to avoid for every team to work out their own customised solution. Some teams might do a great job, but others might struggle as their expertise is quite rightly focused elsewhere. From a business perspective, you don’t really want to hear that your teams have stored their plaintext passwords on S3 because they felt this was an appropriate secret management strategy!
