Overview

In companies, particularly big corporates, where security is paramount and best practice must be adhered to, an increasingly common requirement is for applications to access secrets in other AWS accounts — database passwords, api keys, etc.

Doing this in kubernetes has, in the past, been difficult. Thankfully, the addition of fine grained IAM roles for service accounts in the last few years has brought a much more structured and automatable approach to configuring this access securely.

This post looks at how we can utilise AWS’s IAM roles for service accounts (IRSA) for maximum security when accessing secrets in different AWS accounts, from pods running on EKS. Although quite a specific use case, it’s possibly the most complex, given that the KMS key used to encrypt a secret may well exist in another AWS account, which means we must consider access to resources across three different accounts. This complexity makes this scenario arguably the best one to consider — hopefully, if we can make the most complicated scenerio clear, it should enable us to understand other scenarios more easily.

So what do we need to do?

Let’s say we have a pod on an EKS cluster, running an instance of our application. And the application needs to access a database on an RDS instance in another AWS account. To do this, our app must retrieve the database password, which is stored in a secret within the other account. Additionally, the secret is encrypted using a customer managed key (CMK) stored in KMS in the account of our cyber security department. To retrieve the secret value and decrypt it, our application needs access not only to the secret itself, but also the key.

Let’s break down exactly what is required:

1. A pod, running on an EKS cluster with a service account, which I will use to test.
2. An IAM role, the arn of which will be annotated to the service account on Kubernetes, which is running our pod.
3. A secret, which is stored in Secrets Manager in a separate AWS account. An access policy will be applied to this secret, which will allow access only from the pod IAM role.
4. A KMS key, which will be used to encrypt, and decrypt our secret. This key should also have a policy attached, allowing access only from the pod IAM role.

(NOTE: For the obvious purpose of not exposing details of our infrastructure, names for all the following resources have been replaced with dummy names. The resources are also edited for readability, showing only the key details.)

The Pod and Service Account

Our service account is configured as follows:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam:accountid1:role/pod-role
  name: testapp
  namespace: testnamespace

And our pod:

apiVersion: v1
kind: Pod
metadata:
  name: testapp
  namespace: testnamespace
spec:
  containers:
  - env:
    - name: AWS_ROLE_ARN
      value: arn:aws:iam:accountid1:role/pod-role
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    image: accountid1.dkr.ecr.eu-west-1.amazonaws.com/testapp:1.0.1
    name: testapp
    ...
    volumeMounts:
    - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
      name: aws-iam-token
      readOnly: true
  ...
  serviceAccount: testapp
  serviceAccountName: testapp
  ...
  volumes:
  - name: aws-iam-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
        audience: sts.amazonaws.com
        expirationSeconds: 86400
        path: token
...

Importantly, the environment variables AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE , along with the volume aws-iam-token and corresponding volume-mount, did not need to be included in the original pod config. They were autogenerated by the mutating admission webhook, which has been developed by AWS and is deployed on the EKS api server. This occurs automatically when the service account is annotated with the IAM role.

The IAM role for the Pod

The IAM role pod-role has a policy attached as follows;

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "testappSecretAccess",
            "Effect": "Allow",
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "arn:aws:secretsmanager:eu-west-1:accountid2:secret:testsecret"
        },
        {
            "Sid": "testappKmsAccess",
            "Effect": "Allow",
            "Action": "kms:Decrypt",
            "Resource": "arn:aws:kms:eu-west-1:accountid3:key/test-key-id"
        }
    ]
}

and a trust policy …

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam:accountid1:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/CLUSTERNODETESTID"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringLike": {
                    "oidc.eks.eu-west-1.amazonaws.com/id/CLUSTERNODETESTID:sub": "system:serviceaccount:testnamespace:testapp"
                }
            }
        }
    ]
}

The trust policy must be configured this way to allow the service account to assume the role. As can be seen, this access is also limited to our testappservice account running pods in our testnamespace.

The KMS Key

In our cyber security account, accountid3, we have a symmetric CMK created with arn arn:aws:kms:eu-west-1:accountid3:key/test-key-id , and a key policy as follows;

{
    "Version": "2012-10-17",
    "Id": "project_account_accountid1",
    "Statement": [
        {
            "Sid": "testappKmsAccess",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam:accountid1:role/pod-role"
                ]
            },
            "Action": "kms:Decrypt",
            "Resource": "*"
        }
    ]
}

The Secret

Our secret is created in accountid2with name testsecret , and using the KMS key arn:aws:kms:eu-west-1:accountid3:key/test-key-id to encrypt it on creation. It has the following resource permissions policy set;

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam:accountid1:role/pod-role"
            },
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "*"
        }
    ]
}

Retrieving the Secret …

So first, I log into the pod, and then assume the role. Ordinarily an application would do this programmatically, using the injected environment variables. My test container image is simply one with the awscli tools installed, so for the purposes of this example, I’ll do this on the command line.

> kubectl -n testnamespace exec -it testapp /bin/bash
testapp> 
testapp> aws sts assume-role-with-web-identity --role-arn $AWS_ROLE_ARN --role-session-name testsession --web-identity-token-file file://$AWS_WEB_IDENTITY_TOKEN_FILE --duration-seconds 1000 > /tmp/tempcreds.txt

This has generated the temporary credentials AccessKeyId, SecretAccessKey and SessionToken in the file /tmp/tempcreds.txt, and I can now use the values for them, when finally retrieving our secret (edited here for readability).

testapp> export AWS_ACCESS_KEY_ID=
testapp> export AWS_SECRET_ACCESS_KEY=
testapp> export AWS_SESSION_TOKEN=
testapp>
testapp> aws secretsmanager get-secret-value --secret-id arn:aws:secretsmanager:eu-west-1:accountid2:secret:testsecret
{
    "ARN": "arn:aws:secretsmanager:eu-west-1:accountid2:secret:testsecret",
    "Name" "testsecret",
    "VersionId": "cdd0c402-33b8-4d18-9365-b4ed0dd3e33f",
    "SecretString": "testpassword",
    "VersionStages": [
        "AWSCURRENT"
    ],
    "CreatedDate": 1620897362.815
}

So there we have it. In this case our application is able to programmatically and dynamically retrieve the necessary secret value (testpassword in this case), and use it as needed.

Caveats and Gotchas …

• This walk-through assumes a default region of eu-west-1 . If you are unable to retrieve your secret, you may just need to specify --region eu-west-1 in your get-secret-value command.
• Secrets can also have multiple versions when their value has been updated multiple times. So you may also need to specify --version-stage AWSCURRENT to retrieve a value.
• For the purposes of simplicity in this example, I didn’t set a permissions boundary on the IAM role. These are a good extra layer of security, which limit the permissions on a role, so are a good idea. If applied, however, it may prevent the app from retrieving the secret if configured wrongly.
• It’s very much worth knowing that if the IAM role is deleted, it will break the policy applied to the KMS key. Since the role no longer exists, its ARN is replaced in the key policy by its AWS unique identifier. If the role is recreated, it will have a new unique identifier, and so, what looks like the same role will NOT be replaced in the key policy, meaning it must be corrected.
• If the IAM role is not assumed by the application, it will assume the role assigned to the EC2 instance by default, so it’s advisable to be careful what policy is applied to that role and possibly investigate what can be done to mitigate the risk as outlined by AWS here

Conclusions

Fine grained IAM roles are extremely useful in enabling pods in EKS to access resources outside of clusters, and the fact that they are so granular means we can give permissions for the pod to access only what is needed, which is very important.

The fact that they are so granular, however, also means there is a lot of complexity in configuring them. Everything has to be set, and everything must be correct, which is why it’s useful to lay it all out in blog posts like this one, so it’s clear how we at Airwalk Reply have been able to achieve this for our clients, and how you can too.