How a Pod assumes an AWS identity

Overview

In my previous post, I talked about how an AWS identity can be delegated the permission to manage a Kubernetes namespace. This time, I am going to do the reverse, implement a role based access for a pod in AWS EKS to assume an IAM role for consuming AWS services.

This is achieved by associating an IAM role with a Kubernetes service account. By authenticating with an OpenID Connect (OIDC) that supports federated identities, a valid OIDC JSON web token (JWT) is returned and passed to AWS STS AssumeRoleWithWebIdentity API operation, where an IAM temporary role credential is generated in return.

After that, the K8s mutating admission controller in EKS (via a webhook) will inject the temporary role credentials in the form of environment variables AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE into pods based on the annotation on the Service Account used by the pod. Thus the pods are given the credentials for making subsequent AWS API calls.

Example setup

For illustration, I will make use of official AWS CLI docker image ( amazon/aws-cli) to demonstrate the IAM permission. I will grant the pod the needed IAM permission to make AWS CLI commands for EC2.

I will focus on the permission setup and will not go into the details of how to run the image in a pod container. If you are new to Kubernetes, I strongly suggest you visiting https://kubernetes.io/docs/concepts/workloads/pods/

Some commands are making use of eksctl which you can find at https://eksctl.io/

Before the permission setup, let’s see whether our docker container can make any AWS EC2 calls.

The aws-cli container will try to describe ec2 instances, but it failed:

Next we proceed our setup.

1. Create an OIDC provider for your cluster.

eksctl utils associate-iam-oidc-provider --cluster --approve

Note the OIDC provider:

2. Create a new namespace for running docker pod. (Optional)

kubectl create ns

3. Setup a Kubernetes service account for the namespace.

Every namespace has a default service account resource called default. When a pod is created, if a service account is not specified, it is automatically assigned to run under the default service account in the same namespace. You can use the default service account for learning purpose. But it is advisable to create a separate service account for each of your application for better management.

Create a new service account:

Example yaml file for specifying the service account to use for running a pod:

4. Create an IAM role and attach IAM policy for designated AWS services access. (In our case, we will use the AWS managed policy named “arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess”)

Copy the following code block to a file named trust.json

Replace with your ACCOUNT_ID, OIDC_PROVIDER, serviceAccountNamespace and service AccountName with the above values. Note the StringEquals statement will confine the access to match a particular service account in that namespace only.

Next assign the “AmazonEC2ReadOnlyAccess” policy to this new IAM role.

5. Associate the IAM role with the service account by adding the following annotation to the service account.

When we describe the service account again, we see that the service account include the ARN of the IAM role as an annotation.

5. The Amazon EKS Pod Identity Webhook on the cluster will apply the aforementioned environment variables AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILEto the new pods that are running under this service account with this annotation.

You can examine the AWS credential environment variables set in the pod.

6. Relaunch your aws-cli pod and verify the pod’s IAM permission by issuing AWS CLI command to verify EC2 permission.

You should be able to see a list of EC2 instances if any.

This conclude the EKS identity mapping to AWS IAM role.

Photo by Brett Jordan on Unsplash