In this post, I’m going to showcase how you can build just enough PAM using only AWS services. Phil Longley, Airwalk’s CTO, thought that this might be a good application of AWS Change Manager. Here we are a month later with a proof of concept.

Here’s a setting that might sound familiar. You’re in your infrastructure development and cloud migration journey, most things are automated, you’ve got Infrastructure as Code, maybe looking into policy as code, your CI/CD pipelines are maturing, and all is going better in your product delivery world. However, from time to time a person needs to get elevated access to perform some action. Maybe to login to a “pet” server, to investigate a bug that only manifests itself on the production environment, or some manual intervention to an AWS service. It might be a service specific break-glass situation or something simpler like write access to the configuration of a service.

Here are three common use cases:

1. A person needs privileged access to an AWS Service, e.g. giving rds:PromoteReadReplicaDBCluster. You might need this to start your fail-over process.

2. A person needs to connect to an EC2 Instance. The environment requires manual intervention.

3. A person needs to retrieve a secret, maybe some credentials to another service, that have been created just in time. For example, generating an account for a service and sharing the secret on the fly, with a specific user.

These are some of the scenarios that you’ve faced lately. On top of that, your change process and security requirements, dictate that access to any of the above must be:

1. Privileged — no normal operations access

2. Time limited — temporary access

3. Peer reviewed

4. Justified — for a specific purpose with an action plan

You realise this sounds like Privileged Access Management. PAM is a multi-billion dollar market, with a plethora of solutions. However, you don’t have the capacity to investigate, and implement everything that comes with an enterprise PAM solution, outside of a cloud native toolset context. Another advantage of cloud native in this context is that you can avoid going through the procurement process, and have a consolidated bill.

Summarising your requirements, you need to allow a role to assume another privileged role, for a limited amount of time, whilst having the change tracked, and approved by someone else, with a justification.

We can achieve all three of these use cases by applying time bound and effective IAM policies, modified by a Lambda, just in time, to allow an identity to assume a privileged role. However, the approval and change management process were a little bit more complicated to implement using cloud native tools only. Enforcing a change request process and review required integrations with third party tools and/or building custom applications. That was until AWS Systems Manager Change Manager was announced.

What is Systems Manager Change Manager

Systems Manager Change Manager is a change management capability which allows you to implement change management processes. You can create change templates, change requests, set a list of approvers, and at the end of it execute a set of runbooks; Automation Documents. The change requests are captured in a Change Calendar, in addition to allowing you to specify maintenance windows. Finally, there’s a tight integration with AWS Organisations and AWS SSO, that allows you to set Groups or Users as change request and change template reviewers.

For each change request, we get a detailed view and timeline of our changes, like the one in the screenshot above. This is exactly the service we need, in order to wrap a change management process around our automation that will manage the IAM policy of our PAM role.

Here’s how we can solve all these common problems — mentioned above — with AWS Change Manager

1. Privileged IAM for an AWS Service — Create a privileged role that can only be assumed by a specific role

2. Remote EC2 — Attach the SSM Core Instance Policy to an EC2 instance and grant Session Manager to a specific role

3. Generate and retrieve a secret — Secrets Manager Resource Policy with the relevant role policy

To prove that we can have just enough PAM, we’re going to build the solution for our first case and take a look at how you can setup AWS Change Manager. The process for the other two cases remains largely the same. You can skip to the code at https://github.com/AirWalk-Digital/AWS-ChangeManager-PAM where you can set this up in your own environment.

What we are going to build

This is what the process will look like. A PAM Requestor — the person who requires privileged IAM permissions — will submit a change request that they require PAM. After the change request is approved, the automation document is going to invoke a Lambda. The Lambda will modify the trust policy of the PAM IAM Role, to allow it to be assumed for the duration of time by the PAM Requestor. Once the change request completes successfully, the PAM Requestor can use the AWS CLI to assume the PAM Role for one hour. In this example, the PAM Requestor can only assume the role using AWS CLI, due to a limitation in sts:SourceIdentity when assuming the role. You can remove this condition from the IAM policy, if you are confident that you can monitor AWS actions taken during the session.

Let’s start building our solution. You can find all the components, in the companion repo.

Setting up “just enough PAM”

In this example we’re going to be setting up Change Manager assuming that there’s only one AWS Account. When running a change request, Change Manager, will require an IAM role to assume, to execute the runbook. In a single account setting you can setup your own service role. However, if your account is already in an organisation, you must enable a delegated administrator account for Systems Manager. Effectively you’ll need to make an AWS account the centre of all Systems Manager operations. The management account will provision the relevant service roles to the AWS delegated administrator account.

The next step of our setup is to define the pool of users that will be able to initiate and authorise changes. The pool of users can originate either from IAM Users or SSO Users and Groups.

This is a global setting and you will need to select which pool of users, you’d prefer in your Change Manager settings.

The Change Request Template and Automation Document

The change request template can be a YAML or JSON document that follows the Systems Manager Document schema. It has an approval and rejection process of its own and it forms the blueprint of change requests.

In a change template the runbooks, description, monitoring, and a set of approvers is described. You can use the Document Builder to help you construct the change template. Once a change request template is approved, our PAM Requestor can use it to create a change request. When the change request is approved our automation document can be executed. Our Systems Manager Automation Document is a YAML document that will invoke a Lambda function, using the PAM Requestor’s role ARN as input.

At this point you might ask: “Can I make my own template, approve it, and then approve my own change request?”

The answer is no.

As it stands you cannot change the automation that will be executed without requiring the template to be approved again, however you can change the version of the automation, which will not affect the change template.

This means that if a different IAM identity has access to the automation document, they can make changes to the automation document that will be executed. It’s important that your roles cannot access or modify arbitrary Systems Manager Documents or run Automation Executions.

Setting up IAM access for the PAM Role

We’re also going to be using the source identity attribute. This is a new AWS STS attribute, that makes monitoring the source identity of role-chaining activities, a lot more convenient.

We can track changes with assumed roles, as this attribute is going to be present in requests for any AWS action taken during the role session, in CloudTrail. Two caveats around source identity. Firstly, as it stands the source identity attribute is not captured by CloudTrail when an AWS Service or service-linked role carries an action on your behalf. However it still improves the visibility of role-chaining events . See here for more information https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_control-access_monitor.html

Secondly, source identity cannot be used when assuming a role in AWS Console. However, you can still use it with the AWS CLI or AWS API, like so:

aws sts assume-role -role-arn arn:aws:iam::accountid:role/PAM-Automation -role-session-name PAMRoleAutomation -source-identity PAMRoleAutomation

At the core of our PAM process is our PAM role. We are going to modify this role, with the help of a Lambda. This is what our TrustPolicy will look like:

In this policy we are:

1. Restricting the Principal that can assume the PAM role. I.e. only the PAM Requestor should be able to assume the role
2. Restricting the PrincipalOrgID that can assume the PAM role. I.e. only Principals from this AWS Organisation as a guardrail

3. Give the role a 20 minute window to assume the role. The session duration from AssumeRole will then last for the minimum period of one hour

4. Make the source identity have a specific name so that we can track it in CloudTrail.

Auditing and Monitoring

The Change Template is an SSM Document, you will be able to see any new change templates in CloudTrail under the event name of `CreateDocument`. A change template that has been submitted for review will show under `UpdateDocumentMetadata` with the relevant request parameters of either Approve, Reject or SendForReview. Once a change Request has been approved you will also be able to track its execution.

As I mentioned earlier, with the introduction of the SourceIdentity attribute in our Trust Policy, it will be easier to track the events of the PAM Role. You can always combine any of the above with CloudWatch, and the already existing SNS integrations of change templates.

You can do so much more

Now that you have all the core pieces in place, the world is your oyster. You can use this pattern to build just in time access to EC2 instances, implement PAM password retrieval, trigger events via your CI/CD pipeline, and so much more. AWS Systems Manager Change Manager is a versatile service. It wraps existing Systems Manager services effectively and offers a cloud native way of managing change requests.