DevOps and automation are two key components that help organizations accelerate Cloud adoption. Our Continuous Compliance Framework (CCF) in the AWS Solutions Library helps organizations to accelerate AWS adoption and harden security posture through a fully-automated cloud controls framework.

Most DevOps teams will leverage Infrastructure as Code (IaC) for resource provisioning on Cloud using continuous integration and continuous delivery (CI/CD) pipelines. Continuous delivery done right, therefore, is very essential to DevOps practice.

Why Policy-as-Code ?

Given the complexity and scale of the platforms on Cloud and how the resources created by DevOps teams are evolving, it can be a challenge for the teams to manually apply or validate security and compliance policies in every IaC change, particularly when the policies may stored in Cyber security’s PDFs, Change Advisory Board (CAB)’s Word documents, or engineers’ brains.

Policy as Code allows you to translate the policies into machine-readable definition files, and use them to enforce and validate that the resources provisioned meet those policies. As such, the challenge that Policy as Code (PaC) is trying to solve is the challenge of as-much-as-possible automation in DevOps practice. PaC automates the policy enforcement while extending the benefits of IaC.

Having an efficient feedback loop from CAB, Cyber security and compliance teams to the development teams is very important. We can achieve it by embedding PaC in the integration and delivery pipelines. This article will show you how we can implement “shift-left” PaC in the continuous delivery pipelines.

Instead of a word document, it is more efficient to translate the policies into machine-readable definition files. The policies can cover these categories:

1. Compliance policies: the Payment Card Industry Data Security Standard (PCI-DSS), for example, is a set of technical and operational requirements. PCI data that resides in a MySQL or MariaDB database must adhere to these requirements and follow best practices to ensure the data is secured and compliant.
2. Security policies: these could be corporate security policies that require employing encryption for data at rest; or a rule to only allow connections from whitelisted IP addresses.
3. Operational best practice: the policies from site reliability engineering (SRE) teams aimed to ensure the high availability of the services in a production environment even when a cloud Availability Zone (AZ) is down.
4. Cost optimisation: there could be policies that your DevOps teams must shut down the development environments outside of the business hours, or must tag the AWS resources created with a cost centre.
5. Pod security standards: these could be Kubernetes Pod Security Policies (PSPs) we have to implement on the clusters in order to improve the security posture.

Shifting the Policy as Code left

We are going to use Styra’s Open Policy Agent (OPA) and Armory’s Spinnaker for the use cases.

OPA is a policy engine that provides a policy language Rego, it is a language that allows for declarative fine-grained control. OPA evaluates the policies against input data, which could be a Terraform plan, a Kubernetes resource or any JSON file.

OPA decouples policy decision-making from policy enforcement. When your software needs to make policy decisions it queries OPA and supplies structured data as input. OPA generates policy decisions by evaluating the query input and comparing against policies and data. Decision making with OPA consists of three components:

• Data: the first component is the information about your domain that OPA will use to make decisions.
• Query Input: the second element is the question your application sends to OPA.
• Policy: the final component is the policy which OPA uses to make a decision and return the result to you. It might be a simple “true” or “false”.

It has more than one option to deploy OPA depending on the specific scenario:

• As a Go library: if an application is written in Golang, you can implement OPA as a third-party library in the application.
• As a daemon: you can deploy OPA just like any other service as a daemon.

Spinnaker is an open-source continuous delivery platform for releasing software changes. It empowers developers to own their application code from commit to delivery. Spinnaker provides two core sets of features:

• Application management
• Application deployment

In addition, Spinnaker provides another feature - Managed delivery - focusing on application requirements specified in a declarative format. Armory’s Spinnaker is an extension of open source Spinnaker, an enterprise-grade distribution of Spinnaker.

OPA and Armory Spinnaker installation

There are several methods, for example, using Operators to install Armory enterprise or open source Spinnaker. In this post, we will be setting up Minnaker: Spinnaker on Lightweight Kubernetes for demo purposes.

Spinnaker concepts: before we start building the pipeline, we’d better have a general understanding of certain Spinnaker concepts:

Application — An application represents the service you are going to deploy using Spinnaker, all configuration for that service, and all the infrastructure on which it will run. You will typically create a different application for each service.

Pipeline — A pipeline is a sequence of stages provided by Spinnaker, ranging from functions that manipulate infrastructure to utility scaffolding functions (manual judgment, wait, run Jenkins job).

Stage — A Stage in Spinnaker is an atomic building block for a pipeline, describing an action that the pipeline will perform. You can sequence stages in a Pipeline in any order. Spinnaker provides a number of stages such as Deploy, Resize, Disable, Manual Judgment, and many more.

Artifact — In Spinnaker, an artifact is an object that references an external resource.

Now let’s proceed with the application creation and setting up the pipelines. For demo purposes, we will be using Spinnaker web console. Armory provides pipeline-as-code (Dinghy) which allows you to store Spinnaker pipelines in Git repository and manage them like you would manage code.
 

1. Create application

Log in to Spinnaker console, you will be able to create a new application from Applications -> Create Application:

2. Create pipelines

Application is the placeholder for pipelines, once created you will see the application infrastructure page:

3. Pipeline configuration

Now we are at pipeline configuration page. We will add some stages to the pipeline configuration for the demo. We create a stage Deploy Dev which will deploy to Dev environment. And add a manual Approval stage before it is allowed to deploy to Prod environment.

OPA server is installed as a Pod and exposed as a Service in the same Minnaker deployment. We will be using Kubernetes ConfigMaps for OPA policies. Let’s use the following two example use cases to demonstrate how we can implement “shift-left” PaC in the continuous delivery pipelines.

Many organizations run security scanning software to detect common vulnerabilities, for example a recent CVE-2021–31440 vulnerability allows a Kubernetes container local privilege escalation using eBPF and resulted in an out-of-bounds (OOB) access in the Linux kernel.

In Kubernetes, privileged containers inherit CAP_SYS_ADMIN privileges by default. As a result, unprivileged users inside a privileged container can escalate privileges to the node and break out of the container. We will create a OPA policy to disallow privileged pods and embed that in the continuous delivery pipelines.

Another use-case is a multi-tenant Amazon EKS cluster, teams share the same cluster resources with some level of logical or physical isolation. DevOps teams wanted to enforce policy to prevent developers from creating arbitrary AWS Load Balancer (ALB/NLB) services.
 

1. Writing the policies

We write the deny-privileged-containers and deny-eks-loadbalancer-service policies using OPA’s native query language Rego:

# deny-privileged-containers.rego
package spinnaker.deployment.tasks.deployManifestdeny[msg] {
    manifests := input.deploy.manifests
    manifest := manifests[_]    manifest.kind == "Pod"
    c := input_containers[_]
    c.securityContext.privileged
    msg := "Privileged Pods disable most security mechanisms and is not allowed"
}input_containers[c] {
    c := input.deploy.manifests[_].spec.containers[_]
}input_containers[c] {
    c := input.deploy.manifests[_].spec.initContainers[_]
}

# deny-eks-loadbalancer-service.rego
package spinnaker.deployment.tasks.deployManifestdeny[msg] {
    manifests := input.deploy.manifests
    manifest := manifests[_]    manifest.kind == "Service"
    manifest.spec.type == "LoadBalancer"
    msg := "AWS LoadBalancer Service provisoning is not allowed"
}

2. Enforce the policies

We enforce the OPA policies by creating the ConfigMaps and apply the label openpolicyagent.org/policy=rego to the ConfigMaps created

 

3. Testing the policies

We test the continuous delivery pipelines with the polices created by pushing commits to Git repository if a build trigger is configured. For the demo we start the pipeline execution in Spinnaker console.

When a developer is pushing a commit to create a privileged container in the Kubernetes cluster. Error message: “Privileged Pods disable most security mechanisms and is not allowed”, returned by the OPA policy engine, will be displayed in the pipeline UI immediately.

apiVersion: v1
kind: Pod
metadata:
  name: pac-demo-app
  namespace: dev
spec:
  containers:
    - command:
        - sh
        - '-c'
        - echo The app is running! && sleep 3600
      image: 'busybox:1.32.1'
      name: demo
      resources:
        limits:
          cpu: '1'
          memory: 256Mi
        requests:
          cpu: 100m
          memory: 128Mi
      securityContext:
        privileged: true
  initContainers:
    - image: 'busybox:1.32.1'
      name: initdemo
      resources:
        limits:
          cpu: '1'
          memory: 256Mi
        requests:
          cpu: 100m
          memory: 128Mi

When a developer is pushing a commit to create an AWS NLB in a multi-tenant Amazon EKS cluster, where arbitrary load balancer service is disallowed, the developer will get the following error message returned by the OPA policy engine in the pipeline UI immediately.

Final thoughts

DevOps teams are adapting GitOps to accelerate development timeframes. It is time for DevOps teams to adopt a Policy as Code approach to policy enforcement. Static analysis by embedding PaC in the continuous delivery is a good start to “shift-left”. All they have to gain is a scalable way to manage policies throughout the application lifecycle and distribute them across every pipelines, clusters and multi-cloud environments in the organization.

References

• https://spinnaker.io/concepts/
• https://docs.armory.io/
• https://www.openpolicyagent.org/docs/latest/