OPA Gatekeeper: Bringing Law and Order to Kubernetes

Introduction

Open Policy Agent (OPA) is a policy based control agent that is able to be integrated on various platforms. For the sake of this document we will be talking about its implementation in K8s via the Gatekeeper – Policy Controller for Kubernetes.

In Kubernetes, OPA is implemented as an Admission Controller, the admission controller enforces polices on objects, during create, update and delete operations.

There are two versions of OPA that work with K8s, OPA with sidecar kube-mgmt and Gatekeeper this document will focus primary on the Gatekeeper implementation.

Gatekeeper is an addition on top of plain OPA and adds the following:

  • An extensible, parameterized policy library.
  • Native Kubernetes CRDs for instantiating the policy library (aka “constraints”).
  • Native Kubernetes CRDs for extending the policy library (aka “constraint templates”).
  • Audit functionality.

Kubernetes allows decoupling policy decisions from the inner workings of the API Server by means of admission controller webhooks, which are executed whenever a resource is created, updated or deleted. Gatekeeper is a validating (mutating TBA) webhook that enforces CRD-based policies executed by Open Policy Agent, a policy engine for Cloud Native environments hosted by CNCF as an incubation-level project.

In addition to the admission scenario, Gatekeeper’s audit functionality allows administrators to see what resources are currently violating any given policy.

Finally, Gatekeeper’s engine is designed to be portable, allowing administrators to detect and reject non-compliant commits to an infrastructure-as-code system’s source-of-truth, further strengthening compliance efforts and preventing bad state from slowing down the organization.

Action on Failure

Currently Gatekeeper is defaulting to using failurePolicy: Ignore for admission request web-hook errors. The impact of this is that when the web-hook is down, or otherwise unreachable, constraints will not be enforced. Audit is expected to pick up any slack in enforcement by highlighting invalid resources that made it into the cluster.

If you would like to switch to fail closed, please see our the OPA documentation on how to do so and some things you should consider before doing so.

Policies and Constraints

With the integration of the OPA Constraint Framework, a Constraint is a declaration that its author wants a system to meet a given set of requirements. Each Constraint is written with Rego, a declarative query language used by OPA to enumerate instances of data that violate the expected state of the system. All Constraints are evaluated as a logical AND. If one Constraint is not satisfied, then the whole request is rejected.

Policy Configuration Overview

The enforcement of Gatekeeper depends on two declarative configurations:

  • kind: ConstraintTemplate
    • This describes the base rules for what web-hook inputs to evaluate and the course of action; allow or deny based the evaluation.
    • Gatekeeper comes standard with many pre-defined Constraint Templates, generally your use case will fall with in these templates. However you can make your own constraint templates.
    • A Constraint Template consists of the kind which will directly be referenced in the kind: Constraintto define an application based on the template and a rego code that defines the logic to evaluate.
  • kind: Constraint
    • This accompanies the kind: ConstraintTemplate object and defines the set of rules that the Constraint Template will evaluate.
    • Normally when implementing, you will create a Constraint that directly uses a predefined Constraint Template and apply a custom set of rules.

Constraint Templates

Below is an example of the pre-packaged k8srequiredlabels Constraint Template.

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
      validation:
        # Schema for the `parameters` field
        openAPIV3Schema:
          properties:
            labels:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels

        violation[{"msg": msg, "details": {"missing_labels": missing}}] {
          provided := {label | input.review.object.metadata.labels[label]}
          required := {label | label := input.parameters.labels[_]}
          missing := required - provided
          count(missing) > 0
          msg := sprintf("you must provide labels: %v", [missing])
        }

This constraint template defines what properties are passed in through the web-hook (labels) and the rego code that is used to evaluate if a violation is found. Documentation on rego can be found here.

Packaged Constant Templates

General Constraint Templates

Gatekeeper is pre-packaged with the following general Constraint Templates.

Control AspectConstraint Template KindGatekeeper Constraint and Constraint Template
Allowed Reposk8sallowedreposallowedrepos
Block Nodeport Servicesk8sblocknodeportblock-nodeport-services
Container Resource Limits (cpu, mem)k8scontainerlimitscontainerlimits
Container Resource Ratios (mem/cpu)k8scontainerratioscontainerresourceratios
Disallowed Tagsk8sdisallowedtagsdisallowedtags
Allowed External IPk8sexternalipsexternalip
HTTPS Onlyk8shttpsonly*httpsonly
Image Digestsk8simagedigestsimagedigests
Replica Limitesk8sreplicalimitsreplicalimits
Required Annotationsk8srequiredannotationsrequiredannotations
Required Labelsk8srequiredlabelsrequiredlabels
Required Probesk8srequiredprobesrequiredprobes
Unique Ingress Hostk8suniqueingresshost*uniqueingresshost
Unique Service Selectork8suniqueserviceselectoruniqueserviceselector

*Sync Required, some constraints must access other data sources via sync to confirm constraint. See https://open-policy-agent.github.io/gatekeeper/website/docs/sync for more documentation.

Pod Security Constraint Templates

Pod Security Policy is a cluster-level resource that controls security sensitive aspects of the pod specification. The PodSecurityPolicy objects define a set of conditions that a pod must run with in order to be accepted into the system, as well as defaults for the related fields.

An administrator can control the following by setting the field in PSP or by deploying the corresponding Gatekeeper constraint and constraint templates:

Control AspectField Names in PSPGatekeeper Constraint and Constraint Template
Running of privileged containersprivilegedprivileged-containers
Usage of host namespaceshostPIDhostIPChost-namespaces
Usage of host networking and portshostNetworkhostPortshost-network-ports
Usage of volume typesvolumesvolumes
Usage of the host filesystemallowedHostPathshost-filesystem
White list of Flexvolume driversallowedFlexVolumesflexvolume-drivers
Requiring the use of a read only root file systemreadOnlyRootFilesystemread-only-root-filesystem
The user and group IDs of the containerrunAsUserrunAsGroupsupplementalGroupsfsgroupusers*
Restricting escalation to root privilegesallowPrivilegeEscalationdefaultAllowPrivilegeEscalationallow-privilege-escalation
Linux capabilitiesdefaultAddCapabilitiesrequiredDropCapabilitiesallowedCapabilitiescapabilities
The SELinux context of the containerseLinuxseLinux
The Allowed Proc Mount types for the containerallowedProcMountTypesproc-mount
The AppArmor profile used by containersannotationsapparmor
The seccomp profile used by containersannotationsseccomp
The sysctl profile used by containersforbiddenSysctls, allowedUnsafeSysctlsforbidden-sysctls

* For PSP rules that apply default value or mutations, Gatekeeper v3 currently cannot apply mutation.

Constraints

kind: Constraints uses the above constraint template kind as the kind field in the configuration to directly reference a kind: ConstraintTemplate. A parameter can then be passed in to be evaluated against for the specific Constraint Template definition.

Below is an example of a kind: Constaint that restricts that a namespace must have a label of key: owner to be in the sada.com domain. This example uses a regex as a Constraint parameter:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: all-must-have-owner
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Namespace"]
  parameters:
    message: "All namespaces must have an 'owner' label that points to your company username"
    labels:
      - key: owner
        allowedRegex: "^ [a-zA-Z]+.sada.com$"

Here is another example of a kind: Constraint that restricts the pod resources to 200m CPU and 1Gi of mem, this time the Constraint parameters are static references.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sContainerLimits
metadata:
  name: container-must-have-limits
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    cpu: "200m"
    memory: "1Gi"

The links above in the table column Gatekeeper Constraint and Constraint Template give example Constraints as well as example cases where they would be allowed or disallowed.

Audits

The gatekeeper system allows for auditing, which is a constant periodic evaluation of resources against the policies. More information on audits and how to configure them can be found at this link here.

Other Functionality

Gatekeeper provides more functionality than described in this overview, please visit the github page to find out about more advanced functionality here.

References

https://www.openpolicyagent.org/docs/latest/

https://www.openpolicyagent.org/docs/latest/kubernetes-introduction/

https://kubernetes.io/blog/2019/08/06/opa-gatekeeper-policy-and-governance-for-kubernetes/

https://open-policy-agent.github.io/gatekeeper/website/docs/

Rego

https://github.com/open-policy-agent/gatekeeper-library

Related Posts

Terraform Tips & Tricks – Part 1 – Building A Constant Reference

One of the most common problems I see in large organizations when working with terraform is consistency. When we have a large amount of resources being managed…

Everything You Ever Wanted to Know About Istio but Were Afraid to Ask

Istio is a powerful service mesh that integrates natively with Kubernetes, I have been using Istio as my service mesh, ingress, and egress gateways on my personal…

How to Monitor Your Enphase Home Solar System with Telegraf

How to collect metrics from an Enphase Envoy PV system, with telegraf and influxdb.

How to Deploy Anthos on Bare Metal On-Prem

Introduction The main advantage of Anthos on BM over Anthos on VMWare for on-prem deployments is the ability to run Anthos clusters without a hypervisor license. Cluster…

How to Setup Anthos on GKE Autopilot with Private Certificate Authority

What You Will Create The guide will set up the following: 2 Private GKE autopilot clusters with master global access ASM with multicluster mesh IstioIngress gateway to…

How to Setup a pfSense IPsec VPN Connection Behind a Router to Google Cloud

A tutorial on how to setup a virtual pfSense router behind a main router for VPN communications with Google Cloud Platform.

Leave a ReplyCancel reply

Exit mobile version