# Policy Enforcement

At Fairwinds, we encourage organizations to take a phased approach to rolling out policy enforcement using the Insights Admission Controller. Specifically, we believe in the following stages:

Stage 1: Awareness
Stage 2: Enforcement
Stage 3: Compliance

Many organizations subscribe to the idea of policy enforcement, but may struggle with the implementation strategy. This is where Fairwinds can help. Using the stages above, policy enforcement increases over time, helping platform engineering teams to balance shipping fast while driving security and reliability improvements.

# Define Policy "Groups"

We recommend two groups of policies:

Policy Group	Policies in scope (EventType)
Cluster-wide Policies	Best practices that should apply to all workloads. General recommendation includes: - Memory requests should be set (memoryRequestsMissing) - CPU requests should be set (cpuRequestsMissing) - Liveness probes should be set (livenessProbeMissing) - Readiness probes should be set (readinessProbeMissing) - Image pull policy should be “Always” (pullPolicyNotAlways) - Container should not have dangerous capabilities (dangerousCapabilities) - Image tag should be specified (tagNotSpecified)
Scoped Policies	Policies that may be scoped/targeted to specific resources - such as specific workload labels, namespaces, etc. - Container should not have insecure capabilities (insecureCapabilities) - Host IPC should not be configured (hostIPCSet) - Host PID should not be configured (hostPIDSet) - Privilege escalation should not be allowed (privilegeEscalationAllowed) - Should not be running as privileged (runAsPrivileged) - Container should not be running as root (runAsRootAllowed) Fairwinds supports additional policies, including those references under Profile Level 1 in the CNCF Mult-Tenancy Benchmark (opens new window)

Policy Group

Policies in scope (EventType)

Cluster-wide Policies

Best practices that should apply to all workloads. General recommendation includes:
- Memory requests should be set (memoryRequestsMissing)
- CPU requests should be set (cpuRequestsMissing)
- Liveness probes should be set (livenessProbeMissing)
- Readiness probes should be set (readinessProbeMissing)
- Image pull policy should be “Always” (pullPolicyNotAlways)
- Container should not have dangerous capabilities (dangerousCapabilities)
- Image tag should be specified (tagNotSpecified)

Scoped Policies

Policies that may be scoped/targeted to specific resources - such as specific workload labels, namespaces, etc.
- Container should not have insecure capabilities (insecureCapabilities)
- Host IPC should not be configured (hostIPCSet)
- Host PID should not be configured (hostPIDSet)
- Privilege escalation should not be allowed (privilegeEscalationAllowed)
- Should not be running as privileged (runAsPrivileged)
- Container should not be running as root (runAsRootAllowed)

Fairwinds supports additional policies, including those references under Profile Level 1 in the CNCF Mult-Tenancy Benchmark

# Policy Enforcement Stages

Below is an example strategy for gradually rolling out Policy Enforcement in your organization.

Generally speaking, a brand new cluster without active usage is usually the best time to introduce a policy enforcement strategy. This establishes a strong baseline of "best practices" from the start — lowering the cost of compliance on a go-forward basis.

That said, the below strategy is designed for more real-world scenarios where policy enforcement is needed for existing, running clusters with active usage. The three enforcement stages allow you to gradually roll out policy to these users/teams so best practices can be applied over time.

Stage	Admission Controller Mode	Policy Group	Report Violations for Running Workloads?	Enforce in CI?	Enforce in Admission Controller?	Tips
Stage 1: Awareness Alert on policy violations, but do not block.	Passive	Cluster-wide Policies AND Scoped Policies	Yes	Warn (do not block)	Warn (do not block)	Consider integrating Insights in CI to provide these notifications earlier in the development process.
Stage 2: Enforcement Begin blocking on Cluster-wide Policies.	Active	Cluster-wide Policies	Yes	Warn (do not block)	Block	Consider integrating Insights in CI to provide these notifications earlier in the development process.
		Scoped Policies	Yes	Warn (do not block)	Warn (do not block)	Consider integrating Insights in CI to provide these notifications earlier in the development process.
Stage 3: Compliance Begin blocking on both Cluster-wide and Scoped Policies.	Active	Cluster-wide Policies	Yes	Block	Block	Consider integrating Insights in CI to enforce Cluster-wide Policies
		Scoped Policies	Yes	Warn (do not block)	Block	Warning in CI ensures teams can get feedback on code, but begin enforcing Scoped Policies at time of Admission

# Implementing Enforcement Stages in Insights

In this guide, you will learn how to:

Enable/disable Admission Controller
Setup Insights in your CI/CD pipeline
Use the Insights CLI to manage Policy Configuration
Create Automation Rules to fine-tune your policy enforcement behaviors in CI and at time of Admission

# Implementing Stage 1: Awareness

In this stage, we want to warn users when their YAML code or Admission requests contain misconfigurations found in our Cluster-wide Policies or Scoped Policies.

The simplest way to do this is enabling Admission Controller in your cluster and setting it to Passive Mode (which is enabled by default). Additionally, you can install Insights in your CI pipeline to surface misconfigurations sooner in the process.

# 1. Install Admission Controller

You can install the Admission Controller using the Install Hub. Learn how to install Admission Controller here.

# 2. Set Admission Controller to Passive Mode

Admission Controller is set to Passive Mode by default. Learn more how to enable/disable Passive Mode in Admission Controller here.

# 3. OPTIONAL: Install Insights in your CI pipeline to warn users of Policy violations

Learn how to setup the Insights Continuous Integration (CI) feature here.

By default, the Insights CI integration is configured NOT to fail CI pipelines.

Verify that your fairwinds-insights.yaml file at the root of your repo has the options.setExitCode set to false. Learn more about gating pull requests with the Insights CI integration here.

# Implementing Stage 2: Enforcement

In this stage, we want to enforce our Cluster-wide Policies at time of Admission, but only notify on Scoped Policy violations.

Before proceeding, please verify:

Admission Controller is installed and set to Passive Mode
The Insights CI integration continues to warn when Action Items are found in CI pipelines -- but will not block pipelines. Verify you have options.setExitCode set to false .

# 1. Disable Passive Mode in Admission Controller

Disabling Admission Controller is done on a per-cluster basis. Learn more how to enable/disable Passive Mode in Admission Controller here.

# 2. Use Policy Configuration to enforce Cluster-wide Policies at Admission

Download the Insights CLI
Create a settings.yaml file with the configuration below.
Upload settings.yaml using the Insights CLI. Learn how to manage Policy Configuration using the Insights CLI here.

Example settings.yaml:

checks:
  polaris:
    memoryRequestsMissing:
      admission:
        block: true
    cpuRequestsMissing:
      admission:
        block: true
    pullPolicyNotAlways:
      admission:
        block: true
    livenessProbeMissing:
      admission:
        block: true
    readinessProbeMissing:
      admission:
        block: true
    dangerousCapabilities:
      admission:
        block: true
    tagNotSpecified:
      admission:
        block: true

# 3. Create Automation Rule to keep blocking mode focused on Cluster-wide Policies only

Navigate to the Automation page
Click 'Create Custom Rule'
Create a new Automation Rule with the following settings:

Automation Rule: 0002-enforce-cluster-wide-policies-only

Context: Admission Controller

//Note: Admission Controller will automatically block when Action Items with a 0.7 severity or higher (High and Critical) are found. This lowers severities to avoid unncessary enforcement.
if (ActionItem.Severity >= 0.7) {
  ActionItem.Severity = MEDIUM_SEVERITY;
}

# Implementing Stage 3: Compliance

In this stage, we want to enforce our Cluster-wide Policies and Scoped policies

Optionally, we can enable the Insights CI script to fail pipelines where Cluster-wide Policy issues are present.

Before proceeding, please verify:

Admission Controller is installed and Passive Mode is disabled
The Insights CI integration is installed on at least one pipeline and continues to have options.setExitCode set to false

# 1. Disable the '`0002-enforce-cluster-wide-policies-only`' Automation Rule

To disable the Automation Rule:

Ensure you have Owner permissions in Insights
Navigate to the Automation page
Click on the 0002-enforce-cluster-wide-policies-only Automation Rule
In the upper-right, toggle off the 'Enable' switch
Click 'Update Rule'

In the next step, you will create a new Automation Rule that enforces Scoped Policies for specific Namespaces.

# 2. Create Automation Rule to enforce Cluster-wide and Scoped Policies at Admission

Navigate to the Automation page
Click 'Create Custom Rule'
Create a new Automation Rule with the following settings:

Automation Rule: 0003-enforce-all-policies

Context: Admission Controller

//Define Namespaces to apply policy to, as well as the specific policies to enforce.
namespaceScope = new Array("business-app", "production");
policyScope = new Array(
  "insecureCapabilities",
  "hostIPCSet",
  "hostPIDSet",
  "privilegeEscalationAllowed",
  "runAsPrivileged",
  "runAsRootAllowed"
);

//If Namespace Annotations are set, use this to define specific policies to enforce. Else, use what's configured in the policyScope variable.
policiesToEnforce = new Array();
if (ActionItem.NamespaceAnnotations["insights.fairwinds.com/enforce"]) {
  policiesToEnforce = JSON.parse(
    ActionItem.NamespaceAnnotations["insights.fairwinds.com/enforce"]
  );
} else if (namespaceScope.indexOf(ActionItem.ResourceNamespace) !== -1) {
  policiesToEnforce = policyScope;
}

//Policy Enforcement logic
//Note: Admission Controller will automatically block when Action Items with a 0.7 severity or higher (High and Critical) are found. This lowers severities to avoid unncessary enforcement.
if (ActionItem.Severity >= 0.7) {
  ActionItem.Severity = MEDIUM_SEVERITY;
}

if (policiesToEnforce.length > 0) {
  if (policiesToEnforce.indexOf(ActionItem.EventType) !== -1) {
    //Enforce the policy specified in the Namespace Annotation
    ActionItem.Severity = CRITICAL_SEVERITY;
    //Send optional Slack notification
    //sendSlackNotification("infra-notifications", "A deployment failed due to the following issue: "+JSON.stringify(ActionItem));
  }
}

OPTIONAL: Use Namespace Annotations to customize which policies you'd like to enforce at the Namespace level

The above 0003-enforce-all-policies Automation Rule allows you to define a custom list of policies you'd like to enforce for a given namespace using the insights.fairwinds.com/enforce annotation.

Here's an example of how to use this:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    insights.fairwinds.com/enforce: ['runAsRootAllowed','insecureCapabilities'];

# 3. OPTIONAL: Configure Insights to enforce Cluster-wide Policies in CI

# 3A) Set the Insights CI integration to gate pull requests

Open the fairwinds-insights.yaml file at the root of your Git repo and set options.setExitCode to true.

# 3B) Update Policy Configuration to enforce Cluster-wide Policies in CI

Download the Insights CLI
Update your existing settings.yaml file with the configuration below.
Upload settings.yaml using the Insights CLI. Learn how to manage Policy Configuration using the Insights CLI here.

Example settings.yaml:

checks:
  polaris:
    memoryRequestsMissing:
      ci:
        block: true
      admission:
        block: true
    cpuRequestsMissing:
      ci:
        block: true
      admission:
        block: true
    pullPolicyNotAlways:
      ci:
        block: true
      admission:
        block: true
    livenessProbeMissing:
      ci:
        block: true
      admission:
        block: true
    readinessProbeMissing:
      ci:
        block: true
      admission:
        block: true
    dangerousCapabilities:
      ci:
        block: true
      admission:
        block: true
    tagNotSpecified:
      ci:
        block: true
      admission:
        block: true

# Managing Exceptions

To configure a workload to bypass the Admission Controller, there are two possible options:

# RECOMMENDED: Bypass Admission Controller using YAML Annotations

Add a specific annotation to that workload, and use an Automation Rule to bypass Admission Controller

With this option, security engineers and platform engineers are encouraged to use Git workflows to review/approve annotations added by development teams. This provides an audit trail for exceptions, and follows policy-as-code best practices.

Automation Rule: ignore-via-annotation

Context: Admission Controller

//Bypass Admission Controller if an annotation like this exists: insights.fairwinds.com/ignore: "[\"runAsRootAllowed\"]"
policyException =
  ActionItem.ResourceAnnotations["insights.fairwinds.com/ignore"];

if (policyException) {
  exceptions = JSON.parse(policyException);
  if (exceptions.indexOf(ActionItem.EventType) !== -1) {
    //Reduce severity and resolve this ActionItem so it can bypass the Admission Controller
    ActionItem.Severity = LOW_SEVERITY;
    ActionItem.Resolution = WORKING_AS_INTENDED_RESOLUTION;
  }
}

# Bypass Admission Controller using an Allow List

Create an "allow list" within an Insights Automation Rule that forces the Severity of that Action Item to Low.

With this option, security engineers and platform engineers carry the burden of maintaining an exception list. This is useful if development teams are still not fully responsible for their Kubernetes workload configurations.

Automation Rule: ignore-via-allowlist

Context: Admission Controller

exceptionList = new Array();

//Create an allowlist using EventType, ResourceNamespace, and ResourceName combination.
//Below is an exception example for kube-system workloads that need to run as root
exceptionList["runAsRootAllowed"] = new Array(
  "kube-system/cert-manager",
  "kube-system/external-dns"
);

//Enforcement logic for the Allow List
if (exceptionList[ActionItem.EventType].length > 0) {
  if (
    exceptionList[ActionItem.EventType][
      ActionItem.ResourceNamespace + "/" + ActionItem.ResourceName
    ]
  ) {
    //Reduce severity and resolve this ActionItem so it can bypass the Admission Controller
    ActionItem.Severity = LOW_SEVERITY;
    ActionItem.Resolution = WORKING_AS_INTENDED_RESOLUTION;
  }
}

← Cost Efficiency In-Cluster Scanning →

# Policy Enforcement

# Define Policy "Groups"

# Policy Enforcement Stages

# Implementing Enforcement Stages in Insights

# Implementing Stage 1: Awareness

# 1. Install Admission Controller

# 2. Set Admission Controller to Passive Mode

# 3. OPTIONAL: Install Insights in your CI pipeline to warn users of Policy violations

# Implementing Stage 2: Enforcement

# 1. Disable Passive Mode in Admission Controller

# 2. Use Policy Configuration to enforce Cluster-wide Policies at Admission

# 3. Create Automation Rule to keep blocking mode focused on Cluster-wide Policies only

# Implementing Stage 3: Compliance

# 1. Disable the '0002-enforce-cluster-wide-policies-only' Automation Rule

# 2. Create Automation Rule to enforce Cluster-wide and Scoped Policies at Admission

# 3. OPTIONAL: Configure Insights to enforce Cluster-wide Policies in CI

# 3A) Set the Insights CI integration to gate pull requests

# 3B) Update Policy Configuration to enforce Cluster-wide Policies in CI

# Managing Exceptions

# RECOMMENDED: Bypass Admission Controller using YAML Annotations

# Bypass Admission Controller using an Allow List

# 1. Disable the '`0002-enforce-cluster-wide-policies-only`' Automation Rule