Kurtis Velarde

In the dynamic landscape of cloud-native applications, securing production environments is paramount. Proper threat modeling of workflows is essential to establish a baseline security posture. Tools like Falco and Gatekeeper are indispensable in this context. They provide powerful rule syntax that helps define and enforce the desired state of runtime workloads, alerting administrators whenever a container enters an undesired state. In today’s multi-cloud environments, where the origins of packages are often obscure, such capabilities are crucial for maintaining compliance and ensuring security. These tools are fundamental for professionals in the modern cloud-native ecosystem, covering essential aspects of security as defined by Gartner’s quadrants of “Cloud Configuration: Kubernetes Security Posture Management” and “Runtime Protection.”

Policy test for potential Kubernetes malicious use cases for policy testing in repository: https://github.com/kurtiepie/k8s_test_cases. Configs test malicious workloads to assess your policy defenses effectively.

Introduction to the Tools:

Gatekeeper: Ensuring Compliance through Policy Enforcement

Gatekeeper is an open-source policy engine that enhances Kubernetes security by enforcing policies at the admission stage. Here’s how Gatekeeper can be used to ensure that your Kubernetes clusters are secure and compliant. https://open-policy-agent.github.io/gatekeeper-library/website/ contains rules for building admission assurance policies include:

Image Management: Gatekeeper ensures that only images from approved registries are pulled and run, thereby preventing unauthorized or potentially harmful code from entering your production environment.
Security Context Enforcement: It checks and enforces security contexts and configurations such as privileges, capabilities, and access controls to prevent privilege escalation.
User and Group Management: Gatekeeper ensures that workloads are run with predefined user and group settings, minimizing the risk of unauthorized access.
Labeling and Auditing: Proper labeling (e.g., env: prod, Owner: Web) is enforced to ensure workloads are correctly identified and managed by auditing tools.
Volume Mount Restrictions: It can restrict the use of volume mounts, reducing the risk of data leakage or unauthorized data access.
Container Provenance: Gatekeeper can enforce the use of signed images, ensuring that only verified containers are deployed, thereby supporting a secure supply chain.

Falco: Real-Time Threat Detection with eBPF

Falco, leveraging eBPF (Extended Berkeley Packet Filter), is a cutting-edge intrusion and anomaly detection tool for real-time monitoring of system events in Kubernetes. https://github.com/falcosecurity/rules Open source rule set for runtime assurance polices include:

Shell Execution Monitoring: Falco can detect and alert on unauthorized shell executions in production environments, helping prevent potential breaches.
Network Traffic Control: It blocks any network calls to non-internal addresses (outside the 1918 address space), ensuring that internal communications remain secure.
Metadata Service Protection: Falco detects and prevents unauthorized access to cloud instance metadata services from containers, a common target for attackers.
Tool Usage Monitoring: It monitors and alerts on the use of suspicious tools like Nmap or socat within your containers, which might indicate a breach or an internal threat.
Memory Execution Detection: Detects execution of processes directly from memory (using MEM_FD syscalls) or from unusual locations like /dev/shm, which are red flags for malicious activity.
Drift Detection: Falco can alert on the unauthorized introduction and execution of binaries within containers, helping maintain the integrity of running applications.

Simple Threat Model with Malicious Pod’s example

The following threat model illustrates various activities observed at runtime in Kubernetes, including shell execution commands and unauthorized registry pulls during workload admission. This model depicts a typical attack scenario: deploying a privileged container from a malicious container registry, searching for privilege escalation patterns within the running container, and establishing communication with a remote command and control server.

Malicious workloads may initially be deployed on the Kubernetes control plane or within the standard runtime environment, seeking to move laterally thereafter. Such behavior generates numerous indicators of compromise, which can be promptly tracked, correlated, and alerted to our security and incident response teams in real-time.

Ideally, sufficient attention has been dedicated to assessing standard application activity patterns so that automated responses, such as API calls, can be initiated to halt malicious workloads automatically. Leveraging Falco and Gatekeeper’s API integrations facilitates not only alerting but also the mitigation of attacks in progress, offering a straightforward solution to enhance our security posture. Pod manifiest examples from the great https://github.com/BishopFox/badPods

Gatekeeper Admission Controller and Falco in Kubernetes Request Context

The following diagram illustrates the API request lifecycle in Kubernetes and identifies points where security controls can be implemented. Specifically, we focus on where policies can be applied during the initialization and execution phases of a workload within your cluster.

During initialization, admission control mechanisms enable us to define the types of workflows permissible for deployment into the cluster. These admission policies establish a baseline security posture by specifying parameters such as user and group settings for containers, container capability sets, Mandatory Access Control (MAC) profiles, volume mounts, container registries, environmental variables, and more.

Once the baseline security measures are defined during workload admission, runtime policies come into play to detect and alert deviations from this baseline. Runtime indicators of compromise include alerts for atypical file integrity monitoring, unexpected egress network traffic, container breakouts, malicious system calls, and other anomalous behaviors. These runtime policies serve as an active defense mechanism, enabling prompt detection and response to potential security threats within the Kubernetes environment

Guard Rail 1: Admission Controller Polices

The Guard Duty admission controller enables inspection of potential workloads defined in code and deployed within a running cluster. If a workload fails to meet the criteria defined in Kubernetes Custom Resource Definitions, admission into the cluster is denied. Leveraging Guard Duty with its flexible Rego Domain-Specific Language, any Kubernetes field can be assessed for compliance.

The following diagram outlines a policy designed to detect CronJob objects attempting to run with privileged mode enabled. Given that privileged mode disables all container-based security controls, such workloads typically warrant rejection. When such a policy violation occurs, it triggers a sequence of actions: notification within Slack, graphing of the event, and firing off an alert to the security team. This proactive approach ensures that potential security risks are promptly identified and addressed before they can compromise the integrity of the Kubernetes environment.

Guard Rail 2: Runtime Polices

With the admission controller enabling administrators to establish a baseline, such as requiring all containers to possess a specific set of labels indicating their running environment, runtime monitoring and alerting become pivotal. For instance, we can assert at runtime that any containers labeled { “env”: “production” } executing commands like execve (/bin/bash, /bin/sh, /bin/csh) deviate from our baseline behavior and should trigger alerts.

Upon detection of such anomalies, immediate actions, such as pausing the containers, mitigating the threat, and notifying the incident response teams, are essential. This exemplifies just one rule among many that admission and runtime policies empower us with to defend against attackers, serving as the final frontier in safeguarding our workloads.

Conclusion

Incorporating Falco and Gatekeeper into your Kubernetes clusters not only strengthens your security posture but also aligns with industry standards for runtime protection and configuration management. While these tools demand a notable investment in setup and maintenance, they are indispensable in the cloud-native landscape. They furnish a critical layer of security that evolves alongside the dynamic threat landscape, making them essential components for safeguarding Kubernetes environments.

Gatekeeper and Falco Rules

Example Makefile to Deploy Example with Supply chain and runtime security tools:

REG_URL := "192.168.64.12:30241"
NAME := "myalpineimage"
VERSION := "0.1"

falco: ## Deploy Falco registiry
  @helm install falco falcosecurity/falco --namespace falco --set falcosidekick.
enabled=true --set falcosidekick.webui.enabled=true --set auditLog.enabled=true

gatekeeper: ## Deploy Gatekeeper
  @helm install -n gatekeeper-system gatekeeper gatekeeper/gatekeeper --create-namespace

build: ## Docker Build
  @docker build . -t $(REG_URL)/$(NAME):$(VERSION)

push: ## Push to local registry
  @docker push $(REG_URL)/$(NAME):$(VERSION)

trivy-scan: ## SAST Scan for High and Critical CVEs
  @trivy image --severity HIGH,CRITICAL $(REG_URL)/$(NAME):$(VERSION)

test: ## Push to local registry
  kubectl run -it  --rm --restart=Never $(REG_URL)/$(NAME):$(VERSION) -- /bin/sh
 -c "uptime"