In today's container-centric infrastructure landscape, DevOps professionals need a clear understanding of every tool and process involved in securing workloads — from the first line of code to a running container in production. This article walks through both halves of the security lifecycle: the left side (build-time SDLC) and the right side (runtime defense).
The mental model is simple: everything to the left happens before deployment — code scanning, image building, vulnerability analysis, IaC validation. Everything to the right happens after deployment — runtime monitoring, behavioral detection, incident response.
A secure container workflow touches these stages in order:
Stages 1–4 are the left side (shift left). Stages 5–6 are the right side (runtime).
"Shift left" means catching security issues as early as possible in the development pipeline — before code reaches production. Every stage acts as a gate: if a check fails, the pipeline stops.
Your Git repository is an ecosystem, not just code storage:
Every one of these files is a security surface. A misconfigured Terraform module can expose an RDS instance to the internet. A Dockerfile running as root bypasses container isolation. A Helm chart without resource limits enables denial-of-service.
Before any image reaches a registry, the pipeline should enforce:
docker build, scan the resulting image for vulnerabilitiesEach gate uses an exit code to halt the pipeline on failure. In the Jenkinsfile below, --exit-code 192 tells Trivy to return a non-zero exit code when high-severity CVEs are found.
The Jenkinsfile defines the CI/CD pipeline as code. Each stage is a security gate:
pipeline {
agent any
environment {
APP = 'headers'
VERSION = "0.0.1"
GIT_HASH = """${sh(
returnStdout: true,
script: 'git rev-parse --short HEAD'
)}"""
dockerhub = credentials('dockerhub')
}
stages {
stage('Remote Code Repo Scan') {
steps {
echo "Running ${env.BUILD_ID} on ${env.JENKINS_URL}"
sh "trivy repo --exit-code 192 https://github.com/kurtiepie/headers.git"
}
}
stage('Code Base Scan') {
steps {
sh "trivy fs --exit-code 192 --severity HIGH,CRITICAL --skip-dirs ssl ."
}
}
stage('Docker Build') {
steps {
sh "docker build . -t ${APP}:${VERSION}-${GIT_HASH}"
}
}
stage('Scan Generated Image') {
steps {
sh "trivy image --exit-code 192 --severity HIGH,CRITICAL ${APP}:${VERSION}-${GIT_HASH}"
}
}
stage('Push to Registry') {
steps {
sh "docker tag ${APP}:${VERSION}-${GIT_HASH} kvad/headers:${VERSION}"
sh 'echo $dockerhub_PSW | docker login -u $dockerhub_USR --password-stdin'
sh "docker push kvad/headers:${VERSION}"
}
}
stage('Scan Helm IaC Files') {
steps {
sh "helm template headerschart/ > temp.yaml"
sh "trivy config --severity HIGH,CRITICAL --exit-code 192 ./temp.yaml"
sh "rm ./temp.yaml"
}
}
}
}
Key patterns:
headers:0.0.1-a3f2b1c) for traceability--exit-code 192 stops the pipeline on any high/critical findingThe Dockerfile is the blueprint for your container image. Security starts here:
FROM golang:1.16-alpine AS builder
WORKDIR /app
COPY go.mod ./
COPY go.sum ./
RUN go mod download
COPY *.go ./
RUN go build -o ./headers
FROM alpine:3.18
COPY --from=builder /app/headers /bin/headers
# Create non-root user and set ownership
RUN adduser -D headeruser && chown headeruser /bin/headers
USER headeruser
CMD ["/bin/headers"]
Secure practices demonstrated:
USER headeruser directive ensures the process doesn't run as root. If an attacker compromises the application, they land as an unprivileged user.alpine:3.18 is a few MB, far less attack surface than ubuntu or debian.golang:1.16-alpine, alpine:3.18) rather than :latest to avoid supply chain drift.| Mistake | Risk | Fix |
|---|---|---|
USER root |
Container escape trivial | Use USER <nonroot> |
FROM ubuntu:latest |
Large image, unpinned | Pin version, use alpine |
COPY . . before dependency install |
Cache invalidation, secrets in layers | Copy dependency files first |
RUN apt-get install -y curl wget |
Unnecessary tools for attackers | Only install what the app needs |
No .dockerignore |
Secrets, git history in image | Add .dockerignore excluding .git, .env |
Trivy's Dockerfile scanning catches many of these automatically:
trivy config --severity HIGH,CRITICAL ./Dockerfile
Once a container is deployed, the left-side gates are behind you. Runtime is where the real threats materialize — an attacker who bypasses your build checks, a zero-day in a dependency, or a compromised supply chain artifact.
Runtime security has three pillars:
Runtime agents observe what containers actually do at the kernel level using eBPF (Extended Berkeley Packet Filter). Every syscall — execve, connect, open, mount — is visible. The agent compares observed behavior against a baseline and fires alerts on deviations.
Key tools:
What to detect at runtime:
Shell spawned in container → execve of bash/sh in a production pod
Outbound connection to C2 → connect() to non-RFC1918 addresses
File written to /tmp → open() with O_WRONLY in writable paths
Privilege escalation attempt → setuid/setgid syscalls
Container drift → Binary executed that wasn't in the original image
Cloud metadata access → connect() to 169.254.169.254
Before a container even starts, admission controllers validate the pod spec against security policies. This is the bridge between left and right — it catches misconfigurations that slipped through CI/CD.
OPA Gatekeeper is the standard admission controller for Kubernetes:
# Block privileged containers
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sBlockPrivileged
metadata:
name: block-privileged-containers
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Pod"]
excludedNamespaces:
- kube-system
With this constraint active, any attempt to deploy a pod with privileged: true is rejected at the API server before the container is scheduled.
When a runtime alert fires, the response workflow should be automated:
The goal is to shrink the gap between detection and response from hours to seconds.
The left and right sides aren't separate — they're a continuous feedback loop:
Code Commit → Scan → Build → Scan → Push → Admission → Deploy → Monitor
↑ │
└──────── Runtime findings feed back into policy updates ──────┘
The organizations that close this loop — where runtime intelligence directly hardens the build pipeline — are the ones operating at a genuinely mature security posture.
trivy image <your-image> against your production images and review the findings