Best Practices for SBOMs, Trivy Scans, and Automated CVE Mitigation in Docker Images¶

Introduction¶

Container security isn't optional. Every Docker image you ship is a potential attack vector. The question isn't whether to scan—it's how to scan systematically, generate Software Bill of Materials (SBOMs) for traceability, and automate remediation without breaking production.

Why SBOMs Matter:

SBOMs provide a complete inventory of software components in your images. They enable: - Supply chain traceability: Know exactly what's in every image - Vulnerability correlation: Map CVEs to specific components - Compliance: Meet regulatory requirements (SLSA, SOC2, ISO 27001) - Incident response: Quickly identify affected images when a CVE is disclosed

Why Trivy (and Similar Scanners):

Trivy is fast, accurate, and integrates well into CI/CD pipelines. It scans: - Container images (OS packages, language dependencies) - Filesystems and repositories - Infrastructure as Code (Dockerfiles, Kubernetes manifests) - Git repositories

But scanning alone isn't enough. You need workflows that automatically reduce vulnerabilities, not just detect them.

The Limitations of "Just Scan":

Running Trivy and failing builds on CVEs creates security theater, not security. Without remediation workflows: - Developers ignore failing builds - Critical vulnerabilities accumulate - Security debt compounds - Compliance becomes a checkbox exercise

This guide shows you how to build pipelines that scan, generate SBOMs, and automatically fix vulnerabilities in a controlled, auditable way.

SBOM Fundamentals for Docker Images¶

What is an SBOM?¶

A Software Bill of Materials (SBOM) is a machine-readable inventory of all software components in a container image. Think of it as a manifest that lists: - Operating system packages (apt, yum, apk) - Language dependencies (Python packages, npm modules, Go modules) - Base image layers - Build-time dependencies

Common Formats:

• CycloneDX: JSON/XML format, widely supported, good for tooling
• SPDX: ISO/IEC standard, comprehensive metadata, good for compliance
• Syft JSON: Syft's native format, easy to parse

Where SBOMs Fit:

1. Build Time: Generate SBOM during image build, store as artifact
2. Release Time: Attach SBOM to image tag, publish to registry
3. Runtime/Deployment: Reference SBOM for vulnerability correlation, compliance audits

Inside Image vs Sidecar Artifact¶

Inside Image (SBOM as label/annotation): - Pros: Always travels with image, no separate artifact management - Cons: Increases image size, harder to update without rebuilding

Sidecar Artifact (SBOM as separate file): - Pros: Smaller images, can update SBOM without rebuilding, easier to version - Cons: Requires artifact management, risk of SBOM/image mismatch

Recommendation: Use sidecar artifacts stored alongside images in your registry or artifact repository. Reference SBOMs via image labels or annotations.

Tools & Ecosystem Overview¶

Trivy¶

Trivy is a comprehensive security scanner that: - Scans container images for OS and language vulnerabilities - Generates SBOMs in multiple formats (CycloneDX, SPDX) - Scans filesystems, Git repos, and IaC - Integrates with CI/CD pipelines - Provides JSON, SARIF, and HTML output formats

Why Focus on Trivy: - Fast scanning (parallel processing, caching) - Accurate results (comprehensive vulnerability databases) - Active development and community - Good documentation and examples - Free and open source

Alternative Tools¶

Syft + Grype (Anchore): - Syft generates SBOMs, Grype scans them - Good alternative if you prefer separate tools - Syft has excellent language dependency detection

Docker Scout: - Docker's native scanning tool - Integrated with Docker Hub and Docker Desktop - Good for Docker-native workflows

GitLab / GitHub Native Scanners: - Built into CI/CD platforms - Convenient but less flexible than Trivy - Good for getting started, consider Trivy for advanced workflows

Recommendation: Start with Trivy for maximum flexibility. Use platform-native scanners as secondary validation.

Generating SBOMs for Docker Images¶

Generating SBOMs from Images¶

Basic Trivy SBOM Generation:

# Generate CycloneDX JSON SBOM
trivy image --format cyclonedx --output sbom.cdx.json myimage:tag

# Generate SPDX JSON SBOM
trivy image --format spdx-json --output sbom.spdx.json myimage:tag

# Generate Syft JSON (using Syft directly)
syft myimage:tag -o json > sbom.syft.json

Tying SBOMs to Image Tags:

Use image labels to reference SBOMs:

# In Dockerfile
LABEL org.opencontainers.image.source="https://github.com/org/repo"
LABEL sbom.digest="sha256:abc123..."
LABEL sbom.format="cyclonedx"
LABEL sbom.location="s3://bucket/sboms/myimage-tag.cdx.json"

Or use OCI annotations:

# After building image
docker buildx imagetools create \
  --annotation "org.opencontainers.image.sbom.digest=sha256:abc123" \
  --annotation "org.opencontainers.image.sbom.format=cyclonedx" \
  myimage:tag

Generating SBOMs at Build Time¶

Multi-Stage Build Pattern:

# Dockerfile
FROM python:3.11-slim AS builder

WORKDIR /app
COPY requirements.txt .
RUN pip install --user -r requirements.txt

# Generate SBOM in build stage
FROM aquasec/trivy:latest AS trivy
COPY --from=builder /root/.local /tmp/deps
RUN trivy fs --format cyclonedx --output /tmp/sbom.cdx.json /tmp/deps

FROM python:3.11-slim
COPY --from=builder /root/.local /root/.local
COPY --from=trivy /tmp/sbom.cdx.json /sbom.cdx.json
ENV PATH=/root/.local/bin:$PATH

COPY app.py .
CMD ["python", "app.py"]

CI/CD Build-Time Generation:

# GitHub Actions example
- name: Build Docker image
  run: docker build -t myimage:${{ github.sha }} .

- name: Generate SBOM
  run: |
    trivy image \
      --format cyclonedx \
      --output sbom.cdx.json \
      myimage:${{ github.sha }}

- name: Upload SBOM artifact
  uses: actions/upload-artifact@v3
  with:
    name: sbom
    path: sbom.cdx.json

Storing SBOMs¶

Option 1: Artifact Repository (S3, GCS, Artifactory):

# Upload SBOM to S3 with versioned naming
aws s3 cp sbom.cdx.json \
  s3://my-artifacts/sboms/myimage-${IMAGE_TAG}-${IMAGE_DIGEST}.cdx.json

# Store with metadata
aws s3 cp sbom.cdx.json \
  s3://my-artifacts/sboms/myimage-${IMAGE_TAG}.cdx.json \
  --metadata "image-digest=${IMAGE_DIGEST},image-tag=${IMAGE_TAG}"

Option 2: OCI Artifacts in Container Registry:

# Push SBOM as OCI artifact (Docker Hub, GHCR, Harbor support this)
oras push \
  myregistry.io/myimage:${TAG} \
  sbom.cdx.json:application/vnd.cyclonedx+json

# Or using Trivy's built-in support
trivy image --format cyclonedx \
  --output sbom.cdx.json \
  --sbom-artifact-type cyclonedx \
  myimage:tag

Best Practices:

1. Always generate SBOMs for release images: Tagged releases, not just dev builds
2. Version SBOMs with image digest: Use image-digest in SBOM filename or metadata
3. Include SBOM references in images: Use labels/annotations to link image to SBOM
4. Store SBOMs immutably: Use content-addressable storage (digest-based naming)
5. Generate multiple formats: CycloneDX for tooling, SPDX for compliance

Scanning Docker Images with Trivy¶

Core Patterns¶

Scanning Local Images:

# Basic scan
trivy image myimage:tag

# Scan with severity filter
trivy image --severity HIGH,CRITICAL myimage:tag

# Scan and fail on policy violations
trivy image --exit-code 1 --severity HIGH,CRITICAL myimage:tag

# Scan with JSON output
trivy image --format json --output scan.json myimage:tag

Scanning Remote Registry Images:

# Scan from Docker Hub
trivy image docker.io/library/nginx:latest

# Scan from private registry (with auth)
trivy image --username $REGISTRY_USER --password $REGISTRY_PASS \
  myregistry.io/myimage:tag

# Scan from AWS ECR
trivy image --aws-region us-east-1 \
  123456789012.dkr.ecr.us-east-1.amazonaws.com/myimage:tag

Scanning Dockerfiles:

# Scan Dockerfile for misconfigurations
trivy config Dockerfile

# Scan with policy checks
trivy config --severity HIGH,CRITICAL Dockerfile

Recommended Configurations¶

Severity Thresholds:

# Fail build on CRITICAL and HIGH only
trivy image \
  --exit-code 1 \
  --severity HIGH,CRITICAL \
  --ignorefile .trivyignore \
  myimage:tag

Ignoring Specific CVEs:

Create .trivyignore file:

# .trivyignore
# CVE-2023-12345: False positive in our use case
# Justification: This CVE affects a feature we don't use
CVE-2023-12345

# CVE-2023-67890: Acceptable risk, will fix in next release
# Justification: Low severity in our deployment context, scheduled fix Q2 2024
CVE-2023-67890

Caching for Performance:

# Use cache directory (persist across CI runs)
trivy image --cache-dir /tmp/trivy-cache myimage:tag

# Or use Trivy DB cache
trivy image --cache-backend redis://redis:6379 myimage:tag

CI/CD Integration Examples¶

GitHub Actions (Basic):

name: Security Scan

on:
  push:
    branches: [main]
  pull_request:

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Build image
        run: docker build -t myimage:${{ github.sha }} .

      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: myimage:${{ github.sha }}
          format: 'sarif'
          output: 'trivy-results.sarif'
          severity: 'CRITICAL,HIGH'

      - name: Upload Trivy results to GitHub Security
        uses: github/codeql-action/upload-sarif@v2
        with:
          sarif_file: 'trivy-results.sarif'

GitHub Actions (Advanced with SBOM):

name: Build, SBOM, and Scan

on:
  push:
    tags:
      - 'v*'

jobs:
  security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: Build image
        uses: docker/build-push-action@v4
        with:
          context: .
          push: true
          tags: myregistry.io/myimage:${{ github.ref_name }}
          cache-from: type=registry,ref=myregistry.io/myimage:buildcache
          cache-to: type=registry,ref=myregistry.io/myimage:buildcache

      - name: Generate SBOM
        run: |
          trivy image \
            --format cyclonedx \
            --output sbom.cdx.json \
            myregistry.io/myimage:${{ github.ref_name }}

      - name: Upload SBOM
        uses: actions/upload-artifact@v3
        with:
          name: sbom
          path: sbom.cdx.json

      - name: Scan image
        run: |
          trivy image \
            --format json \
            --output scan.json \
            --exit-code 1 \
            --severity HIGH,CRITICAL \
            --ignorefile .trivyignore \
            myregistry.io/myimage:${{ github.ref_name }}

      - name: Upload scan results
        uses: actions/upload-artifact@v3
        with:
          name: scan-results
          path: scan.json

      - name: Publish SBOM to registry
        run: |
          oras push \
            myregistry.io/myimage:${{ github.ref_name }} \
            sbom.cdx.json:application/vnd.cyclonedx+json

GitLab CI:

# .gitlab-ci.yml
stages:
  - build
  - security

build:
  stage: build
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

security:
  stage: security
  image: aquasec/trivy:latest
  script:
    # Generate SBOM
    - trivy image --format cyclonedx --output sbom.cdx.json $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

    # Scan image
    - trivy image --format json --output scan.json --exit-code 1 --severity HIGH,CRITICAL $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

    # Upload artifacts
    - |
      if [ -f sbom.cdx.json ]; then
        echo "SBOM generated successfully"
      fi
  artifacts:
    reports:
      sast: scan.json
    paths:
      - sbom.cdx.json
      - scan.json
    expire_in: 30 days

Output Management¶

Storing Reports:

# JSON for programmatic processing
trivy image --format json --output scan.json myimage:tag

# SARIF for GitHub/GitLab integration
trivy image --format sarif --output scan.sarif myimage:tag

# HTML for human-readable reports
trivy image --format template --template "@contrib/html.tpl" --output scan.html myimage:tag

Surfacing Results:

• CI Artifacts: Upload JSON/SARIF/HTML reports as build artifacts
• Security Dashboards: Integrate SARIF with GitHub Security, GitLab Security Dashboard
• Alerting: Parse JSON output, send alerts for new CRITICAL CVEs
• Compliance Reports: Generate periodic HTML reports for audits

Policies & Governance Around CVEs¶

Defining Risk Thresholds¶

Example Policy:

# security-policy.yaml
vulnerability_policy:
  fail_build:
    severity: ["CRITICAL"]
    count_threshold: 0  # Fail on any CRITICAL

  warn_build:
    severity: ["HIGH"]
    count_threshold: 5  # Warn if more than 5 HIGH severity

  ignore_list:
    file: .trivyignore
    require_justification: true
    review_required: true

Implementation:

# Fail on CRITICAL, warn on HIGH
trivy image \
  --exit-code 1 \
  --severity CRITICAL \
  --ignorefile .trivyignore \
  myimage:tag

# Generate report for HIGH (non-blocking)
trivy image \
  --format json \
  --severity HIGH \
  --output high-severity-report.json \
  myimage:tag

Managing False Positives¶

Process:

1. Identify false positive: Verify CVE doesn't affect your deployment
2. Document justification: Add comment in .trivyignore
3. Review: Security team reviews ignore list entries
4. Periodic audit: Re-evaluate ignored CVEs quarterly

Example .trivyignore with Justifications:

# CVE-2023-12345: False positive
# Component: libssl1.1
# Justification: CVE affects TLS 1.0/1.1, we only use TLS 1.2+
# Reviewed: 2024-01-15 by security-team
# Next review: 2024-04-15
CVE-2023-12345

# CVE-2023-67890: Acceptable risk
# Component: python3.9
# Justification: Low severity in containerized context, no network exposure
# Mitigation: Network policies restrict access
# Reviewed: 2024-01-20 by security-team
CVE-2023-67890

OS-Level vs App-Level CVEs¶

OS-Level CVEs: - Fixed by updating base image or running apt-get upgrade - Usually higher priority (affects entire container) - Mitigation: Regular base image updates

App-Level CVEs: - Fixed by updating language dependencies - May require code changes - Mitigation: Dependency update automation

Policy Example:

# Separate handling for OS vs app CVEs
# OS CVEs: Fail on CRITICAL
trivy image --severity CRITICAL --scanners os myimage:tag

# App CVEs: Warn on HIGH, fail on CRITICAL
trivy image --severity HIGH,CRITICAL --scanners lang myimage:tag

Maintaining Allowlists¶

Process:

1. Request: Developer requests CVE addition to ignore list
2. Justification: Require written justification
3. Review: Security team reviews request
4. Approval: Merge to .trivyignore with review comment
5. Audit: Quarterly review of all ignored CVEs

Automation:

# Validate .trivyignore format
trivy image --ignorefile .trivyignore --validate myimage:tag

# Generate report of ignored CVEs
trivy image --format json --ignorefile .trivyignore myimage:tag | \
  jq '.Results[] | select(.Vulnerabilities != null) | .Vulnerabilities[] | select(.VulnerabilityID | IN($ignored[]))'

Designing Automated CVE Mitigation Workflows¶

7.1 Base Image Hygiene¶

Strategy: Regularly update base images and rebuild.

Best Practices:

1. Use minimal base images: -slim, distroless, or alpine
2. Pin to digests: Use @sha256:... for reproducibility
3. Automate updates: Use Dependabot/Renovate to bump base images
4. Rescan after updates: Automatically run Trivy on updated images

Example Workflow:

# .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: "docker"
    directory: "/"
    schedule:
      interval: "weekly"
    open-pull-requests-limit: 5

Automated Base Image Update Flow:

flowchart LR
    Bot["Dependabot/Renovate"] --> PR["Open PR: Update Base Image"]
    PR --> Build["CI: Build Image"]
    Build --> Scan["Trivy Scan"]
    Scan --> Check{"CVEs<br/>Reduced?"}
    Check -->|Yes| Merge["Auto-Merge PR"]
    Check -->|No| Review["Human Review"]
    Merge --> Tag["Tag New Image"]

Implementation:

# GitHub Actions: Auto-merge if CVE count reduced
name: Auto-merge Base Image Updates

on:
  pull_request:
    paths:
      - 'Dockerfile'

jobs:
  check-cves:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Build candidate image
        run: docker build -t candidate:${{ github.sha }} .

      - name: Scan current image
        run: |
          trivy image --format json --output current-scan.json \
            myregistry.io/myimage:latest
          CURRENT_CVES=$(jq '[.Results[].Vulnerabilities[]? | select(.Severity == "CRITICAL" or .Severity == "HIGH")] | length' current-scan.json)
          echo "CURRENT_CVES=$CURRENT_CVES" >> $GITHUB_ENV

      - name: Scan candidate image
        run: |
          trivy image --format json --output candidate-scan.json \
            candidate:${{ github.sha }}
          CANDIDATE_CVES=$(jq '[.Results[].Vulnerabilities[]? | select(.Severity == "CRITICAL" or .Severity == "HIGH")] | length' candidate-scan.json)
          echo "CANDIDATE_CVES=$CANDIDATE_CVES" >> $GITHUB_ENV

      - name: Auto-merge if improved
        if: env.CANDIDATE_CVES < env.CURRENT_CVES
        run: |
          gh pr merge ${{ github.event.pull_request.number }} --auto --squash

7.2 Package-Level Updates in Dockerfiles¶

Strategy: Automatically bump OS packages and language dependencies.

OS Package Updates:

# Option 1: Update at build time (pros: latest packages, cons: non-deterministic)
FROM debian:bookworm-slim
RUN apt-get update && \
    apt-get upgrade -y && \
    apt-get install -y python3 && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Option 2: Pin versions, update via automation (pros: deterministic, cons: manual updates)
FROM debian:bookworm-slim
RUN apt-get update && \
    apt-get install -y \
      python3=3.11.2-1 \
      python3-pip=23.0-1 && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

Language Dependency Updates:

Python (pip/poetry/uv):

# Use lock files for reproducibility
COPY requirements.txt poetry.lock ./
RUN pip install --no-cache-dir -r requirements.txt

# Or with poetry
RUN poetry install --no-dev --no-interaction

Automated Dependency Updates:

# .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: "pip"
    directory: "/"
    schedule:
      interval: "weekly"
    open-pull-requests-limit: 10
    versioning-strategy: increase

Node.js (npm/yarn):

COPY package.json package-lock.json ./
RUN npm ci --only=production

Go Modules:

COPY go.mod go.sum ./
RUN go mod download

Guardrails:

1. Use lock files: requirements.txt, package-lock.json, go.sum
2. Automated PRs: Dependabot/Renovate create PRs for updates
3. CI validation: Rebuild and rescan on dependency updates
4. Test coverage: Ensure tests catch breaking changes

7.3 Autofix Loop with Scanner Feedback¶

Concept: Parse Trivy results, suggest Dockerfile changes, create PRs.

Workflow:

sequenceDiagram
    participant CI as CI Pipeline
    participant Trivy as Trivy Scanner
    participant Parser as CVE Parser
    participant Fixer as Autofix Engine
    participant PR as PR Creator

    CI->>Trivy: Scan image
    Trivy->>Parser: JSON scan results
    Parser->>Parser: Map CVEs to packages
    Parser->>Fixer: CVE + package mappings
    Fixer->>Fixer: Generate Dockerfile changes
    Fixer->>PR: Create PR with fixes
    PR->>CI: Trigger rebuild
    CI->>Trivy: Rescan fixed image
    Trivy->>CI: Verify CVEs reduced

Implementation:

# scripts/autofix.py
import json
import subprocess
import re
from typing import Dict, List

def parse_trivy_results(scan_json: str) -> List[Dict]:
    """Parse Trivy JSON output and extract fixable CVEs."""
    with open(scan_json) as f:
        data = json.load(f)

    fixable_cves = []
    for result in data.get("Results", []):
        for vuln in result.get("Vulnerabilities", []):
            if vuln.get("Severity") in ["CRITICAL", "HIGH"]:
                pkg_name = vuln.get("PkgName")
                installed_version = vuln.get("InstalledVersion")
                fixed_version = vuln.get("FixedVersion")

                if fixed_version:
                    fixable_cves.append({
                        "cve_id": vuln.get("VulnerabilityID"),
                        "package": pkg_name,
                        "current": installed_version,
                        "fixed": fixed_version,
                        "severity": vuln.get("Severity"),
                    })

    return fixable_cves

def generate_dockerfile_fix(dockerfile_path: str, fixes: List[Dict]) -> str:
    """Generate updated Dockerfile with package version fixes."""
    with open(dockerfile_path) as f:
        dockerfile = f.read()

    # Map fixes to Dockerfile changes
    for fix in fixes:
        pkg = fix["package"]
        old_ver = fix["current"]
        new_ver = fix["fixed"]

        # Pattern: apt-get install package=version
        pattern = f"{pkg}={re.escape(old_ver)}"
        replacement = f"{pkg}={new_ver}"
        dockerfile = re.sub(pattern, replacement, dockerfile)

    return dockerfile

def create_fix_pr(dockerfile_changes: str, fixes: List[Dict]) -> None:
    """Create PR with suggested fixes."""
    # Write updated Dockerfile
    with open("Dockerfile.fixed", "w") as f:
        f.write(dockerfile_changes)

    # Create PR via GitHub CLI or API
    subprocess.run([
        "gh", "pr", "create",
        "--title", f"Security: Auto-fix {len(fixes)} CVEs",
        "--body", generate_pr_body(fixes),
        "--file", "Dockerfile.fixed",
    ])

if __name__ == "__main__":
    fixes = parse_trivy_results("scan.json")
    if fixes:
        dockerfile_fixed = generate_dockerfile_fix("Dockerfile", fixes)
        create_fix_pr(dockerfile_fixed, fixes)

CI Integration:

# GitHub Actions: Autofix workflow
name: Autofix CVEs

on:
  schedule:
    - cron: '0 0 * * 1'  # Weekly
  workflow_dispatch:

jobs:
  autofix:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Build and scan
        run: |
          docker build -t testimage .
          trivy image --format json --output scan.json testimage

      - name: Run autofix
        run: python scripts/autofix.py

      - name: Create PR if fixes available
        if: steps.autofix.outputs.has_fixes == 'true'
        uses: peter-evans/create-pull-request@v5
        with:
          title: 'Security: Auto-fix CVEs'
          body: 'Automated CVE fixes from Trivy scan'
          branch: autofix/cves

Important: Autofix should create PRs for review, not mutate main branch directly.

7.4 Golden Base Images¶

Strategy: Maintain blessed base images, periodically update and scan.

Workflow:

flowchart TB
    Base["Golden Base Image"] --> Update["Weekly Update Job"]
    Update --> Build["Rebuild Base Image"]
    Build --> Scan["Trivy Scan"]
    Scan --> Check{"Passes<br/>Policy?"}
    Check -->|Yes| Tag["Tag New Version"]
    Check -->|No| Alert["Alert Security Team"]
    Tag --> Promote["Promote to Golden"]
    Promote --> App["Application Images Inherit"]

Implementation:

# golden-base/Dockerfile
FROM debian:bookworm-slim

# Install common dependencies
RUN apt-get update && \
    apt-get install -y \
      curl \
      ca-certificates \
      && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Label with SBOM reference
LABEL org.opencontainers.image.title="Golden Base Image"
LABEL org.opencontainers.image.version="2024.01"

Golden Image Update Pipeline:

# .github/workflows/update-golden-base.yml
name: Update Golden Base Image

on:
  schedule:
    - cron: '0 0 * * 1'  # Weekly
  workflow_dispatch:

jobs:
  update:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Build golden base
        run: |
          docker build -t golden-base:latest ./golden-base

      - name: Generate SBOM
        run: |
          trivy image --format cyclonedx --output golden-base-sbom.cdx.json \
            golden-base:latest

      - name: Scan golden base
        run: |
          trivy image --format json --output golden-base-scan.json \
            --exit-code 1 --severity CRITICAL \
            golden-base:latest

      - name: Tag and push if clean
        if: success()
        run: |
          VERSION=$(date +%Y.%m.%d)
          docker tag golden-base:latest \
            myregistry.io/golden-base:${VERSION}
          docker tag golden-base:latest \
            myregistry.io/golden-base:latest
          docker push myregistry.io/golden-base:${VERSION}
          docker push myregistry.io/golden-base:latest

          # Push SBOM
          oras push myregistry.io/golden-base:${VERSION} \
            golden-base-sbom.cdx.json:application/vnd.cyclonedx+json

Application Images Inherit Golden Base:

# Application Dockerfile
FROM myregistry.io/golden-base:latest

# Add application-specific dependencies
COPY requirements.txt .
RUN pip install -r requirements.txt

COPY app.py .
CMD ["python", "app.py"]

Example End-to-End CI/CD Pipelines¶

Pipeline 1: Standard Security Gate¶

Flow:

flowchart LR
    Build["Build Image"] --> SBOM["Generate SBOM"]
    SBOM --> Scan["Trivy Scan"]
    Scan --> Policy{"Pass<br/>Policy?"}
    Policy -->|Yes| Push["Push Image + SBOM"]
    Policy -->|No| Fail["Fail Build"]
    Push --> Registry["Registry"]

Implementation:

# .github/workflows/security-gate.yml
name: Security Gate

on:
  push:
    branches: [main]
  pull_request:

jobs:
  security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v4

      - name: Build image
        uses: docker/build-push-action@v4
        with:
          context: .
          push: false
          tags: myimage:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: Generate SBOM
        run: |
          trivy image \
            --format cyclonedx \
            --output sbom.cdx.json \
            myimage:${{ github.sha }}

      - name: Scan image
        run: |
          trivy image \
            --format json \
            --output scan.json \
            --exit-code 1 \
            --severity CRITICAL,HIGH \
            --ignorefile .trivyignore \
            myimage:${{ github.sha }}

      - name: Upload SBOM
        uses: actions/upload-artifact@v3
        with:
          name: sbom
          path: sbom.cdx.json
          retention-days: 90

      - name: Upload scan results
        uses: actions/upload-artifact@v3
        with:
          name: scan-results
          path: scan.json
          retention-days: 90

      - name: Push image and SBOM (on main)
        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
        run: |
          docker push myimage:${{ github.sha }}
          docker tag myimage:${{ github.sha }} myimage:latest
          docker push myimage:latest

          # Push SBOM as OCI artifact
          oras push myregistry.io/myimage:${{ github.sha }} \
            sbom.cdx.json:application/vnd.cyclonedx+json \
            --username ${{ secrets.REGISTRY_USER }} \
            --password ${{ secrets.REGISTRY_PASS }}

Pipeline 2: Autofix Loop¶

Flow:

flowchart TB
    Schedule["Scheduled Job"] --> Update["Update Dependencies"]
    Update --> Build["Build Candidate"]
    Build --> Scan["Trivy Scan"]
    Scan --> Compare["Compare CVE Counts"]
    Compare --> Better{"CVEs<br/>Reduced?"}
    Better -->|Yes| Merge["Auto-Merge PR"]
    Better -->|No| Report["Generate Report"]
    Merge --> Tag["Tag Image"]
    Report --> Review["Human Review"]

Implementation:

# .github/workflows/autofix-loop.yml
name: Autofix Loop

on:
  schedule:
    - cron: '0 2 * * 1'  # Monday 2 AM
  workflow_dispatch:

jobs:
  autofix:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          token: ${{ secrets.GITHUB_TOKEN }}

      - name: Update dependencies
        run: |
          # Update base image (example: Dependabot would do this)
          # Or run: pip install --upgrade -r requirements.txt
          # Generate updated requirements.txt
          pip-compile --upgrade requirements.in

      - name: Build current image
        run: |
          docker build -t current:latest .
          trivy image --format json --output current-scan.json current:latest
          CURRENT_CVES=$(jq '[.Results[].Vulnerabilities[]? | select(.Severity == "CRITICAL" or .Severity == "HIGH")] | length' current-scan.json)
          echo "CURRENT_CVES=$CURRENT_CVES" >> $GITHUB_ENV

      - name: Build candidate image
        run: |
          docker build -t candidate:latest .
          trivy image --format json --output candidate-scan.json candidate:latest
          CANDIDATE_CVES=$(jq '[.Results[].Vulnerabilities[]? | select(.Severity == "CRITICAL" or .Severity == "HIGH")] | length' candidate-scan.json)
          echo "CANDIDATE_CVES=$CANDIDATE_CVES" >> $GITHUB_ENV

      - name: Create PR if improved
        if: env.CANDIDATE_CVES < env.CURRENT_CVES
        run: |
          git config user.name "Autofix Bot"
          git config user.email "autofix@example.com"
          git checkout -b autofix/$(date +%Y%m%d)
          git add requirements.txt Dockerfile
          git commit -m "Security: Auto-fix CVEs (reduced from ${{ env.CURRENT_CVES }} to ${{ env.CANDIDATE_CVES }})"
          git push origin autofix/$(date +%Y%m%d)

          gh pr create \
            --title "Security: Auto-fix CVEs" \
            --body "Automated CVE fixes. Reduced HIGH/CRITICAL CVEs from ${{ env.CURRENT_CVES }} to ${{ env.CANDIDATE_CVES }}" \
            --base main

      - name: Generate report if not improved
        if: env.CANDIDATE_CVES >= env.CURRENT_CVES
        run: |
          echo "CVE count not reduced. Current: ${{ env.CURRENT_CVES }}, Candidate: ${{ env.CANDIDATE_CVES }}"
          trivy image --format template --template "@contrib/html.tpl" \
            --output report.html candidate:latest
          # Upload report or send alert

Integration with Container Registries & Artifact Repositories¶

Attaching SBOMs as OCI Artifacts¶

Docker Hub / GHCR:

# Push SBOM as OCI artifact
oras push \
  ghcr.io/org/repo:tag \
  sbom.cdx.json:application/vnd.cyclonedx+json \
  --username $USERNAME \
  --password $TOKEN

Harbor:

# Harbor supports OCI artifacts natively
oras push \
  harbor.example.com/project/repo:tag \
  sbom.cdx.json:application/vnd.cyclonedx+json

AWS ECR:

# ECR supports OCI artifacts
aws ecr put-image \
  --repository-name myrepo \
  --image-manifest "$(cat sbom.cdx.json | base64)"

Tagging Images with Security Posture¶

Labels for Security Metadata:

# In Dockerfile or via build args
LABEL security.scan.date="2024-01-15"
LABEL security.scan.tool="trivy"
LABEL security.scan.version="0.45.0"
LABEL security.cve.count.critical="0"
LABEL security.cve.count.high="3"
LABEL security.sbom.digest="sha256:abc123..."
LABEL security.sbom.format="cyclonedx"

Setting Labels in CI:

- name: Extract security metadata
  run: |
    SCAN_DATE=$(date -u +%Y-%m-%dT%H:%M:%SZ)
    CRITICAL_COUNT=$(jq '[.Results[].Vulnerabilities[]? | select(.Severity == "CRITICAL")] | length' scan.json)
    HIGH_COUNT=$(jq '[.Results[].Vulnerabilities[]? | select(.Severity == "HIGH")] | length' scan.json)

    docker build \
      --label "security.scan.date=$SCAN_DATE" \
      --label "security.cve.count.critical=$CRITICAL_COUNT" \
      --label "security.cve.count.high=$HIGH_COUNT" \
      -t myimage:tag .

Registry-Native Scanners¶

When to Use Registry Scanners:

• Convenience: Built into platform, no setup
• Compliance: Required by organizational policy
• Baseline: Good starting point

When to Use Trivy:

• Flexibility: More control over scanning policies
• CI/CD Integration: Better integration with build pipelines
• SBOM Generation: Native SBOM support
• Custom Policies: More granular control

Hybrid Approach:

Use both: - Trivy in CI/CD for build-time gates - Registry scanners for runtime monitoring and compliance reporting

Operational Best Practices¶

Regular Scheduled Scans¶

Scan Existing Images in Registry:

# .github/workflows/scan-registry.yml
name: Scan Registry Images

on:
  schedule:
    - cron: '0 0 * * 0'  # Weekly
  workflow_dispatch:

jobs:
  scan-registry:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        image:
          - myregistry.io/app:v1.0
          - myregistry.io/app:v1.1
          - myregistry.io/app:latest
    steps:
      - name: Scan image
        run: |
          trivy image \
            --format json \
            --output scan-${{ matrix.image }}.json \
            ${{ matrix.image }}

      - name: Check for new CRITICAL CVEs
        run: |
          CRITICAL_COUNT=$(jq '[.Results[].Vulnerabilities[]? | select(.Severity == "CRITICAL")] | length' scan-${{ matrix.image }}.json)
          if [ "$CRITICAL_COUNT" -gt 0 ]; then
            echo "::error::Found $CRITICAL_COUNT CRITICAL CVEs in ${{ matrix.image }}"
            exit 1
          fi

Reporting & Alerting¶

Slack Integration:

# scripts/alert_slack.py
import json
import requests
import os

def send_slack_alert(scan_json: str, webhook_url: str):
    with open(scan_json) as f:
        data = json.load(f)

    critical_cves = []
    for result in data.get("Results", []):
        for vuln in result.get("Vulnerabilities", []):
            if vuln.get("Severity") == "CRITICAL":
                critical_cves.append({
                    "id": vuln.get("VulnerabilityID"),
                    "package": vuln.get("PkgName"),
                    "title": vuln.get("Title"),
                })

    if critical_cves:
        message = {
            "text": f"🚨 Found {len(critical_cves)} CRITICAL CVEs",
            "blocks": [
                {
                    "type": "section",
                    "text": {
                        "type": "mrkdwn",
                        "text": f"*{len(critical_cves)} CRITICAL CVEs detected*\n" +
                                "\n".join([f"• {cve['id']}: {cve['package']}" for cve in critical_cves])
                    }
                }
            ]
        }
        requests.post(webhook_url, json=message)

if __name__ == "__main__":
    send_slack_alert("scan.json", os.getenv("SLACK_WEBHOOK_URL"))

Email Alerts:

# In CI pipeline
if [ "$CRITICAL_COUNT" -gt 0 ]; then
  echo "Found $CRITICAL_COUNT CRITICAL CVEs" | \
    mail -s "Security Alert: CRITICAL CVEs in $IMAGE" \
    security-team@example.com
fi

Policy for Deprecating Images¶

Deprecation Workflow:

1. Detect: Scheduled scan finds unresolved CRITICAL CVEs
2. Alert: Notify security team and image owners
3. Grace Period: 30 days to fix or justify
4. Deprecate: Mark image as deprecated in registry
5. Block: Prevent new deployments of deprecated images

Implementation:

# Mark image as deprecated
docker manifest annotate \
  myregistry.io/myimage:tag \
  --annotation org.opencontainers.image.deprecated=true \
  --annotation org.opencontainers.image.deprecated.reason="CRITICAL CVEs unresolved"

# Or use registry API
curl -X PATCH \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"deprecated": true, "reason": "CRITICAL CVEs"}' \
  https://registry.example.com/v2/org/repo/manifests/tag

Storing Historical SBOMs and Scan Results¶

Artifact Repository Pattern:

# Store with versioned paths
aws s3 cp sbom.cdx.json \
  s3://artifacts/sboms/myimage/${IMAGE_TAG}/${IMAGE_DIGEST}.cdx.json

aws s3 cp scan.json \
  s3://artifacts/scans/myimage/${IMAGE_TAG}/${SCAN_DATE}.json

Database Pattern:

-- Store scan results in database
CREATE TABLE image_scans (
    id SERIAL PRIMARY KEY,
    image_name VARCHAR(255),
    image_tag VARCHAR(255),
    image_digest VARCHAR(255),
    scan_date TIMESTAMP,
    critical_count INTEGER,
    high_count INTEGER,
    medium_count INTEGER,
    low_count INTEGER,
    scan_report JSONB,
    sbom_location TEXT
);

CREATE INDEX idx_image_scans_image ON image_scans(image_name, image_tag);
CREATE INDEX idx_image_scans_date ON image_scans(scan_date);

Security & Compliance Considerations¶

Supply Chain Security Frameworks¶

SLSA (Supply-chain Levels for Software Artifacts):

• Level 1: Basic provenance (SBOMs provide this)
• Level 2: Version-controlled builds (CI/CD provides this)
• Level 3: Non-falsifiable provenance (signed SBOMs, attestations)

Mapping to SLSA:

• SBOMs: Provide software provenance (what's in the image)
• Scan Results: Provide security attestations
• Signed Artifacts: Use cosign or similar to sign images + SBOMs

Example: Signed SBOM:

# Generate and sign SBOM
trivy image --format cyclonedx --output sbom.cdx.json myimage:tag
cosign sign-blob --bundle sbom.cdx.json.bundle sbom.cdx.json

Compliance Expectations¶

SOC 2: - Control: Change management, vulnerability management - Evidence: SBOMs, scan reports, remediation workflows

ISO 27001: - Control: A.12.6.1 (Management of technical vulnerabilities) - Evidence: Vulnerability scanning, patch management, SBOMs

Traceability Requirements:

flowchart LR
    Image["Docker Image"] --> Digest["Image Digest"]
    Digest --> SBOM["SBOM"]
    Digest --> Scan["Scan Report"]
    SBOM --> Components["Component List"]
    Scan --> CVEs["CVE List"]
    Components --> CVEs
    CVEs --> Remediation["Remediation History"]

Implementation:

• Store image digest in SBOM metadata
• Link scan reports to image digests
• Maintain audit trail of all scans and fixes

Reproducibility¶

Ensuring Reproducible Builds:

1. Pin base images: Use digests, not tags
2. Lock dependencies: Use lock files (requirements.txt, package-lock.json)
3. Deterministic builds: Set build timestamps, use build args
4. Regenerate SBOMs: Same image should produce same SBOM

Example Reproducible Dockerfile:

# Pin base image to digest
FROM debian@sha256:abc123... AS base

# Use specific package versions
RUN apt-get update && \
    apt-get install -y \
      python3=3.11.2-1 \
      && \
    apt-get clean

# Copy lock file
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

Limitations and Non-Goals¶

What Automation Cannot Safely Do¶

Arbitrary Auto-Edit of Application Code: - Don't automatically modify application source code - Don't change business logic to fix CVEs - Focus on dependency and base image updates only

Auto-Upgrade Major Versions: - Major version upgrades (Python 3.11 → 3.12) require testing - Breaking changes may require code modifications - Use automated PRs for review, not direct merges

Bypassing Security Policies: - Don't auto-ignore CVEs without review - Don't lower severity thresholds automatically - Maintain human oversight for security decisions

What You Still Need¶

Good Tests: - Comprehensive test suites catch breaking changes - Integration tests verify image functionality - Security tests validate fixes don't introduce regressions

Human Review: - Review autofix PRs before merging - Validate CVE ignore list justifications - Approve major version upgrades

Monitoring: - Monitor production for new vulnerabilities - Track CVE trends over time - Alert on critical CVEs in running containers

Conclusion¶

A sane maturity path for container security:

Phase 1: Generate SBOMs + Scan Images¶

Goals: - Generate SBOMs for all release images - Run Trivy scans in CI/CD - Store SBOMs and scan results

Implementation: - Add Trivy to CI/CD pipelines - Generate SBOMs at build time - Store artifacts in registry or S3

Phase 2: Enforce Basic Gating Policy¶

Goals: - Fail builds on CRITICAL CVEs - Warn on HIGH severity - Maintain ignore list with justifications

Implementation: - Configure Trivy exit codes - Create .trivyignore with reviews - Integrate scan results into CI gates

Phase 3: Introduce Golden Base Images¶

Goals: - Maintain blessed base images - Regular updates and scanning - Application images inherit golden bases

Implementation: - Create golden base image repository - Weekly update and scan pipeline - Update application Dockerfiles to use golden bases

Phase 4: Add Dependency Update Automation¶

Goals: - Automatically update dependencies - Validate updates with scans - Create PRs for review

Implementation: - Configure Dependabot/Renovate - CI validates dependency updates - Auto-merge safe updates

Phase 5: Add Autofix Proposal Loops¶

Goals: - Parse Trivy results programmatically - Generate Dockerfile fixes - Create PRs with suggested changes

Implementation: - Build autofix scripts - Integrate with CI/CD - Maintain human review process

Key Principles:

1. Automate the boring: Dependency updates, base image bumps
2. Review the risky: Major version changes, security policy changes
3. Track everything: SBOMs, scans, remediation history
4. Fail fast: Catch vulnerabilities early in CI/CD
5. Fix systematically: Use automation to reduce manual toil

This approach transforms container security from a manual, error-prone process into a systematic, automated workflow that scales with your infrastructure.

See Also¶

• Docker & Compose - Production containerization patterns
• Ansible Security Hardening - Infrastructure security practices
• CI/CD Pipelines - Continuous integration patterns