Cross-Environment Configuration Strategy, Drift Prevention, and Multi-Cluster State Management: Best Practices¶

Objective: Master production-grade configuration governance across multiple environments (dev, test, staging, prod, air-gapped, HPC, RKE2, edge). When you need to prevent drift, ensure consistency, and manage configuration lifecycle—this guide provides complete patterns and implementations.

Introduction¶

Configuration governance is the foundation of reliable, reproducible distributed systems. Without proper configuration management, environments drift, deployments fail, and systems become unpredictable. This guide provides a complete framework for managing configuration across all environments and deployment targets.

What This Guide Covers: - The problem space: why configuration governance is critical - Core principles of environment configuration strategy - Patterns for multi-environment configuration design - Anti-patterns and real-world pitfalls - Environment promotion pipelines - Automated drift detection and enforcement - Configuration metadata and versioning - Secrets lifecycle governance - Tooling and implementation examples - Agentic LLM integration for configuration

Prerequisites: - Understanding of distributed systems and Kubernetes - Familiarity with configuration management and GitOps - Experience with multi-environment deployments

The Problem Space: Why Config Governance Is Critical¶

Configuration Drift Types¶

Schema Drift: Database schemas diverge between environments.

Example:

-- Dev: Has new column
ALTER TABLE users ADD COLUMN email_verified BOOLEAN;

-- Prod: Missing column
-- Schema drift: dev has email_verified, prod doesn't

Impact: Application code works in dev, fails in prod.

Secrets Drift: Secrets differ between environments.

Example:

# Dev: Test credentials
database_password: "test123"

# Prod: Production credentials (different)
database_password: "prod-secret-xyz"

Impact: Security vulnerabilities, authentication failures.

Runtime Drift: Runtime configuration differs.

Example:

# Dev: Low resource limits
resources:
  requests:
    cpu: 100m
    memory: 256Mi

# Prod: High resource limits
resources:
  requests:
    cpu: 1000m
    memory: 2Gi

Impact: Performance differences, resource contention.

Cluster State Drift: Kubernetes cluster state diverges.

Example:

# Dev: 3 replicas
kubectl get deployment app -n dev
# replicas: 3

# Prod: 5 replicas (drifted)
kubectl get deployment app -n prod
# replicas: 5

Impact: Inconsistent scaling, unpredictable behavior.

API Contract Drift: API contracts differ between environments.

Example:

# Dev: API v2 with new field
class UserV2(BaseModel):
    id: int
    name: str
    email: str  # New in v2

# Prod: Still on API v1
class UserV1(BaseModel):
    id: int
    name: str

Impact: Integration failures, breaking changes.

Operator/Default Drift: Operator defaults change over time.

Example:

# Old operator version: default replicas = 1
# New operator version: default replicas = 3
# Deployments drift without explicit config

Impact: Unintended scaling, resource consumption.

Environment Variable Entropy: Environment variables become inconsistent.

Example:

# Dev
DATABASE_URL=postgresql://dev-db:5432/app
LOG_LEVEL=DEBUG

# Prod
DATABASE_URL=postgresql://prod-db:5432/app
LOG_LEVEL=INFO
# Missing: LOG_LEVEL in some pods

Impact: Inconsistent behavior, debugging difficulties.

Config Shadowing: Configs override each other unexpectedly.

Example:

# base.yaml
database:
  host: "localhost"
  port: 5432

# override.yaml
database:
  host: "remote-host"
  # port not specified, but might be overridden by env var
  # Config shadowing: unclear which value wins

Impact: Unpredictable configuration resolution.

Version Skew Between Environments: Different versions deployed.

Example:

# Dev: Latest version
image: app:v1.5.0

# Prod: Older version
image: app:v1.3.0
# Version skew: features available in dev not in prod

Impact: Feature inconsistencies, testing gaps.

Consequences of Configuration Drift¶

Inconsistent Correctness Guarantees: - Dev works, prod fails - Different behavior across environments - Unpredictable outcomes

Broken Reproducibility: - Can't reproduce production issues in staging - Different configurations yield different results - Testing becomes unreliable

Unstable Deployments: - Deployments fail due to config mismatches - Rollbacks don't work because configs differ - Recovery procedures fail

Unpredictable Performance Behavior: - Different resource limits cause performance differences - Caching configurations differ - Load balancing behaves differently

Security Gaps: - Secrets drift creates vulnerabilities - Access controls differ - Audit trails incomplete

Misaligned Audit Trails: - Config changes not tracked - Drift not detected - Compliance violations

"Can't Reproduce in Staging" Anti-Patterns: - Issues only occur in production - Staging doesn't match production - Debugging becomes impossible

Core Principles of Environment Configuration Strategy¶

"Configuration as a Product"¶

Principle: Treat configuration as a first-class product with: - Versioning - Testing - Documentation - Lifecycle management

Example:

# config/product.yaml
apiVersion: v1
kind: ConfigProduct
metadata:
  name: app-config
  version: 1.2.3
spec:
  environments:
    - name: dev
      version: 1.2.3-dev
    - name: prod
      version: 1.2.3

Declarative Configuration First¶

Principle: All configuration should be declarative, not imperative.

Example:

# Declarative: Desired state
apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: app
        image: app:v1.2.3

# Not: kubectl scale deployment app --replicas=3

Immutable Infrastructure and Reproducible Builds¶

Principle: Infrastructure and builds should be immutable and reproducible.

Example:

# Immutable: Pinned base image
FROM python:3.11-slim@sha256:abc123...

# Immutable: Pinned dependencies
COPY requirements-lock.txt .
RUN pip install --no-cache-dir -r requirements-lock.txt

Fail-Fast Configuration Validation¶

Principle: Validate configuration early and fail fast.

Example:

# Fail-fast validation
from pydantic import BaseModel, ValidationError

class AppConfig(BaseModel):
    database_url: str
    redis_url: str
    log_level: str

    @validator('log_level')
    def validate_log_level(cls, v):
        if v not in ['DEBUG', 'INFO', 'WARNING', 'ERROR']:
            raise ValueError(f"Invalid log level: {v}")
        return v

# Validate at startup
try:
    config = AppConfig(**load_config())
except ValidationError as e:
    raise SystemExit(f"Configuration validation failed: {e}")

Explicit Environment Boundaries¶

Principle: Make environment boundaries explicit and enforced.

Example:

# Explicit environment boundaries
environments:
  dev:
    cluster: dev-cluster
    namespace: dev
    config_source: config/dev/

  prod:
    cluster: prod-cluster
    namespace: prod
    config_source: config/prod/

Thin Environment Differences (TED Principle)¶

Principle: Minimize differences between environments.

Example:

# Base config (shared)
base:
  database:
    pool_size: 10
    timeout: 30

  # Environment-specific overrides (thin)
  dev:
    database:
      pool_size: 5  # Only difference

  prod:
    database:
      pool_size: 20  # Only difference

Component Autonomy & Isolation¶

Principle: Components should be autonomous and isolated.

Example:

# Autonomous component config
components:
  user-service:
    config:
      database_url: ${USER_DB_URL}
      redis_url: ${REDIS_URL}
    # No dependencies on other component configs

Configuration Layering¶

Principle: Layer configuration from base to runtime.

Layers: 1. Base: Common configuration 2. Environment: Environment-specific overrides 3. Secrets: Secret values 4. Runtime: Runtime-specific adjustments

Example:

# Layer 1: Base
base:
  app:
    name: myapp
    version: 1.0.0

# Layer 2: Environment
environments:
  dev:
    app:
      debug: true

  prod:
    app:
      debug: false

# Layer 3: Secrets
secrets:
  database_password: ${DB_PASSWORD}

# Layer 4: Runtime
runtime:
  replicas: ${REPLICAS}

Version-Locked Everything¶

Principle: Lock versions of all dependencies.

Example:

# Version-locked config
versions:
  app: "1.2.3"
  postgres: "15.3"
  redis: "7.2.0"
  python: "3.11.5"

  dependencies:
    fastapi: "0.104.0"
    pydantic: "2.5.0"

Patterns for Multi-Environment Configuration Design¶

Canonical Configuration Hierarchy¶

Structure:

config/
├── base/
│   ├── app.yaml
│   ├── database.yaml
│   └── redis.yaml
├── environments/
│   ├── dev/
│   │   ├── app.yaml
│   │   └── database.yaml
│   ├── staging/
│   │   ├── app.yaml
│   │   └── database.yaml
│   └── prod/
│       ├── app.yaml
│       └── database.yaml
└── secrets/
    ├── dev/
    └── prod/

Resolution:

# config_resolver.py
class ConfigResolver:
    def resolve(self, environment: str) -> dict:
        """Resolve configuration for environment"""
        # Load base
        base_config = load_config("config/base/")

        # Override with environment
        env_config = load_config(f"config/environments/{environment}/")

        # Merge
        config = deep_merge(base_config, env_config)

        # Inject secrets
        secrets = load_secrets(f"config/secrets/{environment}/")
        config = inject_secrets(config, secrets)

        return config

Dedicated Schema Repository and Environment Diffs¶

Schema Repository:

schemas/
├── postgres/
│   ├── v1/
│   │   └── schema.sql
│   └── v2/
│       └── schema.sql
├── api/
│   ├── v1/
│   │   └── openapi.yaml
│   └── v2/
│       └── openapi.yaml
└── config/
    ├── v1/
    │   └── config-schema.json
    └── v2/
        └── config-schema.json

Environment Diffing:

# diff_tool.py
class EnvironmentDiffer:
    def diff_environments(self, env1: str, env2: str) -> dict:
        """Diff configurations between environments"""
        config1 = load_config(env1)
        config2 = load_config(env2)

        diff = {
            'added': find_added_keys(config1, config2),
            'removed': find_removed_keys(config1, config2),
            'changed': find_changed_values(config1, config2)
        }

        return diff

Context-Aware Config Resolvers¶

Resolver:

# context_aware_resolver.py
class ContextAwareConfigResolver:
    def resolve(self, context: dict) -> dict:
        """Resolve config based on context"""
        environment = context.get('environment')
        cluster = context.get('cluster')
        region = context.get('region')

        # Load base
        config = load_base_config()

        # Apply environment overrides
        config = apply_environment_overrides(config, environment)

        # Apply cluster overrides
        config = apply_cluster_overrides(config, cluster)

        # Apply region overrides
        config = apply_region_overrides(config, region)

        return config

Strict Type-Checked Configuration Loading¶

Type Checking:

# typed_config.py
from pydantic import BaseModel, Field
from typing import Literal

class DatabaseConfig(BaseModel):
    host: str = Field(..., description="Database host")
    port: int = Field(5432, ge=1, le=65535, description="Database port")
    database: str = Field(..., description="Database name")
    pool_size: int = Field(10, ge=1, le=100, description="Connection pool size")

class AppConfig(BaseModel):
    name: str
    environment: Literal["dev", "staging", "prod"]
    database: DatabaseConfig
    log_level: Literal["DEBUG", "INFO", "WARNING", "ERROR"] = "INFO"

# Load with validation
config = AppConfig(**load_raw_config())

Single-Source-of-Truth Values¶

Pattern:

# Single source of truth
# config/base/app.yaml
app:
  name: myapp
  version: 1.2.3

# Environments reference base
# config/environments/dev/app.yaml
app:
  # Inherit from base
  name: ${base.app.name}
  version: ${base.app.version}
  # Only override what's different
  debug: true

"Configuration Contracts" with Versioned Validation Rules¶

Contract Definition:

# config-contract.yaml
apiVersion: config.example.com/v1
kind: ConfigContract
metadata:
  name: app-config-contract
  version: 1.0.0
spec:
  schema:
    type: object
    properties:
      database_url:
        type: string
        pattern: "^postgresql://"
      log_level:
        type: string
        enum: ["DEBUG", "INFO", "WARNING", "ERROR"]
    required: ["database_url", "log_level"]

  validation_rules:
    - name: "database_url_format"
      rule: "must match postgresql:// pattern"
    - name: "log_level_enum"
      rule: "must be one of DEBUG, INFO, WARNING, ERROR"

Feature Flags Aligned with Environment Tiers¶

Feature Flag Strategy:

# feature-flags.yaml
feature_flags:
  new_ui:
    dev: true
    staging: true
    prod: false  # Not ready for prod

  experimental_api:
    dev: true
    staging: false  # Too risky for staging
    prod: false

  performance_optimization:
    dev: true
    staging: true
    prod: true  # Safe for all environments

Temporal Configuration Behavior¶

Temporal Configs:

# temporal-config.yaml
temporal_configs:
  - name: "maintenance_window"
    schedule: "0 2 * * 0"  # Sunday 2 AM
    config:
      maintenance_mode: true

  - name: "high_traffic_period"
    schedule: "0 9 * * 1-5"  # Weekdays 9 AM
    config:
      replicas: 10
      cache_ttl: 300

Docker Compose Overrides¶

Pattern:

# docker-compose.base.yaml
services:
  app:
    image: app:latest
    environment:
      - DATABASE_URL=${DATABASE_URL}
      - LOG_LEVEL=INFO

# docker-compose.dev.yaml
services:
  app:
    environment:
      - LOG_LEVEL=DEBUG
    volumes:
      - ./src:/app/src

# docker-compose.prod.yaml
services:
  app:
    environment:
      - LOG_LEVEL=INFO
    deploy:
      replicas: 3

Usage:

# Dev
docker-compose -f docker-compose.base.yaml -f docker-compose.dev.yaml up

# Prod
docker-compose -f docker-compose.base.yaml -f docker-compose.prod.yaml up

Helm Charts / Kustomize Overlays¶

Helm Pattern:

# values-base.yaml
replicas: 1
image:
  repository: app
  tag: latest
resources:
  requests:
    cpu: 100m
    memory: 256Mi

# values-dev.yaml
replicas: 1
image:
  tag: dev
resources:
  requests:
    cpu: 100m
    memory: 256Mi

# values-prod.yaml
replicas: 5
image:
  tag: v1.2.3
resources:
  requests:
    cpu: 1000m
    memory: 2Gi

Kustomize Pattern:

kustomize/
├── base/
│   ├── kustomization.yaml
│   └── deployment.yaml
├── dev/
│   ├── kustomization.yaml
│   └── patches/
│       └── replicas.yaml
└── prod/
    ├── kustomization.yaml
    └── patches/
        └── replicas.yaml

Postgres Config Files¶

Postgres Config:

# postgresql.base.conf
shared_buffers = 256MB
max_connections = 100
work_mem = 4MB

# postgresql.dev.conf (override)
shared_buffers = 128MB
max_connections = 50

# postgresql.prod.conf (override)
shared_buffers = 1GB
max_connections = 200

Prefect Flows¶

Prefect Config:

# prefect_config.py
from prefect import flow
from pydantic import BaseSettings

class PrefectConfig(BaseSettings):
    environment: str
    database_url: str
    redis_url: str

    class Config:
        env_file = f".env.{environment}"

@flow
def etl_flow(config: PrefectConfig):
    """ETL flow with environment-aware config"""
    # Use config
    pass

Air-Gapped Configs¶

Air-Gapped Pattern:

# Package configs for air-gapped deployment
tar -czf config-bundle-$(date +%Y%m%d).tar.gz \
    config/ \
    schemas/ \
    secrets-encrypted/ \
    checksums.txt

# Verify on air-gapped cluster
sha256sum -c checksums.txt

Anti-Patterns and Real-World Pitfalls¶

Overloaded .env Files¶

Problem: Single .env file with everything.

Example:

# .env (overloaded)
DATABASE_URL=...
REDIS_URL=...
API_KEY=...
SECRET_KEY=...
LOG_LEVEL=...
DEBUG=...
# 100+ variables, no organization

Fix: Organize by domain and environment.

# .env.database
DATABASE_URL=...
DATABASE_POOL_SIZE=...

# .env.redis
REDIS_URL=...
REDIS_TTL=...

# .env.app
LOG_LEVEL=...
DEBUG=...

Copy-Pasted YAML¶

Problem: YAML files copied between environments.

Example:

# config/dev/app.yaml
app:
  name: myapp
  database_url: postgresql://dev-db/app

# config/prod/app.yaml (copy-pasted, forgot to change)
app:
  name: myapp
  database_url: postgresql://dev-db/app  # Wrong! Still pointing to dev

Fix: Use inheritance and templating.

# config/base/app.yaml
app:
  name: myapp

# config/environments/dev/app.yaml
app:
  database_url: postgresql://dev-db/app

# config/environments/prod/app.yaml
app:
  database_url: postgresql://prod-db/app

Environment-Specific Logic Hidden in Code¶

Problem: Environment checks in application code.

Example:

# Bad: Environment logic in code
if os.getenv("ENVIRONMENT") == "dev":
    database_url = "postgresql://dev-db/app"
elif os.getenv("ENVIRONMENT") == "prod":
    database_url = "postgresql://prod-db/app"

Fix: Externalize to configuration.

# Good: Config-driven
config = load_config()
database_url = config.database_url

Divergent Secrets Management Systems¶

Problem: Different secret management in each environment.

Fix: Standardize on one system.

# Standardized secrets management
secrets_management:
  system: "vault"  # Same for all environments
  backend: "vault-backend"
  paths:
    dev: "secret/data/dev"
    prod: "secret/data/prod"

Stale or Undocumented Toggles¶

Problem: Feature flags never cleaned up.

Fix: Document and audit feature flags.

# feature-flags.yaml
feature_flags:
  old_feature:
    enabled: false
    deprecated: true
    removal_date: "2024-06-01"
    owner: "team-a"

  new_feature:
    enabled: true
    description: "New feature description"
    owner: "team-b"

Per-Developer Configuration Mutations¶

Problem: Each developer has different local config.

Fix: Standardize local development.

# .devcontainer/devcontainer.json
{
  "name": "Standard Dev Environment",
  "dockerComposeFile": "docker-compose.yml",
  "service": "app",
  "workspaceFolder": "/workspace"
}

Configuration with Implicit Default Changes¶

Problem: Defaults change without notice.

Example:

# Old version: default replicas = 1
# New version: default replicas = 3
# Deployments change without explicit config

Fix: Always specify values explicitly.

# Explicit values
replicas: 3  # Never rely on defaults

Manual Patching of RKE2 Nodes¶

Problem: Manual changes to nodes cause drift.

Fix: Use GitOps and automation.

# GitOps: All changes through Git
apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  source:
    repoURL: https://github.com/org/config-repo
    path: k8s/

Drift Introduced by "Snowflake" Servers¶

Problem: Servers configured differently.

Fix: Use infrastructure as code.

# Infrastructure as code
resources:
  - type: server
    name: app-server-1
    config: standard-app-server-config

  - type: server
    name: app-server-2
    config: standard-app-server-config  # Same config

Environment Promotion Pipelines¶

Dev → Test Promotion¶

Pipeline:

graph LR
    Dev["Dev Environment"]
    Validate["Validate Config"]
    Test["Test Environment"]
    Verify["Verify Deployment"]

    Dev --> Validate
    Validate --> Test
    Test --> Verify

    style Dev fill:#e1f5ff
    style Validate fill:#fff4e1
    style Test fill:#e8f5e9
    style Verify fill:#f3e5f5

Implementation:

# .github/workflows/promote-dev-to-test.yml
name: Promote Dev to Test
on:
  workflow_dispatch:

jobs:
  promote:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Validate dev config
        run: |
          python scripts/validate_config.py config/environments/dev/

      - name: Diff dev and test
        run: |
          python scripts/diff_configs.py dev test

      - name: Promote to test
        run: |
          python scripts/promote_config.py dev test

      - name: Deploy to test
        run: |
          kubectl apply -f config/environments/test/

Test → Staging Promotion¶

Pipeline:

# .github/workflows/promote-test-to-staging.yml
name: Promote Test to Staging
on:
  workflow_dispatch:
    inputs:
      approval:
        description: 'Approve promotion'
        required: true
        type: boolean

jobs:
  promote:
    if: github.event.inputs.approval == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Validate test config
        run: |
          python scripts/validate_config.py config/environments/test/

      - name: Schema promotion check
        run: |
          python scripts/check_schema_promotion.py test staging

      - name: Promote to staging
        run: |
          python scripts/promote_config.py test staging

Staging → Prod Promotion¶

Pipeline:

# .github/workflows/promote-staging-to-prod.yml
name: Promote Staging to Prod
on:
  workflow_dispatch:
    inputs:
      approval:
        description: 'Approve production promotion'
        required: true
        type: boolean

jobs:
  promote:
    if: github.event.inputs.approval == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Validate staging config
        run: |
          python scripts/validate_config.py config/environments/staging/

      - name: Production readiness check
        run: |
          python scripts/check_prod_readiness.py staging

      - name: Promote to prod
        run: |
          python scripts/promote_config.py staging prod

      - name: Deploy to prod
        run: |
          kubectl apply -f config/environments/prod/

Automated Diffing¶

Diff Tool:

# scripts/diff_configs.py
import yaml
from deepdiff import DeepDiff

def diff_configs(env1: str, env2: str):
    """Diff configurations between environments"""
    config1 = load_config(env1)
    config2 = load_config(env2)

    diff = DeepDiff(config1, config2, ignore_order=True)

    if diff:
        print(f"Differences between {env1} and {env2}:")
        print(diff.pretty())
        return False
    else:
        print(f"No differences between {env1} and {env2}")
        return True

Schema Promotion Policies¶

Policy:

# schema-promotion-policy.yaml
schema_promotion:
  rules:
    - name: "backward_compatible_only"
      rule: "Only backward-compatible schema changes allowed"
      enforcement: "strict"

    - name: "migration_required"
      rule: "Breaking changes require migration plan"
      enforcement: "strict"

    - name: "version_bump"
      rule: "Schema changes require version bump"
      enforcement: "strict"

Config Gates and Approval Workflows¶

Gate Configuration:

# config-gates.yaml
gates:
  - name: "config_validation"
    type: "automated"
    checks:
      - "schema_validation"
      - "type_validation"
      - "constraint_validation"

  - name: "security_review"
    type: "manual"
    approvers:
      - "security-team"

  - name: "architecture_review"
    type: "manual"
    approvers:
      - "architecture-team"

Reproducible Environment Snapshots¶

Snapshot Creation:

# scripts/create_snapshot.py
def create_environment_snapshot(environment: str) -> dict:
    """Create reproducible environment snapshot"""
    snapshot = {
        'timestamp': datetime.now().isoformat(),
        'environment': environment,
        'config': load_config(environment),
        'schemas': load_schemas(environment),
        'secrets_hash': hash_secrets(environment),
        'images': get_image_versions(environment),
        'checksum': None
    }

    # Calculate checksum
    snapshot['checksum'] = calculate_checksum(snapshot)

    return snapshot

Config Freeze Periods¶

Freeze Policy:

# config-freeze-policy.yaml
freeze_periods:
  - name: "year_end_freeze"
    start: "2024-12-15"
    end: "2025-01-05"
    environments: ["prod"]
    exceptions:
      - "critical_security_patches"
      - "emergency_fixes"

Automated Config Migration Scripts¶

Migration Script:

# scripts/migrate_config.py
class ConfigMigrator:
    def migrate(self, source_env: str, target_env: str):
        """Migrate configuration between environments"""
        source_config = load_config(source_env)
        target_config = load_config(target_env)

        # Apply migrations
        migrations = load_migrations()
        for migration in migrations:
            if migration.applies_to(source_config, target_config):
                target_config = migration.apply(target_config)

        # Save migrated config
        save_config(target_env, target_config)

Config Rollback Procedures¶

Rollback Script:

#!/bin/bash
# scripts/rollback_config.sh

ENVIRONMENT=$1
VERSION=$2

echo "Rolling back ${ENVIRONMENT} to version ${VERSION}"

# Restore config from version
git checkout ${VERSION} -- config/environments/${ENVIRONMENT}/

# Validate restored config
python scripts/validate_config.py config/environments/${ENVIRONMENT}/

# Apply restored config
kubectl apply -f config/environments/${ENVIRONMENT}/

echo "Rollback complete"

Automated Drift Detection & Enforcement¶

GitOps Enforcement Patterns¶

ArgoCD Configuration:

# argocd-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-config
spec:
  source:
    repoURL: https://github.com/org/config-repo
    path: k8s/
    targetRevision: main
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Temporal Auditing of Runtime Configs¶

Audit Script:

# audit/runtime_config_audit.py
class RuntimeConfigAuditor:
    def audit_runtime_configs(self):
        """Audit runtime configurations"""
        # Get desired config from Git
        desired_config = load_config_from_git()

        # Get actual config from cluster
        actual_config = get_config_from_cluster()

        # Compare
        drift = compare_configs(desired_config, actual_config)

        if drift:
            # Alert on drift
            alert_on_drift(drift)

            # Optionally auto-remediate
            if self.auto_remediate_enabled():
                self.remediate_drift(drift)

        return drift

Kubernetes Cluster State Diffing¶

State Diff Tool:

# drift/k8s_state_diff.py
class KubernetesStateDiffer:
    def diff_cluster_state(self, cluster: str):
        """Diff Kubernetes cluster state"""
        # Get desired state from Git
        desired_state = load_kubernetes_manifests()

        # Get actual state from cluster
        actual_state = get_cluster_state(cluster)

        # Diff
        diff = kubectl_diff(desired_state, actual_state)

        return diff

Postgres Config/State Hash Monitors¶

Postgres Monitor:

# monitoring/postgres_config_monitor.py
class PostgresConfigMonitor:
    def monitor_postgres_config(self, instance: str):
        """Monitor Postgres configuration"""
        # Get config hash
        config_hash = get_postgres_config_hash(instance)

        # Compare with expected
        expected_hash = get_expected_config_hash(instance)

        if config_hash != expected_hash:
            alert_on_config_drift(instance, config_hash, expected_hash)

Redis Config Fingerprinting¶

Redis Fingerprint:

# monitoring/redis_fingerprint.py
class RedisConfigFingerprinter:
    def fingerprint_redis_config(self, instance: str) -> str:
        """Generate Redis config fingerprint"""
        config = get_redis_config(instance)

        # Create fingerprint
        fingerprint = hashlib.sha256(
            json.dumps(config, sort_keys=True).encode()
        ).hexdigest()

        return fingerprint

FDW Target Drift Detection¶

FDW Drift Detection:

-- Check FDW target consistency
SELECT 
    srvname,
    srvoptions,
    md5(srvoptions::text) AS config_hash
FROM pg_foreign_server
WHERE srvname LIKE 's3_%';

-- Compare hashes across environments
-- Alert if hashes differ

Docker Image SBOM Diffing¶

SBOM Diff Tool:

# drift/sbom_diff.py
class SBOMDiffer:
    def diff_sboms(self, image1: str, image2: str) -> dict:
        """Diff SBOMs between images"""
        sbom1 = get_sbom(image1)
        sbom2 = get_sbom(image2)

        diff = {
            'added_packages': find_added_packages(sbom1, sbom2),
            'removed_packages': find_removed_packages(sbom1, sbom2),
            'version_changes': find_version_changes(sbom1, sbom2)
        }

        return diff

Runbook Automation for Reconciliation¶

Reconciliation Runbook:

#!/bin/bash
# runbooks/reconcile_config_drift.sh

echo "=== Config Drift Reconciliation ==="

# 1. Detect drift
DRIFT=$(python scripts/detect_drift.py)

if [ -z "$DRIFT" ]; then
    echo "No drift detected"
    exit 0
fi

# 2. Analyze drift
echo "Analyzing drift..."
python scripts/analyze_drift.py "$DRIFT"

# 3. Reconcile
echo "Reconciling drift..."
python scripts/reconcile_drift.py "$DRIFT"

# 4. Verify
echo "Verifying reconciliation..."
python scripts/verify_reconciliation.py

echo "=== Reconciliation complete ==="

Alerting Patterns¶

Prometheus Alerts:

# alerts/config_drift.yaml
groups:
  - name: config_drift
    rules:
      - alert: ConfigDriftDetected
        expr: config_drift_score > 0.1
        for: 5m
        annotations:
          summary: "Configuration drift detected"
          description: "Config drift score: {{ $value }}"

      - alert: PostgresConfigDrift
        expr: postgres_config_hash != postgres_expected_config_hash
        for: 1m
        annotations:
          summary: "Postgres configuration drift"
          description: "Postgres config hash mismatch"

Configuration Metadata & Versioning¶

Config Lineage¶

Lineage Tracking:

# metadata/lineage.py
class ConfigLineageTracker:
    def track_lineage(self, config: dict) -> dict:
        """Track configuration lineage"""
        lineage = {
            'config_id': generate_config_id(config),
            'source': 'git',
            'commit': get_git_commit(),
            'author': get_git_author(),
            'timestamp': datetime.now().isoformat(),
            'parent_configs': get_parent_configs(config),
            'derived_configs': get_derived_configs(config)
        }

        return lineage

Environment Tagging¶

Tagging Strategy:

# config-tags.yaml
tags:
  environment: "prod"
  cluster: "prod-us-east-1"
  region: "us-east-1"
  version: "1.2.3"
  deployment_id: "deploy-20240115-120000"

Build-ID Embedding¶

Build ID Injection:

# build_id_injection.py
class BuildIDInjector:
    def inject_build_id(self, config: dict) -> dict:
        """Inject build ID into configuration"""
        build_id = os.getenv("BUILD_ID") or generate_build_id()

        config['metadata'] = {
            'build_id': build_id,
            'build_timestamp': datetime.now().isoformat(),
            'git_commit': get_git_commit(),
            'git_branch': get_git_branch()
        }

        return config

Schema Versioning¶

Schema Version Strategy:

# schema-versioning.yaml
schemas:
  postgres:
    current_version: "2.0.0"
    versions:
      - version: "1.0.0"
        migration: "migrations/001_initial.sql"
      - version: "2.0.0"
        migration: "migrations/002_add_users.sql"

  api:
    current_version: "1.5.0"
    versions:
      - version: "1.0.0"
        openapi: "api/v1/openapi.yaml"
      - version: "1.5.0"
        openapi: "api/v1.5/openapi.yaml"

Config Fingerprinting¶

Fingerprint Generation:

# fingerprinting.py
class ConfigFingerprinter:
    def generate_fingerprint(self, config: dict) -> str:
        """Generate configuration fingerprint"""
        # Normalize config (remove metadata, sort keys)
        normalized = self.normalize_config(config)

        # Generate hash
        fingerprint = hashlib.sha256(
            json.dumps(normalized, sort_keys=True).encode()
        ).hexdigest()

        return fingerprint

Artifact Provenance¶

Provenance Tracking:

# provenance.yaml
provenance:
  config_id: "config-abc123"
  source:
    repo: "https://github.com/org/config-repo"
    commit: "abc123def456"
    path: "config/environments/prod/"
  build:
    build_id: "build-xyz789"
    build_timestamp: "2024-01-15T12:00:00Z"
  deployment:
    deployed_by: "ci-system"
    deployed_at: "2024-01-15T12:05:00Z"
    deployed_to: "prod-cluster"

In-Cluster Validation Hooks¶

Validation Webhook:

# validation/webhook.py
from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route("/validate", methods=["POST"])
def validate_config():
    """Kubernetes validation webhook"""
    admission_review = request.json

    # Extract config
    config = admission_review["request"]["object"]

    # Validate
    try:
        validate_configuration(config)
        allowed = True
        message = "Configuration valid"
    except ValidationError as e:
        allowed = False
        message = str(e)

    # Return response
    return jsonify({
        "apiVersion": "admission.k8s.io/v1",
        "kind": "AdmissionReview",
        "response": {
            "uid": admission_review["request"]["uid"],
            "allowed": allowed,
            "status": {"message": message}
        }
    })

Environment-Aware Dashboards¶

Dashboard Configuration:

id=__codelineno-74-1 name=__codelineno-74-1 href=#__codelineno-74-1>{ "dashboard": { "title": "Configuration Health by Environment", "panels": [ { "title": "Config Drift by Environment", "targets": [ { "expr": "config_drift_score{environment=\"dev\"}", "legendFormat": "Dev" }, { "expr": "config_drift_score{environment=\"prod\"}", "legendFormat": "Prod" } ] }, { "title": "Config Version Distribution", "targets": [ { "expr": "count by (version) (config_versions)", "legendFormat": "{{version}}" } ] } ] } class=p>}

Secrets Lifecycle Governance¶

Sealed Secrets¶

Sealed Secrets Pattern:

# sealed-secret.yaml
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: database-credentials
spec:
  encryptedData:
    password: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEQAx...

Age/SOPS Tooling¶

SOPS Configuration:

# .sops.yaml
creation_rules:
  - path_regex: secrets/dev/.*
    kms: "arn:aws:kms:us-east-1:123456789012:key/abc123"

  - path_regex: secrets/prod/.*
    kms: "arn:aws:kms:us-east-1:123456789012:key/xyz789"

Usage:

# Encrypt
sops -e -i secrets/dev/database.yaml

# Decrypt
sops -d secrets/dev/database.yaml

HashiCorp Vault Patterns¶

Vault Integration:

# vault-config.yaml
vault:
  address: "https://vault.example.com"
  auth:
    method: "kubernetes"
    role: "app-role"
  paths:
    dev: "secret/data/dev"
    prod: "secret/data/prod"

External Secrets Operator:

# external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: database-credentials
  data:
    - secretKey: password
      remoteRef:
        key: secret/data/prod/database
        property: password

Rotation Schedules¶

Rotation Policy:

# rotation-policy.yaml
rotation:
  schedules:
    database_passwords:
      frequency: "90 days"
      environments:
        - prod
        - staging

    api_keys:
      frequency: "180 days"
      environments:
        - prod

    tls_certificates:
      frequency: "365 days"
      environments:
        - prod

KMS Integration¶

KMS Configuration:

# kms-config.yaml
kms:
  provider: "aws"
  region: "us-east-1"
  key_id: "arn:aws:kms:us-east-1:123456789012:key/abc123"
  encryption_context:
    environment: "prod"
    service: "database"

Ephemeral Secrets for Ephemeral Environments¶

Ephemeral Secret Pattern:

# ephemeral_secrets.py
class EphemeralSecretManager:
    def create_ephemeral_secret(self, environment: str, ttl: int = 3600):
        """Create ephemeral secret for temporary environment"""
        secret = generate_secret()

        # Store with TTL
        store_secret_with_ttl(environment, secret, ttl)

        return secret

Redaction Rules¶

Redaction Configuration:

# redaction-rules.yaml
redaction:
  rules:
    - pattern: "password.*"
      action: "redact"
      replacement: "***REDACTED***"

    - pattern: "api_key.*"
      action: "redact"
      replacement: "***REDACTED***"

    - pattern: "secret.*"
      action: "redact"
      replacement: "***REDACTED***"

Integration with Postgres pgaudit¶

PgAudit Integration:

-- Configure pgaudit for secret access
ALTER SYSTEM SET pgaudit.log = 'all';
ALTER SYSTEM SET pgaudit.log_catalog = off;
ALTER SYSTEM SET pgaudit.log_parameter = on;

-- Audit secret table access
CREATE TABLE secrets (
    id SERIAL PRIMARY KEY,
    key VARCHAR(255),
    value TEXT,
    environment VARCHAR(50)
);

-- Enable auditing
ALTER TABLE secrets ENABLE ROW LEVEL SECURITY;

Tooling & Implementation Examples¶

Helm Hierarchical Values¶

Values Structure:

# values.yaml (base)
replicas: 1
image:
  repository: app
  tag: latest

# values-dev.yaml
replicas: 1
image:
  tag: dev

# values-prod.yaml
replicas: 5
image:
  tag: v1.2.3

Usage:

helm install app ./chart -f values.yaml -f values-dev.yaml

Kustomize Overlays¶

Overlay Structure:

kustomize/
├── base/
│   ├── kustomization.yaml
│   └── deployment.yaml
├── dev/
│   ├── kustomization.yaml
│   └── patches/
│       └── replicas.yaml
└── prod/
    ├── kustomization.yaml
    └── patches/
        └── replicas.yaml

Base Kustomization:

# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml

Dev Overlay:

# dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../base
patches:
  - path: patches/replicas.yaml

Python Pydantic Settings¶

Pydantic Config:

# config.py
from pydantic import BaseSettings, Field
from typing import Literal

class DatabaseSettings(BaseSettings):
    host: str = Field(..., env="DATABASE_HOST")
    port: int = Field(5432, env="DATABASE_PORT")
    database: str = Field(..., env="DATABASE_NAME")
    pool_size: int = Field(10, ge=1, le=100)

class AppSettings(BaseSettings):
    environment: Literal["dev", "staging", "prod"]
    database: DatabaseSettings
    log_level: Literal["DEBUG", "INFO", "WARNING", "ERROR"] = "INFO"

    class Config:
        env_file = f".env.{environment}"
        env_file_encoding = "utf-8"

# Load config
settings = AppSettings(environment="prod")

Go fx/Config Loader Patterns¶

Go Config:

// config.go
package config

import (
    "github.com/spf13/viper"
)

type Config struct {
    Environment string `mapstructure:"environment"`
    Database    DatabaseConfig `mapstructure:"database"`
    LogLevel    string `mapstructure:"log_level"`
}

type DatabaseConfig struct {
    Host     string `mapstructure:"host"`
    Port     int    `mapstructure:"port"`
    Database string `mapstructure:"database"`
}

func LoadConfig(env string) (*Config, error) {
    viper.SetConfigName("config")
    viper.SetConfigType("yaml")
    viper.AddConfigPath(fmt.Sprintf("config/environments/%s/", env))

    if err := viper.ReadInConfig(); err != nil {
        return nil, err
    }

    var config Config
    if err := viper.Unmarshal(&config); err != nil {
        return nil, err
    }

    return &config, nil
}

Rust figment/Config Crates¶

Rust Config:

// config.rs
use figment::{Figment, providers::{Format, Yaml, Env}};
use serde::{Deserialize, Serialize};

#[derive(Debug, Deserialize, Serialize)]
struct Config {
    environment: String,
    database: DatabaseConfig,
    log_level: String,
}

#[derive(Debug, Deserialize, Serialize)]
struct DatabaseConfig {
    host: String,
    port: u16,
    database: String,
}

impl Config {
    fn load(env: &str) -> Result<Self, figment::Error> {
        Figment::new()
            .merge(Yaml::file(format!("config/environments/{}/config.yaml", env)))
            .merge(Env::prefixed("APP_"))
            .extract()
    }
}

Prefect Block Configurations¶

Prefect Blocks:

# prefect_blocks.py
from prefect.blocks.system import Secret
from prefect import flow

@flow
def etl_flow():
    """ETL flow with Prefect blocks"""
    # Load secret block
    db_password = Secret.load("database-password")

    # Use in flow
    database_url = f"postgresql://user:{db_password.get()}@host/db"
    # ...

Postgres Config Templating via Jinja¶

Jinja Template:

# postgresql.conf.j2
# Auto-generated from template
shared_buffers = {{ shared_buffers }}
max_connections = {{ max_connections }}
work_mem = {{ work_mem }}

# Environment-specific
{% if environment == "prod" %}
shared_buffers = 1GB
max_connections = 200
{% else %}
shared_buffers = 256MB
max_connections = 100
{% endif %}

Template Rendering:

# render_postgres_config.py
from jinja2 import Template

def render_postgres_config(environment: str) -> str:
    """Render Postgres config from Jinja template"""
    template = Template(open("postgresql.conf.j2").read())

    context = {
        'environment': environment,
        'shared_buffers': get_shared_buffers(environment),
        'max_connections': get_max_connections(environment),
        'work_mem': get_work_mem(environment)
    }

    return template.render(**context)

GitHub Actions + Mike Versioned Docs¶

GitHub Actions:

# .github/workflows/deploy-docs.yml
name: Deploy Docs
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          pip install mkdocs-material mike

      - name: Deploy docs
        run: |
          mike deploy latest
          mike set-default latest

Air-Gapped Config Packaging¶

Packaging Script:

#!/bin/bash
# scripts/package_airgap_config.sh

PACKAGE_NAME="config-bundle-$(date +%Y%m%d).tar.gz"

# Package configs
tar -czf "$PACKAGE_NAME" \
    config/ \
    schemas/ \
    secrets-encrypted/ \
    checksums.txt

# Sign package
gpg --sign "$PACKAGE_NAME"

# Create manifest
cat > manifest.json <<EOF
{
  "package_name": "$PACKAGE_NAME",
  "created_at": "$(date -Iseconds)",
  "checksum": "$(sha256sum $PACKAGE_NAME | cut -d' ' -f1)",
  "environments": ["dev", "staging", "prod"]
}
EOF

echo "Package created: $PACKAGE_NAME"

Agentic LLM Integration Hooks¶

Generate Safe Environment Configurations¶

LLM Config Generator:

# llm/config_generator.py
class LLMConfigGenerator:
    def generate_config(self, requirements: dict) -> dict:
        """Generate safe environment configuration using LLM"""
        prompt = f"""
        Generate safe environment configuration for:

        {json.dumps(requirements, indent=2)}

        Requirements:
        1. Follow security best practices
        2. Use appropriate resource limits
        3. Include proper validation
        4. Document all settings
        """

        response = self.llm_client.chat.completions.create(
            model="gpt-4",
            messages=[
                {"role": "system", "content": "You are a configuration generation expert."},
                {"role": "user", "content": prompt}
            ]
        )

        return json.loads(response.choices[0].message.content)

Validate Configs for Inconsistencies¶

LLM Config Validator:

# llm/config_validator.py
class LLMConfigValidator:
    def validate_config(self, config: dict) -> List[dict]:
        """Validate config for inconsistencies using LLM"""
        prompt = f"""
        Validate this configuration for inconsistencies:

        {json.dumps(config, indent=2)}

        Check for:
        1. Inconsistent values
        2. Security issues
        3. Performance problems
        4. Missing required fields
        """

        response = self.llm_client.chat.completions.create(
            model="gpt-4",
            messages=[
                {"role": "system", "content": "You are a configuration validation expert."},
                {"role": "user", "content": prompt}
            ]
        )

        return json.loads(response.choices[0].message.content)

Surface Hidden Drift¶

LLM Drift Detector:

# llm/drift_detector.py
class LLMDriftDetector:
    def detect_hidden_drift(self, configs: dict) -> List[dict]:
        """Detect hidden configuration drift using LLM"""
        prompt = f"""
        Analyze these configurations for hidden drift:

        {json.dumps(configs, indent=2)}

        Identify:
        1. Subtle differences
        2. Implicit dependencies
        3. Configuration patterns
        4. Potential drift sources
        """

        response = self.llm_client.chat.completions.create(
            model="gpt-4",
            messages=[
                {"role": "system", "content": "You are a drift detection expert."},
                {"role": "user", "content": prompt}
            ]
        )

        return json.loads(response.choices[0].message.content)

Propose Environment Diffs¶

LLM Diff Proposer:

# llm/diff_proposer.py
class LLMDiffProposer:
    def propose_diff(self, env1: str, env2: str) -> dict:
        """Propose environment diff using LLM"""
        prompt = f"""
        Propose configuration diff between {env1} and {env2}:

        Environment 1 ({env1}):
        {json.dumps(load_config(env1), indent=2)}

        Environment 2 ({env2}):
        {json.dumps(load_config(env2), indent=2)}

        Provide:
        1. Diff summary
        2. Rationale for differences
        3. Recommendations
        """

        response = self.llm_client.chat.completions.create(
            model="gpt-4",
            messages=[
                {"role": "system", "content": "You are a configuration diff expert."},
                {"role": "user", "content": prompt}
            ]
        )

        return json.loads(response.choices[0].message.content)

Generate Config Promotion Plans¶

LLM Promotion Planner:

# llm/promotion_planner.py
class LLMPromotionPlanner:
    def generate_promotion_plan(self, source_env: str, target_env: str) -> dict:
        """Generate config promotion plan using LLM"""
        prompt = f"""
        Generate promotion plan from {source_env} to {target_env}:

        Source environment:
        {json.dumps(load_config(source_env), indent=2)}

        Target environment:
        {json.dumps(load_config(target_env), indent=2)}

        Provide:
        1. Promotion steps
        2. Validation checks
        3. Rollback plan
        4. Risk assessment
        """

        response = self.llm_client.chat.completions.create(
            model="gpt-4",
            messages=[
                {"role": "system", "content": "You are a configuration promotion expert."},
                {"role": "user", "content": prompt}
            ]
        )

        return json.loads(response.choices[0].message.content)

Detect Misunderstood Settings¶

LLM Setting Analyzer:

# llm/setting_analyzer.py
class LLMSettingAnalyzer:
    def analyze_settings(self, config: dict, system_info: dict) -> List[dict]:
        """Detect misunderstood settings using LLM"""
        prompt = f"""
        Analyze these settings for potential issues:

        Configuration:
        {json.dumps(config, indent=2)}

        System Info:
        {json.dumps(system_info, indent=2)}

        Identify:
        1. Settings that don't match system capacity
        2. Incompatible settings
        3. Performance issues
        4. Security concerns
        """

        response = self.llm_client.chat.completions.create(
            model="gpt-4",
            messages=[
                {"role": "system", "content": "You are a configuration analysis expert."},
                {"role": "user", "content": prompt}
            ]
        )

        return json.loads(response.choices[0].message.content)

Produce Diagrams of Config Layering¶

LLM Diagram Generator:

# llm/diagram_generator.py
class LLMDiagramGenerator:
    def generate_layering_diagram(self, config_structure: dict) -> str:
        """Generate Mermaid diagram of config layering"""
        prompt = f"""
        Generate a Mermaid diagram showing configuration layering:

        {json.dumps(config_structure, indent=2)}

        Show:
        1. Base layer
        2. Environment layer
        3. Secrets layer
        4. Runtime layer
        """

        response = self.llm_client.chat.completions.create(
            model="gpt-4",
            messages=[
                {"role": "system", "content": "You are a diagram generation expert."},
                {"role": "user", "content": prompt}
            ]
        )

        return response.choices[0].message.content

Audit Kubernetes Manifests for Environment Divergence¶

LLM K8s Auditor:

# llm/k8s_auditor.py
class LLMKubernetesAuditor:
    def audit_manifests(self, manifests: List[dict]) -> dict:
        """Audit Kubernetes manifests for environment divergence"""
        prompt = f"""
        Audit these Kubernetes manifests for environment divergence:

        {json.dumps(manifests, indent=2)}

        Identify:
        1. Environment-specific differences
        2. Inconsistencies
        3. Security issues
        4. Best practice violations
        """

        response = self.llm_client.chat.completions.create(
            model="gpt-4",
            messages=[
                {"role": "system", "content": "You are a Kubernetes audit expert."},
                {"role": "user", "content": prompt}
            ]
        )

        return json.loads(response.choices[0].message.content)

Auto-Patch and Auto-Refactor Configuration Structure¶

LLM Config Refactorer:

# llm/config_refactorer.py
class LLMConfigRefactorer:
    def refactor_config(self, config: dict, issues: List[str]) -> dict:
        """Auto-refactor configuration structure using LLM"""
        prompt = f"""
        Refactor this configuration to address issues:

        Configuration:
        {json.dumps(config, indent=2)}

        Issues:
        {json.dumps(issues, indent=2)}

        Provide refactored configuration that:
        1. Addresses all issues
        2. Maintains backward compatibility
        3. Follows best practices
        """

        response = self.llm_client.chat.completions.create(
            model="gpt-4",
            messages=[
                {"role": "system", "content": "You are a configuration refactoring expert."},
                {"role": "user", "content": prompt}
            ]
        )

        return json.loads(response.choices[0].message.content)

Configuration Scoring Rubric¶

Scoring Criteria¶

Rubric:

# config-scoring-rubric.yaml
scoring_criteria:
  versioning:
    weight: 0.15
    criteria:
      - "All configs versioned": 10
      - "Most configs versioned": 7
      - "Some configs versioned": 4
      - "No versioning": 0

  validation:
    weight: 0.20
    criteria:
      - "Comprehensive validation": 10
      - "Basic validation": 7
      - "Minimal validation": 4
      - "No validation": 0

  documentation:
    weight: 0.15
    criteria:
      - "Complete documentation": 10
      - "Good documentation": 7
      - "Basic documentation": 4
      - "No documentation": 0

  drift_detection:
    weight: 0.20
    criteria:
      - "Automated drift detection": 10
      - "Manual drift detection": 7
      - "Periodic checks": 4
      - "No drift detection": 0

  secrets_management:
    weight: 0.15
    criteria:
      - "Centralized secrets management": 10
      - "Standardized secrets": 7
      - "Some secrets management": 4
      - "No secrets management": 0

  promotion_process:
    weight: 0.15
    criteria:
      - "Automated promotion": 10
      - "Standardized promotion": 7
      - "Manual promotion": 4
      - "No promotion process": 0

Scoring Calculation¶

Calculator:

# scoring/calculator.py
class ConfigScorer:
    def calculate_score(self, config: dict) -> float:
        """Calculate configuration score"""
        rubric = load_scoring_rubric()

        scores = {}
        for criterion, details in rubric['scoring_criteria'].items():
            score = self.evaluate_criterion(config, criterion, details)
            scores[criterion] = score * details['weight']

        total_score = sum(scores.values())
        return total_score

Template for Environment Configuration Governance¶

Governance Template¶

Template:

# governance-template.yaml
configuration_governance:
  policy:
    name: "Environment Configuration Governance Policy"
    version: "1.0.0"
    effective_date: "2024-01-01"

  principles:
    - "Configuration as a Product"
    - "Declarative Configuration First"
    - "Immutable Infrastructure"
    - "Fail-Fast Validation"
    - "Explicit Environment Boundaries"
    - "Thin Environment Differences"
    - "Version-Locked Everything"

  standards:
    versioning:
      required: true
      format: "semver"

    validation:
      required: true
      tools: ["pydantic", "jsonschema"]

    documentation:
      required: true
      format: "markdown"

    drift_detection:
      required: true
      frequency: "daily"

  processes:
    promotion:
      - "dev → test: Automated"
      - "test → staging: Automated + Approval"
      - "staging → prod: Automated + Approval + Review"

    review:
      frequency: "quarterly"
      participants: ["architecture-team", "platform-team"]

  metrics:
    - "Config drift score"
    - "Promotion success rate"
    - "Validation failure rate"
    - "Documentation coverage"