Skip to content

Docker Compose Setup

Blacklake provides a unified Docker Compose setup with profile-driven architecture for development, testing, and production.

Architecture

The Docker Compose configuration uses profiles to organize services into logical groups:

  • db: Database services (etcd, Patroni, HAProxy, PgBouncer)
  • storage: Object storage (MinIO + bucket creation)
  • search: Search services (Solr)
  • app: Application services (FastAPI, migrations)
  • monitoring: Monitoring stack (Prometheus, Grafana, exporters)
  • backup: Backup services (pgBackRest + cron)

Quick Start

# Copy environment template
cp .env.example .env

# Start core services (database + storage)
make up-core

# Start application
make up-app

# Start search
make up-search

# Start monitoring
make up-monitoring

# Start everything
make up-all

# Check status
make ps

# View logs
make logs

# Stop all
make down

Using Docker Compose Directly

# Start specific profiles
docker compose --profile db --profile storage up -d

# Start all profiles
docker compose --profile db --profile storage --profile search --profile app --profile monitoring up -d

# Check status
docker compose ps

# View logs
docker compose logs -f

Service Profiles

Database Profile (db)

  • etcd: Distributed coordination
  • patroni-pg0: Primary PostgreSQL node
  • patroni-pg1: Replica PostgreSQL node
  • haproxy: Load balancer for database VIP
  • pgbouncer: Connection pooler

Storage Profile (storage)

  • minio: S3-compatible object storage
  • create-buckets: Initialize required buckets
  • solr: Search engine with Blacklake core

Application Profile (app)

  • api: FastAPI REST API server
  • migrate: Database migration runner

Monitoring Profile (monitoring)

  • prometheus: Metrics collection
  • grafana: Monitoring dashboards
  • postgres_exporter: PostgreSQL metrics
  • pgbouncer_exporter: PgBouncer metrics

Backup Profile (backup)

  • pgbackrest: Backup service
  • backup-cron: Automated backup scheduling

Configuration

Environment Variables

Copy .env.example to .env and customize:

# Database Configuration
POSTGRES_SUPERUSER_PASSWORD=superuser123
POSTGRES_REPL_PASSWORD=repl123
POSTGRES_APP_PASSWORD=app123
POSTGRES_PGBOUNCER_PASSWORD=pgbouncer123

# Application Database User
APP_DB_USER=app_user
APP_DB_PASSWORD=app123

# MinIO/S3 Configuration
MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=minioadmin
S3_ENDPOINT=http://minio:9000
S3_BUCKET=blacklake-data

# Solr Configuration
SOLR_CORE=blacklake

# JWT Secret
JWT_SECRET=your-jwt-secret-key-change-this-in-production

Service Configuration Files

Configuration files are mounted from the infra/ directory:

  • Patroni: infra/patroni/patroni-pg0.yml, patroni-pg1.yml
  • PgBouncer: infra/pgbouncer/pgbouncer.ini
  • HAProxy: infra/haproxy/haproxy.cfg
  • pgBackRest: infra/pgbackrest/pgbackrest.conf
  • Prometheus: monitoring/prometheus.yml
  • Grafana: monitoring/grafana/

Make Targets

Core Operations

make up-core        # Start db+storage profiles
make up-app         # Start app profile (api+migrate)
make up-search      # Start search profile (solr)
make up-monitoring  # Start monitoring stack
make up-all         # Start all profiles
make down           # Stop all services
make ps             # Show service status
make logs           # Tail logs

Advanced Operations

make backup         # Run pgBackrest full backup
make smoke          # Run database infrastructure smoke test
make clean          # Remove volumes and containers
make restart        # Restart all services

Custom Profiles

# Start specific profiles
PROFILES=db,storage make up

# Start with custom project name
PROJECT=my-blacklake make up-core

Health Checks

All services include comprehensive health checks:

# Check overall status
make ps

# Check specific service
docker compose exec service-name health-check-command

# Database health
curl http://localhost:8000/health/db

# HA cluster health
curl http://localhost:8000/health/ha

# Patroni cluster status
curl http://localhost:8008/cluster

Development Workflow

1. Start Core Infrastructure

# Start database and storage
make up-core

# Wait for services to be healthy
make ps

2. Start Application

# Start API and run migrations
make up-app

# Check API health
curl http://localhost:8000/health/db

# Start Solr
make up-search

# Check Solr health
curl http://localhost:8983/solr/admin/ping

4. Start Monitoring (Optional)

# Start monitoring stack
make up-monitoring

# Access Grafana
open http://localhost:3000  # admin/admin

Troubleshooting

Common Issues

1. Service Startup Failures

# Check service logs
make logs

# Check specific service
docker compose logs service-name

# Restart specific service
docker compose restart service-name

2. Database Connection Issues

# Test database connectivity
docker compose exec api python -c "from blacklake.db.session import check_database_health; print(check_database_health())"

# Test PgBouncer
docker compose exec pgbouncer psql -h localhost -p 6432 -U app_user -d blacklake -c "SELECT 1;"

3. Storage Issues

# Check MinIO
curl http://localhost:9000/minio/health/live

# List buckets
docker compose exec minio mc ls local/

Debugging Commands

# Inspect configuration
docker compose config

# Check service networks
docker compose exec service-name ip route

# Check service environment
docker compose exec service-name env

# Run smoke test
make smoke

Production Considerations

Security

  • Change all default passwords in .env
  • Use TLS for all connections
  • Restrict network access
  • Enable authentication

Performance

  • Adjust resource limits in docker-compose.yml
  • Configure connection pooling
  • Optimize database settings
  • Monitor resource usage

Monitoring

  • Set up alerting in Grafana
  • Configure log aggregation
  • Monitor key metrics
  • Set up dashboards

Migration from Multi-File Setup

If you were using the previous multi-file setup:

# Old way (deprecated)
docker compose -f infra/docker-compose.db.yml up -d
docker compose -f infra/docker-compose.monitoring.yml up -d

# New way
make up-core
make up-monitoring

Service URLs

Service URL Credentials
MinIO Console http://localhost:9001 minioadmin / minioadmin
PostgreSQL localhost:5432 app_user / app123
PgBouncer localhost:6432 app_user / app123
Solr Admin http://localhost:8983/solr -
Blacklake API http://localhost:8000 -
API Docs http://localhost:8000/docs -
Prometheus http://localhost:9090 -
Grafana http://localhost:3000 admin / admin

Next Steps