Skip to content

Configuration

Blacklake uses a hierarchical configuration system with clear precedence rules.

Configuration Precedence

Configuration is loaded in the following order (later sources override earlier ones):

  1. CLI Arguments - Highest priority
  2. Environment Variables - Second priority
  3. .bl/config.yml - Repository-specific configuration
  4. Default Values - Lowest priority

Configuration Sources

1. CLI Arguments

# Override any configuration via CLI
blacklake init \
  --s3-endpoint http://custom-s3:9000 \
  --s3-bucket my-bucket \
  --s3-region us-west-2

2. Environment Variables

# Set environment variables
export S3_ENDPOINT="http://custom-s3:9000"
export S3_BUCKET="my-bucket"
export S3_REGION="us-west-2"
export S3_ACCESS_KEY="your-access-key"
export S3_SECRET_KEY="your-secret-key"

# Use in commands
blacklake init

3. Repository Configuration

Create .bl/config.yml in your repository:

# .bl/config.yml
s3:
  endpoint: "http://localhost:9000"
  bucket: "blacklake-data"
  region: "us-east-1"
  access_key: "minioadmin"
  secret_key: "minioadmin"
  use_ssl: false
  signature_version: "s3v4"

database:
  url: "postgresql://blacklake:blacklake@localhost:5432/blacklake"
  pool_size: 10
  max_overflow: 20

solr:
  url: "http://localhost:8983/solr/blacklake"
  timeout: 30
  max_retries: 3

server:
  host: "0.0.0.0"
  port: 8000
  workers: 4
  log_level: "INFO"

metadata:
  validation: true
  shacl_shapes: true
  export_formats: ["jsonld", "ttl", "nquads"]

Configuration Examples

Development Setup

# .bl/config.yml for development
s3:
  endpoint: "http://localhost:9000"
  bucket: "blacklake-dev"
  access_key: "minioadmin"
  secret_key: "minioadmin"

database:
  url: "postgresql://blacklake:blacklake@localhost:5432/blacklake_dev"

solr:
  url: "http://localhost:8983/solr/blacklake_dev"

server:
  host: "127.0.0.1"
  port: 8000
  log_level: "DEBUG"

Production Setup

# .bl/config.yml for production
s3:
  endpoint: "https://s3.amazonaws.com"
  bucket: "company-data-lake"
  region: "us-west-2"
  access_key: "${AWS_ACCESS_KEY_ID}"
  secret_key: "${AWS_SECRET_ACCESS_KEY}"
  use_ssl: true
  signature_version: "s3v4"

database:
  url: "${DATABASE_URL}"
  pool_size: 20
  max_overflow: 30

solr:
  url: "${SOLR_URL}"
  timeout: 60
  max_retries: 5

server:
  host: "0.0.0.0"
  port: 8000
  workers: 8
  log_level: "INFO"

metadata:
  validation: true
  shacl_shapes: true
  export_formats: ["jsonld", "ttl", "nquads"]

Multi-Environment Setup

# Development
export BLACKLAKE_ENV="development"
blacklake init

# Staging
export BLACKLAKE_ENV="staging"
blacklake init

# Production
export BLACKLAKE_ENV="production"
blacklake init

Environment Variables Reference

S3/Object Storage

Variable Default Description
S3_ENDPOINT - S3 endpoint URL
S3_BUCKET - S3 bucket name
S3_REGION us-east-1 AWS region
S3_ACCESS_KEY - Access key
S3_SECRET_KEY - Secret key
S3_USE_SSL true Use HTTPS
S3_SIGNATURE_VERSION s3v4 Signature version

Database

Variable Default Description
DATABASE_URL - PostgreSQL connection string
DB_POOL_SIZE 10 Connection pool size
DB_MAX_OVERFLOW 20 Max overflow connections

Solr

Variable Default Description
SOLR_URL - Solr base URL
SOLR_TIMEOUT 30 Request timeout (seconds)
SOLR_MAX_RETRIES 3 Max retry attempts

Server

Variable Default Description
BLACKLAKE_HOST 0.0.0.0 Server host
BLACKLAKE_PORT 8000 Server port
BLACKLAKE_WORKERS 4 Number of workers
BLACKLAKE_LOG_LEVEL INFO Log level

Metadata

Variable Default Description
BLACKLAKE_VALIDATION true Enable SHACL validation
BLACKLAKE_SHACL_SHAPES true Use SHACL shapes
BLACKLAKE_EXPORT_FORMATS jsonld,ttl Export formats

Configuration Validation

Check Configuration

# Validate current configuration
blacklake config validate

# Expected output
✓ Configuration is valid
✓ S3 connection: OK
✓ Database connection: OK
✓ Solr connection: OK

Test Connections

# Test S3 connection
blacklake config test-s3

# Expected output
✓ S3 connection successful
✓ Bucket 'blacklake-data' accessible
✓ Write test successful

# Test database connection
blacklake config test-db

# Expected output
✓ Database connection successful
✓ Schema up to date
✓ Migrations applied

# Test Solr connection
blacklake config test-solr

# Expected output
✓ Solr connection successful
✓ Core 'blacklake' accessible
✓ Index test successful

Advanced Configuration

Custom S3 Providers

# Google Cloud Storage
s3:
  endpoint: "https://storage.googleapis.com"
  bucket: "my-gcs-bucket"
  region: "us-central1"
  access_key: "${GCS_ACCESS_KEY}"
  secret_key: "${GCS_SECRET_KEY}"

# MinIO
s3:
  endpoint: "https://minio.company.com"
  bucket: "data-lake"
  access_key: "${MINIO_ACCESS_KEY}"
  secret_key: "${MINIO_SECRET_KEY}"
  use_ssl: true

Database Configuration

# PostgreSQL with SSL
database:
  url: "postgresql://user:pass@host:5432/db?sslmode=require"
  pool_size: 20
  max_overflow: 30
  pool_timeout: 30
  pool_recycle: 3600

# Connection pooling
database:
  url: "postgresql://user:pass@host:5432/db"
  pool_size: 10
  max_overflow: 20
  pool_timeout: 30
  pool_recycle: 3600

Solr Configuration

# Solr Cloud
solr:
  url: "http://solr-cloud.company.com:8983/solr"
  timeout: 60
  max_retries: 5
  collection: "blacklake"

# Authentication
solr:
  url: "http://solr.company.com:8983/solr/blacklake"
  username: "${SOLR_USERNAME}"
  password: "${SOLR_PASSWORD}"
  timeout: 30

Security Configuration

S3 Security

# IAM Roles (recommended)
s3:
  endpoint: "https://s3.amazonaws.com"
  bucket: "secure-data-lake"
  region: "us-west-2"
  # No access keys - use IAM roles

# KMS Encryption
s3:
  endpoint: "https://s3.amazonaws.com"
  bucket: "secure-data-lake"
  kms_key_id: "arn:aws:kms:us-west-2:123456789012:key/12345678-1234-1234-1234-123456789012"

Database Security

# SSL Connection
database:
  url: "postgresql://user:pass@host:5432/db?sslmode=require&sslcert=/path/to/client-cert.pem&sslkey=/path/to/client-key.pem&sslrootcert=/path/to/ca-cert.pem"

Troubleshooting

Configuration Issues

# Show current configuration
blacklake config show

# Show effective configuration (after precedence)
blacklake config show --effective

# Debug configuration loading
blacklake config debug

Common Issues

S3 Connection Failed 

# Check S3 credentials
aws s3 ls s3://your-bucket

# Test with curl
curl -I https://your-s3-endpoint.com

Database Connection Failed 

# Test PostgreSQL connection
psql -h localhost -U blacklake -d blacklake -c "SELECT 1;"

# Check database URL
echo $DATABASE_URL

Solr Connection Failed 

# Test Solr connection
curl -I http://localhost:8983/solr/blacklake/admin/ping

# Check Solr URL
echo $SOLR_URL

Next Steps