Skip to content

Server Overview

The Blacklake API server provides a RESTful interface for managing data repositories with semantic metadata.

Features

  • RESTful API - Complete HTTP interface for all operations
  • OpenAPI Documentation - Interactive API documentation
  • Authentication - JWT-based authentication (optional)
  • Content Negotiation - JSON-LD support for metadata
  • Async Operations - High-performance async processing
  • Health Checks - Built-in monitoring endpoints

Quick Start

Start Server

# Start with default configuration
blacklake server

# Start with custom host/port
blacklake server --host 127.0.0.1 --port 9000

# Start with auto-reload for development
blacklake server --reload

Expected Output: 

INFO:     Started server process [12345]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Access API Documentation

Open your browser to: - Swagger UI: http://localhost:8000/docs - ReDoc: http://localhost:8000/redoc - OpenAPI JSON: http://localhost:8000/openapi.json

Run Modes

Development Mode

# Start with auto-reload
blacklake server --reload --host 127.0.0.1 --port 8000

Features: - Auto-reload on code changes - Debug logging - CORS enabled - Detailed error messages

Production Mode

# Start with multiple workers
blacklake server --host 0.0.0.0 --port 8000 --workers 4

Features: - Multiple worker processes - Optimized logging - Security headers - Performance monitoring

Docker Mode

# Using docker-compose
docker-compose up -d

# Or using Docker directly
docker run -p 8000:8000 \
  -e S3_ENDPOINT=http://minio:9000 \
  -e S3_BUCKET=blacklake-data \
  sempervent/blacklake:latest

Configuration

Environment Variables

Variable Default Description
BLACKLAKE_HOST 0.0.0.0 Server host
BLACKLAKE_PORT 8000 Server port
BLACKLAKE_WORKERS 4 Number of workers
BLACKLAKE_LOG_LEVEL INFO Log level
DATABASE_URL - PostgreSQL connection string
SOLR_URL - Solr base URL

Configuration File

# .bl/config.yml
server:
  host: "0.0.0.0"
  port: 8000
  workers: 4
  log_level: "INFO"
  cors_origins: ["http://localhost:3000"]
  rate_limit: 100  # requests per minute

API Endpoints

Core Endpoints

Method Endpoint Description
GET / API information
GET /health Health check
GET /repos List repositories
POST /repos Create repository
GET /repos/{id} Get repository
GET /commits List commits
POST /commits Create commit
GET /commits/{id} Get commit
GET /objects/{sha256} Get object
POST /objects Create object
GET /search Search content

Metadata Endpoints

Method Endpoint Description
POST /metadata/objects Create object metadata
GET /metadata/objects/{id} Get object metadata
POST /metadata/agents Create agent metadata
GET /metadata/agents List agents
GET /metadata/export Export metadata

Authentication

JWT Authentication (Optional)

# Get authentication token
curl -X POST http://localhost:8000/auth/token \
  -H "Content-Type: application/json" \
  -d '{"username": "admin", "password": "secret"}'

# Use token in requests
curl -H "Authorization: Bearer <token>" \
  http://localhost:8000/repos

API Key Authentication

# Use API key
curl -H "X-API-Key: your-api-key" \
  http://localhost:8000/repos

Content Types

JSON-LD Support

# Request JSON-LD metadata
curl -H "Accept: application/ld+json" \
  http://localhost:8000/metadata/objects/abc123

# Response
{
  "@context": "https://sempervent.github.io/blacklake/ns#",
  "@id": "bl:obj:sha256:abc123",
  "@type": "Object",
  "title": "Sales Data",
  "creator": {
    "@type": "Person",
    "name": "Data Team"
  }
}

Standard JSON

# Request standard JSON
curl -H "Accept: application/json" \
  http://localhost:8000/metadata/objects/abc123

# Response
{
  "id": "bl:obj:sha256:abc123",
  "title": "Sales Data",
  "creator": {
    "name": "Data Team",
    "email": "data@company.com"
  }
}

Error Handling

Standard Error Format

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Invalid metadata format",
    "details": {
      "field": "title",
      "reason": "Required field missing"
    }
  }
}

HTTP Status Codes

Code Description
200 Success
201 Created
400 Bad Request
401 Unauthorized
403 Forbidden
404 Not Found
422 Validation Error
500 Internal Server Error

Monitoring

Health Checks

# Basic health check
curl http://localhost:8000/health

# Detailed health check
curl http://localhost:8000/health/detailed

Expected Output: 

{
  "status": "healthy",
  "timestamp": "2024-01-15T10:30:00Z",
  "services": {
    "database": "healthy",
    "solr": "healthy",
    "s3": "healthy"
  }
}

Metrics

# Get metrics
curl http://localhost:8000/metrics

# Prometheus format
curl http://localhost:8000/metrics/prometheus

Performance

Caching

# Enable Redis caching
server:
  cache:
    type: "redis"
    url: "redis://localhost:6379"
    ttl: 3600  # seconds

Rate Limiting

# Configure rate limiting
server:
  rate_limit:
    requests: 100
    window: 60  # seconds
    burst: 200

Security

CORS Configuration

# Configure CORS
server:
  cors:
    origins: ["https://app.company.com"]
    methods: ["GET", "POST", "PUT", "DELETE"]
    headers: ["Authorization", "Content-Type"]

Security Headers

# Security headers
server:
  security:
    headers:
      - "X-Content-Type-Options: nosniff"
      - "X-Frame-Options: DENY"
      - "X-XSS-Protection: 1; mode=block"

Deployment

Docker Compose

# docker-compose.yml
version: '3.8'
services:
  api:
    image: sempervent/blacklake:latest
    ports:
      - "8000:8000"
    environment:
      - DATABASE_URL=postgresql://blacklake:blacklake@postgres:5432/blacklake
      - SOLR_URL=http://solr:8983/solr/blacklake
      - S3_ENDPOINT=http://minio:9000
      - S3_BUCKET=blacklake-data
    depends_on:
      - postgres
      - solr
      - minio

Kubernetes

# k8s-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: blacklake-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: blacklake-api
  template:
    metadata:
      labels:
        app: blacklake-api
    spec:
      containers:
      - name: api
        image: sempervent/blacklake:latest
        ports:
        - containerPort: 8000
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: blacklake-secrets
              key: database-url

Troubleshooting

Common Issues

Server Won't Start 

# Check configuration
blacklake config validate

# Check port availability
netstat -tlnp | grep :8000

Database Connection Issues 

# Test database connection
psql -h localhost -U blacklake -d blacklake -c "SELECT 1;"

# Check database URL
echo $DATABASE_URL

High Memory Usage 

# Monitor memory usage
htop

# Adjust worker count
blacklake server --workers 2

Logs

# View server logs
tail -f /var/log/blacklake/server.log

# Debug mode
BLACKLAKE_LOG_LEVEL=DEBUG blacklake server

Next Steps