Skip to content

Quickstart

Get Blacklake running in 5 minutes with a complete example.

Prerequisites

  • Python 3.12+
  • uv package manager
  • Docker (for local services)

Step 1: Install Blacklake

# Install with server extras
uv add blacklake[server]

# Or install from source
git clone https://github.com/sempervent/blacklake.git
cd blacklake
uv pip install -e ".[server]"

Step 2: Start Local Services

# Start MinIO, PostgreSQL, and Solr
docker-compose up -d

# Wait for services to be ready
docker-compose ps

Step 3: Initialize Repository

# Create a new directory
mkdir my-data-repo
cd my-data-repo

# Initialize Blacklake repository
blacklake init \
  --s3-endpoint http://localhost:9000 \
  --s3-bucket blacklake-data \
  --s3-access-key minioadmin \
  --s3-secret-key minioadmin

Expected Output: 

✓ Initialized empty blacklake repository in .bl/

Step 4: Add Data with Metadata

# Create a sample dataset
echo "name,age,city
Alice,30,New York
Bob,25,San Francisco
Charlie,35,Chicago" > people.csv

# Add metadata to the file
blacklake meta add people.csv \
  --title "People Dataset" \
  --description "Sample people data with demographics" \
  --creator "Data Team" \
  --email "data@company.com" \
  --keywords "demographics,people,dataset" \
  --rights "CC BY 4.0" \
  --license "https://creativecommons.org/licenses/by/4.0/"

Expected Output: 

✓ Metadata added for people.csv

Step 5: Stage and Commit

# Add file to staging
blacklake add people.csv

# Check status
blacklake status

Expected Output: 

On branch main
Changes to be committed:
  people.csv (new file)

Untracked files:
  (none)


# Commit changes
blacklake commit -m "Add people dataset with metadata"

Expected Output: 

✓ Changes committed

Step 6: Push to S3

# Push to remote S3 storage
blacklake push origin main

Expected Output: 

Pushing to origin/main...
✓ Pushed 1 objects to S3


# Index the data for search
blacklake index people.csv

Expected Output: 

Indexing: people.csv
✓ Data indexed

Step 8: Search Your Data

# Search for data
blacklake search "people demographics"

Expected Output: 

Searching for: people demographics
Found 1 result:
- people.csv: People Dataset (Sample people data with demographics)

Step 9: View Metadata

# List all metadata
blacklake meta export --format jsonld

Expected Output: 

{
  "@context": "../contexts/blacklake.context.jsonld",
  "@graph": [
    {
      "id": "bl:obj:sha256:abc123...",
      "type": ["Object", "dcterms:Dataset"],
      "title": "People Dataset",
      "description": "Sample people data with demographics",
      "creator": {
        "id": "urn:uuid:...",
        "type": "Person",
        "name": "Data Team",
        "mbox": "mailto:data@company.com"
      },
      "keyword": ["demographics", "people", "dataset"],
      "rights": "CC BY 4.0",
      "license": {
        "id": "https://creativecommons.org/licenses/by/4.0/"
      }
    }
  ]
}

What's Next?

Troubleshooting

Services Not Starting

# Check service status
docker-compose ps

# View logs
docker-compose logs minio
docker-compose logs postgres
docker-compose logs solr

S3 Connection Issues

# Test S3 connection
curl http://localhost:9000/minio/health/live

Database Connection Issues

# Check PostgreSQL
docker-compose exec postgres psql -U blacklake -d blacklake -c "SELECT 1;"

Next Steps