Skip to content

Blacklake

A git-like data storage and retrieval system for vendor-agnostic S3/object stores with semantic metadata using JSON-LD, Dublin Core, FOAF, and OWL.

PyPI version Python 3.12+ Documentation Tests Coverage

Why Blacklake?

Modern data workflows need version control for datasets with semantic metadata that's:

  • 🔗 Linked Data: Objects connected through semantic relationships
  • 📊 Standards-Based: Dublin Core, FOAF, OWL for interoperability
  • ✅ Validated: SHACL shapes ensure metadata quality
  • 🔍 Searchable: Full-text search with semantic fields
  • 🌐 Web-Ready: JSON-LD for web integration

Features

Feature Description
Git-like Operations init, add, commit, push, pull, branch, merge
Content-Addressable SHA-256 hashing for immutable objects
S3-Compatible Works with AWS S3, MinIO, Google Cloud Storage
Semantic Metadata JSON-LD with Dublin Core, FOAF, OWL
Data Validation SHACL shapes for metadata quality
Full-Text Search Solr integration with semantic fields
REST API FastAPI server with OpenAPI documentation
Docker Ready Complete stack with docker-compose

Quick Start

# Install with uv
uv add blacklake[server]

# Initialize repository
blacklake init --s3-endpoint http://localhost:9000 --s3-bucket my-data

# Add metadata to a file
blacklake meta add dataset.csv --title "Sales Data" --creator "Data Team"

# Commit changes
blacklake commit -m "Add sales dataset"

# Push to S3
blacklake push origin main

Architecture

graph TB
    CLI[CLI] --> API[FastAPI Server]
    API --> S3[S3/Object Store]
    API --> PG[(PostgreSQL)]
    API --> SOLR[(Solr)]

    CLI --> BL[.bl Directory]
    BL --> OBJ[Objects]
    BL --> META[Metadata]
    BL --> REFS[References]

    META --> JSONLD[JSON-LD]
    META --> SHACL[SHACL Validation]
    META --> OWL[OWL Ontology]

Use Cases

  • 📚 Research Data Management: Version control for datasets with provenance tracking
  • 🏢 Enterprise Data Lakes: Git-like workflows for data engineering
  • 📊 Data Catalogs: Semantic metadata for data discovery
  • 🔬 Scientific Computing: Reproducible data workflows
  • 📈 Analytics: Versioned datasets with metadata search

Getting Started

  1. Installation - Get Blacklake running in minutes
  2. Quickstart - 5-minute tutorial with real examples
  3. Configuration - Set up S3, database, and search
  4. CLI Reference - Complete command documentation
  5. API Reference - REST API with OpenAPI docs

Community

License

Blacklake is licensed under the MIT License.