Architecture Overview¶

Blacklake's architecture is designed for scalability, interoperability, and semantic data management.

System Components¶

graph TB
    subgraph "Client Layer"
        CLI[CLI Interface]
        API[REST API]
        WEB[Web Interface]
    end

    subgraph "Application Layer"
        SERVER[FastAPI Server]
        METADATA[Metadata Manager]
        VALIDATION[SHACL Validator]
    end

    subgraph "Storage Layer"
        S3[S3/Object Store]
        PG[(PostgreSQL)]
        SOLR[(Solr)]
    end

    subgraph "Repository Layer"
        BL[.bl Directory]
        OBJ[Objects]
        REFS[References]
        META[Metadata]
    end

    CLI --> SERVER
    API --> SERVER
    WEB --> SERVER

    SERVER --> METADATA
    SERVER --> VALIDATION
    SERVER --> S3
    SERVER --> PG
    SERVER --> SOLR

    METADATA --> BL
    VALIDATION --> BL

    BL --> OBJ
    BL --> REFS
    BL --> META

Data Flow¶

1. Object Storage Flow¶

sequenceDiagram
    participant C as CLI
    participant S as Server
    participant M as Metadata
    participant S3 as S3 Storage
    participant P as PostgreSQL
    participant Solr as Solr

    C->>S: Add object with metadata
    S->>M: Create metadata JSON-LD
    M->>M: Validate with SHACL
    S->>S3: Store object content
    S->>P: Store metadata
    S->>Solr: Index for search
    S->>C: Return success

2. Search Flow¶

sequenceDiagram
    participant C as CLI
    participant S as Server
    participant P as PostgreSQL
    participant Solr as Solr

    C->>S: Search query
    S->>Solr: Query index
    Solr->>S: Return results
    S->>P: Get metadata
    P->>S: Return metadata
    S->>C: Return results with metadata

Repository Structure¶

.bl Directory Layout¶

.bl/
├── HEAD                    # Current branch reference
├── config.yml             # Repository configuration
├── refs/
│   ├── heads/            # Branch references
│   └── remotes/          # Remote references
├── objects/              # Content-addressable objects
│   └── <sha256>.jsonld   # Object metadata
├── commits/              # Commit metadata
│   └── <commit-id>.jsonld
├── contexts/             # JSON-LD contexts
│   └── blacklake.context.jsonld
├── ontologies/           # OWL ontologies
│   ├── blacklake.owl.ttl
│   └── object.shape.ttl
└── agents/               # FOAF agents
    └── people.jsonld

Object Model¶

Content-Addressable Storage¶

graph LR
    CONTENT[File Content] --> HASH[SHA-256 Hash]
    HASH --> OBJ[Object Store]
    HASH --> META[Metadata]
    META --> JSONLD[JSON-LD]
    JSONLD --> VALIDATE[SHACL Validation]
    VALIDATE --> STORE[Store in PostgreSQL]

Object Types¶

Metadata Architecture¶

JSON-LD Processing¶

graph TD
    INPUT[Raw Metadata] --> CONTEXT[Apply Context]
    CONTEXT --> EXPAND[Expand JSON-LD]
    EXPAND --> VALIDATE[SHACL Validation]
    VALIDATE --> COMPACT[Compact JSON-LD]
    COMPACT --> STORE[Store in Database]
    STORE --> INDEX[Index in Solr]

Semantic Standards¶

Database Schema¶

Core Tables¶

-- Repositories
CREATE TABLE repositories (
  id SERIAL PRIMARY KEY,
  name VARCHAR(255) NOT NULL,
  description TEXT,
  created_at TIMESTAMP DEFAULT NOW()
);

-- Objects
CREATE TABLE bl_objects (
  id TEXT PRIMARY KEY,
  sha256 TEXT NOT NULL,
  mime TEXT,
  size BIGINT,
  title TEXT,
  description TEXT,
  creator_urn TEXT,
  created TIMESTAMPTZ,
  jsonld JSONB NOT NULL
);

-- Commits
CREATE TABLE bl_commits (
  id TEXT PRIMARY KEY,
  title TEXT,
  created TIMESTAMPTZ,
  creator_urn TEXT,
  jsonld JSONB NOT NULL
);

-- Agents
CREATE TABLE bl_agents (
  id TEXT PRIMARY KEY,
  type TEXT NOT NULL,
  name TEXT NOT NULL,
  email TEXT,
  homepage TEXT,
  jsonld JSONB NOT NULL,
  created TIMESTAMPTZ NOT NULL
);

Indexes¶

-- GIN indexes for JSONB
CREATE INDEX bl_objects_jsonld_gin ON bl_objects USING gin (jsonld);
CREATE INDEX bl_commits_jsonld_gin ON bl_commits USING gin (jsonld);
CREATE INDEX bl_agents_jsonld_gin ON bl_agents USING gin (jsonld);

-- B-tree indexes for common queries
CREATE INDEX bl_objects_sha256_idx ON bl_objects (sha256);
CREATE INDEX bl_objects_creator_idx ON bl_objects (creator_urn);
CREATE INDEX bl_objects_created_idx ON bl_objects (created);

Search Architecture¶

Solr Schema¶

<field name="id" type="string" indexed="true" stored="true"/>
<field name="title_t" type="text_general" indexed="true" stored="true"/>
<field name="description_t" type="text_general" indexed="true" stored="true"/>
<field name="creator_s" type="string" indexed="true" stored="true"/>
<field name="keyword_ss" type="string" indexed="true" stored="true" multiValued="true"/>
<field name="created_dt" type="date" indexed="true" stored="true"/>
<field name="mime_s" type="string" indexed="true" stored="true"/>
<field name="size_l" type="long" indexed="true" stored="true"/>

Search Flow¶

graph TD
    QUERY[Search Query] --> PARSE[Parse Query]
    PARSE --> SOLR[Query Solr]
    SOLR --> RESULTS[Search Results]
    RESULTS --> METADATA[Get Metadata]
    METADATA --> FORMAT[Format Response]
    FORMAT --> RETURN[Return Results]

Security Architecture¶

Authentication¶

graph TD
    REQUEST[API Request] --> AUTH{Authenticated?}
    AUTH -->|No| JWT[Generate JWT]
    AUTH -->|Yes| VALIDATE[Validate Token]
    JWT --> VALIDATE
    VALIDATE --> ALLOW[Allow Request]
    VALIDATE --> DENY[Deny Request]

Data Protection¶

Scalability Considerations¶

Horizontal Scaling¶

graph TB
    subgraph "Load Balancer"
        LB[Load Balancer]
    end

    subgraph "API Servers"
        API1[API Server 1]
        API2[API Server 2]
        API3[API Server 3]
    end

    subgraph "Database Cluster"
        PG1[PostgreSQL Primary]
        PG2[PostgreSQL Replica]
    end

    subgraph "Search Cluster"
        SOLR1[Solr Node 1]
        SOLR2[Solr Node 2]
    end

    LB --> API1
    LB --> API2
    LB --> API3

    API1 --> PG1
    API2 --> PG1
    API3 --> PG1

    API1 --> SOLR1
    API2 --> SOLR2
    API3 --> SOLR1

Performance Optimization¶

Monitoring and Observability¶

Health Checks¶

graph TD
    HEALTH[Health Check] --> S3[S3 Connection]
    HEALTH --> PG[Database Connection]
    HEALTH --> SOLR[Solr Connection]

    S3 --> STATUS[Overall Status]
    PG --> STATUS
    SOLR --> STATUS

Metrics¶

Next Steps¶

• DAG - Commit graph and object relationships
• Server API - REST API documentation
• Operations - Deployment and operations