Metadata Model¶

Blacklake uses a comprehensive metadata model based on web standards for semantic interoperability.

Overview¶

The metadata system is built on four key standards:

Dublin Core - Core metadata properties
FOAF - People and organizations
OWL - Ontology definitions
JSON-LD - Web-friendly serialization

Dublin Core Fields¶

Core Properties¶

Field	Dublin Core Term	Type	Required	Description
`title`	`dcterms:title`	String	Yes	Human-readable title
`description`	`dcterms:description`	String	No	Detailed description
`creator`	`dcterms:creator`	Agent	Yes	Creator of the resource
`created`	`dcterms:created`	DateTime	Yes	Creation date
`modified`	`dcterms:modified`	DateTime	No	Last modification date
`format`	`dcterms:format`	String	Yes	MIME type
`size`	`dcterms:extent`	Integer	No	Size in bytes
`rights`	`dcterms:rights`	String	No	Rights statement
`license`	`dcterms:license`	URI	No	License URL
`subject`	`dcterms:subject`	String[]	No	Keywords/tags
`relation`	`dcterms:relation`	URI[]	No	Related resources

Example Object Metadata¶

{
  "@context": "../contexts/blacklake.context.jsonld",
  "id": "bl:obj:sha256:abc123...",
  "type": ["Object", "dcterms:Dataset"],
  "title": "Sales Data 2024",
  "description": "Q1-Q4 sales data for North America",
  "format": "text/csv",
  "size": 1024000,
  "created": "2024-01-15T10:30:00Z",
  "creator": {
    "id": "urn:uuid:550e8400-e29b-41d4-a716-446655440000",
    "type": "Person",
    "name": "Data Team",
    "mbox": "mailto:data@company.com"
  },
  "rights": "CC BY 4.0",
  "license": {
    "id": "https://creativecommons.org/licenses/by/4.0/"
  },
  "subject": ["sales", "revenue", "2024", "north-america"],
  "relation": ["bl:commit:def456..."]
}

FOAF Agents¶

Person¶

{
  "id": "urn:uuid:550e8400-e29b-41d4-a716-446655440000",
  "type": "Person",
  "name": "John Doe",
  "mbox": "mailto:john@company.com",
  "homepage": "https://company.com/~john"
}

Organization¶

{
  "id": "urn:uuid:550e8400-e29b-41d4-a716-446655440001",
  "type": "Organization",
  "name": "Acme Corporation",
  "homepage": "https://acme.com"
}

OWL Classes and Properties¶

Classes¶

Class	Description	Superclass
`bl:Object`	Blacklake object	`dct:Dataset`
`bl:Commit`	Blacklake commit	-
`bl:Repository`	Blacklake repository	-

Properties¶

Property	Domain	Range	Description
`bl:checksum`	`bl:Object`	`xsd:string`	SHA-256 hash
`bl:algo`	`bl:Object`	`xsd:string`	Hash algorithm
`bl:author`	`bl:Object`	`foaf:Agent`	Object author
`bl:hasObject`	`bl:Repository`	`bl:Object`	Repository contains object
`bl:hasCommit`	`bl:Repository`	`bl:Commit`	Repository contains commit

Object Types¶

Blob Object¶

{
  "id": "bl:obj:sha256:abc123...",
  "type": "Object",
  "title": "Sales Data",
  "format": "text/csv",
  "size": 1024000,
  "checksum": "abc123...",
  "algo": "sha256",
  "created": "2024-01-15T10:30:00Z",
  "creator": {
    "id": "urn:uuid:...",
    "type": "Person",
    "name": "Data Team"
  }
}

Tree Object¶

{
  "id": "bl:tree:sha256:def456...",
  "type": "Object",
  "title": "Data Directory",
  "format": "application/x-blacklake-tree",
  "created": "2024-01-15T10:30:00Z",
  "creator": {
    "id": "urn:uuid:...",
    "type": "Person",
    "name": "Data Team"
  },
  "relation": [
    "bl:obj:sha256:abc123...",
    "bl:obj:sha256:ghi789..."
  ]
}

Commit Object¶

{
  "id": "bl:commit:sha256:jkl012...",
  "type": "Commit",
  "title": "Add sales data",
  "description": "Added Q1-Q4 sales data with metadata",
  "created": "2024-01-15T10:30:00Z",
  "creator": {
    "id": "urn:uuid:...",
    "type": "Person",
    "name": "Data Team"
  },
  "relation": [
    "bl:tree:sha256:def456...",
    "bl:commit:sha256:mno345..."
  ]
}

Metadata Validation¶

SHACL Shapes¶

@prefix sh: <http://www.w3.org/ns/shacl#> .
@prefix dct: <http://purl.org/dc/terms/> .
@prefix bl: <https://sempervent.github.io/blacklake/ns#> .

bl:ObjectShape a sh:NodeShape ;
  sh:targetClass bl:Object ;
  sh:property [
    sh:path dct:title ;
    sh:minCount 1 ;
    sh:datatype xsd:string
  ] ;
  sh:property [
    sh:path dct:creator ;
    sh:minCount 1
  ] ;
  sh:property [
    sh:path bl:checksum ;
    sh:minCount 1 ;
    sh:datatype xsd:string
  ] .

Validation Rules¶

Required Fields: title, creator, checksum, created
Data Types: String fields must be strings, dates must be ISO 8601
Agent Validation: Creators must be valid FOAF Person/Organization
Checksum Validation: SHA-256 hash must be 64 characters
Date Validation: Dates must be valid ISO 8601 format

Metadata Workflow¶

1. Create Object Metadata¶

# Add metadata to file
blacklake meta add data.csv \
  --title "Sales Data 2024" \
  --description "Q1-Q4 sales data" \
  --creator "Data Team" \
  --email "data@company.com" \
  --keywords "sales,revenue,2024"

2. Validate Metadata¶

# Validate specific file
blacklake meta validate data.csv.meta.jsonld

# Validate all metadata
blacklake meta validate --all

3. Export Metadata¶

# Export as JSON-LD
blacklake meta export --format jsonld

# Export as Turtle
blacklake meta export --format ttl

Best Practices¶

1. Consistent Naming¶

Use descriptive titles
Include version information
Use consistent keywords

2. Agent Management¶

Create reusable agent profiles
Use consistent email addresses
Include organizational information

3. Rights and Licensing¶

Always specify rights statements
Use standard license URIs
Include attribution information

4. Relationships¶

Link related objects
Use consistent relationship types
Maintain provenance chains

Examples¶

Scientific Dataset¶

{
  "id": "bl:obj:sha256:abc123...",
  "type": ["Object", "dcterms:Dataset"],
  "title": "Climate Data 2023",
  "description": "Daily temperature and precipitation data for North America",
  "format": "application/netcdf",
  "size": 52428800,
  "created": "2024-01-15T10:30:00Z",
  "creator": {
    "id": "urn:uuid:...",
    "type": "Person",
    "name": "Dr. Jane Smith",
    "mbox": "mailto:jane@university.edu"
  },
  "rights": "CC BY 4.0",
  "license": {
    "id": "https://creativecommons.org/licenses/by/4.0/"
  },
  "subject": ["climate", "temperature", "precipitation", "2023"],
  "relation": ["bl:commit:def456..."]
}

Business Dataset¶

{
  "id": "bl:obj:sha256:def456...",
  "type": ["Object", "dcterms:Dataset"],
  "title": "Customer Analytics 2024",
  "description": "Customer behavior and purchase data",
  "format": "application/parquet",
  "size": 104857600,
  "created": "2024-01-15T10:30:00Z",
  "creator": {
    "id": "urn:uuid:...",
    "type": "Organization",
    "name": "Analytics Team",
    "homepage": "https://company.com/analytics"
  },
  "rights": "Proprietary - Internal Use Only",
  "subject": ["customers", "analytics", "behavior", "2024"],
  "relation": ["bl:commit:ghi789..."]
}

Next Steps¶

JSON-LD - JSON-LD processing and examples
SHACL - Data validation with SHACL
Ontology - OWL ontology definitions