SHACL¶

Shape validation for Blacklake metadata using SHACL (Shapes Constraint Language).

Overview¶

SHACL (Shapes Constraint Language) provides a way to validate RDF data against defined shapes, ensuring metadata quality and consistency.

SHACL Shapes¶

Object Shape¶

@prefix sh: <http://www.w3.org/ns/shacl#> .
@prefix dct: <http://purl.org/dc/terms/> .
@prefix bl: <https://sempervent.github.io/blacklake/ns#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

bl:ObjectShape a sh:NodeShape ;
  sh:targetClass bl:Object ;
  sh:property [
    sh:path dct:title ;
    sh:minCount 1 ;
    sh:datatype xsd:string ;
    sh:message "Object must have a title"
  ] ;
  sh:property [
    sh:path dct:format ;
    sh:minCount 1 ;
    sh:message "Object must have a MIME type"
  ] ;
  sh:property [
    sh:path bl:checksum ;
    sh:minCount 1 ;
    sh:datatype xsd:string ;
    sh:message "Object must have a checksum"
  ] ;
  sh:property [
    sh:path bl:algo ;
    sh:minCount 1 ;
    sh:datatype xsd:string ;
    sh:message "Object must specify hash algorithm"
  ] ;
  sh:property [
    sh:path dct:creator ;
    sh:minCount 1 ;
    sh:message "Object must have a creator"
  ] ;
  sh:property [
    sh:path dct:created ;
    sh:minCount 1 ;
    sh:datatype xsd:dateTime ;
    sh:message "Object must have a creation date"
  ] ;
  sh:property [
    sh:path dct:extent ;
    sh:datatype xsd:integer ;
    sh:message "Object size must be an integer"
  ] .

Commit Shape¶

bl:CommitShape a sh:NodeShape ;
  sh:targetClass bl:Commit ;
  sh:property [
    sh:path dct:title ;
    sh:minCount 1 ;
    sh:datatype xsd:string ;
    sh:message "Commit must have a title"
  ] ;
  sh:property [
    sh:path dct:creator ;
    sh:minCount 1 ;
    sh:message "Commit must have a creator"
  ] ;
  sh:property [
    sh:path dct:created ;
    sh:minCount 1 ;
    sh:datatype xsd:dateTime ;
    sh:message "Commit must have a creation date"
  ] .

Person Shape¶

bl:PersonShape a sh:NodeShape ;
  sh:targetClass foaf:Person ;
  sh:property [
    sh:path foaf:name ;
    sh:minCount 1 ;
    sh:datatype xsd:string ;
    sh:message "Person must have a name"
  ] ;
  sh:property [
    sh:path foaf:mbox ;
    sh:datatype xsd:anyURI ;
    sh:message "Person email must be a valid URI"
  ] .

Validation Process¶

1. Load Metadata¶

import json
from rdflib import Graph
from pyshacl import validate

# Load metadata
with open("metadata.jsonld", "r") as f:
    metadata = json.load(f)

# Convert to RDF
data_graph = Graph()
data_graph.parse(data=json.dumps(metadata), format="json-ld")

2. Load SHACL Shapes¶

# Load SHACL shapes
shacl_graph = Graph()
shacl_graph.parse("object.shape.ttl", format="turtle")

3. Validate¶

# Validate metadata
conforms, results_graph, results_text = validate(
    data_graph=data_graph,
    shacl_graph=shacl_graph,
    ont_graph=None,
    inference='rdfs',
    abort_on_first=False,
    allow_warnings=False,
    meta_shacl=False,
    debug=False,
    advanced=False
)

if conforms:
    print("✓ Metadata is valid")
else:
    print(f"✗ Validation errors: {results_text}")

Validation Examples¶

Valid Object Metadata¶

{
  "@context": "../contexts/blacklake.context.jsonld",
  "id": "bl:obj:sha256:abc123...",
  "type": "Object",
  "title": "Sales Data 2024",
  "format": "text/csv",
  "checksum": "abc123...",
  "algo": "sha256",
  "created": "2024-01-15T10:30:00Z",
  "creator": {
    "type": "Person",
    "name": "Data Team",
    "mbox": "mailto:data@company.com"
  }
}

Validation Result:

✓ Metadata is valid

Invalid Object Metadata¶

{
  "@context": "../contexts/blacklake.context.jsonld",
  "id": "bl:obj:sha256:abc123...",
  "type": "Object",
  "format": "text/csv",
  "checksum": "abc123...",
  "algo": "sha256",
  "created": "2024-01-15T10:30:00Z"
}

Validation Result:

✗ Validation errors:
Constraint Violation in bl:ObjectShape (http://www.w3.org/ns/shacl#Violation):
  Severity: sh:Violation
  Source Shape: bl:ObjectShape
  Focus Node: bl:obj:sha256:abc123...
  Result Path: dct:title
  Message: Object must have a title
  Value: [None]

Constraint Violation in bl:ObjectShape (http://www.w3.org/ns/shacl#Violation):
  Severity: sh:Violation
  Source Shape: bl:ObjectShape
  Focus Node: bl:obj:sha256:abc123...
  Result Path: dct:creator
  Message: Object must have a creator
  Value: [None]

Advanced Validation¶

Custom Constraints¶

bl:CustomObjectShape a sh:NodeShape ;
  sh:targetClass bl:Object ;
  sh:property [
    sh:path dct:title ;
    sh:minLength 5 ;
    sh:maxLength 100 ;
    sh:message "Title must be between 5 and 100 characters"
  ] ;
  sh:property [
    sh:path dct:created ;
    sh:datatype xsd:dateTime ;
    sh:minInclusive "2020-01-01T00:00:00Z"^^xsd:dateTime ;
    sh:message "Creation date must be after 2020"
  ] ;
  sh:property [
    sh:path dct:extent ;
    sh:datatype xsd:integer ;
    sh:minInclusive 0 ;
    sh:message "Size must be non-negative"
  ] .

Conditional Validation¶

bl:ConditionalShape a sh:NodeShape ;
  sh:targetClass bl:Object ;
  sh:property [
    sh:path dct:rights ;
    sh:hasValue "CC BY 4.0" ;
    sh:message "Rights must be CC BY 4.0"
  ] ;
  sh:property [
    sh:path dct:license ;
    sh:minCount 1 ;
    sh:message "License must be specified when rights is CC BY 4.0"
  ] .

Validation Workflow¶

CLI Validation¶

# Validate specific file
blacklake meta validate metadata.jsonld

# Validate all metadata
blacklake meta validate --all

# Validate with custom shapes
blacklake meta validate --shapes custom.shapes.ttl metadata.jsonld

Programmatic Validation¶

from blacklake.metadata import MetadataManager

# Initialize metadata manager
metadata_manager = MetadataManager(repo_path)

# Validate metadata
conforms, message = metadata_manager.validate_metadata(metadata, "object")

if conforms:
    print("✓ Metadata is valid")
else:
    print(f"✗ Validation errors: {message}")

Error Handling¶

Common Validation Errors¶

Error	Cause	Solution
Missing title	No `dct:title` property	Add title field
Invalid date	Malformed date format	Use ISO 8601 format
Missing creator	No `dct:creator` property	Add creator information
Invalid email	Malformed email URI	Use `mailto:` prefix
Missing checksum	No `bl:checksum` property	Add SHA-256 hash

Error Reporting¶

def validate_metadata_detailed(metadata):
    """Validate metadata with detailed error reporting."""
    conforms, results_graph, results_text = validate(
        data_graph=metadata_graph,
        shacl_graph=shacl_shapes
    )

    if not conforms:
        errors = []
        for result in results_graph:
            if result[1] == sh.Violation:
                errors.append({
                    "path": str(result[2]),
                    "message": str(result[3]),
                    "value": str(result[4])
                })
        return False, errors

    return True, []

Best Practices¶

Shape Design¶

Start Simple: Begin with basic required fields
Add Constraints: Gradually add validation rules
Test Thoroughly: Validate with real data
Document Rules: Explain validation requirements

Validation Strategy¶

Early Validation: Validate on input
Batch Validation: Validate collections
Continuous Validation: Regular validation checks
Error Recovery: Graceful error handling

Integration¶

Pre-commit Hooks¶

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: metadata-validation
        name: Validate metadata
        entry: blacklake meta validate --all
        language: system
        files: \.jsonld$

CI/CD Integration¶

# .github/workflows/validation.yml
name: Metadata Validation
on: [push, pull_request]
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Validate metadata
        run: blacklake meta validate --all

Troubleshooting¶

Common Issues¶

SHACL Shapes Not Found

# Check shape file location
ls -la .bl/ontologies/object.shape.ttl

# Verify shape file format
blacklake meta validate --debug metadata.jsonld

Validation Timeout

# Increase timeout
export BLACKLAKE_VALIDATION_TIMEOUT=300
blacklake meta validate --all

Memory Issues

# Validate in batches
blacklake meta validate --batch-size 100 --all

Next Steps¶

Metadata Model - Understanding the metadata model
JSON-LD - JSON-LD processing
Ontology - OWL definitions