Skip to content

Ingest Dataset Tutorial

Complete walkthrough for ingesting a dataset with metadata, validation, indexing, and search.

Overview

This tutorial demonstrates the complete workflow for ingesting a dataset into Blacklake:

  1. Prepare Dataset - Create sample data
  2. Add Metadata - Describe the dataset semantically
  3. Validate Metadata - Ensure quality with SHACL
  4. Commit Changes - Version the dataset
  5. Index for Search - Make it discoverable
  6. Search Dataset - Find it later

Step 1: Prepare Dataset

Create a sample dataset:

# Create sample sales data
cat > sales-2024.csv << EOF
date,product,revenue,units
2024-01-15,Laptop,1500.00,1
2024-01-16,Mouse,25.00,2
2024-01-17,Keyboard,75.00,1
2024-01-18,Monitor,300.00,1
2024-01-19,Headphones,150.00,1
EOF

Step 2: Initialize Repository

# Initialize Blacklake repository
blacklake init \
  --s3-endpoint http://localhost:9000 \
  --s3-bucket blacklake-data \
  --s3-access-key minioadmin \
  --s3-secret-key minioadmin

# Initialize metadata system
blacklake meta init

Expected Output: 

✓ Initialized empty blacklake repository in .bl/
✓ Metadata system initialized

Step 3: Add Metadata

Add comprehensive metadata to the dataset:

blacklake meta add sales-2024.csv \
  --title "Sales Data 2024" \
  --description "Daily sales data for Q1 2024 including product, revenue, and units sold" \
  --creator "Sales Team" \
  --email "sales@company.com" \
  --keywords "sales,revenue,products,2024,q1" \
  --rights "Internal Use Only" \
  --license "https://company.com/license"

Expected Output: 

✓ Metadata added for sales-2024.csv

Step 4: Validate Metadata

Ensure metadata quality:

# Validate the metadata
blacklake meta validate sales-2024.csv.meta.jsonld

Expected Output: 

✓ Metadata is valid

Step 5: Stage and Commit

# Add file to staging
blacklake add sales-2024.csv

# Check status
blacklake status

Expected Output: 

On branch main
Changes to be committed:
  sales-2024.csv (new file)

Untracked files:
  (none)


# Commit changes
blacklake commit -m "Add Q1 2024 sales data with metadata"

Expected Output: 

✓ Changes committed

Make the dataset discoverable:

# Index the dataset
blacklake index sales-2024.csv

Expected Output: 

Indexing: sales-2024.csv
✓ Data indexed

Step 7: Search Dataset

Find the dataset using search:

# Search for sales data
blacklake search "sales revenue"

# Search by year
blacklake search "2024"

# Search by product
blacklake search "laptop mouse"

Expected Output: 

Searching for: sales revenue
Found 1 result:
- sales-2024.csv: Sales Data 2024 (Daily sales data for Q1 2024 including product, revenue, and units sold)

Step 8: View Metadata

Inspect the metadata:

# View metadata as JSON-LD
blacklake meta export --format jsonld

Expected Output: 

{
  "@context": "../contexts/blacklake.context.jsonld",
  "@graph": [
    {
      "id": "bl:obj:sha256:abc123...",
      "type": ["Object", "dcterms:Dataset"],
      "title": "Sales Data 2024",
      "description": "Daily sales data for Q1 2024 including product, revenue, and units sold",
      "creator": {
        "id": "urn:uuid:...",
        "type": "Person",
        "name": "Sales Team",
        "mbox": "mailto:sales@company.com"
      },
      "subject": ["sales", "revenue", "products", "2024", "q1"],
      "rights": "Internal Use Only",
      "license": {
        "id": "https://company.com/license"
      }
    }
  ]
}

Advanced Workflow

Multiple Datasets

# Add multiple datasets
for dataset in sales-*.csv; do
  blacklake meta add "$dataset" \
    --title "$(basename "$dataset" .csv)" \
    --creator "Sales Team" \
    --email "sales@company.com"
  blacklake add "$dataset"
done

blacklake commit -m "Add multiple sales datasets"

Branching Workflow

# Create feature branch
blacklake branch feature/new-analysis

# Switch to branch
blacklake checkout feature/new-analysis

# Add analysis results
blacklake meta add analysis-results.csv \
  --title "Sales Analysis Results" \
  --creator "Analytics Team"

blacklake add analysis-results.csv
blacklake commit -m "Add sales analysis results"

# Merge back to main
blacklake checkout main
blacklake merge feature/new-analysis

Validation Workflow

# Validate all metadata
blacklake meta validate --all

# Expected output
✓ Validated 5 files, 0 errors

Troubleshooting

Common Issues

Metadata Validation Fails 

# Check specific validation errors
blacklake meta validate sales-2024.csv.meta.jsonld --format json

Search Returns No Results 

# Re-index the data
blacklake index sales-2024.csv --force

# Check Solr connection
curl http://localhost:8983/solr/blacklake/admin/ping

Commit Fails 

# Check repository status
blacklake status

# Check S3 connection
curl http://localhost:9000/minio/health/live

Next Steps