Skip to content

Solr Integration

Apache Solr search integration for Blacklake.

Overview

Blacklake integrates with Apache Solr to provide powerful full-text search capabilities for your data repositories.

Schema Fields

Core Fields

Field Type Description
id string Unique object identifier
title_t text_general Object title
description_t text_general Object description
creator_s string Creator name
rights_s string Rights statement
license_s string License URL
keyword_ss string Keywords (multi-valued)
created_dt date Creation date
mime_s string MIME type
size_l long Object size in bytes

Search Examples

# Search for sales data
blacklake search "sales data"

# Search by year
blacklake search "2024"

# Search by creator
blacklake search "Data Team"

# Search with filters
blacklake search "sales" --creator "Data Team" --year "2024"

# Search by MIME type
blacklake search "csv" --format "text/csv"

# Search by size range
blacklake search "large dataset" --size-min "1000000"

Indexing

Automatic Indexing

# Index single file
blacklake index data.csv

# Index directory
blacklake index --recursive data/

# Force re-indexing
blacklake index --force data.csv

Manual Indexing

from blacklake.solr import SolrClient

# Initialize Solr client
solr = SolrClient("http://localhost:8983/solr/blacklake")

# Index document
doc = {
    "id": "bl:obj:sha256:abc123...",
    "title_t": "Sales Data 2024",
    "description_t": "Q1-Q4 sales data",
    "creator_s": "Data Team",
    "created_dt": "2024-01-15T10:30:00Z"
}

solr.add(doc)
solr.commit()

Query Examples

Solr Query Syntax

# Field-specific search
blacklake search "title:sales AND creator:Data Team"

# Date range search
blacklake search "created:[2024-01-01 TO 2024-12-31]"

# Size range search
blacklake search "size:[1000000 TO *]"

# Wildcard search
blacklake search "title:*sales*"

Advanced Queries

# Boolean queries
blacklake search "(sales OR revenue) AND (2024 OR 2023)"

# Phrase search
blacklake search '"sales data"'

# Fuzzy search
blacklake search "sales~"

Configuration

Solr Configuration

# .bl/config.yml
solr:
  url: "http://localhost:8983/solr/blacklake"
  timeout: 30
  max_retries: 3
  batch_size: 100

Schema Configuration

<!-- solr/schema.xml -->
<field name="id" type="string" indexed="true" stored="true"/>
<field name="title_t" type="text_general" indexed="true" stored="true"/>
<field name="description_t" type="text_general" indexed="true" stored="true"/>
<field name="creator_s" type="string" indexed="true" stored="true"/>
<field name="keyword_ss" type="string" indexed="true" stored="true" multiValued="true"/>
<field name="created_dt" type="date" indexed="true" stored="true"/>
<field name="mime_s" type="string" indexed="true" stored="true"/>
<field name="size_l" type="long" indexed="true" stored="true"/>

Performance

Indexing Performance

  • Batch Processing: Index multiple documents at once
  • Async Processing: Non-blocking indexing
  • Incremental Updates: Only index changed content

Search Performance

  • Caching: Query result caching
  • Faceting: Fast faceted search
  • Highlighting: Search result highlighting

Troubleshooting

Common Issues

Search Returns No Results 

# Check Solr status
curl http://localhost:8983/solr/blacklake/admin/ping

# Re-index data
blacklake index --force --all

Slow Search Performance 

# Check index size
curl "http://localhost:8983/solr/blacklake/admin/mbeans?stats=true"

# Optimize index
curl "http://localhost:8983/solr/blacklake/update?optimize=true"

Index Corruption 

# Clear and rebuild index
blacklake index --clear --all

Next Steps