Streaming Architecture Patterns: SAGA, CQRS, and Outbox: Best Practices¶

Objective: Establish comprehensive streaming architecture patterns including SAGA for distributed transactions, CQRS for read/write separation, and Outbox for reliable event publishing. When you need distributed transactions, when you want read/write separation, when you need reliable events—this guide provides the complete framework.

Introduction¶

Streaming architecture patterns enable scalable, reliable distributed systems. Without proper patterns, systems suffer from consistency issues, performance bottlenecks, and reliability problems. This guide establishes patterns for SAGA, CQRS, and Outbox in streaming architectures.

What This Guide Covers: - SAGA pattern for distributed transactions - CQRS pattern for read/write separation - Outbox pattern for reliable event publishing - Event sourcing patterns - Stream processing architectures - Consistency models - Failure handling and compensation

Prerequisites: - Understanding of distributed systems - Familiarity with event-driven architecture - Experience with stream processing

Related Documents: This document integrates with: - Event-Driven Architecture - Event patterns - System Resilience, Rate Limiting, Concurrency Control & Backpressure - Resilience - Operational Risk Modeling, Blast Radius Reduction & Failure Domain Architecture - Risk patterns

The Philosophy of Streaming Patterns¶

Pattern Principles¶

Principle 1: Eventual Consistency - Accept eventual consistency - Design for availability - Compensate for failures

Principle 2: Event-Driven - Events as first-class citizens - Event sourcing - Event replay

Principle 3: Scalability - Read/write separation - Horizontal scaling - Stream processing

SAGA Pattern¶

SAGA Architecture¶

Diagram:

graph LR
    subgraph Orchestrator["SAGA Orchestrator"]
        Start["Start"]
        Step1["Step 1"]
        Step2["Step 2"]
        Step3["Step 3"]
        Compensate["Compensate"]
    end

    Start --> Step1
    Step1 --> Step2
    Step2 --> Step3
    Step3 -->|"Failure"| Compensate

    style Orchestrator fill:#fff4e1

CQRS Pattern¶

CQRS Architecture¶

Diagram:

graph TB
    subgraph Write["Write Side"]
        Command["Commands"]
        WriteDB["Write Database"]
    end

    subgraph Read["Read Side"]
        Query["Queries"]
        ReadDB["Read Database"]
    end

    subgraph Sync["Synchronization"]
        Events["Events"]
    end

    Command --> WriteDB
    WriteDB --> Events
    Events --> ReadDB
    Query --> ReadDB

    style Write fill:#e1f5ff
    style Read fill:#fff4e1
    style Sync fill:#e8f5e9

Outbox Pattern¶

Outbox Architecture¶

Pattern:

# Outbox pattern
class OutboxPattern:
    def process_with_outbox(self, transaction):
        """Process transaction with outbox"""
        with transaction:
            # Write to database
            self.write_to_database(transaction.data)

            # Write to outbox
            self.write_to_outbox(transaction.event)

        # Publish from outbox
        self.publish_from_outbox()

Architecture Fitness Functions¶

Streaming Pattern Fitness Function¶

Definition:

# Streaming pattern fitness function
class StreamingPatternFitnessFunction:
    def evaluate(self, system: System) -> float:
        """Evaluate streaming pattern quality"""
        # Check consistency
        consistency = self.check_consistency(system)

        # Check reliability
        reliability = self.check_reliability(system)

        # Check scalability
        scalability = self.check_scalability(system)

        # Calculate fitness
        fitness = (consistency * 0.4) + \
                  (reliability * 0.3) + \
                  (scalability * 0.3)

        return fitness