Streaming Systems The What, Where, When, and How of Large-Scale Data Processing

Streaming Systems provides a definitive guide to modern large-scale data processing, arguing that well-designed streaming systems are a strict superset of batch systems. The authors, from Google's core data processing teams, demystify the complexities of handling unbounded, out-of-order data by introducing the Beam Model. This model breaks down data processing into four fundamental questions: What results are calculated, Where in event time are they calculated, When in processing time are they materialized, and How do refinements of results relate. Through detailed explanations of event time vs. processing time, windowing, watermarks, and triggers, the book equips engineers and data scientists with the conceptual tools to build systems that balance correctness, latency, and cost. It further grounds these concepts in the practical theory of streams and tables, explores the mechanics of exactly-once processing, and charts the historical evolution of data processing systems, making it an essential resource for anyone serious about big data.

What it argues

This model describes how specific pipeline design choices, corresponding to the 'What, Where, When, and How' of data processing, influence the internal state and temporal processing behavior of a streaming system to achieve desired outcomes related to correctness, latency, and cost for unbounded, out-of-order data.

Key ideas it contributes

• Transformation Logic (What) — The specific business logic or computation applied to the data, such as aggregations, filtering, or element-wise transformations. This corresponds to the 'What' question in the Beam model.
• Windowing Strategy (Where) — The method of partitioning a dataset, typically unbounded, into finite chunks along temporal boundaries in the event-time domain. This corresponds to the 'Where' question in the Beam model.
• Triggering Strategy (When) — The declarative mechanism that specifies when the output for a window should be materialized in processing time, relative to signals like data arrival, watermark progression, or processing-time passage. This corresponds to the 'When' question.
• Accumulation Mode (How) — The specification defining the relationship between multiple results (panes) for the same window when it is materialized multiple times. This corresponds to the 'How' question.
• Allowed Lateness — A configuration that defines a horizon on the allowed lateness of data relative to the watermark, bounding how late a record may be for the system to process it and, consequently, how long state for a window must be retained for potential updates.
• Consistency Mechanism — The underlying system's mechanism for guaranteeing correctness in the face of failures, such as exactly-once processing, which is achieved through techniques like consistent checkpointing, snapshots, and duplicate detection.
• Event-Time Completeness Perception (Watermark) — The system's metric of progress in the event-time domain, representing the point in event time up to which all input data are believed to have been observed. This metric allows the system to reason about the completeness of event-time windows.
• State Management Behavior — The internal process by which the system buffers, aggregates, and persists intermediate results for each key/window grouping. This represents the 'data at rest' in the stream-table duality.