Skip to content

Streaming Lakehouse

A continuous pipeline reads events from a Kafka topic, processes them with Flink or Spark Structured Streaming, and writes to an open table format with sub-minute commit latency. Combines the freshness of streaming with the query semantics of a lakehouse.

When to use

  • Latency requirement is 1–5 minutes (or sub-minute with Flink)
  • Source data is already in Kafka (events, CDC output, IoT telemetry)
  • Reprocessing must replay the Kafka log from a given offset — not re-extract from source
  • Exactly-once semantics are required (Flink variant) or micro-batch is acceptable (Spark variant)

When NOT to use

  • Latency > 15 minutes — use Batch Lakehouse (simpler, cheaper)
  • Source is a live database and you haven't set up Kafka — use CDC Pipeline first, then this pattern downstream
  • Team has no Kafka or stream processing experience — operational complexity is high

Variants

Variant Stack Key idiom Latency floor Use when
flink-iceberg Flink 1.19 + Iceberg 1.5 + Kafka FlinkSink with two-phase commit on checkpoint ~5s (checkpoint interval) Exactly-once, event-time windowing, sub-minute
spark-delta Spark 3.5 + Delta 3.1 + Kafka readStream + foreachBatch with Delta MERGE ~30–60s (trigger interval) Team knows Spark, latency > 30s acceptable