Ingest
Use standard Kafka producers, Arrow FlightRPC, HTTP/2 Arrow payloads, or PostgreSQL COPY. Confluent Schema Registry-compatible Avro is supported on the Kafka path.
Streaming ingestion lakehouse
Stream data into DuckLake without a batch staging layer
BoilStream accepts Kafka-protocol, Arrow FlightRPC, HTTP/2 Arrow, and PostgreSQL COPY traffic, makes rows queryable in a hot DuckDB tier, and writes durable Parquet snapshots into DuckLake-managed object storage.
What is a streaming DuckLake?
It is a DuckLake catalog whose tables receive continuous events. BoilStream keeps recent rows in a hot tier for low-latency SQL while concurrently finalizing older data as Parquet in object storage. The client queries one logical table across both tiers.
One data path from producer to lakehouse
Traditional streaming-to-lakehouse stacks often assemble a broker, a stream processor, object-storage writers, compaction jobs, and a separate query engine. BoilStream combines the ingestion, streaming SQL, hot visibility, and DuckLake write path in one deployable system.
The storage stays open: bulk data is Apache Parquet and the catalog follows the DuckLake specification. Remote DuckDB clients can connect through the BoilStream community extension and receive temporary credentials for the selected catalog.
Query recent rows
Incoming batches become visible in the hot DuckDB tier on a configurable commit interval, so dashboards and agents do not wait for Parquet finalization.
Retain durable history
Completed data is written as Parquet to S3, Azure Blob, GCS, MinIO, or filesystem storage and registered with the DuckLake catalog.
How the streaming and lakehouse layers fit together
| Stage | BoilStream component | Result |
|---|---|---|
| Producer connection | Kafka wire protocol, FlightRPC, HTTP/2 Arrow, or PGWire | Existing clients can send typed event batches. |
| Immediate visibility | Per-topic hot DuckDB tier | Recent data is available to SQL queries before cold files finalize. |
| Continuous processing | Streaming views and windowed materialized views | Filters, transforms, and aggregates create derived topics. |
| Durability | Parquet files plus DuckLake catalog metadata | Open-format history lives in customer-controlled object storage. |
| Remote access | PostgreSQL, FlightSQL, Airport, and the DuckDB BoilStream extension | BI tools, services, and DuckDB clients query the same logical data. |
Create and query a streaming DuckLake
Streaming catalogs use the __stream suffix. Once connected through the BoilStream DuckDB extension, you can create a catalog, use it, and query its tables with standard SQL.
INSTALL boilstream FROM community;
LOAD boilstream;
PRAGMA boilstream_login(
'https://your-server.example/user@example.com',
'password',
'123456'
);
PRAGMA boilstream_create_ducklake(
'events__stream',
'Real-time event catalog'
);
USE events__stream;
SELECT * FROM web_events
ORDER BY event_time DESC
LIMIT 100;For deployment prerequisites and complete connection examples, use the BoilStream Quick Start.
When this architecture is a good fit
- You want DuckDB-compatible analytics over continuously arriving events.
- You need recent data quickly but want durable history in open Parquet files.
- You already have Kafka producers and want to simplify the downstream processing path.
- You want SQL transformations, windowed aggregates, and full-text search in the same system.
When to keep a general-purpose event broker
BoilStream is optimized for the ingestion-to-analytics path. Keep a general-purpose broker when your architecture depends on a broad connector ecosystem, long broker-side retention as the system of record, or application messaging semantics outside BoilStream's supported protocols and processing model.
Streaming DuckLake FAQ
Does DuckLake support streaming data?
DuckLake supports data inlining for small changes. BoilStream adds the continuous ingestion, hot query tier, Parquet finalization, and catalog coordination needed to operate a streaming data path around DuckLake.
Where does BoilStream store the data?
Recent data is held in the hot DuckDB tier. Durable cold data is stored as Parquet in a configured S3-compatible, Azure, GCS, MinIO, or filesystem backend and registered in DuckLake.
Can existing Kafka clients produce to BoilStream?
Yes. BoilStream implements the Kafka wire protocol for producers and supports Confluent Schema Registry-compatible Avro on that ingestion path.
How do clients query the data?
Clients can use PostgreSQL-compatible SQL, FlightSQL, Airport, or the BoilStream DuckDB community extension, depending on the deployment and client.