Full-text search for streaming DuckDB data

BoilStream indexes selected text columns with Tantivy as rows enter a streaming DuckLake table. Query relevance-ranked results from SQL across recent local segments and durable bundles registered in DuckLake.

Read the search guide Try the Quick Start

Can DuckDB do full-text search? DuckDB has an fts extension for indexes inside a DuckDB database. BoilStream serves a different streaming use case: it indexes new rows during ingestion, maintains hot Tantivy segments, stores cold bundles in S3-backed DuckLake tables, and exposes the results through a SQL table function.

Search without a separate indexing pipeline

A conventional analytics-and-search architecture often copies the same events into a columnar store and a separate search cluster. BoilStream can route one incoming batch to both Parquet persistence and a Tantivy index, keeping search configuration with the streaming table.

Enable indexing only for the fields that need tokenized search. Other supported scalar columns are indexed for exact matches or filters, and complex nested or binary values are skipped.

Hot search tier

Sharded local writers index incoming rows and commit segments on a configurable interval. The default commit interval is 30 seconds.

Cold search tier

Committed Tantivy segments are packed into bundle files, uploaded to object storage, and registered in a shadow DuckLake table.

SQL query surface

The multilake_search() table function returns source columns plus a relevance _score for matching rows.

Enable search on a streaming table

Full-text search is available for tables in __stream DuckLake catalogs. The text-field setting selects the string columns that Tantivy tokenizes for full-text queries.

ALTER TABLE knowledge__stream.main.documents SET (
  tantivy_enabled = true,
  tantivy_text_fields = 'title,body'
);

BoilStream derives the Tantivy schema, creates a shadow table named documents__tantivy_idx, and indexes subsequent inserts. Changing the text-field list does not automatically re-index existing rows.

Query full-text results from SQL

-- Search every configured text field and return ten results
SELECT id, title, body, _score
FROM multilake_search(
  'knowledge__stream',
  'documents__tantivy_idx',
  'distributed systems',
  10
)
ORDER BY _score DESC;

-- Restrict the query to one field
SELECT id, title, _score
FROM multilake_search(
  'knowledge__stream',
  'documents__tantivy_idx',
  'title:Rust'
);

Input column	Tantivy mapping	Search behavior
Configured UTF-8 text field	TEXT	Tokenized full-text search
Other UTF-8 field	STRING	Exact-match indexing
Integer or float	Numeric field	Indexed numeric queries and filters
Timestamp	Date field	Indexed date-range queries
Boolean	Boolean field	Indexed boolean filter
Binary, list, or struct	Not indexed	Skipped by the Tantivy mapping

Where integrated streaming search helps

Logs and events

Search error messages, traces, and event payload text while keeping analytical columns in the same ingestion flow.

Knowledge and support

Index documents, articles, or support tickets as they arrive and retrieve relevance-ranked context for people or AI agents.

Product catalogs

Search titles and descriptions while retaining numeric, date, and boolean fields for structured filtering.

When should I still use a dedicated search service? Keep a dedicated service when you need its mature ecosystem, specialized analyzers, complex relevance tooling, or independent search scaling. BoilStream is strongest when the same continuously ingested data needs SQL analytics, DuckLake retention, and integrated full-text retrieval.

DuckDB and Tantivy search FAQ

Does BoilStream require Elasticsearch?

No. Tantivy indexing, hot segment management, cold bundle storage, and SQL query access are built into the BoilStream data path.

How quickly do new rows become searchable?

Rows become searchable after a Tantivy commit. The default commit interval is 30 seconds and can be configured.

Can I use Tantivy without writing Parquet?

Yes. Tantivy-only mode disables Parquet persistence while keeping hot indexes and durable cold Tantivy bundles. It is intended for search-first workloads that do not need columnar analytics.

Where are cold indexes stored?

Committed segments are packed into bundle files in object storage and registered as Tantivy-format files in a shadow DuckLake table.