The ingestion and indexing side of a RAG pipeline
I built the ingestion and indexing half. The comments were sitting in the analytics store as raw data, so the first problem was designing a new structure for them: what becomes a document, and for every field on it, whether it needs to be a SimpleField, a SearchableField, sortable, filterable, or facetable. That schema is what everything downstream runs on. I built a batch process to transform the historical comments and index them, a daily process to keep indexing new ones as they came in, and shipped a trial version so clubs could actually try it before it opened up to everyone.
PROBLEM
Clubs collect thousands of free-text survey comments a year, and reading them all is a job nobody has time for. The product lets someone ask a question about that feedback in plain language, which only works if the right handful of comments can be found first.
DECISIONS I'D DEFEND
One club can never see another's comments
Tenant isolation starts in the schema, which was my half: the account ID is a filterable field on every document, so a search can be scoped to one club before anything is ranked or returned. That makes the isolation a property of the index rather than an instruction in the prompt, and since the model can't issue a query of its own, what reaches it can only ever be one club's documents.
The index stores the text, not just the IDs
Keeping the comment and question text in the index is what makes it a search index rather than a lookup table. Storing IDs alone would mean a second round trip to the source for every hit, and the ranking would have nothing to rank on.
WHAT I BUILT
- ▸The index schema and its semantic configuration. The rule I applied: searchable only where the text is genuinely searched, filterable where something scopes a query, facetable only where the UI offers a breakdown. Every capability costs index size, so you don't switch them all on.
- ▸The pipeline that pulls comments from the analytics store, enriches each one with its club and question context, deduplicates them, and pushes them into the index.
- ▸Batched uploads for the historical backfill: around a thousand documents per batch, three batches running concurrently. Enough to move a lot of data, not so much that the failures become hard to reason about.
- ▸Retry with exponential backoff, and rollback when it keeps failing: the partial documents are deleted from the index and the comment's indexed-at marker is left unset, so a later run picks it up again. A response is never left half-indexed.
- ▸A scheduled function keeping new comments searchable, plus the subscription and trial-quota layer and the APIs behind it.
WHAT WAS MINE
Mine: the index schema and semantic configuration, the ingestion and transform, the batched indexer with retry and rollback, the scheduled refresh, and the trial-quota layer. Not mine: the assistant itself, the prompts, the chat UI and the query endpoint.
STACK