Skip to content
Series
4 posts

Private Corpus Pipeline

How private data enters indexes safely and continuously, remains subject to query-time authorization, and stays consistent when sources change or disappear—focused on the data lifecycle rather than RAG retrieval techniques.

Private Corpus Search Boundaries: Decide Where Data May Go First

The first private-corpus decision is not which vector database to buy. Define data classes, trust zones, policy enforcement points, and freshness SLAs so every index, model, and observability system receives only the minimum data it is allowed to process.

From Source to Index: Sync and Incremental Updates for Private Corpora

Private-corpus sync is not periodic refetching. It requires stable canonical IDs, source versions plus checksums for change detection, idempotent upserts, and tombstones that propagate deletion through every index.

Securing Private-Corpus Queries: ACLs, Deletion Propagation, and Freshness

Authorization must take effect before candidate generation, while ACLs, deletion events, and source versions must propagate to every derived index; freshness needs measurable event-time SLOs too.

Private-Corpus Retrieval Eval: Turning a Traditional Chinese Query Set into a Reproducible Benchmark

The repository has a 20-query Traditional Chinese/English golden dataset, but no document-level qrels, retrieval runs, raw latency data, or executable benchmark script. Reporting Recall@k, MRR, or nDCG as measured results would therefore be dishonest; this article defines the contract needed to run them reproducibly.