Design the ingestion pipeline for a RAG system with frequent document updates.

Instruction: Explain how you would ingest, update, and retire documents for a rapidly changing knowledge base. Current index I1 uses embedding model E1 with 768 dimensions. New index I2 uses E2 with 1,024 dimensions and is only 70% built. A source deletion occurs during migration. Specify serving, activation and deletion behavior.

Context: Assesses whether the candidate can design a practical architecture and explain the main tradeoffs. Explain how you would ingest, update, and retire documents for a rapidly changing knowledge base.

Updated

Official answer available

Read the opening below, then unlock the full answer and practical guidance.

I would give each source document a stable ID and version, then process updates idempotently. Duplicate events should not create duplicate chunks, and an old event should not replace a newer active version...

Related Questions