Implementing Retrieval Architecture for Contract Review and Game Search — Recovery

Index cost changes the recovery design. For a gaming catalog, I would keep a cheap, durable source-of-truth index and rebuild semantic shards in small, replayable batches. The same pattern fits a legal contract review workspace, where stale or incomplete retrieval is a correctness problem rather than a minor outage.

Short answer: treat retrieval as a versioned pipeline with a lexical fallback, explicit freshness states, and an evaluation gate before a recovered shard can serve answers.

Start with a failure contract

“The search is up” is too weak a signal. A useful contract says which corpus version is visible, how many documents were accepted, and which embedding model produced each vector. Every query should carry those identifiers into logs and citations.

In the game-content case, the source records are quests, item descriptions, and patch notes. In contract review, they are clauses and exhibits. Both domains punish silent partial indexing: a missing patch note can mislead a player, while a missing indemnity clause can mislead counsel.

I keep three states per batch: pending, servable, and quarantined. A batch becomes servable only after count checks, checksum checks, and retrieval tests pass. Recovery then means selecting the newest servable version, not guessing which files made it through.

How should retrieval architecture recover a legal contract review workspace after failure?

The recovery path is deliberately boring:

1. Freeze the failed batch and record its input manifest.
2. Re-read source objects by immutable identifier.
3. Chunk with the same rules and write vectors to a new versioned namespace.
4. Run lexical and semantic evaluation against a held-out question set.
5. Promote the namespace atomically, then release the write freeze.

That sequence prevents a half-rebuilt index from mixing with a prior version. It also gives an auditor a finite replay window. For a gaming catalog, I can replay only the changed product records; for contracts, I can replay the affected matter while preserving the previous corpus for review history.

Here is a small Python skeleton for the state transition. It uses generic interfaces so the storage and embedding implementation can be swapped without changing the recovery policy.

from dataclasses import dataclass
from typing import Literal

State = Literal["pending", "servable", "quarantined"]

@dataclass(frozen=True)
class Batch:
    batch_id: str
    source_digest: str
    vector_model: str
    state: State

def promote(batch: Batch, *, accepted: int, expected: int, recall_at_10: float) -> Batch:
    if accepted != expected:
        return Batch(batch.batch_id, batch.source_digest, batch.vector_model, "quarantined")
    if recall_at_10 < 0.90:
        return Batch(batch.batch_id, batch.source_digest, batch.vector_model, "quarantined")
    return Batch(batch.batch_id, batch.source_digest, batch.vector_model, "servable")

The 0.90 threshold is an example policy, not a universal benchmark. Set it from your own labeled queries and document why. I'm not sure any team can pick a meaningful threshold before it has a representative contract set or a realistic product-content query set; your mileage may vary until that evaluation exists.

During rebuild, route exact names, identifiers, and quoted phrases to a lexical index. Return the corpus version and a freshness label with every result. A caller can then avoid presenting an answer as current when it came from the last known-good snapshot. In practice, I expose this as a small response envelope: results, corpus_version, indexed_at, and retrieval_mode. The UI can render a review banner from those fields without knowing which index implementation is running. That separation matters during a recovery drill: the lexical path may answer an exact item-name query while the semantic worker replays a bad batch, yet the same citation renderer still shows the snapshot and mode. For a legal workspace, the banner can require confirmation before a generated summary is exported; for game content, it can point editors to the patch-note queue. The fallback must be intentionally limited. It should not pretend that lexical matching covers paraphrases, translated item names, or a clause whose meaning depends on nearby text. In a contract workspace, a review screen can ask the user to confirm the snapshot before generating a summary. In a game search UI, it can show that new patch notes are still indexing.

The fallback must be intentionally limited. It should not pretend that lexical matching covers paraphrases, translated item names, or a clause whose meaning depends on nearby text. In a contract workspace, a review screen can ask the user to confirm the snapshot before generating a summary. In a game search UI, it can show that new patch notes are still indexing.

One practical mistake is retrying the same failed write forever. Backoff does not repair a bad manifest or a non-deterministic chunker. Persist the input digest, cap retries, and quarantine the batch for inspection when deterministic checks disagree.

Measure it.

Measure recovery, not just relevance

An eval harness should exercise both normal and degraded paths. I use paired questions: one set for semantic intent (“Which weapon grants a bleed bonus?”), and one set for exact evidence (“Find the termination-for-convenience clause”). For each, record recall at 10, citation coverage, freshness lag, and the percentage of queries served by fallback.

Cost belongs in the same report. Count embedding calls, stored vector bytes, and replayed documents per recovered batch. That makes index-cost-at-scale visible without turning a price claim into the architecture decision. A compact table is enough:

Run the same suite before and after promotion. If the recovered version improves one metric while harming citation coverage, keep the old version and investigate; a green ingestion job is not permission to serve it.

Choose boundaries your team can operate

A single service can own manifests, indexing, evaluation, and promotion when the corpus is small. At larger scale, separate workers by stage and pass immutable batch metadata between them. The important boundary is operational: each stage must be replayable and observable with ordinary Python jobs, queue metrics, and object-store logs.

This approach is not suitable when users need cross-document transactional updates in milliseconds; a database with native transactional search may be a better fit. Stick with a simpler lexical design when the content is mostly identifiers and exact names. Semantic retrieval earns its storage and rebuild cost only when the evaluation set demonstrates a material gain.

I started by assuming a failed shard just needed another retry. The useful correction was to make promotion, not writing, the unit of safety. That shift keeps notebook experiments close to production behavior and keeps prompt costs predictable because the generator sees fewer irrelevant chunks.

References

• https://arxiv.org/abs/2005.11401
• https://www.rfc-editor.org/rfc/rfc9110
• https://www.sqlite.org/fts5.html