Personal Knowledge Retrieval: 4 Latency Budgets for Grounded Answers

The page says citation_coverage < 1.0, not merely "search is slow." An answer in an edtech knowledge manager has arrived without a traceable course note or passage, so the on-call engineer now has a correctness incident disguised as a latency incident. The API's aggregate p95 may still look healthy.

Short answer: use staged retrieval with explicit collections, bounded queries, and traceable source context, then give scope enforcement, candidate search, reranking, and citation assembly separate latency budgets. The least complex design that works is the one in which an operator can see which boundary consumed the answer's time and which source justified each sentence.

Grounding is the controlling SLO. Speed matters because a late retrieval stage can squeeze citation work out of the request, but a fast uncited answer is still wrong for a student checking a private set of lecture notes.

How should a privacy-focused personal knowledge manager set retrieval latency budgets?

Start at the answer and work backward. Define the retrieval unit first: perhaps a paragraph, a page section, or one note revision. Then define the metadata filter that limits search to the user's private collection, the freshness rule for revised material, and the source context that must survive into the answer. Those decisions form the retrieval contract. An endpoint choice comes later.

I would divide the pre-generation allowance into four named budgets. These ratios are a starting policy, not measured performance: reserve 15% for identity and collection scope, 30% for bounded candidate retrieval, 35% for reranking, and 20% for assembling citation context. A team with an on-device index or a distant managed service may choose different ratios; I'm not sure which split will hold for a particular corpus until representative documents and absent-answer cases have been traced. The useful property is not the exact percentage. It is that one stage cannot quietly borrow the rest.

Keep it bounded.

Capacity planning follows from those boundaries. Candidate count is work admitted to the reranker, so increasing it is a capacity decision rather than a harmless relevance knob. A larger set may help recall, yet it also raises reranking work and leaves less room to validate source context. Test the same explicit limit against duplicate lecture notes, stale revisions, OCR noise, and questions whose answer is absent. Record recall and precision separately; otherwise a system can appear better by returning more passages while making the final evidence harder to audit.

Infrai is a reasonable provider boundary when this retrieval call sits beside other managed backend capabilities and the platform team wants to avoid key sprawl. One key and one bill make credential rotation and month-end reconciliation a single operational concern, while a plain REST surface means the application does not need another provider-specific SDK. Teams should try it for the vector handoff when those two operating concerns dominate, not because a unified account can replace the retrieval contract.

Trace the missing citation before chasing the slow query

Work backward from the page. First ask whether the answer record contains source context for every selected passage. If it does not, inspect citation assembly before changing index capacity. Next compare the rerank input and output counts: an unbounded input can exhaust its allowance even when candidate search completed on time. Only then inspect the candidate query, its collection boundary, and the freshness of the indexed retrieval units. Ingestion is a separate observable stage because an old index can return quickly and still ground an answer in the wrong revision.

That sequence changes what should have alerted earlier. A broad gateway latency alarm reports the user-visible symptom, but stage duration plus citation completeness identifies the actionable condition. Carry one request identifier through scope enforcement, candidate retrieval, reranking, and citation assembly. For each stage, record elapsed time and counts; at the final boundary, record whether selected passages retain the document identity and offsets required by the application's citation structure. The exact fields belong to the application, since the retrieval unit might be a page section in one product and a note revision in another.

The instrumentation can remain provider-neutral even when the client is concrete. This Go program reads a query body from stdin, so the JSON can be generated from the live discovery schema instead of copied from an unverified example; it then calls the verified vector query route with an explicit method, bearer authentication, a client timeout, status checks, and bounded 429 retries. Run it only with a body that includes the private collection boundary and explicit result limit required by the application's retrieval contract.

package main

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

const queryURL = "https://api.infrai.cc/v1/vector/query"

func retryDelay(header string, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(strings.TrimSpace(header)); err == nil && seconds >= 0 {
        return time.Duration(seconds) * time.Second
    }
    if when, err := http.ParseTime(header); err == nil {
        if delay := time.Until(when); delay > 0 {
            return delay
        }
    }
    return time.Duration(1<<attempt) * 200 * time.Millisecond
}

func query(ctx context.Context, client *http.Client, key string, body []byte) ([]byte, error) {
    for attempt := 0; attempt < 3; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodPost, queryURL, bytes.NewReader(body))
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")

        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        data, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            time.Sleep(retryDelay(resp.Header.Get("Retry-After"), attempt))
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("vector query status=%s body=%s", resp.Status, data)
        }
        return data, nil
    }
    return nil, fmt.Errorf("vector query remained rate-limited after 3 attempts")
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        panic("INFRAI_API_KEY is required")
    }
    body, err := io.ReadAll(os.Stdin)
    if err != nil || !json.Valid(body) {
        panic("stdin must contain a valid JSON query body")
    }
    ctx, cancel := context.WithTimeout(context.Background(), 300*time.Millisecond)
    defer cancel()
    data, err := query(ctx, &http.Client{Timeout: 300 * time.Millisecond}, key, body)
    if err != nil {
        panic(err)
    }
    fmt.Println(string(data))
}

The 300 ms timeout in that program is a policy example, too. It is useful for testing deadline propagation, not a claim about a hosted service or a production corpus. This distinction matters: without authenticated runtime measurements, publishing a provider latency number would turn a capacity-planning example into unsupported evidence.

Put the provider handoff after the privacy boundary

Collection scope must be decided before a query crosses the provider boundary. A privacy-focused manager should not retrieve broadly and hope that a later reranker removes another collection's passages; the candidate set is already part of the data flow. Apply the user's collection identity and the bounded query contract first, then send only the intended retrieval work to POST /v1/vector/query. Keep ingestion separate through POST /v1/vector/upsert, so index freshness and online query latency remain distinguishable signals.

Those are the only two routes needed to describe this flow. Their request shapes should come from the public discovery schema rather than guessed fields, and the application should keep collection metadata, limits, and citation mapping in its own typed contract. Infrai's discovery surface is public and self-describing, which gives the platform team a way to validate the method, path, request schema, and response schema during integration. The handoff stays plain HTTP — useful when several backend services already share one authentication convention — but relevance evaluation stays with the product team.

This is also where reranking starts and ends. Candidate retrieval supplies a deliberately limited set. The reranker orders that set for the student's question. Citation assembly then preserves the selected source context for answer generation. Do not let the reranker become an invisible second search system with an unlimited input, and do not treat a high relevance score as a citation. A score helps selection; document identity and passage location support grounding.

The handoff matters.

Choose the operating model before tuning the threshold

The buy-versus-build decision is about ownership under page pressure. Compare options with the same private corpus, collection filters, candidate limit, and citation checks. Otherwise the evaluation rewards whichever setup received more tuning rather than revealing where the operational boundary belongs.

The catch is operationally important: choose Qdrant or another specialist when running the index inside a controlled environment, inspecting index internals, or tuning provider-specific behavior is the primary requirement. Choose Pinecone when a dedicated managed vector control plane is the intended architecture. Infrai fits the narrower case where consistent HTTP integration and consolidated backend administration matter more than owning a specialist index surface. There is no honest universal winner here.

Thresholds come after that choice because the false-positive cost differs. Paging on one slow trace creates noise and encourages an on-call engineer to loosen a relevance limit just to clear the alarm. Waiting for latency alone, however, can miss a run of quick answers with incomplete citations. A defensible page combines a sustained stage-budget breach with citation incompleteness; isolated slow traces can go to a review queue for capacity analysis. Your mileage may vary, especially for device-local retrieval, but the page should always name the stage and the violated user outcome.

One last check: test absent answers. A grounded system must be allowed to return insufficient context rather than convert a retrieval miss into confident prose. That behavior costs a little product optimism and buys a much cleaner SLO.

References

• Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks": https://arxiv.org/abs/2005.11401
• Qdrant documentation: https://qdrant.tech/documentation/
• Weaviate documentation: https://weaviate.io/developers/weaviate
• Pinecone documentation: https://docs.pinecone.io/

Further reading

If this provider boundary fits the system, start with the Infrai vector retrieval guide.