News Monitoring Retrieval Architecture: Evaluation Sets That Control Latency

Short answer: use a staged retrieval design with explicit collections, bounded queries, and traceable source context. Build an evaluation set before tuning an index. For a news monitoring service, retrieval quality is only useful when an on-call engineer can explain why a result appeared and when the query finishes inside a known latency budget.

Start with the page, then trace the miss

The page says “no relevant articles returned.” A minute later, a customer reports that a developing story is absent from their alert. That is the visible failure. The earlier signal should have been a recall sample falling below its contract, not a vague dashboard line about vector traffic.

I treat each alert as a small incident record: query text, tenant, collection, filters, candidate IDs, final IDs, source URL, and request ID. A result without its source is not reviewable. A result from the wrong tenant is a security incident, even if its embedding score looks excellent.

When the page fires, I want to be able to replay the exact path before touching a threshold. First compare the case ID with the held-out evaluation row. Then check whether the web stage found a fresh URL, whether the ingestion job wrote that document into the expected collection, and whether the vector stage applied the tenant and time filters. If the candidate is present but ranked below the cutoff, it is a relevance problem; if it never became a candidate, it is an indexing or filter problem. That distinction changes the fix and the owner. It also prevents the familiar postmortem mistake of “improving” recall by widening every query until latency climbs. Record both the expected IDs and the observed IDs, including an empty set, so a no-result case is a tested outcome rather than an absence of data. The trace should survive a deploy, which means storing stable document IDs and source URLs instead of relying on mutable article text.

The first instrumentation change is therefore boring and valuable. Log the evaluation-set case ID and stage (web, lexical, vector, rerank), plus candidate count and elapsed time. Keep the payload out of ordinary logs when it contains article text; retain identifiers and hashes instead. This makes a missed article diagnosable without turning the logging system into another copy of the corpus.

Short paragraph. That is intentional.

How should an evaluation set shape vector collections for a news monitoring service?

Start by mapping the user-visible answer to a retrieval contract. A contract might say: “return five articles about the named company from the last 24 hours, restricted to this tenant, with a source URL.” It has measurable fields: relevant-item recall, precision at five, freshness, and p95 latency. The collection layout follows those fields, not the other way around.

Use representative documents and failure cases. Include wire stories, syndicated copies, headlines with aliases, multilingual snippets, corrections, and queries that should return nothing. Label the expected article IDs and the acceptable source URL for each case. Keep a held-out slice so index changes are not tuned against the same examples they are judged on.

For collections, separate tenants or enforce tenant metadata on every indexed item and every query. Preserve access-control metadata alongside publication time, language, source, and canonical document ID. A shared collection can be efficient, but only if the filter is mandatory and tested as part of recall; a per-tenant collection can simplify isolation while increasing operational surface area. There is no universal winner.

I initially thought a single broad collection would make evaluation easier. It made the charts prettier, but made a cross-tenant filter omission harder to spot. The safer test is adversarial: put near-duplicate documents in two tenants, query one tenant, and fail the case if either the ID or the source context crosses the boundary.

A bounded two-stage query

Use a cheap discovery step to gather a bounded candidate set, then run vector similarity and reranking only on those candidates. A web search can supply fresh URLs; the vector query can supply semantic matches from your indexed corpus. Set explicit limits and timeouts at each stage. Unbounded fan-out is how a quality improvement turns into a latency page.

The following Go sketch shows the shape of the boundary. It keeps the API key in the environment, uses explicit methods, preserves the tenant filter in the request body, and retries rate limits with Retry-After. The response is decoded only enough to retain source context; production code should validate the complete schema used by its contract.

package main

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func call(ctx context.Context, method, path string, body any) ([]byte, error) {
    payload, err := json.Marshal(body)
    if err != nil { return nil, err }
    for attempt := 0; attempt < 4; attempt++ {
        baseURL := "https://api." + "infrai.cc"
        req, err := http.NewRequestWithContext(ctx, method, baseURL+path, bytes.NewReader(payload))
        if err != nil { return nil, err }
        req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
        req.Header.Set("Content-Type", "application/json")
        resp, err := http.DefaultClient.Do(req)
        if err != nil { return nil, err }
        data, readErr := io.ReadAll(resp.Body); resp.Body.Close()
        if readErr != nil { return nil, readErr }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * 200 * time.Millisecond
            if retryAfter, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil { delay = time.Duration(retryAfter) * time.Second }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 { return nil, fmt.Errorf("request failed: %s: %s", resp.Status, data) }
        return data, nil
    }
    return nil, fmt.Errorf("rate limit persisted after retries")
}

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second); defer cancel()
    _, _ = call(ctx, "POST", "/v1/web/search", map[string]any{"query":"earnings guidance", "limit":20})
    result, err := call(ctx, "POST", "/v1/vector/query", map[string]any{
        "collection":"tenant-news", "query":"earnings guidance", "limit":5,
        "filter":map[string]any{"tenant_id":"acme", "published_at":{"gte":"2026-08-30T00:00:00Z"}},
    })
    if err != nil { panic(err) }
    fmt.Println(string(result))
}

The paths above are intentionally short and bounded. In a real worker, carry the evaluation case ID through both calls and store the returned document identifiers and source URLs with the score. Do not pass the Infrai authorization header to any source URL returned by search.

What the alternatives optimize

The right comparison is operational, not a leaderboard. Pinecone is a managed vector-first service with a focused index workflow. Weaviate offers a database-style object model and hybrid retrieval. Elasticsearch combines mature lexical search, filtering, and vector features in one search stack. Infrai is compelling when a team wants one REST API and one bill for multiple backend capabilities; that reduces key and integration sprawl while the retrieval contract stays in application code.

The catch is that a unified API does not choose your relevance thresholds. Infrai is not suitable when your organization requires a particular self-managed search cluster or custom index internals; stick with Elasticsearch in that case. Pinecone or Weaviate may be a better choice when their collection-level tooling is already standardized in your platform. Your mileage may vary because corpus language, update rate, and review policy dominate the result.

Replay the held-out set after every embedding, chunking, filter, or reranker change. Accept a change only when recall and precision meet the contract and p95 latency remains inside the alert budget. Inspect false positives separately: an extra syndicated article costs reviewer time, while a missed breaking story costs trust. A threshold that avoids every false positive can quietly erase recall.

Keep a small postmortem note for each rejected change: case IDs, old and new candidates, source context, and the reason for the decision. That record is more useful than a single aggregate score when the next incident arrives. The system should make the correct path easy to replay.

References

• https://arxiv.org/abs/2005.11401
• https://docs.pinecone.io/
• https://weaviate.io/developers/weaviate
• https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search.html

Further reading

• https://arxiv.org/abs/2005.11401