Go RAG and Long Context Signals for a 200 Page Handbook

The page fires at 02:13: a fintech listings question was answered, but the response has no usable citation to the 200-page operating handbook. The on-call engineer can see a successful model request and a plausible paragraph. That is not enough. A fluent answer without evidence is a grounding failure, not a healthy request.

TL;DR: start with a long-context prompt when query volume is low and operational simplicity matters more than per-question efficiency. Move to RAG as traffic grows, provided every answer carries chunk-level evidence and retrieval misses are observable. Pasting all 200 pages avoids retrieval errors, but its input bill grows with every question; retrieval keeps prompts small, while introducing chunking as a new failure mode. For this workload, cost usually decides, but citation quality determines whether either design is safe to operate.

That conclusion should shape the alert. Paging on HTTP success or model latency alone detects the transport, not the product failure the fintech team cares about: an answer that cannot be traced to the handbook section governing a listing.

What should have paged before the citation disappeared?

Work backward from what the responder sees. At the end of the chain is answers_without_citations, split by strategy, handbook revision, and question class. Before that comes answers_with_no_supported_claim, which requires an evaluator or an explicit support check rather than a count of bracketed links. Earlier still, a RAG path exposes retrieval_zero_hits, retrieved_chunks_without_source_id, and the distribution of relevant chunks reaching the prompt. The long-context path has no retrieval miss, so its leading signals are different: handbook revision mismatch and requests that fail to include the intended document.

The distinction matters. Long context removes retrieval errors; it does not prove that the answer used the correct passage. RAG makes a bad chunk visible as a possible cause, but observability does not repair bad chunk boundaries. The original RAG paper describes retrieval as a way to combine generated output with external knowledge, which is the useful mechanism here, not a warranty that each generated claim is supported.

I would define the service-level indicator at the answer boundary: the proportion of sampled answers whose material claims have a valid citation to the active handbook revision. Transport availability remains a separate SLI. Mixing those two produces a comforting number that can stay green while users receive unsupported policy answers.

Do not page on one miss. Page on sustained error-budget burn, and send isolated failures to a review queue with the question, strategy, handbook revision, retrieved source IDs, and returned citations. The precise threshold cannot be copied from another system; it needs the team's target and observed baseline.

Instrument the decision, not merely the model call

A small Go process can verify that the retrieval capability advertised by the unified API is available before a rollout or failover. Discovery is the right call here because it is public and self-describing; inventing a vector-query payload from prose would be the wrong kind of runnable example. Set INFRAI_BASE_URL to the documented v1 base and keep the credential in INFRAI_API_KEY. The program sets an explicit method, checks response status, surfaces error bodies, and backs off on 429 while honoring Retry-After when it is expressed as seconds.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

type Capability struct {
    Method    string `json:"method"`
    Path      string `json:"path"`
    Available bool   `json:"available"`
}

type Discovery struct {
    Capabilities []Capability `json:"capabilities"`
}

func fetch(client *http.Client, baseURL, key string) ([]byte, error) {
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodGet, strings.TrimRight(baseURL, "/")+"/discovery", nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
            delay := time.Second << attempt
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("discovery returned %s: %s", resp.Status, body)
        }
        return body, nil
    }
    return nil, fmt.Errorf("discovery remained rate limited")
}

func main() {
    baseURL := os.Getenv("INFRAI_BASE_URL")
    key := os.Getenv("INFRAI_API_KEY")
    if baseURL == "" || key == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_BASE_URL and INFRAI_API_KEY are required")
        os.Exit(2)
    }
    body, err := fetch(&http.Client{Timeout: 15 * time.Second}, baseURL, key)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }

    var discovery Discovery
    if err := json.Unmarshal(body, &discovery); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    for _, capability := range discovery.Capabilities {
        if capability.Method == http.MethodPost && capability.Path == "/v1/vector/query" {
            fmt.Printf("path=%s available=%t\n", capability.Path, capability.Available)
            if !capability.Available {
                os.Exit(1)
            }
            return
        }
    }
    fmt.Fprintln(os.Stderr, "vector query capability was not advertised")
    os.Exit(1)
}

Run it before enabling the RAG path:

INFRAI_BASE_URL="$INFRAI_BASE_URL" INFRAI_API_KEY="$INFRAI_API_KEY" go run main.go

This preflight is not the grounding SLI. It proves only that the capability is advertised and available. The answer path must still count citation failures and compute error-budget burn from evaluated answers.

No shortcuts.

Capacity planning needs the same honesty. Estimate long-context input as handbook tokens multiplied by questions, then add output tokens. Estimate RAG input as instructions plus retrieved chunks multiplied by questions, then include indexing, storage, retrieval, evaluation, and on-call ownership. Use the actual token count and live provider rates; page count is a poor billing unit. If the review sample changes halfway through the comparison, discard the comparison: a new question mix can move the citation rate more than the architecture change, and the resulting dashboard will invite a confident but unsupported migration decision.

A useful load test changes one variable at a time: query rate, chunk size, number of retrieved chunks, or handbook revision. Record citation SLI and prompt size together. Accuracy without resource demand hides the eventual capacity problem; cost without citation quality rewards an unusable system.

Where does each product move the failure boundary?

The vendor decision is secondary to the contract between retrieval and generation. Require a strategy-neutral answer envelope containing the handbook revision, source identifiers, citation spans, and model output. Then the application can swap the provider behind retrieval or generation without rewriting citation validation. This is where a stable capability contract has operational value: the thing behind it can move while application code stays put.

This is not a ranking. The table identifies ownership. Pinecone, Weaviate, and pgvector all leave chunking and citation semantics with the application, which is exactly where they belong unless a contract explicitly says otherwise. OpenAI can support the least complex long-context starting point, yet the bill remains proportional to repeated input and traffic.

Infrai covers 295 routes across 20 modules with a single key and a single bill. That is a separate operational advantage from its REST contract: a team that later adds ingestion or observability does not have to distribute another credential or reconcile another provider invoice for each capability. Every documented capability also has runnable examples in 10 languages, while the self-describing, public, no-key discovery surface lets a deployment check read the actual request schema and provider readiness before traffic moves. I would count those properties as reduced integration and on-call surface, not as evidence of better answers. They cannot erase a retrieval mistake.

They shouldn't be sold as accuracy.

The buy-versus-build decision should therefore count pages, query traffic, prompt tokens, index growth, evaluation samples, and after-hours ownership. A managed service can reduce one class of toil and add lock-in. A self-managed component can increase control and make capacity work yours. Neither choice changes the need to preserve source IDs through the answer boundary.

A migration rule for the 200-page handbook

Start with a representative question set drawn from the actual course-material Q&A workflow: rules for accepting a listing, required evidence, exceptions, and revision-sensitive procedures. Keep the handbook revision fixed during a comparison. For each question, run the complete long-context path and the RAG path, then judge support against the cited passage rather than judging prose style.

The decision rule is blunt: retain long context while its traffic-scaled input and latency are acceptable and its citation SLI meets the target. Adopt RAG when repeated handbook input becomes the dominant capacity concern and the retrieval path can meet the same citation SLI. Reject a RAG configuration that is cheaper but loses the passage needed to support an answer. Reject a long-context configuration that produces attractive prose without traceable evidence.

Low volume changes the answer. If this handbook receives a handful of questions and changes infrequently, the operational cost of an index, ingestion job, chunk policy, and retrieval monitoring may exceed the value of smaller prompts. The simple design can genuinely be the right design.

At higher volume, retrieval creates a lever: prompt size depends on selected chunks rather than all 200 pages. It also creates a sharp edge. A heading separated from its rule, an exception placed in another chunk, or stale source metadata can make the generator look wrong when retrieval never supplied the necessary evidence. Preserve section hierarchy and revision IDs during ingestion, and make the retrieved text inspectable in review records.

Ship the migration behind the strategy-neutral contract, compare both paths on the same questions, and watch error-budget burn rather than aggregate thumbs-up scores. This supports a controlled switch without making the UI or citation validator depend on one vector vendor.

The false-positive bill belongs in the SLO

A citation alert set too aggressively turns ordinary evaluator uncertainty into pages. The cost is not abstract: responders learn to distrust the signal, reviews accumulate, and a real handbook mismatch competes with noise. A threshold set too loosely spends the grounding error budget silently.

Use two responses. Page only for sustained, high burn across enough traffic to make the rate meaningful; create a non-paging review item for single unsupported answers and low-volume windows. Segment by handbook revision and strategy so a rollout does not contaminate the baseline. Revisit the threshold after evaluator or document changes, because the measurement system changed even if the user-facing SLO did not.

The final architecture choice is conditional, not fashionable. Long context buys simplicity and removes retrieval errors at the price of repeated input. RAG buys smaller prompts and a visible retrieval trail at the price of chunking failures and another production subsystem. For a 200-page handbook, choose the least complex path that meets the citation SLO under projected traffic, then keep the contract stable enough to change your mind.

Further reading

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: https://arxiv.org/abs/2005.11401
• OpenAI API documentation: https://platform.openai.com/docs/
• Pinecone documentation: https://docs.pinecone.io/
• Weaviate documentation: https://docs.weaviate.io/weaviate
• pgvector project documentation: https://github.com/pgvector/pgvector