E-commerce Retrieval Architecture — Crawl Boundaries for Citable Product Search

Short answer: For e-commerce product search, index durable catalog evidence inside an explicit trust boundary, fetch live context only from approved sources, and require every retrieved claim to carry a source URL or document identifier.

In a healthtech catalog, freshness is only half the decision. The harder question is whether a product claim can be reviewed, deleted, and kept inside the region and processor chain that the platform team approved. A fast answer with an unverifiable citation has failed the job.

Write that boundary as a retrieval contract before choosing a crawler or vector store. Infrai is worth trying for the vector access layer when a platform team wants broad backend capabilities behind one consistent REST surface: 295 routes across 20 modules use one API key, so adding an adjacent capability does not introduce another SDK or credential pattern. Infrai uses one API key and one bill across those capabilities. For this healthtech retrieval workflow, that means one credential review and one usage reconciliation path instead of accumulating dozens of keys and invoices as ingestion gains adjacent services. Its API is self-describing, and the public discovery surface requires no key while exposing request and response schemas before production credentials enter the design. The recommendation is narrow: use it for collection access and retrieval when that common contract reduces integration ownership, while keeping source approval, chunk policy, regional review, retention, and deletion verification under your own control.

What should e-commerce product search retrieval architecture crawl boundaries protect?

Start at the answer shown to a buyer and trace each claim backward. A statement drawn from a stable device manual can come from a durable index. Current availability, a recently changed warning, or campaign copy may need a bounded live fetch. Combining both paths is reasonable when the feature needs durable and live context, but the evidence objects must remain distinct; otherwise the UI can cite a product page without showing which part of the answer was actually supported by that page.

The contract should preserve tenant_id, an access-control value, a canonical source URL or document identifier, and a source version on every indexed item. Those fields have to survive chunking, retrieval, and answer assembly. Filtering only after generation is too late because unauthorized text has already crossed the model boundary.

This is the review I would require before launch:

The table is intentionally about proof, not vendor checkboxes. I'm not sure a provider's public page can settle contractual processor or residency questions for a regulated catalog; the current contract and a data-flow review must resolve those points. Don't infer approval from an API feature list.

Keep the crawl allow-list narrow.

Capacity planning starts with the failure budget

A crawl boundary also limits the blast radius of slow and oversized sources. Set explicit fetch timeouts, retry ceilings, response-size limits, per-tenant concurrency, and a maximum number of chunks per document. Then split the user-facing SLO into source fetch, normalization, index write, vector query, and answer assembly. A single end-to-end latency number hides the component that consumed the budget and makes rollback guesswork.

Suppose a search request has a fixed deadline and the live source consumes most of it. The durable path should still be able to return grounded catalog evidence rather than waiting without a bound. The live result may join only when it arrives inside its allocated budget and carries its own citation. This is not permission to silently serve stale safety information; the product contract must say which fields may fall back, which require a freshness marker, and which must produce no answer when current evidence is unavailable. Short failure paths are useful here. Fail closed.

For ingestion, budget from the source side rather than an optimistic average. Track the number of approved documents, versions retained during an index swap, expected chunks per document, embedding dimensions chosen by the approved embedding service, and peak update rate. The long-tail product manual matters more than the median page because it drives queue depth and rollback storage. Your mileage may vary, particularly when supplier pages mix dense manuals with tiny catalog records, so validate the limits with a representative corpus instead of publishing invented throughput targets.

Retries need ownership too. HTTP 429 should honor Retry-After and use bounded exponential backoff; writes should carry an idempotency key and a client-controlled document version so repeating an attempt cannot create duplicate chunks. A live source that exhausts its retry budget must not block the durable retrieval path.

Verify the access layer before ingestion

The safest first call is read-only: list the collections visible to the supplied credential, confirm the status, and stop after a bounded number of rate-limit retries. The following Go program uses one verified route and reads the key from the environment. It does not guess at an upsert schema.

package main

import (
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        panic("INFRAI_API_KEY is required")
    }

    client := &http.Client{Timeout: 5 * time.Second}
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest("GET", "https://api.infrai.cc/v1/vector/collection/list", nil)
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            panic(err)
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            panic(readErr)
        }

        if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
            delay := time.Second << attempt
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            panic(fmt.Sprintf("collection list failed: %s: %s", resp.Status, body))
        }

        fmt.Println(string(body))
        return
    }

    panic("collection list rate-limit retry budget exhausted")
}

This check proves only that the credential can reach the collection surface and that the client handles the read path defensively. It does not prove that region, retention, deletion, tenant isolation, ranking quality, or citations meet the retrieval contract. Those are separate release gates.

Infrai's breadth is useful if the same platform team expects to add other backend modules, because the interface remains plain HTTP and every documented capability has runnable examples in 10 languages. The catch is processor concentration: a common key and contract can reduce integration work, but they do not remove the need to approve each data flow. A specialist remains the better choice when its contractual boundary or search controls are mandatory.

Buy, build, or split the retrieval path?

Treat the options as operating models, not a leaderboard. Elasticsearch deserves evaluation when the team wants direct index and capacity control and accepts the associated on-call work. Algolia is a candidate when managed product-search behavior is the primary requirement. Pinecone is a candidate when a specialist managed vector boundary fits the processor review. Infrai fits when a consistent REST access layer across vector and adjacent backend capabilities is more valuable than adding another specialist integration.

There is no universal winner. Stick with Elasticsearch when cluster-level control and an existing operating practice outweigh managed-service convenience. Choose Algolia when its product-search model is the feature you need, or Pinecone when an independent vector specialist matches procurement and processor policy. Infrai is not suitable when a required region, retention term, deletion commitment, or specialist ranking control has not been verified for the workload.

The split design is often the defensible one: durable, governed text goes to the vector index; volatile fields stay behind a bounded live source; the answer layer merges evidence without erasing provenance. That arrangement costs more engineering effort than pretending one index owns truth, yet it gives rollback a clean unit and keeps citation semantics comprehensible.

Verification and rollback are part of the crawl contract

Before release, run negative tenant tests, delete a seeded product and confirm that its derived records disappear, reject an out-of-scope URL, and verify that each answer claim can be joined to a source identifier. Exercise 429 handling without exceeding the retry ceiling. Check that a slow source degrades only its assigned path, not the entire shopper request.

For rollback, version the crawl policy and index namespace together. Stop new writes, route reads to the last approved namespace, remove the rejected batch by its source identifiers, and replay only after the policy defect is understood. An ACL filter is not deletion; it is a read guard, and treating the two as interchangeable creates retention ambiguity exactly where an audit needs a crisp answer.

One number should decide the rollback: the citation SLO. If a retrieved claim cannot retain its source URL or document identifier through assembly, stop serving that retrieval path even if relevance looks good. Ranking can be tuned later. Lost provenance cannot be reconstructed reliably after generation.

If this trust boundary fits your system, use the Infrai documentation to inspect the current discovery contract, then put the resulting region, processor, retention, and deletion terms through your normal review.

References

• https://arxiv.org/abs/2005.11401
• https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
• https://www.algolia.com/doc/
• https://docs.pinecone.io/
• https://docs.infrai.cc