AI Crawlers Are Expensive. Agent Builders Should Care.

On August 29, kernel.org administrator Konstantin Ryabitsev published an unusually concrete account of what automated traffic can cost a website.

Across five geographically distributed servers with 90 CPU cores, he reported that 14–16 cores were continuously occupied rendering Git commit pages for scrapers. The same repositories were already available for cloning, but automated clients were requesting individual HTML pages and repeatedly collecting overlapping content across repository forks.

Kernel.org remained responsive. This was not a story about AI crawlers taking the site offline. It was a story about persistent, avoidable infrastructure work—and about how easily that work disappears from the application developer's view.

The underlying problem should sound familiar to anyone building a research agent or AI search product: what happens when retrieving evidence is cheap for your application but expensive for the source providing it?

You can read his original account in “Creepy crawlies”.

Not All AI-Related Traffic Does the Same Job

Discussions about AI crawlers often combine several activities that have different purposes.

Training crawlers collect material that may contribute to future model development. Search crawlers discover or refresh pages so they can be found later. User-triggered retrieval fetches a page in response to a particular question or task.

The distinction is visible in provider documentation. OpenAI separately documents GPTBot, OAI-SearchBot, and ChatGPT-User. Site owners can express different preferences for training, search discovery, and certain user-initiated visits.

For developers, the important question is not simply whether a request is related to AI. It is who initiated it, why the content is being requested, how often it is fetched, and whether a better access method already exists.

A website might welcome search discovery while objecting to bulk collection. An agent might need three relevant pages without acquiring the surrounding site.

The Cost Is Missing from Your Agent Dashboard

Most AI application dashboards show model tokens, tool calls, latency, and perhaps the price of a search or extraction request. They rarely show what the source website had to do before returning the evidence.

A dynamic page may require database queries, template rendering, permission checks, and substantial server work before it produces a small amount of useful text. Ten URLs can also contain nine copies of the same information.

This is a familiar constraint in search infrastructure. Google's crawl-budget documentation explains that crawling has to respond to server capacity and that duplicate URLs can waste resources.

Agentic workflows can amplify that waste. Imagine a research agent dividing one question among four subagents. Each searches independently, finds the same report, and downloads it. A later verification stage retrieves the report again because it cannot see the earlier fetches.

The extra work does not improve the answer. It adds latency and cost to the application while sending redundant requests to the publisher.

A useful retrieval system therefore needs to optimise for distinct, relevant evidence—not raw page count.

Inefficient Retrieval Also Weakens the Answer

Source cost is only half of the problem. Retrieval behaviour shapes which information reaches the model.

When a primary source becomes unavailable or expensive to access, an agent may quietly substitute a secondary article. It may use an older cached copy or find several sites repeating the same claim. The final response can remain fluent even while its evidence becomes weaker.

That creates a source-selection bias: information that is easy to retrieve may receive more weight than information that is authoritative.

Consider an agent investigating a software vulnerability. A maintainer's explanation, a vendor advisory, and a general news report serve different purposes. If the maintainer's page cannot be accessed, the news report may still provide context. It should not silently become evidence the system never inspected.

Retrieval failures need to remain visible. “The source could not be accessed” is different from “the source was checked and contained no relevant information.” Several syndicated copies are also different from independent confirmation.

Without those distinctions, adding more results can create the appearance of confidence without adding evidence.

Search First, Then Read Selectively

A responsible workflow separates source discovery from deeper extraction.

Discovery identifies likely sources. Extraction reads the selected pages or documents closely enough to support the answer. This lets the agent narrow the evidence set before asking websites to render and deliver more content.

Cloudsway Search follows this pattern through SmartSearch and Reader. SmartSearch discovers relevant, source-backed information, while Reader extracts structured content from selected pages and documents. The application still has to decide which results deserve deeper reading and whether an earlier retrieval can be reused.

A shared retrieval record can be simple:

{
  "url": "https://example.com/source",
  "retrieved_at": "2026-09-01T08:30:00Z",
  "published_at": null,
  "content_hash": "sha256:...",
  "status": "available",
  "evidence_group": "vendor-advisory-42"
}

Parallel agents should be able to see that a URL has already been retrieved, whether the content changed, and which evidence group it belongs to.

This also helps prevent a common mistake: treating retrieval time as publication time. A page fetched today may contain information written months ago. Both timestamps can matter, but they answer different questions.

Fresh Does Not Always Mean Fetch Again

Real-time search is sometimes implemented as though every task requires downloading every source from scratch. Most information does not change at the same rate.

A service-status page may become stale within minutes. A historical specification might remain useful for years. Your freshness policy should reflect that difference.

Caching can reduce unnecessary work when the stored content is still suitable for the task and its use respects the applicable conditions. Conditional requests are another useful option: a server can report that content has not changed without sending the entire response again. Google's crawl guidance explicitly recommends supporting HTTP caching headers such as ETag and Last-Modified.

Where a publisher provides an API, feed, repository clone, or bulk download, that interface may be more efficient than repeatedly processing presentation-oriented pages. It is not automatically the right choice—APIs have limits, and cached data can be too old—but the access method should be an explicit design decision.

The goal is sufficiently current evidence for the task, rather than maximum retrieval activity.

A Practical Retrieval Review

Before deploying a research agent at scale, inspect one complete task and ask:

• How many unique sources did the agent discover?
• How many pages did it actually retrieve?
• How many retrievals repeated an earlier request?
• Which results were syndicated copies of the same evidence?
• Which primary sources were unavailable?
• Why was each source refreshed instead of served from suitable stored content?

These questions reveal waste that token and latency dashboards miss.

Robots.txt belongs in this review, but its scope should remain clear. RFC 9309 states that robots rules are not a form of access authorisation. Authentication, rate limits, contractual permissions, and downstream reuse remain separate concerns.

A search API does not automatically solve those concerns. It does not guarantee access to every source, reduce total crawling by itself, or establish permission to reuse every result. The outcome depends on both the provider's infrastructure and the way the application coordinates retrieval.

Retrieval Efficiency Is Part of Answer Quality

Web-connected AI depends on people and organisations continuing to publish useful information. Those sources need an affordable way to serve readers, search systems, and increasingly capable agents.

For developers, this makes retrieval efficiency more than infrastructure housekeeping. It affects which sources remain available, which evidence reaches the model, and whether the final answer can be verified.

A workflow that appears harmless in a demonstration can generate substantial duplicate traffic when multiplied across thousands of users and parallel agents. Shared retrieval state, selective reading, sensible caching, explicit failures, and evidence deduplication help prevent that outcome.

AI search will continue to need fresh information. Its long-term reliability depends partly on whether we retrieve that information with enough care to leave the original sources available for the next person—or agent—that needs them.