A Complete Web Retrieval Route for AI Agents: When to Use Search, Fetch, Crawlers, and Browsers

🌏 中文版

The previous article in this series covers how a research agent plans work and combines sources. This article narrows the scope to the retrieval layer: when an agent receives “find this information” or “read this URL,” should it call Search, direct Fetch, a Crawler, a Browser, or a controlled stealth browser?

The answer is not “use the strongest tool.” Build a route that escalates from cheap, predictable, and auditable operations to expensive operations with more side-effect risk. Success cannot mean HTTP 200; login forms, challenge pages, soft 404s, and empty JavaScript shells can all return 200.

Five routes, one responsibility each

Search and Fetch are alternative entry points, not fixed consecutive steps. If the user supplies a URL, Fetch first. If the user supplies only a question, Search first. Crawlers and Browsers are escalations. Stealth is not a fifth level with a higher success rate; it is a separate branch that requires explicit source authorization and policy approval.

This is also why SearXNG and Crawl4AI do not replace each other. The former answers “where might the answer be?” while the latter answers “what readable material exists inside this known site?”

Place seven common tools in the right layer first

The table assigns each tool its primary identity in the router; it is not an inventory of every feature the product offers. Decide whether the missing capability is candidate discovery, single-page reading, site traversal, or managed website automation before choosing a product. Do not arrange all seven names into one fixed fallback chain.

Valid combinations usually cross layers. Exa/Tavily/Linkup → Jina Reader fits “discover sources, then read a small set of public pages.” A Search API followed by Firecrawl or Crawl4AI fits “the candidate sites are known, but the answer spans pages.” Apify belongs on the branch where the work has become site-specific, repeatable, and suited to managed execution.

Every layer inherits this article's exit conditions: stop when content passes, when authorization or policy denies access, or when the remaining budget cannot finish the next stage.

Classify the task before classifying the failure

The router's first decision is the task shape, not the tool:

• discover: no trusted URL; find candidate sources.
• read: a URL exists; retrieve one page.
• traverse: read a section, documentation tree, or paginated set.
• interact: the answer appears only after client-side rendering or safe UI interaction.
• authenticated: use a user-authorized session after verifying scope and data isolation.

The second decision is why the previous stage failed. Do not compress every condition into fetch_failed; that leaves the agent blindly swapping tools.

RFC 9110 defines HTTP status semantics, but status is not a content-quality classifier. For example, 403 means that the server understood the request and refuses to fulfill it; it does not mean “try a browser that looks more human.” RFC 6585 states that a 429 response may include Retry-After; the router should use it instead of immediately resending.

A 200 response still needs content acceptance

Every retrieval should pass four layers:

1. Transport: the response is readable, redirects do not loop, and the content type is supported.
2. Page identity: canonical URL, title, primary language, and expected origin agree.
3. Content quality: body length, text density, required fields, and query relevance meet task-specific thresholds.
4. Policy: no auth boundary was crossed, no form was submitted, no prohibited path was accessed, and no challenge page was cited.

Thresholds belong to the task. A global minChars = 500 is wrong: an exchange-rate lookup may need one number, while a specification task needs sections and a version. Let the task declare required_fields, expected_content_types, and quality_checks; the router should execute that contract.

A Browser should not use “sleep three seconds” as its success condition. Playwright auto-waiting checks element state before actions. For retrieval, wait for an observable selector, response, or page state. The readiness condition belongs to the task contract, not a hard-coded delay.

Retry only errors that can recover on their own

Retryable conditions commonly include timeouts, connection resets, 408, 429, and selected 5xx responses. Non-retryable conditions commonly include unsupported formats, explicit 401/403/404/410/451 responses, policy denial, and confirmed login or challenge pages.

Even retryable conditions need all of the following:

• A per-stage maxAttempts, such as two Fetch attempts and one Browser attempt.
• Exponential backoff with jitter so workers do not collide again in lockstep.
• A task deadline; do not begin an attempt that cannot finish in the remaining time.
• An idempotency constraint; this route is read-only by default and must not replay state-changing actions.

Switching tools is not a retry. Escalating Fetch to Browser adds latency, CPU, memory, cookies, and interaction risk, so record it as an escalation.

Budget and depth define when to stop

A fallback tree without budgets becomes brute-force search. Every task needs at least:

type RetrievalBudget = {
  deadlineMs: number
  maxRequests: number
  maxSearchQueries: number
  maxPages: number
  maxDepth: number
  maxBrowserStarts: number
  maxCostUsd?: number
}

maxDepth limits how far a crawler walks from its seed. maxPages limits total expansion. maxBrowserStarts prevents launching a browser for every candidate. On exhaustion, return budget_exhausted with partial evidence; do not disguise it as “not found.”

Define positive stopping conditions too. Stop when required facts, source count, and freshness all pass. Reading more pages does not automatically make an answer more reliable.

Cache, deduplication, and provenance belong in the router

A cache key needs at least canonical URL, representation variant, and auth scope. URL alone is unsafe for authenticated content because it can return user A's page to user B. RFC 9111 separates fresh, stale, and validated responses. Prefer conditional requests with ETag or Last-Modified instead of always refetching or always trusting old data.

Deduplicate at three levels:

• URL normalization removes tracking parameters and fragments while preserving query parameters that change content.
• Redirect and canonical identity merge aliases while retaining the original chain.
• Content fingerprints process the same body once while preserving provenance when multiple origins syndicate it.

Every artifact should record requested_url, final_url, retrieval time, route, status, content type, content hash, cache state, auth scope, parent page, and search query. This provenance lets a claim trace back to a specific representation. W3C PROV-O provides a general model for entities, activities, agents, and derivations.

An executable router shape

The code below omits provider SDKs but retains the contracts that control behavior: classification, budget, policy, and trace.

async function retrieve(task: Task, ctx: Context): Promise<Result> {
  const trace = ctx.trace.start(task)
  const candidates = task.url
    ? [{ url: normalize(task.url), discoveredBy: "user" }]
    : await searchWithBudget(task.query, ctx.budget, trace)

  for (const candidate of dedupe(candidates)) {
    if (!ctx.policy.mayFetch(candidate.url)) continue

    const cached = await ctx.cache.get(candidate.url, ctx.authScope)
    const fetched = await boundedFetch(candidate, cached, ctx, trace)
    const fetchVerdict = classify(fetched, task)

    if (fetchVerdict.kind === "usable") {
      const result = task.mode === "traverse"
        ? await crawlWithinBudget(candidate, task, ctx, trace)
        : toArtifact(fetched, trace)
      if (satisfies(result, task)) return trace.complete(result)
    }

    if (fetchVerdict.kind === "js-shell" && ctx.budget.browserStartsLeft > 0) {
      const rendered = await browseReadOnly(candidate, task.waitFor, ctx, trace)
      const browserVerdict = classify(rendered, task)
      if (browserVerdict.kind === "usable" && satisfies(rendered, task)) {
        return trace.complete(toArtifact(rendered, trace))
      }
      if (browserVerdict.kind === "challenge") {
        return trace.stop("challenge", { retryable: false })
      }
    }

    if (fetchVerdict.kind === "auth-required") {
      if (!task.requiresAuth || !ctx.authScope) continue
      const authorized = await browseWithAuthorizedSession(candidate, ctx, trace)
      if (satisfies(authorized, task)) return trace.complete(authorized)
    }

    if (!fetchVerdict.retryable) trace.recordStop(candidate, fetchVerdict.kind)
    if (ctx.budget.exhausted()) return trace.stop("budget_exhausted")
  }

  return trace.stop("no_acceptable_source")
}

In production, boundedFetch owns redirect caps, timeout, Retry-After, backoff, and cache validation. classify owns soft 404, login-page, challenge, JS-shell, and content-quality detection. Keeping them separate prevents transport retries and tool escalation from collapsing into one ambiguous loop.

The default route

A safe, practical default is:

Question only → Search → Fetch candidates
Known URL → Fetch
Site section → Fetch seed → Crawler
JS shell or safe interaction → Browser
Authenticated content → Authorized Browser session with auth-scoped cache
Challenge or explicit denial → Stop or switch to a permitted public source
Every stage → Stop when content passes; also stop when budget is exhausted

The goal is not to make an agent “get into every site.” The goal is to give every escalation an explainable failure signal, cost, and policy reason. The next article turns these contracts into a fixed corpus and regression gates: How to Evaluate Agent Search Quality: Building a Web Retrieval Benchmark.

Update record

• 2026-08-22: Added a cross-layer selection matrix, valid combinations, and exit conditions for Exa, Tavily, Linkup, Jina Reader, Firecrawl, Crawl4AI, and Apify.

References

• RFC 9110: HTTP Semantics
• RFC 9111: HTTP Caching
• RFC 6585: Additional HTTP Status Codes
• RFC 9309: Robots Exclusion Protocol
• Playwright: Auto-waiting
• W3C PROV-O: The PROV Ontology
• Exa Search API Guide
• Tavily Search API
• Linkup Search Overview
• Jina AI Reader API
• Firecrawl Advanced Scraping Guide
• Crawl4AI Quick Start
• Apify Actors
• On this site: Building a Web Retrieval Benchmark