Writing Search Queries for Agents: Keywords, Semantic Descriptions, Decomposition, and Rewriting

🌏 中文版

Search quality does not come from the search engine alone. When an agent forwards “please look this up” unchanged, it treats conversational language as a retrieval language. That occasionally works, but it is not controllable.

A more reliable approach treats the query as compiled output. First determine whether the task seeks an identifier, a particular kind of page, an unnamed concept, or a set of scholarly records. Then translate that need into syntax the target provider understands. This article covers query generation and correction before an individual search call. It does not cover planning or stopping a multi-step research task. Multi-path retrieval inside a vector database is covered separately in Multi-Query Expansion.

Decide which query you are writing

The same information need can compile into at least four targets:

Length does not determine the class. CVE-2026-1234 is short, but every character matters. “An engineering article about observability for AI agents operated by small teams on Cloudflare Workers” is long but suited to semantic retrieval. “Published in the last five years, retrieval in the title, excluding reviews” should compile into fields and filters in a scholarly database.

Before searching, an agent should at least produce this intermediate representation:

{
  "intent": "exact_lookup | keyword | semantic | fielded",
  "must_keep": ["product names, error strings, or identifiers kept verbatim"],
  "concepts": ["concepts that may be expanded with synonyms"],
  "source_preference": ["official", "primary"],
  "filters": {
    "domains": [],
    "published_after": null,
    "language": null,
    "document_type": null
  },
  "queries": []
}

Separating must_keep from concepts matters. A model may expand “error handling” into exception handling, but it should not replace CrawlerRunConfig with a class name it guessed.

Keyword queries: preserve discrimination, then remove words

A keyword query is not merely a question shortened to three nouns. It retains the lexical anchors that exclude the wrong pages.

Apply four operations in order:

1. Preserve error messages, symbols, product names, versions, and identifiers verbatim.
2. Remove conversational framing such as “please find” or “what is the latest.”
3. Add one term that identifies the page type, such as release notes, API reference, or security advisory.
4. Put source, time, language, and file-type restrictions in native parameters when the provider offers them.

Google's official documentation confirms that site: restricts a domain or URL prefix and filetype: restricts a file type. These operators are not a universal standard. The SearXNG Search API passes queries to external search services and explicitly warns that syntax understood by one upstream may not be honored by another. With SearXNG, prefer API parameters such as engines, language, and time_range. If you depend on an upstream operator, pin the engine and record it.

{
  "q": "\"CrawlerRunConfig\" \"wait_for\"",
  "engines": "google",
  "language": "en",
  "format": "json"
}

More quotation marks do not automatically mean more precision. Exact strings are appropriate for errors and symbols. Quoting every ordinary concept can block synonyms, inflections, and alternative word orders. Lock one genuinely immutable anchor first and leave the remaining concepts available for recall.

Semantic queries: describe the page you want

For semantic search, write how an ideal result would be introduced rather than supplying a bag of keywords. Exa's Search Best Practices says that query supports long, semantically rich descriptions and uses blog post about embeddings and vector search as an example.

A useful semantic query usually contains four parts:

[page type] + [topic] + [required conditions] + [desired evidence]

Instead of:

React Vue performance

write:

An engineering article comparing React and Vue rendering performance,
with a reproducible benchmark setup, measured results, and stated limitations.

engineering article, reproducible benchmark setup, and stated limitations describe properties of the ideal page. If the API has includeDomains, date, or category parameters, continue to use them. Do not blend “official sites only, after 2026, in English” into natural language and expect identical interpretation on every call.

Decomposition is not twelve paraphrases

One query should own one verifiable information need. This question should be decomposed:

Compare Tavily and Exa on pricing, data retention, search capability, and self-hosting.

A useful decomposition is:

1. Tavily official pricing API credits
2. Exa official pricing API credits
3. Tavily official privacy data retention API queries
4. Exa official privacy zero data retention API queries
5. Tavily self-hosted official
6. Exa self-hosted official

The reason is not that more calls are inherently stronger. Different claims have different authoritative sources. Pricing belongs on pricing pages, retention belongs in privacy or security documentation, and self-hosting requires deployment documentation or a repository. When six needs are placed in one sentence, ranking must guess which one matters most.

Query expansion is a different operation. It preserves one information need while changing terms likely to appear in documents. data retention may get separate variants for query storage and zero data retention; the Taiwanese term 隨身碟 may get an English USB flash drive variant. Every variant should retain the same claim_id, so merged results do not masquerade as independent evidence.

A practical budget is: begin with one precise query and add two variants only when there is a concrete sign of missed recall. This is a cost guardrail, not a claim that three queries are universally optimal.

Scholarly search: compile natural language into database grammar

Scholarly APIs do not share one query syntax. Applying Web-search habits to them often does not produce an error; it quietly searches the wrong fields.

• PubMed Help explains that untagged terms go through Automatic Term Mapping, while tags such as [tiab], [mh], and [dp] change processing and Boolean operators must be uppercase. It also warns that a quoted phrase absent from the phrase index does not necessarily behave like exact match in a general Web engine.
• OpenAlex Search currently exposes search, search.exact, and search.semantic, with only one allowed per request. Boolean, phrase, proximity, and wildcard syntax have their own rules.
• The Crossref REST API separates text queries from exact filters. Publication dates belong in from-pub-date and until-pub-date, not merely as a year inside query.bibliographic.
• The Semantic Scholar Academic Graph API distinguishes paper relevance search from bulk search. Bulk text queries match title and abstract with Boolean support, while filters and returned fields are separate parameters.

An agent should therefore not generate one “scholarly search string” and paste it everywhere. It should first represent concept groups, then let each adapter compile them:

concept A: retrieval-augmented generation OR RAG
concept B: query rewriting OR query reformulation
date: 2023-01-01..
document type: research article

The same intermediate form must produce different output for PubMed, OpenAlex, and Semantic Scholar.

Twelve fixed cases: a rerunnable specification, not a leaderboard

The table below is a query regression set. As of 2026-08-22, this site has not stored cross-provider raw results for these twelve cases. “Expected change” is therefore a direction to validate, not an observed ranking or success rate. A real run should retain the provider, full request, timestamp, first results, and human judgment before making any quality claim.

Q10–Q12 are syntax examples, not complete systematic-review strategies. A real review must also address controlled vocabulary, database coverage, deduplication, and human screening; those concerns belong in the later scholarly-search-pipeline article.

Rewrite after failure: change one control at a time

When results are poor, do not merely ask an LLM to “rewrite this better.” Classify the symptom first.

Changing one control at a time preserves causality. A useful trace retains at least:

{
  "case_id": "Q03",
  "claim_id": "astro-current-stable-version",
  "provider": "provider-name",
  "request": {},
  "rewrite_reason": "source-mismatch",
  "parent_query_id": "q-001",
  "searched_at": "ISO-8601 timestamp",
  "raw_result_path": "artifacts/search/q-002.json"
}

Without the raw response, do not retain only “version two was better.” Search indexes, rankings, and APIs change. A rerunnable query plus the response at the time is the artifact that can actually be debugged.

A sufficient agent loop

User need
  ↓
Extract must_keep, concepts, source, time, language, and document type
  ↓
Classify exact / keyword / semantic / fielded
  ↓
Compile provider query + native filters
  ↓
Inspect the first results: on-topic, primary, and fresh enough?
  ↓
Relax, narrow, expand, or decompose according to the symptom—one at a time
  ↓
Store request + raw results + judgment

Define the stopping condition at the query layer as well. Stop once a primary source directly supports the claim. If two consecutive synonym-only rewrites add no qualified source, do not continue indefinitely. Switch providers, query a known database, or report the evidence gap.

Overall

The central skill in agent search-query writing is not asking an LLM for more keywords. It is separating what cannot change, what may expand, what belongs in a filter, and what kind of page or record the task actually needs.

Keyword queries depend on lexical anchors. Semantic queries describe ideal pages. Scholarly queries use the target database's fields and Boolean grammar. After failure, relax or tighten one condition at a time, retaining the parent query and raw results. The query then stops being a string that disappears inside a prompt and becomes a retrieval artifact that can be versioned, regression-tested, and audited.

The next article moves to “fetching correctly”: a fixed-URL comparison of how Crawl4AI, Firecrawl, Jina Reader, and Readability preserve body text, tables, code, and metadata.

References

• SearXNG Search API: queries and upstream syntax
• Google Search Central: site: and filetype: operators
• Exa Search Best Practices: long semantic queries, filters, and search types
• OpenAlex Search: exact, Boolean, phrase, proximity, and semantic search
• Crossref REST API Filters
• Crossref REST API Tips: query, filter, and select
• Semantic Scholar Academic Graph API
• PubMed Help: Automatic Term Mapping, field tags, and Boolean operators
• Related on this site: Exa guide, SearXNG guide, Web Retrieval Fallback Routing, and Multi-Query Expansion