Table of Contents
- Agenda: structured queries and hybrid retrieval
- The query is an executable hypothesis
- Evaluation goes beyond SQL strings
- The complete database-agent dataflow
- Few-shot parsing on small schemas
- Enumerated values connect schema and content
- Empty results are a diagnostic branch
- Execution metrics and spot checks
- From routing to hybrid QA
- A maintainable test suite
- Monitor schema drift and data updates separately
- When not to use an LLM semantic parser
- Route hybrid data before combining it
- A practical implementation exercise
- Material gaps
- References
🌏 中文版
This guide reconstructs the lecture from the official Fall 2025 deck; system descriptions and reported results below are attributed to that historical course material unless a paper is linked at the claim.
Lecture 6 replaces free text with databases. The main challenge is no longer placing relevant prose in context. It is translating user constraints into a precise query that matches the schema. This remains the Fall 2025 Conversational Virtual Assistants course, not the renamed Autumn 2026 syllabus.
Agenda: structured queries and hybrid retrieval
The deck defines relational databases, schemas, and NL-to-SQL semantic parsing; then covers small and large enumerated domains, empty results, and evaluation traps. Its second half expands to text, tables, and Wikidata through query classification, retrieve-and-read, and multi-source combination. (lecture source)
The query is an executable hypothesis
For “a Japanese restaurant in Palo Alto,” a parser must identify location and cuisine fields, know whether cuisine is an array, and respect types. SQL is both an intermediate representation and an inspectable hypothesis. Syntax failure, an empty set, and an incorrectly filtered set demand different fixes. (lecture source)
Small enumerations can fit in the schema description. When choices are too numerous, the lecture introduces enum classification or retrieval before parsing. Mapping language to a legal value prevents the model from generating a string absent from the database.
Evaluation goes beyond SQL strings
Different SQL programs can return the same correct result, so exact match rejects valid queries. Identical results can also conceal an incorrect condition. The lecture combines execution results, query precision, and manual spot checks. Empty results require diagnosis: absent data, wrong field, or wrong enumeration? (lecture source)
The complete database-agent dataflow
The lecture separates language, a schema-aware parser, SQL compilation, DB/API execution, results, and response generation. Each boundary exposes an artifact: formal query, returned rows, and the fields used in the answer. Schema descriptions must include names, types, relationships, and local conventions without dumping all data into context. (lecture source)
Formal queries also create a permission boundary. A read-only assistant can permit only SELECT against approved views, validate the AST, and cap runtime and result size. Language flexibility does not imply flexible database authority.
Few-shot parsing on small schemas
Examples teach schema-specific mappings—array membership, local distance functions, and range conventions—not basic SQL syntax. Select examples by operators: equality, membership, range, ordering, aggregation, and conjunction. (lecture source)
Test unseen values and spelling variants because a model can copy entities from demonstrations. This exposes the enumerated-value problem rather than rewarding memorization.
Enumerated values connect schema and content
A type does not reveal whether the database stores Japanese, japanese_food, or an internal ID. Ten legal values can fit in a schema prompt; thousands of brands and locations cannot. The lecture adds enumeration classification or retrieval before parsing. (lecture source)
The resolver should return candidates and confidence. Ambiguous names can refer to organizations, places, or several category tags. Clarification is safer than silently collapsing ambiguity to one value.
Empty results are a diagnostic branch
Zero rows can mean absent data, overly strict constraints, bad enum mapping, or bad parsing. A runtime should preserve the query, test targeted relaxation, and ask a specific clarification. Repair attempts need explicit changes and budgets so the agent does not delete constraints until something appears. (lecture source)
No-result cases belong in evaluation. Gold behavior includes constraints that must remain and clarifications that are allowed, not only an empty set.
Execution metrics and spot checks
Execution accuracy accepts equivalent SQL but can pass an incorrect query by accident when the test database lacks distinguishing rows. Manual review and counterexample rows help expose omitted constraints. (lecture source)
Query precision and recall matter for recommendations. Returning one correct item among many invalid ones is not success. Result-set quality and ranking should remain separate from final prose.
From routing to hybrid QA
A binary classifier can route fixed-field questions to semantic parsing and review questions to textual QA. Questions combining price filters and review meaning require composition. Multi-source retrieve-and-read must retain source-specific subqueries and evidence. (lecture source)
HybridQA often hops from a table cell to a passage or extracts a textual value before comparing table rows. A one-time router cannot represent that sequence. SUQL, in the next lecture, preserves it inside an executable language.
A maintainable test suite
Build a matrix covering fields, operators, joins, and enumerations, then add paraphrases, misspellings, and unseen entities. Save gold intent, result set, required constraints, and accepted equivalent queries. (lecture source)
Author extension: Add empty results, invalid enums, ambiguous entities, schema drift, timeout, oversized results, and injection-like requests. Hybrid tests should identify each source and the handoff between steps. Production traces need schema version, rewritten question, enum candidates, query, execution status, row count, and cited fields.
Monitor schema drift and data updates separately
Schema drift breaks parsing when columns, types, or enumerated values change; data refresh changes answers even when a query remains valid. Version the schema snapshot used by the parser and the data snapshot used for execution so a legitimate new result is not mislabeled as a parser regression. (lecture source)
When not to use an LLM semantic parser
For a small fixed menu, templates or a typed form may be cheaper and easier to prove correct. High-impact writes also require explicit authorization and deterministic validation beyond plausible SQL. Linguistic variability should justify the model. (lecture source)
Route hybrid data before combining it
Some questions need SQL filters and aggregation; others require reading reviews; still others combine tables, text, and a knowledge graph. The deck compares binary routing and multi-source retrieve-and-read. Combining sources does not mean concatenating everything: each sub-answer still needs source provenance. (lecture source)
A practical implementation exercise
Author extension: The following is an implementation or review method derived from the lecture, not a result reported by the deck.
Write ten questions for one restaurant table and save gold result sets rather than only gold SQL. Include enumeration synonyms, no-result cases, ordering, aggregation, and questions requiring review text. Label each failure schema, enum, routing, execution, or generation before changing a component.
Material gaps
Author extension: The following is an implementation or review method derived from the lecture, not a result reported by the deck.
The slides survey architectures and literature without a single complete reference implementation. Their benchmark tables do not establish cross-dataset generalization. No recording or full speaker notes are public.
References
Loading...