Accuracy Benchmark Design for Text-to-SQL Analytical Tools

Text-to-SQL accuracy benchmarks test whether tools can turn plain-language questions into database queries. But their scores often promise more than they prove. A reliable benchmark must resemble the enterprise where the tool will run, use trustworthy answers, and measure more than whether two queries happen to return the same rows.

Why text-to-SQL benchmark scores don't mean what they appear to mean

The Spider leaderboard tells an encouraging story. Execution accuracy for state-of-the-art systems climbed from roughly half to over 90% between 2020 and late 2023. That looks suspiciously like a solved problem. Yet those scores came from conditions that differ sharply from enterprise production, where column names can be cryptic, join keys implicit, and documentation absent. Public benchmarks served a real purpose, but success on their clean schemas is not a passport to every warehouse.

The dominant failure is semantic. A model can fabricate column names, table joins, filter values, or aggregation logic that look plausible but do not fit the schema or what the user is asking. The SQL may look perfectly respectable in a meeting. That does not make it right.

Analysis of thousands of incorrect queries across four state-of-the-art systems on Spider and BIRD finds that schema errors cause the large majority of execution failures. Semantic errors, in which a model misreads the question or schema, account for more than a third of BIRD errors. These are not merely punctuation mistakes that a parser can catch.

How annotation errors corrupt the scores further

Jin et al.’s CIDR 2026 analysis finds many annotation errors in BIRD Mini-Dev and Spider 2.0-Snow. BIRD Mini-Dev’s error rate is 52.8%, with a comparably high rate in Spider 2.0-Snow. The trouble starts even before the test has to deal with the complexity of an enterprise. The reference answers used to judge the tools are themselves unreliable.

The analysis identifies four patterns, including gold SQL that mismatches the question’s intended logic, gold SQL that mismatches the actual data, misannotated domain knowledge, and ambiguous questions. An ambiguous question, for instance, leaves the intended answer unsettled even before a model writes SQL. Calling one query “gold” does not settle the dispute.

Correcting the annotations substantially shifts individual agents’ scores and moves leaderboard positions by multiple places. One agent ranked fourth among five examined agents jumps to first. That is more than a cosmetic adjustment. The system itself stays the same, but the change to the answer key makes it look stronger. For a buyer interpreting a vendor’s BIRD or Spider 2.0-Snow ranking, the unresolved question is therefore whether the tool understood the task or merely agreed with the answer key’s mistakes.

What execution accuracy, the field's primary metric, structurally cannot measure

Execution accuracy, usually shortened to EX, asks whether a generated query returns the same result set as the gold query. That sounds admirably practical. But agreement on one database is not the same as agreement in logic. Different queries can happen to return the same results on the test data, so a wrong query can get a passing grade.

An EDBT 2025 analysis distinguishes that false-positive problem from the false negatives produced by exact matching. Exact matching can reject logically equivalent queries because they are written differently, while execution-based comparison can accept different logic because its results coincide. The scoring problems pull in opposite directions. Cleaning the answer key cannot fix a rule that confuses accidental agreement with correctness, and an aggregate score conceals which judgments went wrong.

FLEX, or False-Less EXecution, uses large language models to emulate expert evaluation of SQL queries. It achieves stronger agreement with human experts, with Cohen’s kappa of 87.04 compared with EX’s 62.00. That makes it evidence for richer query comparison.

Correctness also leaves another production question unanswered. How much work did the query and its generation require? Structural efficiency, complexity inflation, execution speed, and token consumption disappear inside a single pass/fail score. SQLyzr reports correctness along with efficiency, structural complexity, and generation cost, and breaks the results down by category.

What happens when enterprise schema complexity is actually tested

BEAVER, published in PVLDB by researchers from MIT, Harvard, Intel, TU Munich, and AWS AI, is the first text-to-SQL benchmark derived from private data warehouses. It uses thousands of question-SQL pairs from real query logs that span hundreds of tables and more than one domain. The schemas are large, heterogeneous, and continuously evolving. Cryptic names, implicit joins, and domain knowledge are part of the task.

State-of-the-art agentic frameworks achieve only 10.8% accuracy on BEAVER, far below the scores that the same model class earns on Spider and BIRD. What happens if the model gets help? Even when oracle hints solve every hard subtask, accuracy improves greatly but stays far below the level reached on simpler benchmarks. Finding the right schema is therefore only part of the difficulty, because the SQL logic remains demanding after those clues are supplied.

BEAVER separates evaluation into multi-table retrieval, join key detection, column mapping, domain knowledge extraction, and query decomposition. Scoring each of these on its own shows where things go wrong. A single failed query otherwise says little about whether the system chose the wrong tables, misunderstood a column, or could not break down the question.

Healthcare evaluations reveal a related weakness in generalization. Models that perform strongly on original dataset splits drop sharply on new splits designed to measure it. Those results indicate memorization of dataset patterns. A benchmark can therefore look demanding while still rewarding familiarity, making the choice of test questions as consequential as the size of the schema.

The dimension standard benchmarks ignore entirely: access control and security

A query can answer a question correctly and still be forbidden. Standard benchmarks ask whether the result is right. Syntactically valid SQL can expose restricted email addresses, phone numbers, salaries, account balances, or health data.

A user may be allowed to query a table but barred from particular columns, so field-level authorization cannot be inferred from the query’s correctness. When the platform enforces row-level and column-level permissions, it guards against part of this risk regardless of what the model does. An evaluation that includes such controls is testing a different safety boundary from one that tests the model alone.

Schema inference research demonstrates another route to exposure. Attacks on text-to-SQL systems can reconstruct table names with near-perfect accuracy by exploiting their dependence on schema information. The warehouse’s rows are not the only sensitive asset. Catalog descriptions, metric definitions, and query history can also reveal business information, so authorization, audit, retention, and minimization policies need to cover that metadata.

No current public benchmark measures these risks. That omission limits what a high score can establish about deployment readiness. A tool can excel on every leaderboard and still require additional controls for a regulated or security-conscious environment. Accuracy and permission are separate questions, and neither answers the other by implication.

The practitioner debate that benchmark design must settle: raw text-to-SQL versus a governed semantic layer

What should the model be allowed to query? Raw text-to-SQL works against tables, while a governed semantic layer places a defined business model between the question and the database. The choice of which one a benchmark tests shapes what its accuracy claim says before scoring even starts.

dbt Labs’ 2026 benchmark finds notably higher accuracy for semantic-layer-grounded queries than for raw text-to-SQL on the same frontier models. In that comparison, architecture becomes a stronger determinant of accuracy than model choice. A 2026 arXiv paper on a semantic-layer-mediated agent also reports markedly high execution accuracy on a text-to-SQL benchmark.

Jason Ganz of dbt Labs frames the trade-off through how failure appears. Raw text-to-SQL can produce a plausible but incorrect answer, while a semantic layer can produce an error message. The visible refusal has an advantage for non-technical users. But a semantic layer must be modeled ahead of time, so it leaves out undefined ad hoc questions and may leave gaps that go unnoticed. Raw text-to-SQL remains appropriate for prototyping and for technical users who can inspect generated SQL and catch a bad join.

An evaluation needs to state what it assumes about the architecture being tested. Tests of bare-schema generation leave out the strongest architecture for governed self-serve analytics, while tests of only modeled questions leave out open-ended technical exploration. Vendor scores become interpretable only within that distinction. Without that distinction, a comparison may make the model look better when the architecture is a better fit.

Schema alone was not sufficient context. Historical query patterns supplied information needed for the task, making a schema-only evaluation a stripped-down test.

BenchPress, presented at CIDR 2026 by researchers from MIT, TU Munich, Intel, and AWS, tackles the corresponding problem of building trustworthy test questions. Given SQL from an enterprise log, it uses retrieval-augmented generation and LLMs to propose multiple natural-language descriptions. Human experts then select, rank, or edit those drafts for accuracy and domain alignment. The division of labor is deliberate. SQL logs are abundant, but matching natural-language annotations are scarce, and fully manual curation is too expensive at scale.

The five design choices that close the gap between benchmark scores and production accuracy

The first design criterion is schema realism. A production-oriented benchmark should use actual warehouse or production schemas, retaining large table counts, cryptic names, implicit join keys, and domain-specific abbreviations. Those difficulties belong in the test. BEAVER provides the model for this choice, while BenchPress offers a way to reduce the annotation burden without substituting a tidier task.

The second criterion is annotation quality control. Expert curation, AI-assisted draft generation, and human review should form multiple layers of verification rather than leaving ground truth to a single annotator. The question needs scrutiny as well as the SQL. An accurately transcribed query is not a reliable reference if its paired question asks for something else.

The third criterion is measurement beyond all-or-nothing execution accuracy. Evaluation should reject coincidentally matching results, credit logically equivalent queries, and report performance across query-complexity categories. Subtask scoring adds the missing diagnosis. BEAVER’s breakdown shows where reasoning fails, while FLEX shows how expert-style comparison can handle more queries without a person reviewing every query pair. Efficiency and generation cost belong in the report.

The fourth criterion is workload alignment. Questions should come from actual query logs or representative users. Realism includes frequency and difficulty. Historical patterns help define the workload, while taxonomy-guided synthesis can cover rare but important cases that log sampling misses by combining query intent types, statement types, syntactic structures, and key actions.

The fifth criterion is security and access-control coverage. Evaluation should test field-level permissions, restricted-column exposure, and resistance to schema inference attacks as first-class requirements. A system guarded by row-level and column-level controls deserves evaluation as a complete system, but the model’s behavior without those controls still needs to be measured.

Teams deploying self-serve analytics need not wait for a universal benchmark. BenchPress’s pipeline and BEAVER’s subtask framework provide a route to domain-specific evaluation against their own schemas and query history. The useful accuracy claim is then about the questions, data, architecture, and permissions the deployment will actually face.