Uber QueryGPT: natural language to SQL — 10 minutes down to 3

• QueryGPT — Natural Language to SQL at Uber — Uber Engineering Blog, 2024-09-19
• A Deep Dive into QueryGPT: Uber's AI-Powered SQL Generator — Sequel (независимый разбор), 2024

Background

Uber is a data-driven company in the literal sense: pricing, routing, driver payouts, and demand balancing all run on a data platform that handles approximately 1.2 million interactive SQL queries per month. The biggest consumer is the operations organization, contributing about 36% of all queries. That is thousands of employees who need data daily but for whom SQL is not their core profession.

Internal analytics at Uber rests on thousands of datasets, and the key (tier-1) tables have grown to 200+ columns each. Authoring a query is not just syntax: you must find the right tables among thousands and understand the schema and the business logic of fields — how dates are computed in a given domain, or what a particular flag means.

QueryGPT was born bottom-up, not by executive decree: the idea was proposed at the internal Generative AI Hackdays hackathon in May 2023, when Uber was systematically hunting for LLM applications across the company. From the hackathon prototype to the production service described in the September 2024 engineering blog post, the Data Platform team went through more than 20 architecture iterations.

The case is valuable precisely for its engineering honesty: Uber published not a marketing announcement but a detailed teardown with version evolution, evaluation metrics, and unsolved problems — including hallucinations and evaluation non-determinism. The write-up is bylined by a team of eight Data Platform engineers — from a principal engineer to developers focused on LLM productivity; it is the product of an engineering organization, not an AI lab. For anyone building text-to-SQL for their own company, it is one of the most useful public documents in the industry.

Problem

An average query took its author about 10 minutes. The anatomy of those minutes: find the right tables among thousands of datasets, make sense of a two-hundred-column schema, recall domain conventions (how to filter test orders, which timezone dates are in) — and only then write the SQL itself. At a scale of hundreds of thousands of queries per month this is a huge productivity tax: operations teams spent time on query mechanics rather than analysis and decisions.

The naive approach — 'just give the LLM the schema and the question' — hit fundamental limits. The schema of a single large table consumed 40–60 thousand tokens, while model context windows before 128K-era models were 32 thousand. Queries often join several tables: the schema physically did not fit into the model, to say nothing of the cost and latency of such calls.

The second problem was people. Real user prompts range from detailed, keyword-rich formulations to five-word questions with typos. A naive pipeline expecting tidy input falls apart on live traffic: a short, context-free question carries too little signal either for table selection or for generating correct SQL.

And third: the cost of error. Generated SQL that looks plausible but uses a non-existent column or wrong business logic is worse than no answer — it either fails or, more dangerously, returns wrong numbers that decisions get made on. Users, as the team puts it, hold a high bar: queries should 'just work'.

Solution

The first version, assembled at the hackathon, was classic RAG: k-nearest-neighbor vector search over a small set of 7 tier-1 tables and 20 reference SQL queries, of which 3 tables and 7 samples were pulled into the prompt, plus instructions with Uber-specific conventions (date handling, business terms). The prototype proved demand, but accuracy fell on the real variety of questions and tables — and the next twenty-plus iterations turned 'one prompt with RAG' into a pipeline of specialized agents.

The first structural move was 'workspaces': curated sets of tables and SQL samples for a specific business domain. Production runs 12+ system workspaces (Mobility, Core Services, Platform Engineering, IT, Ads, and others), plus custom workspaces for narrow use cases. Curation is manual work by domain experts, and it — not the model — delivers most of the accuracy gain: search runs not across the whole platform but within a pre-digested subset.

A question then passes through the pipeline. The Intent Agent classifies the query into business domains and picks a workspace, narrowing the RAG search radius. The Table Agent selects specific tables — and shows the list to the user, who can confirm or edit it: the team added this human-in-the-loop step after complaints about wrong table selection. The Column Prune Agent strips irrelevant columns from schemas — this solved the token problem: even with GPT-4 Turbo's 128K window (model 1106), full schemas were expensive and slow, and pruning cut both latency and call costs substantially. A separate 'prompt enhancer' enriches terse user phrasings with context before generation.

Against hallucinations (non-existent tables and columns are the main failure mode) the team uses an iterative chat mode where the user refines the query, and experiments with a validation agent that recursively finds and fixes errors in the generated SQL.

The evaluation system deserves special attention. The team assembled a 'golden' set of question→SQL pairs from the service's real logs across domains and runs it in two modes: Vanilla (the full path from question to SQL) and Decoupled (intent and tables pre-set, to measure components in isolation). The signals: intent accuracy, overlap of selected tables with the reference (0–1), query execution success, non-empty result return, and an LLM-based similarity score against the reference SQL. A separate finding: evaluation runs are non-deterministic, with up to ~5% variance between runs with no code changes — so the team tracks error patterns over long periods rather than reacting to metric wobble.

Result

QueryGPT produces sufficiently reliable queries in about 3 minutes versus ~10 minutes of manual authoring — roughly 70% time saved per query. In limited release the service averages about 300 daily active users, 78% of whom say the generated queries reduce the time they would have spent writing by hand. At the platform's scale of 1.2 million queries per month the scaling potential is obvious — but the team is deliberately rolling out gradually, and in its published learnings explicitly names choosing the right initial audience (personas) as a lesson of its own: start where the benefit is highest and SQL needs are routine — with operations teams, not with data engineers, who need an LLM draft least of all.

The team's published learnings matter just as much. First: LLMs work excellently as classifiers on narrow tasks — a pipeline of specialized agents (intent → tables → columns) consistently beats one big prompt. Second: the user's question alone is insufficient input for generation; it must be enriched with context before the model call. Third: answer multiplicity (the same question is correctly solved with different tables and SQL styles) makes automated evaluation inherently fuzzy — hence an LLM judge and visual comparison against the reference instead of a binary match/no-match.

Frame the numbers correctly: 10 and 3 minutes are the Uber team's estimates; 78% is user self-report, not a stopwatch measurement; the service was in limited release at publication. Uber publishes neither a generation-accuracy percentage nor a financial impact — and, in our view, that is more honest than third-party blog extrapolations that paint the case with hundreds of thousands of saved hours.

In our view, QueryGPT's main transferable lesson is that production text-to-SQL is 20% model and 80% context engineering: curated domain workspaces, schema pruning, prompt enrichment, and evaluation discipline on a golden set. All of that transfers to any company with a large data platform — and requires neither your own model nor Uber's scale. The second lesson is pace: from hackathon to production took over a year and 20+ iterations. Teams expecting text-to-SQL to 'work within a sprint' underestimate precisely the long tail of domain tuning, not the difficulty of LLMs.

Lessons learned

1. Production text-to-SQL is a pipeline of specialized agents (intent → tables → columns), not one big prompt: LLMs are most reliable as classifiers on narrow tasks.
2. Curated workspaces with per-domain SQL samples beat feeding the model the entire schema: the main investment is manual curation by domain experts.
3. Schema pruning (Column Prune) is about both accuracy and cost: fewer tokens — cheaper, faster, better; even a 128K window does not cancel context economy.
4. User input cannot go into the model as-is: real questions range from verbose to five words with typos, and a 'prompt enhancer' before generation is mandatory.
5. Human-in-the-loop in the right place: user confirmation of table selection was added after real complaints — a cheap step that removes the top error source.
6. Evaluate on a 'golden set' built from real logs, measuring components in isolation; and mind LLM evaluation non-determinism (~5% between runs) — track error patterns, not metric wobble.
7. The honest quality framing is 'a sufficiently reliable draft in 3 minutes', not 'perfect SQL': a human stays as the reviewer — a condition of trust, not a product weakness.

Frequently asked questions