Grab: LLM data classification — 20,000+ entities in the first month and 360 man-days saved per year

• LLM-powered data classification for data entities at scale — Grab Engineering Blog (Caspian & Data Governance), 2023-10
• Metasense V2: Enhancing, improving and productionisation of LLM powered data governance — Grab Engineering Blog (Buhrer, Parbat, Zeng), 2024-11-14
• Grab: LLM-Powered Data Classification System for Enterprise-Scale Metadata Generation (сторонний разбор) — ZenML LLMOps Database

Background

Grab is Southeast Asia's leading superapp: ride-hailing, food delivery, and financial services in one application. The superapp model means superapp data: the company manages a petabyte-scale landscape — countless database tables and Kafka message schemas continuously created and modified by dozens of product teams.

For regulatory compliance and internal policy, every data entity must be classified by sensitivity: does it contain personal data, and which confidentiality tier does it belong to. This is not bureaucratic formality but the foundation of the whole protection system: at Grab, sensitivity tags determine table access levels and drive attribute-based access control (ABAC) policies and dynamic data masking in queries. An unclassified or misclassified table is either a hole in personal data protection or excessive restrictions blocking teams' work.

The problem was tackled by the Caspian (data engineering) and Data Governance teams. In October 2023 they described on the engineering blog how they moved data classification from a manual process to LLMs — one of the first public cases of applying generative models to data governance. A telling detail: the orchestration service that embeds the LLM into the data platforms is called Gemini — no relation to Google's model of the same name, as the post predates its announcement. In November 2024 the team published a sequel — Metasense V2 — on upgrading the system to cover the entire data lake, making this a rare documented example of an LLM system evolving across two generations.

Problem

Manual data classification does not scale. Data owners had to review every table and every column and assign sensitivity tags by hand — slow, costly, and error-prone: an engineer labeling their hundredth table of the week inevitably loses focus. On a petabyte-scale landscape where new entities appear daily, the backlog of unclassified tables and Kafka schemas grows faster than it can be cleared — and compliance risk grows with it.

The team's alternatives were imperfect. Hard heuristics (regular expressions over column names) don't understand semantics: a user_note column can contain anything from a technical comment to an address and a phone number. The third-party classification service the team evaluated in parallel with LLMs required predefined rules and struggled with column semantics in the context of Grab's business. Training an in-house ML model would mean first assembling a large labeled dataset — that is, solving the same labeling problem, with ML infrastructure on top.

Generative models offered a third path: an LLM has enough 'general knowledge' to tell from column names and descriptions whether it is looking at a phone number, an address, or a device identifier, and classification requirements can be expressed in plain prompt text — no code, no model training. But that path came with engineering constraints: GPT-3.5's context is just 4,000 tokens (roughly 3,000 words), and the Azure OpenAI quota is 240K tokens per minute shared across all of the company's deployments.

Solution

The team wired GPT-3.5 into Gemini — an orchestration service that takes classification requests from data platforms. The architecture is built for quotas and scale: requests are aggregated in message queues, a workflow-level rate limiter keeps the stream within the Azure OpenAI limit (240K tokens per minute), and an entity's metadata — column names and descriptions — is packed into a prompt within the 4,000-token context; overly wide tables are processed in parts. In parallel with the LLM, a third-party classifier ran during the evaluation period — a direct engine comparison on the same data stream.

Quality was achieved through prompt engineering, with no custom model training: clear requirements per tag, few-shot examples, output constrained to a strict schema via DTO definitions, and — an important detail — a default <None> tag for uncertain cases so the model doesn't force dubious classifications. The team described the resulting accuracy as 'surprisingly accurate'. Results flow back to the data platforms via Kafka, and data owners receive weekly verification notifications to confirm or correct the proposed tags. The human stayed in the loop, but the role changed from labeler to validator.

The whole protection stack runs on the tags: table sensitivity tier determination, attribute-based access control policies, dynamic data masking in queries, and data discovery. The team also built analytics pipelines to score the quality of every prompt version — prompts are versioned and tested like code.

The system's second generation — Metasense V2 (November 2024) — fixed weaknesses found on live traffic: the model was confused by large mixed samples where personal data hid among non-personal (business emails with legal names, nested JSON, passenger communications). The fixes are notable for their simplicity: one model was split into two — PII tags separately from the rest; the tag count in the PII part was cut from 21 to 8; the prompt was trimmed from 1,254 to 737 words; tables wider than 150 columns are split into chunks. For rapid prompt experimentation the team adopted LangChain and LangSmith, and set up automated alerts on misclassification thresholds in case of quality degradation.

Result

In the first month after rollout the system scanned more than 20,000 data entities — averaging 300–400 per day, a pace physically unreachable for a manual process. In a September 2023 survey, 80% of data owners said the new process helped them tag their entities, and for acknowledged tables users changed less than one tag on average — meaning the vast majority of the model's proposals were accepted unchanged. At two minutes of manual classification per entity, the automation saves roughly 360 man-days per year. By V2 the system covered Grab's entire data lake with what the team calls 'exceptionally low' misclassification rates — though the company publishes no exact percentages, and all the case's figures are self-reported without external audit.

In our view, this case is a model of sober task selection for LLMs. Metadata classification is a scenario where a generative model is nearly ideal: the input is compact (column names and descriptions), the output is structured (tags from a fixed taxonomy), the cost of a single error is bounded (a human validator and weekly notifications), and the alternative is not 'another model' but thousands of hours of manual labor. The economics is calculated conservatively and legibly: 2 minutes × 20,000+ entities a month is arithmetic any CFO will accept — unlike abstract 'productivity percentages'.

Our second observation: the V1→V2 history honestly shows that an LLM system is not 'set and forget'. The first version, 'surprisingly accurate' in 2023, accumulated a list of weaknesses on live traffic — and they were cured not with a more powerful model but with task decomposition, halving the prompt, and observability (LangSmith, threshold alerts). That is perhaps the case's most transferable lesson: in production LLM systems, the engineering around the model — orchestration, quotas, output schemas, prompt versioning, monitoring — matters more than the choice of the model itself.

Lessons learned

1. Data governance is an underrated LLM use case: sensitivity classification from metadata needed no custom models — GPT-3.5 plus prompt engineering was enough.
2. Count the impact in man-days: 2 minutes per entity × 20,000+ entities a month = ~360 man-days a year — arithmetic any executive understands.
3. Keep a human in the loop as validator: data owners confirm tags, and the 'less than one correction per table' metric shows system maturity.
4. Design for API quotas from day one: Azure OpenAI limits (240K tokens/min) and the 4K-token context are hard architectural constraints, not fine print — hence Gemini's queues and rate limiter.
5. Give the model the right to say 'I don't know': a default <None> tag for uncertain cases protects against forced classifications better than demanding an answer.
6. Version prompts like code: analytics pipelines scoring every prompt version, plus LangSmith in V2, are the basis of controlled LLM system evolution.
7. An LLM system needs a second iteration: V2 cured V1's real weaknesses with task decomposition (PII/non-PII), cutting the prompt from 1,254 to 737 words, and misclassification alerts — not with a bigger model.

Frequently asked questions