SK Telecom: a telco LLM on fine-tuned Claude — 68% fewer low-quality support responses

• SK Telecom transforms customer service with Claude (customer story) — Anthropic / Claude (кейс-стади)
• SK Telecom improves telco-specific Q&A by fine-tuning Anthropic's Claude models in Amazon Bedrock — AWS Machine Learning Blog, 2024-10-10
• SKT Partnership Announcement — Anthropic (news), 2023-08
• SKT Invests USD 100 Million in Anthropic and Joins Forces with the Company to Promote AI Innovation — SK Telecom Newsroom, 2023-08

Background

SK Telecom is South Korea's largest mobile operator, ranked #1 in the National Customer Satisfaction Index (NCSI) for 27 consecutive years. The Korean market does not forgive slowness: "Korea is a very urgent society. If you're one minute late, you're ten minutes late," says Eric Davis, Head of the AI Tech Collaboration Group at SKT. Customer service here is not a cost center but part of a national brand competition where speed, personalization, and cultural nuance matter.

SKT's relationship with Anthropic goes deeper than ordinary vendorship. In August 2023 the operator announced a $100 million investment in Anthropic (on top of an earlier investment by its venture arm SKTVC) and joint development of a multilingual LLM for the telecom industry — supporting Korean, English, German, Japanese, Arabic, and Spanish. The deal's context is the Global Telco AI Alliance — SKT, Deutsche Telekom, e&, and Singtel — created to accelerate operators' AI transformation: the idea is that telcos get customized AI capabilities without building their own models from scratch.

The case documented by Anthropic and the AWS technical blog is the first major practical embodiment of that bet: an industry telco LLM based on fine-tuned Claude for SKT's contact centers. Davis describes the project's philosophy with the term "IA" — intelligence augmentation: "This is not meant to replace jobs—it's meant to augment the experience."

For the market, the case is valuable for its rare technical transparency: beyond marketing percentages, the methodology details are published — from training data composition and synthetic data experiments to blind human evaluation methodology. It is one of the few cases you can actually learn to reproduce results from.

Problem

Contact center agents are overwhelmed. During a call they must quickly find precise answers in telco-specific documentation — with the customer on the line and the Korean speed standard allowing no pauses. After the call, they manually write up its outcomes: summary, topic classification, action items. The routine consumes cognitive resources that should go to the customer.

Generic LLMs without adaptation stumbled on two things. First — industry terminology: precise answers about tariffs, devices, and procedures require knowledge of SKT's documentation corpus, not "general telecom knowledge." Second — cultural and linguistic specificity: Korean customer service demands calibrated empathy and politeness, and service spans channels (phone, chat), regional dialects, and age demographics. A model that speaks Korean "reasonably well" does not pass that bar.

Brand requirements added rigor: "For us, brand safety is crucial—anything that tarnishes the brand is just a no-go. Anthropic's huge focus on safety really stood out," says Davis. And it is not only about answer ethics: "Performance wasn't just about model accuracy—the throughput and latency had to meet our standards." Finally, there was an organizational constraint: traditional service runs weekdays 9 AM to 6 PM, so anything that offloads live agents directly extends service availability. The task crystallized as: not replacing agents with a bot, but building an industry model that relieves people in real time and after the call — at a quality level worthy of a 27-year NCSI leader.

Solution

SKT fine-tuned Claude 3 Sonnet in Amazon Bedrock on its documentation and wired the model to its RAG system into a single end-to-end loop: per the AWS technical blog, the architecture uses Amazon Bedrock Knowledge Bases for domain retrieval, Bedrock Agents for extended scenarios, and Bedrock Guardrails for safety rails.

Two products were deployed on this foundation. In-Call Assist helps the agent during the call: it searches documentation in real time and suggests answers, lifting cognitive load off the overwhelmed employee. Post-Call Processing automates the write-up: conversation summarization, topic classification, to-do list generation, sentiment analysis — with human oversight of the results retained.

The most instructive part of the case is the quality-improvement stack, documented with numbers. The first layer is prompt optimization: alone it delivered 35–40% improvements; per the AWS blog's metrics, the prompt update by itself improved ROUGE-3 by 38.3% and citation accuracy by 52.94% over the base model. The second layer is fine-tuning: the "prompt + fine-tuning" combination pushed ROUGE-3 improvement to 58.1%, ROUGE-L to 26.8%, and citation accuracy to 70.59%; in select metrics the combined gains reached 50–60%. The third layer is synthetic data augmentation to fight the shortage of training examples.

The synthetic data experiment deserves a retelling because it answers the question every team with a small dataset asks. In blind human evaluation (ranking four variants, from 4 points for best to 1 for worst), the model trained on 2,000 original samples scored 114 of 160; the model on 500 original + 1,500 synthetic — 112; the model on only 500 original — 85; the base model without fine-tuning — 84. SKT's conclusion: synthetically augmented data performs almost like an equivalent volume of original data — meaning annotation scarcity stops being a blocker.

The application focus recorded in the AWS blog is grounded Q&A: source-grounded answers over technical telecom documents, in Korean. That explains why citation accuracy became one of the project's headline metrics: a suggestion to an agent is only useful when it can be instantly verified against the document.

Quality evaluation is also built as a system: blind rankings by human evaluators with a preference scale — a methodology that measures usefulness to humans, not just automated metrics. Davis adds a subjective model-selection argument: "Claude is more relatable and creative" — for a service where empathy is part of the standard, that is a requirement, not cosmetics.

Result

LLM response quality ratings from live agents rose 34% (in-call assistance). The share of low-quality responses fell 68% — the fine-tuned telco model versus the base model. In post-call processing, quality reached ~89% of human-agent level, and the telco LLM's overall score improved from 3.3 to over 4.3. Per the AWS technical blog, fine-tuning Claude 3 Sonnet combined with prompt optimization improved ROUGE-3 by 58.1%, ROUGE-L by 26.8%, and citation accuracy by 70.59% over the baseline.

Framing. All percentages are relative improvements on SKT's internal benchmarks; absolute quality values are not disclosed. ROUGE measures textual overlap, not semantic correctness, and guarantees nothing by itself (which the blind human evaluations compensate for). Both primary sources are parties to the deal: Anthropic (in which SKT invested $100M) and AWS (whose platform hosts it all). No data on contact center business metrics — handling time, satisfaction, savings — has been published.

Editorial analysis. First: the case effectively publishes an industry LLM "maturity ladder" with the price of each rung — prompts deliver the first 35–40% almost for free, fine-tuning adds the next jump, synthetic data removes the data shortage. The practical takeaway for any team: don't start with the expensive stage — most of the effect goes to disciplined prompt work, and fine-tuning makes sense once the prompt ceiling is genuinely reached and measured. Second: the "89% of human level" figure in post-call processing is a model of honest metrics: it shows both readiness (routine write-ups can be handed to the model under supervision) and the boundary (the 10+ percent shortfall is why the human stays in the loop). Third: the SKT–Anthropic investor link makes the case simultaneously stronger and weaker: stronger because SKT risks its own money and builds an operator alliance (GTAA) on this bet; weaker because every published metric passes through two financially interested parties. The result has no external replication yet — though the very design of the Global Telco AI Alliance implies the approach will be replicated across the alliance's other operators, and that will be the real test of whether a telco LLM transfers beyond the Korean market.

Lessons learned

1. An industry LLM is a stack of techniques, not one fine-tune: prompt optimization (35–40% gains alone) → fine-tuning → synthetic data; only together did they cut bad responses by 68%.
2. Start with the cheap rung: the prompt update alone improved citation accuracy by 52.94% — before paying for fine-tuning, reach and measure the prompt ceiling.
3. Synthetic data genuinely works when training examples are scarce: SKT's model on 500 original + 1,500 synthetic samples (112/160) nearly matched the model on 2,000 originals (114/160).
4. The '% of human level' metric (89% in post-call processing) is more honest than abstract benchmarks — it shows exactly where AI is ready to take over routine work and why the human stays in the loop.
5. Evaluate blind, with humans: preference ranking by live evaluators catches what ROUGE misses — politeness, empathy, answer applicability.
6. Positioning as 'augmentation, not replacement' (IA — intelligence augmentation) lowers staff resistance: the goal is to relieve overwhelmed agents, not cut them.
7. Cultural context is part of quality: for Korean service, empathy, politeness, dialects, and age demographics proved as decisive in model selection as accuracy, throughput, and latency.

Frequently asked questions