Enterprise AI: why 10,000 queries hide just 14 real users

A technology analyst described on Habr a pattern he observed in four companies with different budgets and industries in 2025: corporate reports on "AI transformation" systematically hide failing indicators of actual tool usage.

The scheme that repeats in every company

In all four cases, the scenario unfolded according to a single template. From the top came a request: "we need AI, everyone already has it". Teams rushed to deploy a corporate chat on top of an LLM — usually in the company's brand colors. After a month, the board of directors saw a slide: "AI assistant was used 10,000 times". The board remained satisfied. In a letter to investors, a line appeared about "AI transformation". The product team received a bonus. The author provides specific numbers from one of the analyzed cases:

• 10,000 — total number of assistant requests during the reporting period
• 9,000 of them — one-time visits from users who never returned
• approximately 500 — forced testing by employees and demo sessions for management
• 14 — real weekly active users a quarter after launch
• 4,000 — total number of company employees

The report still listed "10,000". The slide was shown to the board twice.

Why the "requests" metric says nothing about value

The author describes three questions that make it possible to distinguish real implementation from a showcase. First: how many users returned a second time? Second: who uses the tool once a week a month after launch? Third: what exactly did these people stop doing by hand? In real cases, the answers to all three turned out to be uncomfortable. A significant share of requests from the pretty slide belonged to people whom management asked to "test it", and demo sessions before top management. Organically returned users — a handful.

"The report kept "10,000".

The slide was shown twice more at board meetings," — the author's observation.

The mechanism is simple: the metric "number of requests" is convenient for those who want to show progress without proving value. It is easily accumulated through initial curiosity, forced onboarding, and demo traffic. Meanwhile, pressure on teams only intensifies: competitors "implemented AI", investors expect a line about transformation, and success is measured by the first obvious number — number of requests.

How to distinguish real implementation from reporting

The formulation of the original task itself is instructive: when "we need AI, like everyone else" comes from the top, no one clarifies what problem the tool should solve and how to measure that it solved it. Without an answer to this question, any implementation is doomed to be a showcase. The real metric of AI implementation is changed employee behavior.

Not "how many times it was used", but "what they stopped doing by hand". If a user applied the assistant and returned the next week without anyone's request — this is a signal of real value. If 14 people out of 4,000 come weekly over a quarter — this is a failure, it is incorrect to call it transformation.

Companies that are truly changing processes with AI and companies that just launched a chat in brand colors look the same in current reports. The difference will manifest itself later — in productivity, competitive advantages, and ability to reduce operating costs.

What does this mean

Most corporate reports on "AI implementation" in 2025 document the fact of launching a tool, not actual change in work. The true benchmark of successful implementation is the weekly active user, retention after 30 days, and tasks that employees stopped performing by hand.