95% of AI pilots do not pay 📉
TL;DR: an AI pilot does not create value just because it uses a model. It needs a concrete problem, an observable result, and a final decision.
The pilot as waiting room 🪑
An AI pilot should help decide whether a solution is worth adopting. Without a deadline and a way to close it, it becomes a permanent trial: a tool is purchased, a demo is shown, and the important question is postponed.
The problem is not experimentation. The problem is experimenting without knowing which decision should follow. The pilot then continues through inertia, even when daily work does not improve.
Ghost productivity 👻
Activity is easy to see: more accepted suggestions, more documents produced, more generated code. Value is less visible. It requires checking whether work reaches its destination sooner, whether errors decrease, or whether a queue actually gets shorter.
This is ghost productivity: lots of movement, little change. AI can speed up the production of material without improving the process that is supposed to use it.
Faster output is not automatically a better result.
The problem is not only the model 🧾
A model can be useful while the pilot still fails. This happens when technology is added to an unclear process, without clear ownership or a shared way to evaluate the result.
Value is not an intrinsic property of the model. It depends on the chosen problem, the context in which it is used, and the decisions around the experiment. Without these elements, debating model quality becomes a way to avoid discussing the actual work.
Why we measure usage instead of value 🎰
Measuring usage is convenient: licenses, requests, and active users produce numbers that are easy to report. Measuring value requires deciding what should improve and accepting that the result may be negative.
A serious pilot must allow three outcomes: adopt, change, or stop. If stopping is not an option, it is not an experiment. It is maintenance.
Measuring first costs less 🔍
Describe the problem before choosing a model. Define what should improve, how improvement will be recognised, and which limits must not be exceeded. Then decide when to make the call.
Tokens, lines produced, and time saved can be useful signals, but they are not value by themselves. AI helps when it improves the flow of work, not when it adds more work to review.
Measuring does not reduce ambition. It stops a trial from becoming internal folklore.
We have already touched on the theme of metrics that measure usage instead of value when talking about intrinsic motivation and where the money really goes: organizations tend to optimize what they know how to count.
The guide Measuring without illusions shows how to build a useful baseline; FOCUS helps choose a use case for the problem it solves, not for the most impressive demo.