Imagine a demo that gets everything right. A team shows how generative AI can turn a long technical report into a useful decision brief in minutes. The quality is convincing. The leadership team nods. The pilot gets the green light.
Six months later, the report is still produced much as it was before. There is now an extra AI tool, a set of guidelines and a few enthusiastic users. But the approval loops have not changed. Nobody is quite sure who owns a faulty summary. The experts check every sentence because they do not trust the system. Management tracks logins and prompts rather than better decisions.
The pilot worked. The change did not.
There is no contradiction. A pilot answers a technical question: can the system perform the task at all? Companies often turn this too quickly into an organisational claim: if it can, we will work faster, better or at lower cost.
A technical system changes an organisation only when leaders also redesign decisions, work and accountability.
Adoption is not value creation
AI is no longer a fringe activity in business. According to Stanford's 2026 AI Index, opens an external URL in a new window, 88 per cent of surveyed organisations used AI in at least one function in 2025; 70 per cent used generative AI. Yet in a global McKinsey survey, more than 80 per cent of respondents said they were not seeing a tangible enterprise-wide EBIT impact. Only 21 per cent said their organisations had fundamentally redesigned at least some workflows.
That gap tells us more than another debate about the “best” model. Companies have accelerated access to AI. Many have not changed the way work moves through the organisation at the same pace.
The result is digital double work. People run the old process and feed a new tool on top of it. They copy data into a chat window, check the response, transfer it into the familiar template and seek the same approvals as before. A task may become faster while the end-to-end process barely moves.
Local productivity may even increase system-wide cost. If AI produces more drafts, analyses or recommendations than downstream teams can review properly, the control burden grows. The organisation produces work faster, but not necessarily value.
A faster step is not yet a faster value stream.
We are changing the wrong unit
Many AI programmes begin with a list of functions: summarise, classify, predict, write. That makes sense because functions are easy to demonstrate. Economic value, however, rarely lives in an isolated function. It emerges from a chain of decisions.
Consider predictive maintenance. A forecast of possible equipment failure has value only if someone knows who evaluates the signal, what threshold triggers an intervention, whether spare parts are available and who has the authority to stop production. Without those decisions, the model mainly produces another alert.
Or consider an AI assistant in sales. A proposal drafted in minutes will save little if price approval, technical review and contract negotiation still take days. The bottleneck may even move: the team creates more proposals than downstream functions can review.
The useful unit of AI transformation is therefore not the use case alone. It is the workflow in which the use case changes a decision.
The technology has a jagged frontier
Even powerful models are not uniformly capable. In a field experiment involving 758 consultants, opens an external URL in a new window, participants using GPT-4 completed more tasks within the technology's capability frontier, worked 25.1 per cent faster on average and produced higher-quality results. Yet on a complex managerial task outside that frontier, AI users were 19 per cent less likely to reach the correct solution than the control group.
The “jagged technological frontier” is a more useful idea for managers than asking whether AI works in general. Within the same job, a system can be excellent at one task and unreliable at another that looks remarkably similar.
Nor is the frontier clearly marked. A model does not reliably announce: “You should stop trusting me here.” A wrong answer can sound as confident and complete as a correct one.
Teaching people to prompt is therefore not enough. They need to recognise what kind of task they are facing, what evidence is missing and when a plausible output calls for a deeper check.
Five decisions that turn a pilot into change
1. Start with the bottleneck, not the tool
Do not begin by asking, “Where could we use generative AI?” Ask, “Which recurring decision takes too long, creates too much rework or relies on too little evidence?”
This wording forces a connection between technology and business. It also stops an impressive prototype from going in search of a problem.
A good starting point has four features: the work recurs, its quality can be observed at least in part, the required data is accessible and an improvement would have visible consequences for customers, employees or financial performance.
2. Map the real workflow, including the exceptions
Process maps tend to show the normal route. Much of an organisation's expertise sits in the exceptions. What happens when data is incomplete? When does a case need escalation? Which customers should never be handled automatically? Which shortcut works only because an experienced person understands the context.
Ignore those exceptions and you automate an idealised version of the work. The system looks clean in the demo and then meets an organisation that never existed.
Involve the people who handle rework, spot edge cases and absorb informal decisions. This is not a polite change-management exercise. It is research into how the process actually works.
3. Design human judgement as a function, not a reassurance
“Human in the loop” sounds responsible. Too often, it means that somebody clicks “approve” at the end.
Effective oversight requires more. The reviewer must know what to look for, have access to the relevant information and possess the authority to disagree. The organisation must also define which cases can never proceed without a human decision.
The best division of labour is rarely “human or machine”. A machine may search for patterns, produce alternatives or prioritise cases. A person weighs competing objectives, unusual situations and consequences that cannot sensibly be compressed into one metric.
4. Measure process outcomes rather than activity
Login counts, generated texts and minutes saved show usage. They do not yet show business value.
Before rollout, define a small and demanding measurement logic. Depending on the workflow, it might include cycle time, error rate, rework, escalations, conversion, downtime or decision quality. Add at least one risk measure: missed errors, complaints, false escalations or cases in which people accepted an output without checking it.
If a company never defines impact, it will end up debating anecdotes.
5. Decide in advance when to stop
AI portfolios accumulate projects because nobody wants to end a visible experiment. Stopping is treated as failure, even when it is the soundest economic choice.
Set clear stopping conditions before the pilot begins. The data may not be good enough. The process may happen too rarely. Review may cost more than automation saves. The risk may be impossible to control at a reasonable cost. Users may bypass the system for understandable reasons.
A stopped pilot can be a good result. It protects resources for applications where technology, process and responsibility genuinely fit together.
A 30-minute test for your current AI projects
Choose one active pilot and answer six questions with the leadership team:
Which decision or bottleneck is meant to change?
Which measure must improve for the use case to make economic sense?
Which exceptions make the real process harder than the demo suggests?
Where is the jagged frontier: in which cases does the output become unreliable?
Who can challenge, stop or escalate the system, and does that person have enough information and authority?
Under which conditions will we end the project?
If the answers are vague, the project probably does not need a better model. It needs a better management decision.
AI capability sits between technology and management
The ILO's global analysis, opens an external URL in a new window concludes that generative AI is more likely to transform tasks than to make entire occupations redundant.[6] That reframes the leadership challenge: what should be automated, and how should tasks, roles, learning, control and accountability be put back together afterwards?
Neither pure technical expertise nor conventional general management is enough. Technical teams need to understand business value, costs and risk. Executives need to judge data quality, model limits and system dependencies without becoming model builders themselves.
This is where an AI tool either becomes a better process or remains an impressive demo.
Continue the discussion at TU Wien Academy
The Executive MBA Operational Excellence & AI connects data and AI foundations with processes, operating models and the strategic implementation of participants' own AI initiatives. The Certified AI Manager programme covers use-case prioritisation, data requirements, CRISP-DM, economics, risk and a project drawn from the participant's own practice. For leaders who first need a focused portfolio and credible roadmap, the short programme “AI in Business: Use Cases & Roadmap” offers a compact starting point.
These programmes do not begin with whichever tool happens to be popular. They begin where technical possibilities must be translated into decisions, processes and responsibility.
About Cookies and other techniques
Our website uses cookies and integrates content from third-party providers to ensure you get the best experience on our website, for analytical purposes, to provide social media features, and for targeted advertising. This it is necessary in order to pass information on to respective service providers. If you would like additional information about cookies and content from third-party providers on this website, please see our Data protection declaration.