Most enterprise AI programmes do not fail on the model. They fail on seven decisions that are made badly, made late, or never assigned to anyone. This is what the evidence says about each one as of September 2026, including where the evidence contradicts itself, and who in your organisation has to own the answer.
How data and knowledge become trusted decisions. Data comes in on the left, is governed and given shared meaning in the middle layers, is reasoned over by the models at the core, and leaves on the right as answers, insights and actions for people and agents. Judgment and trust controls sit around the edge; the control plane runs underneath. Read it as the destination, not the starting point: each use case lights up only the parts it needs, and the gate in decision 02 tells you which part to fund next. Every decision below names where it sits on this picture.
Seven considerations, in the order they actually bite. Four of them have to be settled before a pilot starts, because getting them wrong is what makes the pilot unrepeatable. One governs the crossing itself. Two only begin once the system is carrying real traffic.
| Phase | Consideration | The question it actually answers | Where it sits on the reference architecture |
|---|---|---|---|
| Before the pilot | 01 Objectives, ROI and ownership | What are we measuring, and whose name is against the number? | Human / Business Judgment · Observability & Evaluation · Continuous Learning & Feedback |
| 02 Data readiness and integration | Is the data good enough for this one use case, and how would we know? | Data In · Ingest & Prepare · Data Acquisition & Integration · Data Governance & Trust · Data & Knowledge Layer | |
| 03 Infrastructure and architecture | What is the simplest shape that could work, and when does it stop working? | Reasoning & Memory · Model Compute & Orchestration · AI Model Core · the control plane | |
| 04 Security, compliance and trust | What breaks, who is liable, and what evidence would we need? | Identity & Access · Policy & Risk · Judgment & Trust Gate · Evidence & Provenance | |
| Pilot to production | 05 Organisational readiness | Who does the work differently on Monday, and who reviews the output? | Deliver & Act · Intelligent Consumption · Human Decision Makers |
| In production | 06 Governance and risk management | How much human attention does each decision deserve? | Judgment & Trust Gate · Decision Traces · Governance & Guardrails · Observability & Evaluation |
| 07 Scale and evolution | How does this survive the model changing underneath it? | Model Registry · Environment & Deployment · Model Gateway · Continuous Learning & Feedback |
A pilot proves the model can do it once. Production proves the organisation can do it every day, at cost, with someone accountable.
The second use case should be cheaper than the first. If it is not, you built a project, not a platform.
Managed means someone is watching. Resilient means the system degrades safely when nobody is.
The published evidence on enterprise AI is unusually contaminated, and knowing how tells you which numbers to trust. Four things explain nearly every apparent contradiction in this field:
Where I cite a number, I name the source and the date. Where sources disagree, I state both cases before taking a position. Where I could not verify something, I say so rather than rounding it into a claim.
If nobody can name the person accountable for what this costs and what it returns, the programme has already chosen its outcome.
Ownership and cost sit around and beneath the rings, because no layer can supply them. Cost is measured in the control plane's observability and evaluation service; the outcome it is divided by is captured in the continuous learning and feedback loop along the bottom; the person accountable for both is the human and business judgment at the edge.
This is the decision most organisations skip, and it is the one with the cleanest evidence behind it. Scaling has moved. Margin has not.
Adoption rose roughly six points year on year while EBIT attribution stayed flat. That gap is the entire problem. More organisations are running AI in more functions and the same proportion can point to a margin outcome. Scale is not converting.
The strongest single finding in the 2026 evidence base is not about models or use cases. It is about accounting. Organisations with strong visibility into AI spend are five times more likely to have established ROI, 15 percent against 3 percent (KPMG, June 2026). The mechanism is not mysterious: you cannot establish a return on a cost you cannot see.
Token cost has become a board-level line item faster than most finance functions adapted. In EY's May 2026 wave of 534 senior leaders, 82 percent were concerned about token consumption and cost, and 98 percent of those using token-consuming tools said cost had made them reconsider their approach: 37 percent reassessing functional priorities, 35 percent reconsidering use cases, 32 percent reconsidering who gets access. Only 64 percent actively monitor token usage against a budget. The FinOps Foundation's 2026 survey found that 98 percent of FinOps practitioners now manage AI spend, up from 63 percent in 2025 and 31 percent in 2024, the fastest scope expansion in the survey's history.
KPMG's June 2026 study is the closest thing to a controlled test of single-owner accountability. Where the CEO is the named owner of AI outcomes, against where accountability sits diffusely across the C-suite:
| Outcome measure | CEO is named owner | Accountability is diffuse | Spread |
|---|---|---|---|
| Confidence in AI strategy | 60% | 22% | 2.7x |
| Realising meaningful business value | 57% | 21% | 2.7x |
| Established ROI | 14% | 4% | 3.5x |
Two structural findings sit alongside it. Accenture's September 2026 work found 42 percent of organisations have no single owner of AI cost and outcomes, relying on shared IT and finance accountability. BCG's 2025 survey of 1,250 executives found the highest performers are not centralised or federated but both: 43 percent have joint business and IT ownership, they are three times more likely to have a chief AI officer, and C-suite engagement runs near 100 percent against 8 percent for laggards.
This is the most actionable disagreement in the whole evidence base, and almost every enterprise dashboard is currently on the wrong side of it.
Uber published the clearest working model in August 2026. They decompose AI cost multiplicatively, giving each term a named optimisation owner:
Users × Sessions per user × Turns per session × Requests per turn × Tokens per request × Price per token
Their production metrics are denominated in outcomes, not tokens: cost per merged pull request, cost per review, cost per alert, cost per cleanup. Result over February to August 2026: cost per thousand model requests down 34 percent from peak, cost per session down 52 percent from its June peak, while weekly active employees grew 7x and weekly agent requests grew 9.4x, with total spend broadly flat.
DoiT's August 2026 research gives the arithmetic for why token denomination misprices reliability:
Cost per solved task = expected cost per attempt ÷ success probabilityRisk-adjusted CPT = CPT + (leak rate × cleanup cost)On their measured workloads the ranking between two frontier models flips depending on whether failure cleanup is priced, and again depending on whether the workload is coding or tool use. Run-to-run variance on identical agentic tasks reached 30x. Tokenizer differences alone shift token counts by around 30 percent for the same text. Any chargeback model denominated in tokens will systematically reward the model that fails cheaply over the model that succeeds.
Before renegotiating anything, three engineering levers have published, verifiable economics:
| Lever | Published economics | Break-even |
|---|---|---|
| Prompt caching | Cache read at 0.1x base input price. Five-minute cache write 1.25x, one-hour write 2x. | Five-minute cache pays back after one hit; one-hour after two |
| Batch processing | 50 percent discount on both input and output tokens for non-realtime work | Immediate, where latency permits |
| Context compression | Shopify: system prompt 6,000 tokens down to 1,500 gist tokens, end-to-end latency down 38 percent, GPU allocation down 14 percent | Compression cost paid once at training time |
Cache time-to-live is a real decision with a published rationale. Uber runs a one-hour TTL for interactive developer sessions because engineers pause for more than five minutes while thinking, and a five-minute TTL for subagents because they run single tasks. The break-even is governed by the expected idle gap, not by prompt size.
Is 95 percent of enterprise AI failing, or is 80 percent of it working?
MIT NANDA (2025): 95 percent of organisations getting zero return on 30 to 40 billion dollars. S&P Global (March 2025): 42 percent scrapped most AI initiatives, up from 17 percent a year earlier; 46 percent of proofs of concept abandoned. McKinsey: EBIT attribution flat across two waves, 6 percent high performers. KPMG: 7 percent with established ROI. Gartner (June 2025): over 40 percent of agentic AI projects cancelled by end 2027, on "escalating costs, unclear business value or inadequate risk controls."
Bain (November 2025, n=197): 80 percent of generative AI use cases met or exceeded expectations, 78 percent of those with measurable revenue or cost movement. IDC and Microsoft (2024): 3.70 dollars returned per dollar spent, top performers 10.30. Omdia and Snowflake (2025, n=2,050): 92 percent of early adopters report positive return. EY (May 2026): 98 percent report positive ROI.
What actually explains the gap. MIT's 95 percent counts all surveyed organisations including the roughly 80 percent that never ran a custom pilot; on its own data, around 25 percent of custom pilots succeeded, which is inside Bain's 20 to 40 percent range by function. MIT required sustained P&L impact within six months; IDC measured value realisation at thirteen. Omdia is explicit that only 40 percent of its respondents actually quantified the return behind the 92 percent. EY's own data undercuts its 98 percent: 82 percent are worried about token cost and only 3 percent hit their planned budget allocation. IDC was commissioned by Microsoft, Omdia by Snowflake, and MIT NANDA's authors recommended their own project as the remedy.
Open and closed model selection, the monthly cost model, and the evaluation self-assessment. This is where the unit economics above get turned into a defensible model choice rather than a preference.
Read the paperData readiness is a gate on one use case, not a programme that precedes all of them. Treating it as a programme is its own failure mode.
The left half of the picture: where data comes in, how it is ingested and prepared, and the two layers that make it trustworthy and meaningful. The eight-test gate below decides which of these parts a single use case actually needs, so the foundation is funded by use cases rather than ahead of them.
This is the consideration most often quoted and least often understood. The famous statistic is real. The reading most people take from it is wrong.
The exact wording: "Through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data" (Gartner, February 2025). Note the scope. It is not 60 percent of AI projects. It is 60 percent of AI projects unsupported by AI-ready data. Secondary coverage drops that clause almost universally.
Gartner's definition of AI-ready is also not what most people assume: "Data must be representative of the use case, of every pattern, errors, outliers and unexpected emergence that is needed to train or run an AI model for a specific use." Gartner is explicit that high data quality is not the same as AI-ready, and that representative data "may include poor-quality data, too." Melody Chien of Gartner puts the scoping plainly: "AI-ready data means that data is ready to support certain AI cases." The same data can be ready for one use case and not another.
Three independent studies converge on the same narrow band: 7 percent of enterprises say their data is completely ready for AI (Cloudera and Harvard Business Review Analytic Services, March 2026), 6 percent say fully ready to support AI at scale (Dun & Bradstreet, July 2026, n=10,000 businesses across 32 countries), 7 percent qualify as Accenture's data reinventors. The reinventor group carries a 4.5 percentage point EBIT margin advantage over peers.
The Precisely and Drexel LeBow study (n=505 data and analytics leaders, published January 2026) found 88 percent believe their data is AI-ready while 43 percent simultaneously cite data readiness as a barrier, in the same survey. The same pattern repeats on infrastructure (87 percent claim it, 42 percent cite it as a challenge) and skills (86 percent claim sufficiency, 41 percent identify a gap).
This is why the 7 percent figures and the 88 percent figures are both true. The dispersion is mostly an artifact of what "ready" was allowed to mean. Anyone quoting "only 7 percent" owes you the definition.
Most data readiness claims are correlational. This one is not. Across three independent production settings, the accuracy lift from semantic and metadata grounding is larger than the lift from upgrading the model.
| Setting | Without grounding | With semantic layer or knowledge graph | Source |
|---|---|---|---|
| Snowflake Cortex Analyst, 150 real BI questions | 51% | 90%+ | Snowflake, Aug 2024 |
| LinkedIn SQL Bot, overall correctness | 9% | 53% | LinkedIn engineering |
| dbt Labs benchmark, modelled data, Claude Sonnet 4.6 | 90.0% | 98.2% | dbt Labs, Apr 2026 |
| dbt Labs benchmark, modelled data, GPT-5.3 Codex | 84.1% | 100.0% | dbt Labs, Apr 2026 |
LinkedIn's own conclusion is worth stating exactly: the jump from 9 percent to 53 percent came "almost entirely from the knowledge graph. Not from better prompting. Not from smarter agents." Their graph combines catalog metadata, query logs for popularity and join patterns, domain knowledge collected through the UI, and certified example queries, refreshed weekly. Table recall moved 45 to 78 percent, column recall 24 to 56 percent, and invalid table or column references fell from 23 percent to 1 percent.
The benchmark ladder shows why this matters more on your schema than on anyone's demo:
| Benchmark | What it tests | Best reported accuracy |
|---|---|---|
| Spider 1.0 | Textbook schemas, around 4 tables per database | 91.2% |
| BIRD | Larger, dirtier academic schemas | 73.0% |
| Spider 2.0 | Real enterprise workflows, 1,000+ columns, 100+ line SQL, multiple dialects | GPT-4o: 10.1% |
| BEAVER | Two genuinely private warehouses the models cannot have seen, 99 and 366 tables | 0.0% end to end |
BEAVER is the fact that no amount of model scale accommodates. "None of the off-the-shelf LLMs can answer any question correctly." Given gold tables and column mappings, GPT-4o rose from 2.1 percent to 4.2 percent. The models cannot have memorised what they have never seen, and your warehouse is something they have never seen.
Chroma Research reports 91.9 percent token-level retrieval recall for semantic chunking on general text. FloTorch reports 54 percent end-to-end accuracy for the same strategy. Both are true and they are not the same measurement. High retrieval recall does not imply correct answers. Insist on end-to-end numbers in every vendor conversation.
Embeddings are personal data. Research on embedding inversion reconstructs 92 percent of 32-token text inputs exactly from their vectors. Independent replication recovered 40 percent of personally identifiable information exactly from sentence-length text. Redaction before embedding drove recovery to zero. A vector index inherits the classification of its source text and must carry the same access control and residency constraints.
Deleting from a vector store usually does not delete. The June 2026 "Ghost Vectors" research found that vector databases typically implement deletion as a metadata soft delete, leaving the embedding physically on disk. Measured recovery from soft-deleted vectors: 25.5 percent of person names and 46.4 percent of geographic locations from Wikipedia biographical data, 49.9 percent patient surname recovery from synthetic clinical data, 99.17 percent identity recovery from facial embeddings. A deletion request satisfied at the API layer is very likely not satisfied at the storage layer. If you have committed to an erasure right, you need cryptographic overwrite or media destruction, not a delete call.
Permissions are the third one, and it is the one that has already embarrassed people publicly. Retrieval assistants honour existing permissions exactly. They do not create a leak; they make a latent one legible at conversational speed. Remediate sharing before indexing, not by filtering after retrieval.
Is data readiness the real blocker, or is it consultant product that delays everything?
BEAVER's 0 percent on private warehouses. The semantic-layer lift exceeding the model-upgrade lift across three independent settings. Accenture's 72 percent lacking trusted data and 80 percent altering initiatives because of data risk. McKinsey (April 2026): eight in ten companies cite data limitations as a roadblock to scaling agents, and fewer than 10 percent have scaled them.
dbt Labs' own benchmark shows raw text-to-SQL on an unmodelled schema rising from 32.7 percent in 2023 to 64.5 percent in 2026 with no data work at all. Models are absorbing some of the burden, measurably. McKinsey's own August 2026 survey does not name data as the top barrier, attributing constraint to organisational absorption capacity instead. And the argument from practice: perfect data does not exist anywhere, including inside the most data-sophisticated companies in the world.
Honest assessment of the counter-case: weak on evidence, right on framing. Most of it is anecdote or vendor content. The one rigorously measured piece, the dbt doubling, comes from a study whose own conclusion is pro-semantic-layer, and in the same table the semantic layer still wins by eight to sixteen points and reaches near 100 percent where raw generation reaches 84 to 90. But the counter-case is correct about one thing that the readiness camp consistently gets wrong.
Not "improve customer service." A person, a task they do, the artefact produced, and the accuracy below which it is worthless.
Gartner's definition, not a cleanliness score. Representative beats clean.
If the definition of active customer lives in a dashboard or an analyst's head, the agent will invent its own.
This is where LinkedIn's 9 to 53 percent came from. It is cheap and it is the highest-yield work available.
Not filtered after retrieval. Broad grants get fixed or the content does not get indexed.
Overwrite, not soft delete. Confirm with your vector store vendor in writing.
Automated, sampled, reviewed or advisory. See consideration 06.
42 percent of organisations do not have one. That is the cheapest gap on this list to close.
Data becomes information becomes knowledge becomes wisdom becomes inference becomes decision, and the decision feeds back. The interactive version shows what moves between each stage and where the judgment gates sit.
Open the loopStart with the simplest shape that could work. Complexity should be something the system earned, not something it started with.
The inner rings and the control plane: context retrieval and memory, the model gateway, tool use and agent runtime, and the frontier and domain models at the core. The workflow-versus-agent choice below is made in workflow orchestration; the context engineering techniques live in reasoning and memory.
The published engineering position from the people building these systems is consistently anti-complexity. That is worth stating loudly, because the market incentive runs the other way.
Anthropic's own guidance: "When building applications with LLMs, we recommend finding the simplest solution possible, and only increasing complexity when needed" and "For many applications, optimizing single LLM calls with retrieval and in-context examples is usually enough." The distinction that matters is not agent versus no agent. It is workflow versus agent:
| Shape | Definition | Use when |
|---|---|---|
| Workflow | LLMs and tools orchestrated through predefined code paths: prompt chaining, routing, parallelisation, orchestrator-workers, evaluator-optimiser | The steps are predictable. This covers most enterprise work, and it is cheaper, testable and debuggable. |
| Agent | The model dynamically directs its own process and tool use, controlling how it accomplishes the task | Open-ended problems where you cannot predict the number of steps or hardcode the path, and the task value justifies the cost. |
Gartner's agent-washing finding belongs here rather than in a governance section, because it is an architecture problem: "Many use cases positioned as agentic today don't require agentic implementations." Menlo Ventures found only 16 percent of enterprise deployments qualify as true agents; the rest are fixed-sequence or routing-based workflows. That is not a criticism. It is the correct architecture being mislabelled by its vendors.
Large context windows did not remove the constraint. They moved it. Models have an attention budget, and "every new token introduced depletes this budget by some amount," because transformer attention creates n-squared pairwise relationships across n tokens.
Chroma's context rot research, across 18 models, found performance degrades significantly as input length grows even on simple tasks. Low semantic similarity between question and target degrades faster than high similarity. Distractors compound non-uniformly. All models performed significantly better on focused prompts than on full ones.
The single largest waste in most enterprise agent deployments is tool definitions. Loading a full set of connector tool definitions up front against progressive disclosure through code execution: 150,000 tokens down to 2,000, a 98.7 percent reduction. Uber independently measured the same thing on their MCP estate, cutting 50,000 to 70,000 tokens per request to near zero, with batched tool calls in code reducing token usage 55 to 100 percent.
The tau-bench work introduced pass^k to measure whether an agent succeeds repeatedly. The finding: state-of-the-art function-calling agents succeed on under 50 percent of tasks on a single attempt, and pass^8 falls below 25 percent in the retail domain. For production, pass^k is the honest metric. Single-attempt accuracy tells you what the demo will look like.
The compounding is simple arithmetic and it is worth doing out loud with any vendor. At 98 percent per-step success, a 50-step task succeeds about 36 percent of the time. At 99 percent, about 61 percent. This is why the production response to unreliability is not a better model. It is checkpointing and verification that reset the error budget: commits per increment, durable execution that resumes at the failed step rather than restarting, circuit breakers on step count and cost.
On capability trajectory, use the primary source and its confidence intervals rather than the headline. METR's revised time-horizon work (January 2026) puts the doubling time of task length that frontier agents complete at 50 percent reliability at 196.5 days across the full period, 130.8 days since 2023, and 88.6 days since 2024. Their reported 50 percent horizons carry wide intervals: Claude Opus 4.5 at 320 minutes with a 50 percent interval of 170 to 729. That is a 4.3x spread. Plan against the bottom of the interval.
Buy the application, centralise the access. That is what the market has converged on and what the published platform teams actually own. Uber runs an AI Gateway and a separate MCP Gateway fronting more than 1,000 MCP servers behind unified authentication and policy. Netflix built in-house serving on vLLM behind an OpenAI-compatible interface specifically to get deep customisation of constrained decoding and output validation inside their existing production environment.
The argument for the gateway is not elegance. It is that the cost equation, the model-churn migration, the authentication and policy boundary, and the evaluation gate are all cross-cutting concerns. Uber holding total spend broadly flat through 9.4x request growth is an argument for the gateway, not for any individual optimisation. On the reference architecture, that centralised access is the model gateway, and the cross-cutting concerns are the control plane along the bottom.
Should you build multi-agent systems at all? Two well-argued positions, published one day apart in June 2025, and the reconciliation that arrived ten months later.
Cognition: "Don't build multi-agents." Decision-making becomes too dispersed and context cannot be shared thoroughly enough between agents. Actions carry implicit decisions, and conflicting decisions carry bad results. Recommendation: single-threaded linear agents with context compression for long tasks.
Anthropic: a multi-agent research system outperformed the single-agent equivalent by 90.2 percent on their research evaluation. But the same post scopes it tightly: good for breadth-first parallelisable search, bad for anything needing shared context, heavy interdependencies, real-time coordination, or most coding tasks. And it costs roughly 15x the tokens of chat.
The reconciling variable is whether subagents write. Cognition's own April 2026 follow-up states it: "writes stay single-threaded and the additional agents contribute intelligence rather than actions." Their working exceptions are a review agent with no prior context catching around two bugs per pull request, 58 percent of them severe; a weaker primary calling a stronger model on hard cases; and hierarchical delegation, which needed more context engineering than expected.
Attention and the KV cache, mixture-of-experts routing, the training pipeline, quantisation, the evaluation layers, agentic memory and protocols. The long answer to why the architecture choices above have the consequences they do.
Read the paperContain at the environment layer first, then steer behaviour at the model layer. Probabilistic defences cannot provide complete protection, and the incident record says so.
The controls around the edge of the picture. One thing a layer diagram cannot show is the environment boundary itself: the sandboxes and egress limits this section argues come first. Treat them as the container the whole picture runs inside.
This section has real incidents with dates and identifiers, because generic prompt-injection warnings have stopped moving anyone.
| Incident | Product | Identifier | Severity | What it did |
|---|---|---|---|---|
| EchoLeak | Microsoft 365 Copilot | CVE-2025-32711 | 9.3 | Zero-click indirect prompt injection, data exfiltration from a productivity assistant. Patched June 2025. |
| ForcedLeak | Salesforce Agentforce | not assigned | 9.4 | Disclosed September 2025. Patched via trusted URL allowlists. |
| ShareLeak | Microsoft Copilot Studio | CVE-2026-21520 | 7.5 | Customer data exfiltrated to an attacker-controlled address. Found November 2025, patched January 2026. |
| PipeLeak | Salesforce Agentforce | none | n/a | CRM exfiltration with no volume cap. Bypasses the ForcedLeak allowlist patch by moving to the email channel. |
The last row is the one to take to your security architect. A URL allowlist was deployed as the control against prompt injection, and it was defeated by changing egress channel. That is the empirical case for controlling egress at the environment layer rather than filtering content at the model layer.
Anthropic's published containment principle says the same thing as a design rule: "Design for containment at the environment layer first, then steer behavior at the model layer," on the explicit basis that probabilistic model defences cannot provide complete protection. The mechanisms are ephemeral containers, OS-level sandboxes, full VM isolation with read-only and read-write-no-delete mount modes, and egress allowlists enforced by proxy interception.
Introducing OS-level sandboxing produced an 84 percent reduction in human permission prompts. Better containment removed human-in-the-loop friction rather than adding it. Containment and speed were complements, not a trade-off. If your security conversation is framed as safety against velocity, that framing is wrong and this is the counter-example.
In September 2025 a malicious MCP server was published to npm. The backdoor was present in versions 1.0.16 through 1.0.18 and silently added a blind-copy recipient to every outgoing email, harvesting full contents, attachments and headers. The publisher owned 31 other packages at the time of disclosure.
MCP servers and agent plugins are a software supply chain with tool-level trust, no provenance requirement, and in most enterprises no inventory. MITRE ATLAS v2026.08 now carries 16 tactics, 114 techniques and 83 sub-techniques, having added 16 agent-specific techniques in a single month, including autonomous reconnaissance and autonomous attack-path adaptation, plus new mitigations for agent authority and scope drift. Agent authority creep is now a named threat category, not a governance worry.
The EU AI Omnibus was published in the Official Journal on 24 July 2026 as Regulation (EU) 2026/1744 and entered into force on 27 July 2026. Stand-alone Annex III high-risk obligations moved from 2 August 2026 to 2 December 2027. Embedded high-risk under Annex I moved from 2 August 2027 to 2 August 2028.
What did not move. Article 50 transparency obligations still applied from 2 August 2026. The deferral covers Chapter III only. Anyone who deferred their whole programme still has live transparency exposure today, and anyone who built a 2026 conformity programme now has sixteen months of slack to use properly.
Article 14 is the human oversight clause and its wording is worth having exactly: high-risk systems must be designed "in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons during the period in which they are in use." Three details most summaries omit:
Article 12 requires automatic event logging over the lifetime of the system, with a six-month minimum retention. It specifies no format, no required fields and no tamper-proofing requirement. Logs that can be edited without detection have no evidentiary value. The regulation creates a logging duty without creating an integrity duty, so append-only or cryptographically chained logging is your decision to make, not the regulator's.
Model risk management has a hole in it. On 17 April 2026 the US banking agencies issued revised model risk management guidance superseding the 2011 originals. The revised guidance explicitly excludes generative and agentic AI from its scope on the grounds that these technologies are novel and rapidly evolving. Annual validation mandates were dropped, vendor contingency plan requirements eliminated, and a materiality concept introduced. The regulators looked at LLMs, concluded the 2011 validation paradigm does not fit, and removed them from scope rather than adapting it. The classic validation triad assumes a stable artefact you can re-run; a hosted model is non-deterministic, silently re-versioned and opaque as to training data. That leaves a governance vacuum exactly where the novel risk sits, and filling it is now the institution's problem rather than the supervisor's.
The insurance backstop is being withdrawn. In January 2026 a generative AI exclusion was introduced for commercial general liability policies, excluding coverage for bodily injury, property damage and personal or advertising injury arising out of generative AI. Broad AI exclusions are appearing in directors and officers, employment practices and fiduciary policies with language expansive enough to capture ordinary business activities involving AI. This is the strongest non-regulatory argument for governance available: if AI failure has moved from an insured loss to an uninsured balance-sheet loss, governance is no longer a compliance cost. It is the substitute for a risk transfer mechanism that has been taken away.
And on whether the vendor is responsible rather than you: in Moffatt v. Air Canada (February 2024) the tribunal rejected the argument that the chatbot was a separate entity accountable for its own actions. "While a chatbot has an interactive component, it is still just a part of Air Canada's website." And: "It should be obvious to Air Canada that it is responsible for all the information on its website." No novel AI liability theory was required. Existing misrepresentation law reached the outcome.
Why coordination does not improve as models get more capable, how agents collude without a shared channel, and why credulity is the property that makes a helpful agent exploitable. The failure modes behind the incident table above.
Read the paperThe bottleneck does not disappear when a task is automated. It moves one level up, to whoever reviews the output and carries the accountability.
The right-hand side: where answers, insights and actions are delivered, and the people who receive them. The review capacity this section asks you to fund is the human decision makers box, staffed and given the time.
This is where most published advice is weakest, because the people who publish on it sell change programmes. So here is the evidence, including the case against the advice.
The closest quantified analogue comes from Microsoft's 2026 Work Trend Index (n=20,000 knowledge workers, 10 countries): organisational factors such as culture, manager support and talent practices drive 67 percent of reported AI impact, against 32 percent for individual mindset and behaviour. Managers who model AI use produce a 17-point lift in perceived value; managers who set quality standards produce a 22-point increase in critical thinking. Note the vendor interest: this is a conclusion Microsoft benefits from.
McKinsey's 2026 survey is cleaner on the mechanism: 73 percent of high performers are redesigning workflows against 25 percent of everyone else, and high performers are 3.3x more likely to expect business transformation within three years. Deloitte's parallel finding is that only 30 percent are redesigning key processes and 37 percent use AI with minimal process change at all.
| Study | Finding | Population |
|---|---|---|
| METR randomised trial, July 2025 | Developers took 19 percent longer with AI. They expected 24 percent faster, and afterwards still believed they had been 20 percent faster. | 16 experienced open-source developers, mature codebases they had worked on for years |
| GitHub with Accenture | Pull requests up 8.69 percent, merge rate up 15 percent, successful builds up 84 percent, 81.4 percent same-day install | Enterprise developers. Sample size never disclosed. |
| Atlassian internal, June 2026 | Adopting repositories merged 19 percent more pull requests per month; 2 to 3 hours saved per developer per week | 6,093 of 6,200+ developers |
| Dell'Acqua et al., peer reviewed | Inside the capability frontier: 12.2 percent more tasks, 25.1 percent faster, 40 percent higher quality. Outside it: 19 percentage points less likely to be correct. | 758 consultants, pre-registered |
The jagged frontier finding resolves the contradiction. Gains concentrate in less experienced people on tasks inside the model's competence. Penalties concentrate in experts working outside it, through over-reliance on plausible but wrong output. The levelling effect is consistent: below-average performers gained 43 percent against 17 percent for above-average, and the largest coding gains went to the least experienced. Targeting enablement at your high performers inverts the evidence.
The finding that survives all of this is DORA's, and it is the one to design around: AI adoption now shows a positive relationship with delivery throughput and a continuing negative relationship with delivery stability. More output, more failures. Thirty percent of respondents report little or no trust in AI-generated code. AI is an amplifier of whatever the team already is.
Harvard Business Review research (June 2026, 18 interviews across two consulting firms) found a consistent pattern: senior leaders used AI strategically for wider scope and smaller teams, junior staff achieved dramatic productivity gains, and middle managers absorbed three new unfunded duties: validating AI-generated output, identifying errors, and coaching AI competence, with unchanged or increased delivery pressure and no formal support.
Put that next to DORA's stability finding and the conclusion is uncomfortable. More throughput with worse stability means the review, test and rollback functions need more capacity, not less, precisely when headcount logic says cut them. The sample is small and consulting-specific, but it is the only direct evidence on this question and it points the same way as the delivery metrics.
Roughly half of employees report using AI contrary to company policy, including uploading sensitive data to public tools. Zylo's spend telemetry corroborates this behaviourally rather than by self-report: expense-based SaaS spend, meaning employees buying it themselves, rose 267 percent year on year, with a consumer AI assistant the single most-expensed application across the estate. Only 40 percent report any workplace guidance on generative AI use.
Klarna is quoted constantly and usually wrongly. The sequence: February 2024, an assistant handling work equivalent to 700 agents across 2.3 million conversations a month in 35+ languages. December 2024, hiring frozen, headcount down 22 percent through attrition. May 2025, the CEO announces recruitment of human agents again: "I just think it's so critical that you are clear to your customer that there will always be a human if you want," and he characterised the chatbot output as lower quality than human staff despite being cheaper.
What actually changed is narrower than the headlines. Klarna did not rehire 700 people; it restarted hiring for a remote, flexible pool. The AI still handles work equivalent to roughly 850 agents. Headcount is down around 50 percent since 2022 and revenue per employee rose from 300,000 to 1.3 million dollars. The cost case held. The last-mile service quality and customer-choice case did not. Both readings are defensible; say which one you mean.
Is change management essential, or is it consultant product for a problem good tools solve by themselves?
Shadow adoption is enormous and entirely unmanaged: around 90 percent personal AI use for work, 66 percent regular use, a consumer assistant topping expense reports across 40 million licences. None of that was produced by a change programme. 81.4 percent of developers installed a coding assistant the same day they got access. Shopify's result came from removing procurement and legal friction and building an internal proxy, not from communications. MIT's own diagnosis of failure is integration and contextual adaptation, explicitly not a talent or skills gap.
Individual adoption is not organisational value capture: 90 percent personal use coexisted with most businesses seeing no return. Organisational factors explain 67 percent of impact variance against 32 percent individual. DORA's seven team archetypes from the same tools are a direct rebuttal of tool determinism. 73 percent of high performers redesign workflows against 25 percent of the rest. And managers break before tools do.
Five real decisions where the intelligent answer and the sensible answer diverge. Data foundation, decision intelligence, AI-led engineering, trust and compliance, adoption. Pick your option before you see what happened.
Walk through itThe failure being corrected across 2026 is not oversight. It is uniform oversight, which spends the review budget on low-consequence work and produces rubber-stamping where it matters.
The judgment and trust gate at the top of the picture is the oversight tiers below, made concrete. The decision traces in reasoning and memory, and the observability and evaluation service in the control plane, are the evidence those tiers depend on.
Tier on three variables: how far the consequence reaches, whether it can be reversed, and whether the action was authenticated.
That triple comes from the Cloud Security Alliance's agentic profile work (April 2026). It is the most usable tier-assignment rule published, and it is compatible with the only proportionality language in the EU AI Act, Article 14(3), which ties oversight intensity to risk, autonomy level and use context.
| Tier | What it means | Assign when | Review cadence and evidence |
|---|---|---|---|
| Automated | The system acts. No human sees the individual output. | Consequence is contained, the action is cheaply reversible, and the trigger is authenticated. | Quarterly audit of a sampled population. Logs retained and tamper-evident. Drift alarms on the input distribution. |
| Sampled | The system acts. A random subset is reviewed after the fact. | Consequence is moderate, reversal is possible but costs something, volume makes full review impossible. | Weekly review of a random subset sized for the error rate you need to detect, not for what reviewers have time for. |
| Reviewed | A human approves every output before it takes effect. | Consequence reaches a customer, a regulator or the balance sheet, and reversal is expensive or public. | Every item. Reviewer must be able to see the inputs, the retrieved evidence and the tool calls, or the review is theatre. |
| Advisory | The system analyses. A human decides and owns the decision. | Reversal is impossible, or accountability is personal and non-delegable under regulation. | Every item, with the human's reasoning recorded separately from the system's. Two-person verification where the law requires it. |
If you have delegated quality measurement to a model judging another model, this is the most important research published in 2026 on the subject. A systematic evaluation of 21 judge models across 9 providers (June 2026) found:
There is a second measurement problem underneath: reference instability. Airbnb's evaluation engineering work found around 75 percent of generated references differ across labelling runs on identical inputs, while the true signal they were trying to detect was 1 to 3 percent. Most observed variance is noise. If your evaluation sample is sized for convenience, you are measuring your own jitter.
And golden sets decay as control artefacts even when the model has not changed. Evaluation datasets need versioning and expiry the same way models do. There is currently no NIST, ISO or regulatory-published evaluation methodology for production LLM systems. NIST's generative AI profile tells you what to measure conceptually across its twelve risk categories; it does not tell you how to gate a deployment. That gap is yours to fill and to document.
Article 14(4) requires the overseer to correctly interpret the output. Colorado's replacement law requires a plain-language explanation of a decision and the role of the automated system within 30 days of an adverse outcome. Both presume the decision can be explained afterwards. For a non-deterministic model on a vendor endpoint that may have been silently re-versioned, you cannot re-run the decision.
The only defensible substitute is capturing, at decision time: the full input context, the retrieved documents, every tool call and its result, the exact model identifier, and the sampling parameters. That is a design decision made before the incident, not after. ISO/IEC 42001 is what auditors are increasingly asking for as evidence of a governed management system, and it is certifiable, which matters when the question is whether you can show a third party rather than assert it. On the reference architecture these are the decision traces, and they are what the evidence and provenance control draws on.
On agents specifically, the standards work is real and immature. Decentralised identifiers and workload identity credentials issued through an agent name service, agents registered in existing identity infrastructure, a delegation register tracking authority lineage, and pre-authorised automatic suspension for high-severity incidents. Several IETF drafts exist on agent authentication and trust negotiation; all are individual submissions with no consensus status, so do not cite them as standards. Agent spend limits are a genuine white space: no published framework treats them as a first-class control, and they are trivially easy to implement and easy to forget.
Does governance accelerate delivery or kill it?
Databricks telemetry across more than 20,000 organisations including over 60 percent of the Fortune 500: companies actively using AI governance tooling put 12 times more AI projects into production, and evaluation and monitoring practices correlate with roughly 6 times higher production rates. Organisations with formalised AI risk and compliance policies show ROI at 2.2 times the rate of those without. And the 84 percent reduction in permission prompts after sandboxing is a mechanistic example of containment removing friction.
Four regulators moved in the deregulatory direction within fifteen months. The EU deferred Chapter III by sixteen months citing innovation and competitiveness. Colorado repealed its own duty-of-care and impact-assessment regime before it took effect. The US banking agencies dropped annual validation and excluded generative AI from scope. BCG's own diagnosis: existing risk programmes were designed for the world three or four years ago and fall apart at scale.
Name the confound before using the 12x. Databricks, Dataiku and the survey houses producing these multiples all sell into the governance market, and the figures are correlational with an obvious confound: organisations mature enough to deploy governance tooling are mature enough to ship. Nobody has published a controlled study. Cite the 12x with that attached, or it will be dismissed by the first person in the room who knows.
The model underneath you will change every few weeks. Build the capability to absorb that, not a project that assumes it will not.
The control plane along the bottom: the registry that pins model versions, the environment and deployment service that shadows and promotes them, the gateway that routes between them, and the feedback loop that keeps the whole picture current as the models at the core change.
This is the pivot from managed to resilient, and it is the one that separates a platform from a collection of successful pilots.
Uber states the reality plainly: models are continuously re-evaluated because the frontier shifts every few weeks. Every managed agent carries its own real-work benchmark precisely so re-evaluation is mechanical rather than a judgment call each time. The practices that hold up:
A model version change invalidates the entire prompt cache prefix. Every request that was reading cache at 0.1x becomes a cache write at 1.25x or 2x, simultaneously, across all traffic. Uber's session-analysis dashboard flags cache expiration inefficiency as one of sixteen tracked anti-patterns, which tells you it is a real operational cost. I could not find a published post-mortem quantifying the spike, so treat the magnitude as unquantified and the direction as certain. Model your migration day, do not discover it.
| Organisation | What changed | Measured result |
|---|---|---|
| Uber | Gateway, shared spend tiers, code-mode tool calls, subagents defaulted to cheaper models, 400K auto-compaction | Cost per 1,000 requests down 34 percent, cost per session down 52 percent, through 9.4x request growth with spend broadly flat |
| Robinhood | Hierarchical tuning: prompt optimisation, then trajectory tuning, then a small fine-tuned model | P90 latency 55 seconds to under 1 second, with an 8B fine-tune matching frontier quality for that use case |
| Care Access | Separated static records (cached) from dynamic questions | 86 percent cost reduction and 3x speedup |
| Shopify | Prompt compression to learned gist tokens | Latency down 38 percent, GPU allocation down 14 percent, compression cost paid once at training time |
Every one of these is a platform capability, not a project outcome. That is the point of the second pivot. If your second use case did not cost less than your first, nothing was industrialised.
There is no controlled comparison of centre-of-excellence against federated against hub-and-spoke for AI. Anyone claiming one performs better with evidence is overstating. What is citable:
The clearest signal from the LangChain production survey is the spread between 89 percent with observability and 37.3 percent with online evaluation. Tracing is a solved problem with a competitive tool market. Measuring whether the thing is still right, in production, on live traffic, is not. That gap is the highest-value platform investment available right now, and it is what converts consideration 06's governance model from a document into a control.
Agentic memory, protocols, and the argument that the word to watch is coupling. What you are actually deciding when you choose how tightly the application binds to the model underneath it.
Read the paperOrdered by the ratio of evidence behind them to effort required.
Forty-two percent of organisations have nobody. Where a single executive owns it, established ROI runs 3.5 times higher. This costs a meeting.
On the reference architecture: Human / Business JudgmentCost per resolved case, per merged change, per document. A token-denominated chargeback systematically rewards the model that fails cheaply.
On the reference architecture: Continuous Learning & FeedbackThe measured lift from semantic grounding exceeds the lift from upgrading the model, in three independent settings. Certified example queries and join metadata are the cheapest part and carried most of the gain at LinkedIn.
On the reference architecture: Data & Knowledge LayerUniform oversight produces rubber-stamping, which Article 14(4) treats as a compliance failure in itself. Tier on consequence, reversibility and authentication, then actually pay for the review capacity at the top two tiers.
On the reference architecture: Judgment & Trust GateIt does not give you a maturity score. Scores create the boil-the-ocean failure mode that Gartner's own definition of AI-ready data argues against, and they let everyone agree on a number without anyone owning a decision. Every section here ends in two columns instead: what a group works out, and what one named person owns. If you cannot fill the right-hand column, the left-hand column will not save you.
Primary where possible. Where a finding routes through a secondary aggregation or where I could not reach the original, it is flagged in the text rather than laundered into a clean citation.
McKinsey, The State of AI in 2026: On the road to ROI, Aug 2026, n=1,719
KPMG Global AI Pulse, Jun 2026, n=2,145
EY US AI Pulse Survey Wave 5, May 2026, n=534
Bain, Executive Survey: AI Moves from Pilots to Production, Nov 2025
FinOps Foundation, State of FinOps 2026, n=1,192
FinOps Foundation, FinOps for AI working group guidance, Feb 2026
Uber Engineering, Running a Software Factory Efficiently at Uber Scale, Aug 2026
DoiT, Cost Per Task, Not Cost Per Token, Aug 2026
Gartner, Over 40% of Agentic AI Projects Will Be Canceled by End of 2027, Jun 2025
Gartner, Lack of AI-Ready Data Puts AI Projects at Risk, Feb 2025
Accenture, AI-Ready Data and the data reinventors, May 2026, n=2,000
Anthropic and Accenture, Deploying AI from pilot to production, 2026
Precisely and Drexel LeBow, State of Data Integrity and AI Readiness 2026, n=505
Cloudera and Harvard Business Review Analytic Services, Mar 2026
dbt Labs, Semantic Layer vs Text-to-SQL benchmark, Apr 2026
Snowflake, Cortex Analyst text-to-SQL accuracy, Aug 2024
Spider 2.0 (Lei et al.), ICLR 2025
BEAVER: An Enterprise Benchmark for Text-to-SQL, 2024
LinkedIn SQL Bot, knowledge graph accuracy analysis
Ghost Vectors: Soft-Deleted Embeddings Remain Reconstructible, Jun 2026
Morris et al., Text Embeddings Reveal (Almost) As Much As Text
Anthropic, Building Effective Agents, Dec 2024
Anthropic, How we built our multi-agent research system, Jun 2025
Cognition, Don't Build Multi-Agents, Jun 2025
Cognition, Multi-Agents: What's Actually Working, Apr 2026
Anthropic, Effective context engineering for AI agents, Sep 2025
Chroma Research, Context Rot, Jul 2025
Anthropic, Code execution with MCP, Nov 2025
tau-bench and the pass^k metric, 2024
METR, Time Horizon 1.1, Jan 2026
METR, Measuring the Impact of Early-2025 AI on Experienced Developers, Jul 2025
METR, We are Changing our Developer Productivity Experiment Design, Feb 2026
Menlo Ventures, State of Generative AI in the Enterprise, Dec 2025
LangChain, State of Agent Engineering 2026, n=1,340
Netflix, In-House LLM Serving, Jul 2026
Shopify Engineering, Gisting, Aug 2026
EchoLeak, ForcedLeak, ShareLeak and PipeLeak incident analysis, 2026
Snyk, Malicious MCP server on npm, Sep 2025
MITRE ATLAS v2026.08, Sep 2026
Anthropic, How we contain Claude, May 2026
EU AI Act, Article 14, Human oversight
Regulation (EU) 2026/1744, the AI Omnibus, Jul 2026
OCC, Federal Reserve and FDIC, revised model risk management guidance, Apr 2026
Reliability without Validity: LLM-as-a-Judge across agreement, consistency and bias, Jun 2026
NIST AI 600-1, Generative AI Profile, Jul 2024
Cloud Security Alliance, Agentic Profile v1 (draft), Apr 2026
Moffatt v. Air Canada, 2024 BCCRT 149
Fenwick, The end of silent AI: emerging AI exclusions in insurance, 2026
DORA, 2025 State of AI-assisted Software Development, Sep 2025
Microsoft Work Trend Index 2026, n=20,000
Dell'Acqua et al., Navigating the Jagged Technological Frontier, Organization Science 2025
KPMG and University of Melbourne, Trust, Attitudes and Use of AI, n=48,000+, 47 countries
Harvard Business Review, AI Adoption Is Overloading Your Middle Managers, Jun 2026
Zylo, 2026 SaaS Management Index, 40M licences
TIME and Charter, What Klarna learned from its AI rollout
Deloitte, State of AI in the Enterprise 2026, n=3,235
BCG, How to speed up AI risk management, Sep 2026
Airbnb, Making LLM evaluation fast enough to iterate on, Jul 2026
a16z, Enterprise AI arms race, Jan 2026, n=100 Global 2000