Understanding Enterprise AI Risk

An employee pastes a client contract into a chat window

That is the whole subject in one gesture. Nothing about it is exotic. There is no AGI in it, no rogue agent, no existential scenario. There is a person with a deadline, a tool that is genuinely useful, a company that has probably written a policy about this and almost certainly cannot enforce it, a vendor whose terms say one thing about training and something narrower about retention, and a general counsel who will not learn any of it happened unless something goes wrong. Every expensive problem in enterprise AI is a variation on that scene — mundane, already occurring, and structural.

The problems that actually cost enterprises money are unpredictable unit economics (what a single unit of work — a query, a document, a resolved ticket — actually costs to run), data-governance regimes built for a pre-LLM world (before large language models: the text-in, text-out systems behind ChatGPT and its rivals), and organizations that bought capability faster than they built the muscle to own it — not the frontier scenarios that dominate the discourse.

This primer is written for someone operating as — or becoming — the person accountable for AI strategy inside a company. It is not investment research and contains no view on any company as a security. Model vendors, cloud providers, integrators and regulators appear here as actors in a system: what they do, what they control, and why their behaviour propagates into a buyer's risk register.

The organizing argument is causal, not descriptive. A pilot succeeding is not the same as a deployment being governed. A written AI policy is not the same as an enforceable control. A cheaper model is not the same as a predictable bill. And "your data is not used for training" is a considerably narrower promise than most buyers hear. Each of those gaps is a mechanism, and each one has a documented failure record and — in most cases, though not all — a named response pattern that some organizations are demonstrably running. Where no reliable response pattern exists, this primer says so; an unsolved problem flagged honestly is more useful to a playbook than a manufactured best practice.

1. 1Falling token prices and rising task costs are two different curves. A token is the chunk of text a model meters and bills — roughly three-quarters of an English word. The same 50k-input, 2k-output job on one model spans $0.06 to $0.55 — a 9.2× swing driven purely by caching (reusing already-processed prompt text at a discount), batching (accepting slower delivery for a cheaper rate), reasoning tokens (the model's hidden working-out, billed as output), tokenizer (how text is cut into tokens) and inference-geography settings (where the computation physically runs), before a single line of business logic changes. Known Known
2. 2Better cost measurement surfaces overruns; it does not stop them. In a February 2026 survey of 500 finance leaders, organizations with mature FinOps practice — the discipline of tracking, allocating and controlling cloud and AI spend — reported AI overruns more often (89%, mean overspend 30.9%) than early-stage ones (69%, 16.1%). Estimated
3. 3Shadow AI is now the majority data-exposure channel, not an edge case. Shadow AI — employees putting company data into AI tools the company holds no contract over — featured in 43% of breached organizations in IBM's 2026 study, roughly double the prior year, at an average $5.39m per incident. Known Known
4. 4Prompt injection is bought off with capability, not patched away. Prompt injection is text planted in a document, email or web page that the model reads as an instruction rather than as content. The strongest published defence, CaMeL (a research design that keeps untrusted text from steering what the program does), holds 77% of task utility against 84% undefended and costs roughly 2.7× the tokens; OWASP (the open security-community project behind the standard application-security risk lists) states plainly that fool-proof prevention is unclear, and OpenAI runs a standing automated adversarial attacker against its own browser agent. Known Known
5. 5The EU obligations biting a deployer today are the cheap ones, not the high-risk regime. A deployer, in the Act's language, is any organization using an AI system in its own operations, as distinct from the firm that built it. Annex III high-risk duties (the heaviest conformity regime, attaching to the uses the Act classes as high-risk) slipped to 2 December 2027 under Regulation (EU) 2026/1744, while AI-literacy (Art. 4) has bound every deployer since February 2025 — softened by the same instrument from an obligation of result to one of means, with its scope untouched — and Article 50 transparency (telling people they are dealing with an AI, and marking synthetic content) has applied since 2 August 2026. Known Known
6. 6Adoption is near-universal; scaled operation is not. 88% of organizations use AI in at least one function, 39% report enterprise-level EBIT impact, 23% are scaling an agentic system — one that plans and takes actions with tools, not merely answers — anywhere, and no more than 10% in any single function. Estimated
7. 7The gap between a large and a small company is not risk appetite — it is who absorbs the vendor's decisions. A 250-plus-employee US firm is roughly twice as likely to use AI as a sub-20-employee one (37% vs under 20%), and is the only one of the two with the leverage to negotiate retention terms. Known Known

§The stakes, quantified

Three numbers set the scale. Enterprise spending on generative AI reached roughly $37 billion in 2025 on Menlo Ventures' bottoms-up estimate, up from $11.5bn in 2024 and $1.7bn in 2023 — of which foundation-model APIsA foundation model is a large general-purpose model trained once, at great expense, and then adapted to many uses. An API — application programming interface — is the machine-to-machine door into it: a company's own software sends text in and gets text back, with no person opening a chat window. alone accounted for $12.5bn (Menlo Ventures, 9 Dec 2025). That figure deliberately excludes chips, model-serving infrastructure and AI features bundled into existing software, so it understates total outlay; it is a measure of net-new discretionary AI budget, not of what AI costs a company.

Second, exposure. IBM's 2026 Cost of a Data Breach study — 602 breached organizations across 17 industries and 16 countries, breaches occurring March 2025 to February 2026 — found shadow AI implicated in 43% of breached organizations, up from 20% a year earlier, with those incidents averaging $5.39m against a global all-breach average of $4.99m, itself up 12% year on year (Cybersecurity Dive, 29 Jul 2026; population and headline cost confirmed in IBM's own release, 29 Jul 2026, which adds that roughly one in four malicious breaches is now AI-enabled, at about $6m each — some $1m above the average, with such attacks up 56% year on year). Note the denominator: this is 43% of organizations that suffered a breach, not 43% of all organizations. It measures how often ungoverned AI shows up at a crime scene, not how often ungoverned AI causes one.

Third, the deployment gap. Stanford HAI's 2026 AI Index puts organizational AI adoption at 88% and generative-AI use in at least one function at 70% (AI Index 2026, Economy chapter), while McKinsey's State of AI work finds only 39% reporting EBIT impact at the enterprise level and 23% scaling an agentic system anywhere in the business, with no function above 10% (Forbes reporting McKinsey, 22 Mar 2026). Gartner's read is blunter still: at least half of generative-AI projects will overrun their budgets on architectural and operational grounds, and most organizations attempting custom models will abandon them to cost, complexity and technical debt (The Register on Gartner's 2026 Hype Cycle, 28 May 2026).

Sources, left to right: Menlo Ventures (Dec 2025); IBM Cost of a Data Breach 2026 via Cybersecurity Dive (Jul 2026); Stanford HAI AI Index 2026 and McKinsey State of AI; DoiT/Sapio Research survey of 500 finance leaders (Jun 2026). Populations differ — see each section for scope.

Put those side by side and the shape of the problem appears. Money is going in at a rate that tripled in a year. Value is coming out for a minority. And the loss events cluster not around the technology's frontier capabilities but around its most boring properties: that it is metered, that it ingests whatever you give it, and that it does not behave the same way twice.

By the end of this part a reader should be able to sit across from a sceptical CFO or general counsel and explain, without hand-waving, exactly what happens to a piece of data the moment an employee sends it to an AI tool, and exactly why the invoice for doing so is hard to forecast. Everything in Part II is a consequence of the machinery described here.

1.1The unit of account is the token, and the token is not a stable unit

Start with the invoice, because that is where most executive misunderstanding lives. A language-model API charges separately for tokens sent in and tokens generated out, and output is dearer — typically by a factor of five. Anthropic's published rates put Claude Opus 5 at $5 per million input tokens and $25 per million output; Sonnet 5 at $2 and $10; Haiku 4.5 at $1 and $5 (Anthropic pricing documentation). OpenAI's published table puts gpt-5.6-sol at $5 in and $30 out, gpt-5.6-terra at $2 and $12, and gpt-5.6-luna at $0.20 and $1.20, with cached input at roughly a tenth of standard input and a 50% batch discountWork submitted as a batch is queued to be completed whenever the vendor has spare capacity, rather than answered immediately. The buyer trades latency for a lower rate; the vendor gets to fill troughs in demand. It only helps workloads nobody is waiting on — overnight document processing, back-catalogue classification — and is useless for anything a customer is watching. (OpenAI API pricing). Those numbers are public, stable enough to plan against, and almost entirely useless on their own for forecasting a bill.

They are useless because the token count, not the token price, is the volatile term — and four separate mechanisms move it.

The context windowThe total amount of text a model can hold in view for a single request — the prompt, the conversation so far, the documents attached, everything. It is short-term working memory that is rebuilt from scratch on every call and forgotten the moment the call ends. Measured in tokens, and paid for in tokens. accumulates. Everything in a request counts: the system promptThe standing instruction the developer puts in front of every request — the model's job description, tone, rules and constraints. The user never sees it, and it is re-sent and re-billed on every single call., every prior message, every tool definition, every tool result, every attached document. Anthropic's own documentation is explicit that "everything in the request counts toward the context window" and that cached prefixes "still occupy the context window: prompt caching changes what you pay for those tokens, not whether they count" (Anthropic, context windows). A ten-turn conversation is not ten requests of equal size; it is ten requests of monotonically increasing size. In an agent loop where each tool call appends its output to the transcript, cost grows roughly with the square of the number of steps.

Reasoning tokens bill as output. Models that "think" before answering generate a hidden intermediate trace. Those tokens are billed at the output rate, and on newer Anthropic models the thinking blocks from previous turns are retained by default and then billed again as input on subsequent requests: "thinking tokens are billed as output tokens once, when they are generated. On models that keep previous thinking blocks, the kept blocks are then part of later requests' input and are billed as input tokens." A workload that migrates from a non-reasoning to a reasoning model can multiply its output token count severalfold with no change to the visible answer length.

The tokenizer itself changes. Least appreciated of the four, and the most instructive. Anthropic's pricing page carries a note that Claude 4.7 and later models "use a newer tokenizer that contributes to their improved performance" and that this tokenizer "produces approximately 30% more tokens for the same text." A buyer who moved from an older model at $3/$15 to a newer one at $2/$10 would read a 33% headline price cut. On the same corpus of text, the metered volume rises about 30%, and the realized saving collapses to roughly a tenth of what the price sheet implied. Nothing was hidden — the disclosure is on the vendor's own page — but nothing in a standard procurement comparison would catch it either.

Where the inferenceRunning a finished model to produce an answer, as against training it in the first place. Training is a one-off capital event borne by the vendor; inference is what happens every time an employee sends a prompt, and it is the part an enterprise pays for continuously. physically runs is a price modifier. Both major vendors now charge a premium for constrained routing. Anthropic applies a 1.1× multiplier across all token categories when a customer pins inference to the United States, and partner clouds apply a 10% premium on regional versus global endpoints. OpenAI's pricing page notes that "regional processing (data residency) endpoints are charged a 10% uplift." Data residency, in other words, is not a compliance checkbox that is free once negotiated; it is a standing 10% tax on the AI line, and it interacts with everything else on this list multiplicatively.

That figure holds the task constant and varies configuration, which is the point. It does not compare vendors, because a like-for-like vendor comparison would need identical tokenizers, identical caching semantics and identical reasoning behaviour, and none of those hold. Cross-vendor cost-per-token tables are among the most misleading artefacts in enterprise AI procurement for exactly this reason.

Context rot: why bigger windows do not solve the problem they appear to solve

Context windows have grown to a million tokens on frontier models, available by default and billed at standard rates (Anthropic, context windows). The intuitive conclusion — just put everything in — is wrong, and the vendor says so. The same documentation states that "as token count grows, accuracy and recall degrade, a phenomenon known as context rot," and Anthropic's engineering write-up attributes it to the transformer's n² pairwise attention relationships and to a training distribution dominated by shorter sequences, noting that needle-in-a-haystack benchmarking "has uncovered context rot across all models" (Anthropic engineering, 29 Sep 2025).

The naive response to a quality problem — give the model more context — raises cost and can lower accuracy simultaneously. The disciplined response is retrieval and compactionPeriodically replacing a long transcript with a shorter summary of it, so the working context stays small while the thread of the task survives. The cost is whatever detail the summary drops.: load the smallest high-signal set of tokens, summarize history server-side, and let sub-agents return condensed results. Every one of those is engineering work that a pilot does not need and a production system cannot do without. It is the first concrete example of the pattern that runs through this entire primer: the thing that makes a demo work is not the thing that makes a deployment work, and the difference is paid for in engineering, not licence fees.

1.2Why the same workload's bill moves month to month

A traditional software line item is a licence: negotiated, annual, boring. An AI line item is a utility meter attached to a system whose consumption depends on user behaviour, model version, retry logic, and the length of documents people happen to upload that month. Each mechanism has a different owner inside a company, and a cost problem attributed to the wrong owner never gets fixed.

Cache hit rateVendors will store the already-processed front portion of a prompt — the prefix — and charge a fraction of the normal rate to reuse it on the next call. A hit is a call that reuses it; the hit rate is the share of traffic that does. The catch is that the match must be exact from the first character, so a single changed word near the front throws the whole prefix away. is an engineering property, and it swings the bill by a factor of two to three. Anthropic's caching economics are explicit: a cache read costs 10% of the standard input price, a five-minute cache write costs 1.25×, and a one-hour write 2× — so "caching pays off after one cache read for the 5-minute duration, or after two cache reads for the 1-hour duration." A system whose prompts vary slightly at the front — a timestamp, a user name, a reordered instruction — invalidates the prefix and silently pays full freight. Nobody gets an alert. The bill simply arrives higher.

Retry and fan-out logic is an architecture property. An orchestration that calls three models and picks the best answer costs three times a single call plus the judge. An agent that retries on tool failure has an unbounded worst case unless someone bounded it.

Server-side tool use introduces charges that are not tokens at all. Anthropic bills web search at $10 per 1,000 searches on top of token costs, and prices code-execution containers at $0.05 per hour beyond a free allowance, with a five-minute minimum per execution. A research agent that issues ten searches per run has added a dime to that run before it generates a word. At a million runs a month that is $100,000 in a line item that no token forecast contains.

Vendor pricing changes are the term the buyer controls least. These move in both directions and both directions are disruptive. Anthropic's pricing page records that the introductory $2/$10 rate for Claude Sonnet 5 — originally scheduled to rise to $3/$15 on 1 September 2026 — was made permanent instead, and that "the previously scheduled increase will not occur." That is a buyer-favourable change, and it still breaks a budget model built on the announced increase. The general lesson is that a company's AI cost base is partly set by decisions made by three or four firms that do not consult it.

The surveys have caught up with the mechanism, and they are not describing a theoretical problem. A February 2026 survey of 500 finance leaders at 1,000-plus-employee organizations in the US and UK, run by Sapio Research for DoiT, found 79% reported AI cost overruns in the prior twelve months, and only 15% of finance leaders could calculate AI ROI without significant obstacles (DoiT/Sapio, 9 Jun 2026). A separate survey of 300 executives by WitnessAI found 68% reporting at least some AI initiatives over budget and 33% saying overruns happened "mostly or always," with only 9% saying more than three-quarters of their AI initiatives delivered measurable financial returns (CFO Dive, 22 Jul 2026). Flexera's 2026 data adds the visibility dimension: 59% report increased wasted spend on AI software year over year — a higher rate than for public cloud (44%) or SaaS (43%) — and only 31% claim accurate visibility into AI software spending (Flexera, 20 Jul 2026).

The obvious reading of that comparison is wrong, and the correct one is more useful. The DoiT data shows organizations with mature FinOps practice reporting overruns more often and larger than early-stage ones. The naive inference — that FinOps maturity causes overspending — is absurd. Two better explanations compete. The first is measurement: a mature practice has the instrumentation to detect an overrun and a baseline to detect it against; an immature one has an invoice and a shrug. The second is selection: organizations that build mature AI FinOps are disproportionately those with large, volatile AI estates, which are exactly the estates that overrun. Both are probably operating. Either way the practical implication for a strategy leader is the same and it is not the comfortable one: standing up AI cost governance should be budgeted as a discovery exercise whose first output is bad news, not as a control that will bend the curve in its first year.

The FinOps Foundation's own community data confirms how fast this became everyone's problem. Its sixth annual survey — 1,192 respondents representing more than $83bn in annual cloud spend — reports that 98% now manage AI spend, up from 63% in 2025 and 31% in 2024, that "FinOps for AI" is the top forward-looking priority, and that AI cost management is the single most-wanted skill across organizations of every size (State of FinOps 2026). The named difficulties are precisely the mechanisms above: visibility into costs across providers with different pricing models, allocation of AI cost to business units, and determining value while investments remain exploratory.

The response pattern, and its honest limit

Organizations that handle AI cost competently do three concrete things. They route all model traffic through a gateway so that spend is attributable before it is aggregated. They tag and chargebackPassing the cost of shared infrastructure back to the internal team that consumed it, so the spend lands on the budget of whoever caused it rather than in a central pool nobody owns. by team, feature and environment. And for high, steady volumes they buy reserved capacity instead of paying per token.

The third of those has a limit that the vendor documents and almost nobody reads. Microsoft's provisioned throughput is the canonical instrument: a fixed allocation of model processing capacity billed per PTU-hourA provisioned throughput unit is a block of model-serving capacity held open for one customer. Buying PTU-hours is renting a slice of the vendor's machines by the hour, the way one reserves a server, instead of paying by the token the way one pays for electricity. rather than per token, with Azure Reservations giving a discounted rate in exchange for a one-month or one-year commitment. It converts a variable cost into a fixed one and adds a latency target. But Microsoft states in the same document that "having PTU quota doesn't guarantee that capacity is available. If capacity in the region is insufficient for the requested PTU count, the deployment fails," that capacity availability "changes throughout the day based on customer demand," and that "reservations don't guarantee capacity" — the recommended sequence is to create the deployment first, confirm capacity exists, and only then buy the reservation (Microsoft Foundry, provisioned throughput, updated 15 Jul 2026).

The standard enterprise instrument for making AI spend predictable buys a price, not a supply. The buyer converts price risk into availability risk and keeps the latter. For a bank running a customer-facing assistant on provisioned capacity, the residual exposure is not a surprise invoice; it is a failed scale-up during a demand spike in a region where the hyperscalerOne of the handful of cloud providers operating data centres at global scale — Amazon Web Services, Microsoft Azure, Google Cloud. They sit between the model vendor and the buyer, and they are where the physical capacity actually is. has no spare acceleratorsThe specialized chips that run model inference — GPUs and their equivalents — as opposed to ordinary server processors. When a region "has no capacity," these are the thing it has none of.. That risk sits nowhere on a traditional software risk register, because traditional software does not have one.

Evidence: Microsoft Foundry provisioned-throughput documentation (updated 15 Jul 2026); Anthropic and OpenAI published pricing pages. Method: qualitative contrast of risk allocation under the two commercial models. Synthesis: the AI line item transfers two risks to the buyer that enterprise software historically did not.

1.3Build, buy, fine-tune, retrieve — what each one actually does with your data

Four architectures dominate enterprise deployments, and they differ less in capability than in what they do to a company's data, its cost structure and its exit options. Confusing them is the most common source of a governance conversation that goes nowhere, because the legal team's question — "does our data leave, and does it stay?" — has a different answer for each.

Prompting an API. Data leaves the building for the duration of the request and, under standard commercial terms, is retained briefly for abuse monitoring and then deleted. Nothing is learned. Cheapest to start, cheapest to abandon, and the pattern with the fewest lineage problems.

Retrieval-augmented generation (RAG). The company indexes its own content, retrieves the relevant fragments at query time, and pastes them into the prompt. The model learns nothing; the index does the remembering. This is the dominant enterprise pattern and it relocates the governance problem rather than solving it. Microsoft's own RAG documentation names the security challenge in a single line — "opening private content to LLMs requires granular access control. Users and agents must only retrieve authorized content" — and lists the mitigations: document-level security trimmingFiltering search results against the asking user's own permissions before anything is returned, so a person only ever sees passages from documents they were already entitled to open. Without it, the index answers on behalf of everyone at once., inherited Entra IDMicrosoft's corporate identity directory — the system that knows who each employee is, which groups they belong to and what they may open. "Inherited permission metadata" means the search index carries those entitlements alongside each document instead of forgetting them. permission metadata, filter-based security at query time, and network isolation (Azure AI Search, RAG overview, updated 4 Aug 2026). The failure mode is exact and worth memorizing: a RAG index that does not carry source permissions turns every access-control mistake in the underlying repository into a conversational search engine. The finance folder that was technically readable by everyone but that nobody ever found is now one natural-language question away.

Fine-tuning. The company's data is used to adjust a model's weightsThe billions of numerical parameters inside a trained model. They are what learning changes, and they encode statistical patterns rather than a filing cabinet of the text they were trained on — which is why a single document cannot be surgically pulled back out of them.. This is the only one of the four where the data genuinely becomes part of an artefact, and the governance consequences differ in kind: a fine-tuned model cannot be un-taught a document the way an index can have a row deleted, and the retention question shifts from "how long do you keep my prompt" to "what happens to my derived weights." Microsoft's lifecycle documentation makes the operational cost concrete — fine-tuned models retire in two phases, training and deployment, and a base model's retirement caps the life of everything trained on it.

Agents. The model is given tools and permission to act. A tool is a function the model is allowed to call — search this index, send this email, run this query, refund this order — and an agent is a model in a loop: it picks a tool, reads what comes back, and decides from that what to do next, over and over, until it judges the task done. Nobody writes that sequence in advance. The model composes it at runtime, and that is the whole difference between an agent and a chatbot with a plug-in. Everything above still applies, plus a new class of exposure that Part II treats at length. For the purpose of this section, the important property is that an agent's data flow is not knowable in advance: which documents it retrieves, which systems it touches and what it sends outward are determined at runtime by the model. That single property is what breaks the standard data-protection impact assessmentThe formal written analysis European data-protection law requires before a risky processing activity begins. It sets out what personal data will be handled, by what operations, for what purpose and with what safeguards — a description that has to be written down in advance and kept accurate., which assumes an enumerable set of processing operations.

The live debate here is whether fine-tuning is worth its governance cost at all in 2026. The case for: a tuned smaller model can be dramatically cheaper per task at high volume and can encode format and tone in ways prompting handles clumsily. The case against: base-model improvement has repeatedly outrun the gains from tuning an older model, the artefact's life is capped by the base model's retirement schedule, and Gartner's read is that most organizations attempting custom models "will abandon their efforts due to costs, complexity and technical debt." The evidence tilts against fine-tuning as a default and leaves it defensible for narrow, high-volume, stable tasks — classification, extraction, structured formatting — where the base model's improvement curve is least relevant. That is a real position, and it is falsifiable: if a serious longitudinal study shows tuned-model deployments outliving two base-model generations at lower total cost than prompting equivalents, it should be abandoned.

1.4What actually happens to a piece of data when an employee sends it

Here is the answer to give the general counsel. It has five stages, and the confusion in most boardrooms comes from collapsing stages two, three and four into a single question about "training."

1. 01
   Egress

   The text leaves the corporate boundary. Which boundary it crosses depends entirely on the account: a corporate API key, a licensed enterprise seat, or an employee's personal login.
2. 02
   Inference

   The text is tokenized and processed. It is held in memory for the request. On global endpoints it may be processed in any region unless the customer has paid the residency premium.
3. 03
   Retention

   Under commercial terms, prompts and outputs are typically stored ~30 days for abuse and safety monitoring, then deleted — unless flagged, or unless the law requires longer.
4. 04
   Training use

   Under commercial terms, none. Under consumer terms, potentially yes, with retention that can extend to years. This is the stage everyone asks about and the least likely to be the actual exposure.
5. 05
   Compulsion

   Retained data can be preserved and produced under a court order in litigation the customer is not party to. This stage is absent from almost every vendor risk assessment.

Evidence: OpenAI API data-usage guide (abuse-monitoring logs retained up to 30 days "unless longer retention is required by law"); Anthropic data-retention practices for covered models (9 Jul 2026); Google Gemini API Additional Terms (effective 23 Mar 2026); OpenAI's account of the preservation order in NYT v OpenAI. Method: Stages reconstructed from the three vendors' own published terms; stage 5 from the litigation record. Synthesis: Stages 3 and 5 carry the enterprise exposure; stage 4 carries the boardroom attention.

Take the stages in turn against the actual documents.

Training use is genuinely off by default on commercial tiers — and genuinely on by default on some consumer ones. OpenAI's developer documentation states that "data sent to the OpenAI API is not used to train or improve OpenAI models (unless you explicitly opt in)" (OpenAI, your data). Anthropic's commercial position is the same, and its consumer position changed in 2025: from 28 August 2025, data from Claude Free, Pro and Max accounts is used to train models unless the user opts out, with retention extended to five years for users who allow it, and users given until 8 October 2025 to choose — while the update explicitly does not apply to Claude for Work, the API, Bedrock, Vertex, Claude Gov or Claude for Education (Anthropic, 28 Aug 2025). Google draws the line by payment rather than by product family: for unpaid services, "Google uses the content you submit to the Services and any generated responses to provide, improve, and develop Google products" and "human reviewers may read, annotate, and process your API input and output," with an instruction not to submit sensitive or confidential information; for paid services, Google "doesn't use your prompts... or responses to improve our products" but still logs them for a limited period for policy enforcement (Gemini API Additional Terms, effective 23 Mar 2026).

The training guarantee attaches to the contract, not to the model. Same vendor, same model, different tier, opposite answer. An employee who uses the corporate API through an approved application is covered; the same employee, on the same model, through a personal login at home, is not. Shadow AI is not primarily a story about employees using forbidden tools. It is a story about employees using the same tools under a different contract.

Retention is a shorter and more conditional promise than most buyers register. Anthropic's privacy documentation states that prompts and outputs for covered models "are retained for 30 days," after which "the data is deleted automatically, except in the rare cases where it's been flagged by our automated trust and safety systems" or where Anthropic is "legally required to keep it" — and that this applies across the direct API, Bedrock, Google Cloud, Microsoft Foundry and Claude Code with Enterprise (Anthropic, 9 Jul 2026). OpenAI's documentation is parallel: "abuse monitoring logs are generated for all API feature usage and retained for up to 30 days, unless longer retention is required by law."

Zero data retention is the real control, and it has holes exactly where the industry is heading. Zero data retention, ZDR for short, means the vendor holds a request in memory long enough to answer it and writes nothing to disk afterwards. The phrase to hold onto is at rest: the data still travels and is still processed, it simply is never parked anywhere. That is what makes it the strong control — there is no stored copy to leak in a breach, to be read by a reviewer, or to be produced under an order months later. ZDR eliminates storage at rest, and OpenAI documents that with ZDR enabled "the store parameter will always be treated as false, even if the request attempts to set the value to true." But eligibility is per endpoint, not per account. OpenAI lists chat completions, responses, embeddings, moderation, audio and image endpoints as ZDR-eligible while assistants and conversation endpoints are not. Anthropic's public documentation notes that existing ZDR agreements do not extend to its most capable designated models. The pattern is consistent across vendors and follows directly from the mechanics: statefulnessA stateless service forgets everything the instant a request ends; a stateful one remembers between calls — a saved conversation, an uploaded file, an agent's running memory of what it has already done. Remembering is not a feature that can be bolted on without storage: it is storage. and zero retention are in tension. A batch job, a files API, a conversation object and a managed agent all need to hold something somewhere between calls. As enterprises move from stateless chat completions to stateful agents — which is precisely the 2026 direction of travel — the strongest available data control quietly stops applying.

And then there is stage five, which almost no vendor questionnaire asks about. On 13 May 2025, in The New York Times v OpenAI, Magistrate Judge Ona Wang ordered OpenAI "to preserve and segregate all output log data that would otherwise be deleted on a going forward basis until further order of the Court" — an order upheld by the district judge in June and reaching, on one contemporaneous estimate, some 60 billion conversations. ChatGPT Enterprise and educational accounts were excluded. OpenAI objected that fewer than 0.010% of the retained records would prove relevant to the copyright claims and that users expected deleted chats to be gone within thirty days per its own policy (National Law Review, 25 Jul 2025). The going-forward obligation was terminated as of 26 September 2025 by an order filed 9 October, but logs already preserved remain held, along with data tied to accounts the plaintiff has flagged — a list the plaintiff may still expand (Engadget, 11 Oct 2025).

A company's deleted AI conversations can be preserved by a court in litigation between two other parties, and the exemptions ran along contract lines — enterprise agreements and zero-retention terms — that most mid-market buyers do not have and often do not know to ask for. The implication is uncomfortable and it generalizes far beyond this one case. This is the clearest example in the whole subject of a risk invisible to the standard control framework: it is not a security failure, not a privacy breach, not a vendor breaking its word. Every party behaved correctly and the data was retained anyway. A retention policy that a court can suspend is a commitment about ordinary operations, not a guarantee — and it should be described that way to a board.

1.5The actor map: who controls what, and why it propagates

Most of the dependencies in enterprise AI become obvious once the actors are laid out and each one's actual lever is named. The system has seven of them, and the load does not sit where the org chart suggests.

1. 01
   Model vendors

   OpenAI, Anthropic, Google, Microsoft, Meta, Mistral and the open-weightModels whose trained parameters are published for anyone to download and run on their own hardware. Note the word: open weights is not the same as open-source software. The file is yours to run, but the licence attached to it can still restrict how you use it. ecosystem. They set token price, tokenizer, context limits, data-use terms, deprecation schedules and the capability frontier. Their commercial incentive is usage growth now and margin later, which makes today's price a policy choice rather than a cost-plus outcome.

   Sets the terms unilaterally
2. 02
   Cloud / capacity layer

   AWS, Azure, Google Cloud and the accelerator supply beneath them. They mediate availability, regional residency and the reservation instruments. Microsoft's own documentation warns that quota is not capacity and reservations do not guarantee it.

   Controls supply, not just price
3. 03
   IT and security (the CISO)

   Inherits an attack surface it did not design and often cannot see. Gravitee's 2026 survey of 900-plus executives and practitioners found only 14.4% of organizations said all AI agents went live with full security or IT approval.

   Accountable without control
4. 04
   Legal and compliance

   Asked to underwrite probabilistic output against deterministic obligations. Owns the vendor terms, the indemnity conditions, the retention exposure and — after Moffatt — the company's liability for what its automated systems tell customers.

   Real leverage, applied late
5. 05
   The CFO's office

   Faces a cost line that behaves like a utility bill attached to a research project. 98% of FinOps practitioners now manage AI spend; only 31% of organizations claim accurate visibility into it.

   Owns the meter, not the dial
6. 06
   Business units and employees

   The actual point of adoption and the actual point of exposure. Netskope's 2026 data has 47% of workplace generative-AI users on personal accounts. Employees are not adversaries here; they are a demand signal that arrived before the supply of governed tools.

   Where the data actually leaves
7. 07
   Regulators, standards bodies, integrators

   The EU AI Act and its enforcement machinery, NIST, ISO/IEC, sector supervisors — and the consultancies who sell the response and shape how organizations structure it. Rules are being written while adoption outruns them; the omnibus deferral of the high-risk regime is the clearest evidence that the rule-writers know it.

   Setting the clock, not the pace

Evidence: Anthropic and OpenAI pricing/terms pages; Microsoft Foundry provisioned-throughput doc (15 Jul 2026); Gravitee State of AI Agent Security 2026 (4 Feb 2026); Netskope Cloud and Threat Report 2026 via Infosecurity Magazine (7 Jan 2026); FinOps Foundation State of FinOps 2026; Flexera (20 Jul 2026); Regulation (EU) 2026/1744. Method: Actors ordered by distance from the buyer's control, with each verdict stating the specific lever that actor holds. Synthesis: Accountability concentrates at layers 3–5; control concentrates at layers 1–2. That mismatch is the structural cause of most of Part II.

One consequence of that map reframes almost every problem in Part II. The control plane is the set of switches that determine how a running system behaves — which version is executing, where it executes, what gets recorded and for how long, and when any of that is allowed to change. It is distinct from the work the system does; it is who gets to set the terms on which the work is done. In classic enterprise IT, the buyer owns the control plane: it decides when to upgrade, what version runs, how long logs are kept and where the workload executes. In enterprise AI, the control plane sits with a third party, changes on its schedule, and is priced on a meter the buyer cannot audit token by token. Every governance framework the industry is reaching for — NIST's, ISO's, the EU's — was designed for a world in which the deployer controls the system. That assumption is now false, and the frameworks are quietly straining against it.

Seven layers, each treated the same way: the mechanism that produces the failure, who inside the organization discovers it first and how long that takes, the evidence that it is real and costly, the response pattern that competent organizations are observed running, and the honest limit of that response. Then, at every layer, how the shape of the problem differs for a large enterprise and a small one — because it differs at every layer, in different directions, and flattening that into a single "SMEs have fewer resources" line would waste the most useful distinction in the subject.

2.1Economics: the line item that does not behave like software

Part I established the mechanisms. What breaks in practice is the budgeting cycle itself. A traditional software cost is committed once a year and consumed evenly. An AI cost is committed continuously and consumed at a rate set by whoever is using the product that week — which means the finance function loses the two properties it depends on: a forecast it can defend and a variance it can explain.

Who finds out first, and when. Almost always engineering, from a dashboard, weeks after the fact — or finance, from an invoice, a month after that. Neither is the person accountable for the budget. In the DoiT survey, the named barriers to calculating AI ROI were the pace of technological change (40%), finance and engineering defining success differently (37%), and lack of clear financial attribution (36%). That middle item is the diagnostic one: the disagreement is not about the number, it is about what the number is supposed to mean.

What it costs to remediate versus prevent. Prevention is cheap and boring: a gateway, tags, budgets, alerts, a caching policy — weeks of platform-engineering work. Remediation is a mid-year budget conversation, a feature freeze, and in the documented worst cases a scale-back. Flexera's 2026 data reports 49% of organizations having delayed or scaled back AI because of cost. The asymmetry is roughly the same as any other observability investment, with one difference: because AI spend can accelerate non-linearly with adoption, the window between "fine" and "not fine" is measured in weeks rather than quarters.

The response pattern: the gateway, and what it actually buys

The architecture that competent organizations converge on is an LLM gateway — a proxy every model call passes through. Cloudflare's product describes the function precisely: "connect to any model, dynamically route requests, and manage usage, billing, and logs from one unified gateway," with response caching to "reduce redundant API calls — leading to direct cost savings," rate limiting, per-request logging with "token counts, prompt performance, and pattern analysis," and configurable fallbacks between providers (Cloudflare AI Gateway). The self-hosted equivalents add hierarchical budgets and per-team virtual keys.

What the gateway genuinely buys is attribution and enforcement at a single choke point: you cannot chargeback what you cannot attribute, and you cannot cap what you cannot intercept. What it does not buy is demand management. A gateway will tell you, precisely and in real time, that the customer-support team's agent is burning $40,000 a month on retries. It will not tell you whether that is worth it. The FinOps Foundation's own 2026 framing is an admission of that: a new "Executive Strategy Alignment" capability, and a mission change from managing "the Value of Cloud" to "the Value of Technology." The discipline's centre of gravity has moved from cost reduction to value attribution. That is a much harder problem, and the tooling does not solve it.

Large versus small. The large enterprise has the negotiating leverage — volume discounts, private offers, committed-spend agreements, and the ability to demand the reservation instruments at all. It also has the harder allocation problem: a shared gateway serving forty product teams needs a chargeback model, and building one is a political exercise. The small company has no leverage and no allocation problem. It pays list price, absorbs vendor pricing changes without notice or recourse, and has neither a FinOps function nor a platform team to build the gateway. Its practical mitigation is the crude one: a hard spend cap on the vendor console and a willingness to have the product stop working. That is a genuine control — it is simply a much blunter one, and it converts a cost risk into an availability risk in a way the large enterprise can usually avoid.

2.2Data exposure and shadow AI: the same tool, a different contract

Strip away the phrase and shadow AI has a precise mechanical definition: employees sending corporate data to model endpoints governed by a contract the company is not party to. Not rogue tools — usually the same tools, reached through a personal login, a browser extension, a free tier, or an AI feature quietly added to a SaaS product the company already licenses. That last channel is the one most often missed, because it does not look like adoption at all; it looks like a version upgrade.

The volume data is now unambiguous. Netskope's 2026 Cloud and Threat Report found 47% of workplace generative-AI users on personal accounts — an improvement from 78% a year earlier, which is the encouraging half of the number — while total users tripled and monthly prompt volume rose sixfold, from about 3,000 to 18,000 prompts per organization, with the largest adopters exceeding 1.4 million prompts a month. Roughly 3% of gen-AI users generated an average of 223 data-policy violations per organization per month, rising to about 2,100 monthly incidents in the top quartile, with exposed material including source code, confidential business information, intellectual property and credentials (Infosecurity Magazine on Netskope, 7 Jan 2026).

Two findings inside the IBM data invert the usual framing. First, 68% of breached organizations had no AI governance in place to manage AI or detect shadow AI, up from 63% the prior year — governance coverage went backwards against a doubling of shadow-AI incidence. Second, 92% of organizations hit by AI-model attacks lacked basic access controls. Neither of those is a story about sophisticated adversaries. They are stories about a control gap widening because adoption outran the control function, which is the thesis of this primer stated in someone else's data.

Who finds out first, and how long it takes. In the classic security problem, the security function detects an anomaly. Here, the discovery is usually accidental and late: a customer notices their data in a competitor's output, a regulator asks a question, a departing employee's browser history surfaces in an unrelated investigation, or — most commonly — nobody ever finds out at all. There is no telemetry on the personal-account channel by construction; that is what makes it shadow. Detection time for the unmanaged channel is not long, it is undefined, and any risk register that assigns it a mean-time-to-detect is inventing a number.

The response pattern: provisioning, the gateway again, and posture management

The observed response has three parts and they work in a specific order. First, supply a governed alternative. Netskope's telemetry contains the best available evidence that this works: personal-account use among workplace gen-AI users fell from 78% to 47% in a single year, against tripling user numbers and a sixfold rise in prompt volume. One qualification matters and is usually dropped when this number is quoted: Netskope counts a user in the 47% if they use a personal account "either exclusively or alongside company-approved tools," so the two channels overlap and the residual is not a clean 53% of governed users. Alongside it, the share of organizations blocking at least one gen-AI application rose from 80% to 90%, with an average of ten applications blocked (SecurityBrief on Netskope, 6 Jan 2026). Provision and block moved together, and the exposure channel narrowed. The practical lesson holds either way: a licensed enterprise seat for the tool people already want, provisioned quickly, is the most effective shadow-AI control available, because policy without provisioning simply moves the traffic to a channel with no logging.

Second, intercept at the network and the gateway. Egress controls and DLPData-loss prevention — software sitting on the way out of a company that inspects traffic and blocks or logs anything matching a sensitive pattern: card numbers, contract language, source code, customer records. It only sees traffic that passes through it, which is the whole difficulty with the personal-account channel. inspection on AI endpoints convert an invisible channel into a logged one; the gateway does the same for sanctioned application traffic. This is the part most organizations do partially — blocking a handful of consumer domains while a hundred SaaS products quietly ship AI features that call the same models from their own infrastructure.

Third, fix the substrate, which is where the most credible tooling now points. Microsoft's Purview Data Security Posture ManagementTooling that inspects where sensitive data actually sits and who can reach it, rather than watching it move. Where DLP guards the exits, posture management audits the building: its output is a ranked list of over-exposed places to go and fix. frames the problem correctly: "because of the power and speed AI can proactively surface content that might be obsolete, over-permissioned, or lack governance controls, generative AI amplifies the problem of oversharing data." The default data risk assessment runs weekly against the top 100 SharePoint sites by usage, surfacing sensitive-data counts and links shared with anyone, and offers concrete remediations — DLP policies that stop Copilot summarizing labelled content, SharePoint Restricted Content Discovery to exempt sites from Copilot entirely, auto-labellingHaving software read a document and stamp it with a sensitivity classification — public, internal, confidential — instead of waiting for the person who wrote it to do so. Every downstream control that acts on sensitivity needs the stamp to exist first. for unlabelled sensitive files, and retention policies to delete content untouched for three years (Microsoft Purview, updated 25 Jun 2026).

The assistant did not create the oversharing. It made twenty years of accumulated permission drift searchable in natural language. The remediation Microsoft recommends is not an AI control at all — it is auto-labelling, access review and records retention, the unglamorous information-governance backlog that every large organization has been deferring since the file server. AI deployment is, in this precise sense, an information-governance audit that arrives whether or not the organization asked for one.

The honest limits. Microsoft publishes them: item-level scanning for data risk assessments is capped at 10 SharePoint sites, a maximum of 200,000 items per location with file counts possibly inaccurate above 100,000, and OneDrive is not supported for item-level scanning at all. That is not a criticism of the product; it is a measure of how large the underlying problem is relative to the tooling. And no posture-management product addresses the personal-account channel, which is precisely where Netskope still finds 47% of users.

Large versus small. Divergence here is sharper than at any other layer, and it runs in both directions. The large enterprise can negotiate zero-retention terms, buy an enterprise tenant with audit logs, run DLP at the egress point and — crucially — has the volume to make a vendor answer a data-processing questionnaire honestly. It also has a vastly larger surface: tens of thousands of employees, hundreds of SaaS products shipping AI features, and a decade of permission drift across content stores that no single team owns. The small company has almost none of that surface and almost none of the instruments. It takes the vendor's standard terms because there is no negotiation to be had, cannot generally obtain ZDR, has no DLP, and typically discovers its exposure only if a customer or an auditor raises it. Its one structural advantage is real and underrated: a fifty-person company can credibly know what tools its people use and can standardize on one governed provider in an afternoon. A fifty-thousand-person company cannot do that at any price.

2.3Security: prompt injection, the agentic surface, and the model supply chain

Prompt injection sits at the top of the OWASP Top 10 for LLM Applications for the third consecutive edition, ahead of sensitive-information disclosure, in the 2026 list published on 3 August 2026 (OWASP GenAI, LLM Top 10 2026). Why it holds that position is architectural rather than incidental, and the reason determines what a mitigation can and cannot achieve.

The 2026 revision itself repays a strategy leader's attention, and not because the top two moved — they did not. What changed is how the list was built, and what rose. For the first time the ranking was not purely a practitioner vote: the community ballot carries 75% of the weight and the remaining 25% comes from 6,639 real incidents drawn from public vulnerability databases and an AI-harm database (Help Net Security, 6 Aug 2026). Against that evidence base, eight of the ten positions changed. Excessive agency — damage done by an over-permissioned system with no attacker involved — climbed from sixth to third, on the explicit reasoning that the 2024 list described chat interfaces while by 2026 "agents had become the norm — systems with persistent memory, tool access, file and API permissions" (ReversingLabs, 12 Aug 2026). Improper output handling fell from fifth to tenth. The category the industry treated as the classic AI security problem lost ground to the category created by giving the model a job.

OWASP's own framing of what to do about it is the same conclusion this section reaches from the other direction: "Stop trying to build a model that cannot be fooled. Build the system around it, so that when the model is fooled, and it will be, nothing important breaks." That is a design instruction, not a control recommendation.

A language model receives instructions and data through the same channel. There is no equivalent of a parameterized SQL query — no syntactic boundary that marks "this is a command" and "this is content to be processed." When an assistant reads an email, a web page or a PDF, the text of that document arrives in the same token stream as the system prompt, and the model has only statistical priors, not a trust boundary, to distinguish them. OWASP's own guidance says the quiet part directly: "it is unclear if there are fool-proof methods of prevention for prompt injection," attributing this to the stochastic nature of the models (OWASP LLM01:2025). NIST's updated adversarial-machine-learning taxonomy, AI 100-2 E2025 (March 2025), extends its coverage to direct and indirect prompt injection and agent security and pairs each attack category with mitigations "along with an assessment of the limitations of those techniques" (OECD.AI catalogue entry for NIST AI 100-2 E2025). And OpenAI, shipping a security update for its browser agent in December 2025, described prompt injection as "one of the most significant risks we actively defend against" and disclosed that it had built an automated LLM-driven attacker with reinforcement learning to find injection strategies before release — the posture of a continuing defensive programme, not a closed problem (CyberScoop, 30 Dec 2025).

Simon Willison's formulation is the most useful operational frame available, and it has been adopted widely because it converts an unbounded problem into a bounded design decision. The "lethal trifecta" is the conjunction of three capabilities: access to private data, exposure to untrusted content, and the ability to communicate externally. Any system with all three can be made to exfiltrateTo move data out of the organization to somewhere the attacker can read it. In this setting the exit is rarely dramatic — a rendered image whose web address carries the stolen text, a link, an outbound message — which is why it leaves nothing that looks like an intrusion.. Willison's argument against defensive prompting is the practical one — "LLMs are unable to reliably distinguish the importance of instructions based on where they came from," and a filter that stops 95% of attacks is a failure in a security context, because an attacker retries. The recommended mitigation is not a better filter. It is to remove one of the three legs (Simon Willison, 16 Jun 2025).

The canonical demonstration that this is not theoretical is EchoLeak, CVE-2025-32711A CVE — Common Vulnerabilities and Exposures — identifier is the public catalogue number assigned to a disclosed software flaw, so that researchers, vendors and defenders all refer to the same bug by the same name. Being assigned one is the marker that a finding has left the realm of a research demo., a zero-click indirect prompt injection in Microsoft 365 Copilot disclosed in 2025. The published case study describes an attack chain that required no user interaction at all: a crafted email carrying a hidden payload, evading Microsoft's cross-prompt-injection classifier, circumventing link redaction with reference-style Markdown, exploiting auto-fetched images, and abusing a Teams proxy permitted by the content security policyA browser-enforced allowlist naming which outside destinations a page is permitted to contact. It is the last barrier between a rendered image and an exfiltration channel, and an attacker needs only one destination on the list that they can also reach. — achieving "full privilege escalation across LLM trust boundaries without user interaction" and remote, unauthenticated data exfiltration (Reddy & Gujral, arXiv:2509.10540, 6 Sep 2025). The authors' conclusion is the one that matters for a strategy leader: this establishes "prompt injection as a practical, high-severity vulnerability class in production AI systems," not a research curiosity.

1. 01
   Untrusted content arrives

   An email, web page, ticket, PDF or tool result enters the assistant's context. No user action required.
2. 02
   Instructions and data merge

   The model has no syntactic trust boundary. Hidden text — HTML comments, white-on-white — is read as instruction.
3. 03
   Private context is in scope

   The same session holds mailbox, documents, tickets — everything the assistant was connected to in order to be useful.
4. 04
   An outbound channel exists

   A rendered image URL, a link, a tool call, a message send. Exfiltration needs only one egress path the CSP permits.
5. 05
   Loss occurs silently

   No credential was stolen, no software vulnerability exploited in the traditional sense, and no log entry looks anomalous.

Evidence: EchoLeak case study (arXiv:2509.10540, 6 Sep 2025) for the concrete chain; Willison's lethal-trifecta framing (16 Jun 2025) for the general condition; OWASP LLM01:2025 for the classification. Method: Sequence abstracted from the documented EchoLeak chain and generalized to the trifecta conditions; each step is a necessary condition, and removing any one breaks the chain. Synthesis: This is a design-level exposure. It is created by connecting an assistant to useful things, which is the entire point of deploying one.

The response pattern — and, unusually, a published price for it

OWASP's LLM01 lists seven mitigations: constrain model behaviour through system prompts, define and validate expected output formats, filter input and output, enforce least privilegeThe rule that a component is granted exactly the access its job requires and not one permission more. Applied to an agent, it is the difference between a summarizer that can read the mailbox and one that can also empty it., require human approval for high-risk actions, segregate and identify external content, and conduct adversarial testing. Those are sound and none of them is sufficient, which OWASP says.

The more interesting development is a research line arguing that provable resistance is achievable by constraining what the agent can do rather than by improving what it can detect. Sixteen researchers from Google, IBM, Microsoft, ETH Zurich and elsewhere set out six design patterns for agents "with provable resistance to prompt injection," explicitly framing the trade-off: the patterns "mitigate the risk of prompt injections by constraining agent actions to explicitly prevent them from solving arbitrary tasks" (Beurer-Kellner et al., arXiv:2506.08837, 10 Jun 2025). Google DeepMind's CaMeL is the fullest instantiation: it "explicitly extracts the control and data flows from the (trusted) query" so that untrusted retrieved data "can never impact the program flow," and uses capability tokensUnforgeable permission slips attached to a piece of data or an action. The system checks the slip before anything runs, so an instruction the model picked up from untrusted text cannot conjure the authority to act on it — the authority has to have been granted upstream, by the trusted part of the program. to block exfiltration over unauthorized data flows (Debenedetti et al., arXiv:2503.18813, v2 24 Jun 2025).

And here is the number that a strategy leader should carry into every agent-deployment conversation. On the AgentDojo benchmark, CaMeL solved 77% of tasks with provable security, against 84% for an undefended system. Independent reporting of the same work puts the token overhead at roughly 2.7–2.8×. So the strongest published defence costs about seven points of task success and something close to triple the inference bill. That is not a criticism of CaMeL — it is the first real price tag anyone has put on this class of security, and it is far cheaper than a breach. But it settles a question that vendor material tends to leave open: prompt-injection security is purchased with utility and money, not with a patch. Any roadmap that assumes the problem gets fixed upstream at zero marginal cost is planning on a fact not in evidence.

Excessive agency: the failure that needs no attacker

Excessive agency was the layer's most underrated entry for two editions, and the 2026 ranking has stopped underrating it — sixth to third, as Figure 5 shows. It earns the position because it does not require an adversary at all. OWASP defines it as damaging action arising from three separable root causes: excessive functionality (an extension that summarizes email but also retains the ability to send and delete), excessive permissions (a read-only task granted broad database access), and excessive autonomy (a high-impact action taken without verification). The triggering condition can be "hallucination from poorly-engineered prompts" just as easily as prompt injection (OWASP LLM06:2025). An agent with delete permissions it never needed will eventually use them for a reason nobody predicted, and no attacker will have been involved.

The model supply chain: executable files that look like data

A model file does not look like software. It is — which is why the third security mechanism here is the one most familiar to a traditional security team and the one most often skipped in AI reviews. OWASP's LLM03 catalogues nine categories of exposure including vulnerable pre-trained models — "binary black boxes potentially containing hidden biases, backdoors, or other malicious features" — malicious LoRA adaptersA small add-on file that nudges a base model's behaviour without retraining the whole thing — cheap to make, cheap to share, and loaded with the same trust as the model it attaches to., weak provenance, and weaponized model-merge services, and recommends signed inventories, SBOMsA software bill of materials is a machine-readable list of every component inside a piece of software. Its whole purpose is the day after: when one component turns out to be compromised, you can answer in minutes whether you shipped it., integrity checks and red-teaming beyond published benchmarks (OWASP LLM03:2025).

The mechanism is concrete and old. PyTorch's traditional serialization uses Python's pickle format, and "unpickling can run code hidden in the file, which makes it dangerous to use with data from untrusted sources" — loading a model with torch.load() executes whatever the file's author embedded (Rapid7, 1 Jul 2025). In February 2025, ReversingLabs found two models on Hugging Face carrying platform-aware reverse shells, evading the platform's Picklescan tool by using broken pickle files compressed with 7z rather than the standard ZIP, with the payload placed at the start of the stream so it executed before the corruption was detected (The Hacker News, 8 Feb 2025). The researchers assessed them as proofs of concept rather than active attacks — and that is exactly the point. The technique works; the demonstration is public; the control is a scanner that had a gap.

The 2026 version of this problem is the tool-connection layer. A Cloud Security Alliance research note published 4 May 2026 describes a systemic flaw in the Model Context ProtocolMCP is the de-facto standard by which an AI assistant reaches outside tools and data sources — a plug socket for capabilities. Each "server" on the far end of the socket is third-party code, and it runs with whatever access the assistant was given.'s STDIO transportThe simplest of the ways an assistant talks to an MCP server: launch the server as a local program on the same machine and pipe text back and forth through its standard input and output. Convenient, and it means the assistant is starting operating-system processes., which "executes operating system commands without sanitization or validation," present in every official MCP SDK across Python, TypeScript, Java and Rust rather than as isolated coding errors. OX Security's April 2026 disclosure estimated 200,000 vulnerable instances across a supply chain of more than 150 million package downloads; at least 1,862 publicly accessible MCP servers were found responding to unauthenticated requests in July 2025; and seven confirmed high- or critical-severity CVEs were documented across major platforms. Notably, "Anthropic confirmed the behavior was intentional and declined to modify the protocol architecture, instead assigning sanitization responsibility to downstream developers" (CSA, 4 May 2026).

That last clause is the governance lesson, and it is not really about MCP. A protocol designer's decision to place a security responsibility downstream is a legitimate engineering choice and a material risk transfer to every enterprise that adopts the protocol. The recommended controls — process isolation for MCP servers, allowlist validation of STDIO parameters, runtime behavioural monitoring, explicit version pinning with change alerts, and treating each server as untrusted — are exactly the controls a mature organization applies to any third-party executable. The failure is not technical sophistication. It is that model files and tool servers are being onboarded through data-procurement paths rather than software-procurement paths.

Large versus small. The large enterprise has a security function, can run red teamsA team paid to attack the organization's own systems the way a real adversary would, and to write down what worked. The point is not a pass mark; it is a list of the things nobody thought to defend., can mandate signed models and an internal model registry, and can afford the utility cost of scoped agents. Its problem is scale and shadow deployment: Gravitee's 2026 survey found only 14.4% of organizations reporting that all AI agents went live with full security or IT approval, and an average of just 47.1% of agents actively monitored — with 82% of executives nonetheless confident their policies protect them from unauthorized agent actions (Gravitee, 4 Feb 2026). That confidence gap is the finding. The small company has no red team, consumes third-party agents and MCP servers wholesale, and inherits every upstream design decision without the capacity to evaluate it. Its realistic posture is not to secure the agent; it is to refuse the trifecta — no agent that simultaneously touches sensitive data, ingests untrusted content and can act outward. That is a real, cheap, enforceable control, and it is available precisely because a small firm's use cases are simple enough to draw the line.

2.4Reliability: wrong answers, unrepeatable answers, and models that move

Three distinct failures get bundled under "the model isn't accurate enough." They have different causes, different owners and different fixes.

HallucinationA model stating something false in exactly the register it uses for something true — an invented citation, a plausible quotation nobody said, a case that does not exist. It is not a malfunction in the ordinary sense: the system has no internal line between what it knows and what it has assembled, so nothing inside it flags the difference. is an incentive problem before it is a capability problem

The rates are worse than most executives assume and better than the sceptics claim, and the variance across settings is the useful signal. Stanford RegLab's Large Legal Fictions, published in the Journal of Legal Analysis, found hallucination rates of 69% for GPT-3.5 and 88% for Llama 2 on specific, verifiable questions about random federal court cases, and concluded that models "cannot always predict, or do not always know, when they are producing legal hallucinations" — cautioning "against the rapid and unsupervised integration of popular LLMs into legal tasks" (Dahl, Magesh, Suzgun & Ho, Jan 2024). Those numbers describe general-purpose models on an adversarially hard task, and they should not be quoted as production rates for a well-built RAG system, which are far lower. But the mechanism they expose — confident error, undetected by the model itself — does not go away with retrieval; it becomes rarer and correspondingly harder to catch.

Why it persists is now reasonably well understood, and the explanation is unflattering to the industry rather than to the technology. OpenAI researchers argue that models hallucinate because "training and evaluation procedures reward guessing over acknowledging uncertainty" — models are "optimized to be good test-takers, and guessing when uncertain improves test performance" — and propose a socio-technical fix: rescore the dominant benchmarks so that appropriate expressions of uncertainty are not penalized (Kalai, Nachum, Vempala & Zhang, arXiv:2509.04664, 4 Sep 2025). If that analysis is right, hallucination is partly an artefact of how the industry grades itself, which means it is partly fixable by changing the grading — and also that it will not be fixed by any single vendor unilaterally, because the leaderboards are shared.

The production cost is documented. In August 2025, academics identified fabricated references and a misattributed Federal Court quotation in a Deloitte report for Australia's Department of Employment and Workplace Relations, on a contract worth around A$440,000; three nonexistent academic references and a falsely attributed legal quote were found, and Deloitte issued a partial refund after acknowledging limited use of generative AI (AI Incident Database, incident 1193). Note the discovery path: not the firm's quality process, not the client's review, but an outside academic who happened to recognize that a paper attributed to a colleague did not exist. That is the characteristic detection mechanism for hallucination in professional work — external, accidental, and after publication.

Non-determinism breaks the QA model, and the fix has a measured price

Conventional software testing rests on an assumption so basic it is rarely stated: the same input produces the same output. LLM inference violates it even at temperature zeroThe setting that tells a model to always take its single most likely next word rather than sampling among the plausible ones. Turning it to zero removes the deliberate randomness, and is what an engineer reaches for when they want the same answer twice., and the reason is not the one most engineers assume. Two ideas carry the explanation. A batch is the group of unrelated customer requests a server processes together to keep its chips busy, and how many are in it depends on who else happens to be asking at that instant. Batch invariance is the property that the arithmetic comes out identical whichever batch a request lands in — and without it, part of the answer depends on the company a request keeps. Thinking Machines Lab's analysis attributes it not to floating-point non-associativity alone but to a lack of batch invariance in inference kernels — "the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies," and normalization, matrix multiplication and attention all produce slightly different results depending on how many other requests share the batch. Their experiment is precise: 1,000 completions at temperature 0 from Qwen3-235B produced 80 unique outputs, identical for the first 102 tokens and diverging at token 103. With batch-invariant kernels, all 1,000 were identical — at a cost of 1.6× to 2.1× in latency, depending on the attention implementation (Horace He, Thinking Machines Lab, 10 Sep 2025).

That result is genuinely important for governance and is almost entirely absent from enterprise discourse. It establishes three things at once. Determinism is achievable, so "LLMs are inherently random" is no longer an acceptable answer to an auditor. It is not free. And critically, it is available at the serving layer but not in the commercial relationship.

The serving-layer half has moved fast. vLLM now ships batch invariance as a beta feature — enabled with a single environment variable, validated across DeepSeek, Qwen, Llama, Mistral, Phi and GPT-OSS families on NVIDIA GPUs of compute capability 8.0 or higher — guaranteeing "that the output of a model is deterministic and independent of the batch size or the order of requests in a batch," while documenting the cost plainly: "enabling batch invariance may impact performance compared to the default non-deterministic mode. This trade-off is intentional to guarantee reproducibility," and noting that it disables optimizations including custom all-reduce in tensor-parallel mode (vLLM, batch invariance).

The commercial half has not. Hosted APIs offer, at best, best-effort determinism: OpenAI documents a seed parameter with an explicit warning that determinism is not guaranteed and that backend changes can affect outputs. The distinction is the whole point. An enterprise that needs reproducible inference for a regulated model-validation process today faces a choice it should make consciously: self-host on an invariant stack and accept the throughput penalty and the operational burden, or use a hosted API and accept that its production system cannot be exactly re-run. There is no third option, and no amount of vendor-questionnaire diligence produces one. Model-risk regimes built on reproducible validation have not caught up with that fact, which is the more interesting half of the problem.

The model moves underneath you, on the vendor's calendar

Models are retired. That is the third reliability mechanism, the most mundane of them and the most reliably underestimated. The commitments are published, they differ materially by vendor, and the observed windows are short.

OpenAI's policy states minimum notice periods before retirement: "Generally available models: At least 6 months," unless safety or compliance requires faster (OpenAI deprecations). Its 2026 announcements include GPT-5 and o3 snapshots retiring 11 December 2026, legacy GPT snapshots including gpt-3.5-turbo, gpt-4 and o1 retiring 23 October 2026, the Assistants API retiring 26 August 2026, and — notably for teams that built on them — the Evals platform, Agent Builder and reusable prompts API all shutting down 30 November 2026. Anthropic commits to "at least 60 days' notice before model retirement for publicly released models" (Anthropic model deprecations). Microsoft Foundry sets a programmatic 18-month lifecycle for GA models from launch, with at least 60 days' notice before retirement and 30 days for preview models, notes that generally available models from Anthropic, DeepSeek, Fireworks and Mistral follow a 12-month rather than 18-month lifecycle, and states flatly that retirement dates "aren't extendable" (Microsoft Foundry lifecycle policy, 24 Jul 2026).

On Microsoft's standard deployment types, model versions are auto-upgraded when retired, "on a rolling, region-by-region basis," and upgrades "can occur even if the new model version isn't yet separately available in that region." Provisioned deployments are explicitly not auto-upgraded and must be migrated by hand. So the same organization, running the same workload on two deployment types, faces opposite failure modes: silent behavioural change on one, hard failure on the other. Microsoft does expose a versionUpgradeOption property with a NoAutoUpgrade setting — whose documented consequence is that the "deployment stops working at retirement." Those are the three available outcomes, and a team that has not consciously chosen among them has chosen the first.

The response pattern here is well established and cheap: pin explicit model versions rather than floating aliasesA model name like "latest" that points at whichever version the vendor currently ships. Convenient — and it means the engine underneath a live product can be swapped without anyone on the buyer's side doing, or noticing, anything., maintain a golden-set regression suiteA fixed set of test inputs with known-good answers, kept as the standing benchmark and re-run against every candidate model. If a new version drifts, the suite finds out before customers do. It is the same artefact that lets a team certify a cheaper model as good enough, which is why it earns its keep twice. that runs against every candidate replacement, subscribe the platform team to the vendor's retirement notifications, and keep a dated migration calendar as a standing item. Anthropic even documents the audit path — export API usage by key and model from the console to locate deprecated-model usage. None of this is difficult. It is simply nobody's job in most organizations until the first forced migration.

Large versus small. A large enterprise has more workloads to migrate but also has an evaluation harness, a platform team and, at sufficient volume, a vendor relationship warm enough to get advance notice and migration support. It also has the worse tail risk: a validated model in a regulated process cannot simply be swapped, and 60 days does not accommodate a model-risk revalidation. The small company usually runs one or two workloads and can migrate in an afternoon — but it typically has no evaluation suite at all, so it discovers behavioural regressions through customer complaints. It is also more likely to be building on a reseller or an application vendor that swapped the underlying model without telling anyone, which is the same risk one level of indirection further away and with no notification path at all.

2.5Governance and legal: what is actually enforceable, and when

The most useful thing a strategy leader can do with the regulatory layer is separate what is in force from what is coming, because the two are usually discussed together and the practical implications are opposite. One distinction governs everything that follows. European AI law splits its duties between the provider — whoever develops a system and puts it on the market under their own name — and the deployer, whoever uses one under their own authority in the course of business. A company that buys an assistant and switches it on is a deployer, and the Act binds it directly rather than through its vendor. Almost every organization reading this sits on that side of the line. As of August 2026 the European position has just moved, and it moved in the direction of delay for the heaviest obligations and no delay at all for the light ones.

The EU AI Act, after the omnibus

"High-risk" is the Act's own classification, not a description of how powerful a model is. It attaches to the use a system is put to, and it is the trigger for the Act's heaviest duties — conformity assessmentThe European pre-market compliance procedure: documenting that a product meets the applicable requirements before it may be placed on the market, and in some categories having a notified third party check the documentation. It is the same machinery that governs medical devices and lifts, pointed at software., risk management, data governance, logging, human oversight and technical documentation. Two annexes carry the lists: Annex III for standalone uses, Annex I for AI built into products that are already regulated. Nothing about the model itself decides the question.

Regulation (EU) 2026/1744 — the Digital Omnibus on AI, adopted 8 July 2026, published in the Official Journal on 24 July and in force from 27 July — amends the AI Act to postpone application of the Chapter III high-risk obligations: standalone Annex III high-risk systems move to 2 December 2027, and Annex I product-embedded high-risk systems to 2 August 2028 (EUR-Lex, Regulation (EU) 2026/1744). The same instrument adds two prohibited practices to Article 5 — AI systems generating non-consensual intimate imagery and child sexual abuse material — with a transitional grace period for the associated technical safeguards, and extends simplifications to SMEs, notably making the simplified quality-management system available to all small and medium enterprises rather than only microenterprises, and mandating simplified technical-documentation forms for SMEs and small mid-caps under Article 11(1).

What did not get deferred is the part that already binds ordinary deployers — though one of those obligations was rewritten in place, and the rewrite is easy to miss because it did not change a date. Article 4, the AI-literacy duty, has applied since 2 February 2025 in its original form, which required that providers and deployers "take measures to ensure, to their best extent, a sufficient level of AI literacy" of their staff and of others operating systems on their behalf (AI Act Article 4, original text). The omnibus replaced that with a duty to "take measures to support the development of AI literacy," and added an express clarification that the obligation "does not require providers or deployers to guarantee any specific level of AI literacy of any individual" — a change applying from 27 July 2026 (Law & Technology, on the Article 4 rewrite; NicFab, Regulation (EU) 2026/1744 in the Official Journal).

The practical consequence is narrower than the headline "the EU softened AI literacy" suggests. What changed is the standard: an obligation of result became an obligation of means. What did not change is the scope: it still binds every provider and every deployer, at every risk tier, with no headcount threshold and no exemption for a company that merely bought an off-the-shelf assistant. A firm that had done nothing is still in breach; a firm that ran a documented, role-differentiated programme and can show it is now safer than it was, because it no longer has to defend the competence of every individual it trained. The compliance artefact that satisfies the amended article is the same artefact — evidence that measures were taken — which is another instance of the pattern this section is about to name. Article 50's transparency duties — disclosing that a person is interacting with an AI system, marking synthetic content, disclosing deepfakes — took effect on schedule on 2 August 2026, with the omnibus giving providers whose systems were already on the market before that date until 2 December 2026 to meet the machine-readable marking requirements.

The penalty tiers are unchanged and sit in Article 99, though the article itself was amended by the omnibus to require Member States to lay down rules on penalties "and other enforcement measures" — warnings and non-monetary measures among them — and to have regard to the economic viability of SMEs and small mid-caps. The tiers: up to €35,000,000 or 7% of total worldwide annual turnover, whichever is higher, for breaching the Article 5 prohibitions; up to €15,000,000 or 3% for most other operator obligations including the transparency duties; and up to €7,500,000 or 1% for supplying incorrect or misleading information to authorities. For SMEs including start-ups, each cap is "the percentages or amount referred to in paragraphs 3, 4 and 5, whichever thereof is lower" (AI Act Article 99). That SME provision is one of the few places anywhere in AI regulation where small firms get a structurally better deal rather than a worse one.

The deferral changes a planning assumption that many boards have already baked in. Organizations that budgeted a 2026 high-risk conformity programme now have sixteen extra months — and a decision to make about whether to spend them. The instinctive answer is to slow down. The better answer is probably not, for two reasons that are both evidenced. First, the deferral is of application, not of the underlying engineering: data governance, logging, human oversight and technical documentation take longer to retrofit than to build in, and every month of new deployment on ungoverned foundations increases the retrofit. Second, the direction of national and sectoral rule-making has not paused. Colorado repealed and replaced its AI Act via SB 26-189, signed 14 May 2026, substituting the narrower Automated Decision-Making Technology Act effective 1 January 2027, which drops the algorithmic-discrimination duties but imposes transparency, consumer notification before consequential decisions, adverse-outcome disclosure within 30 days, three-year record retention, and designated trained personnel for human review and override (Skadden, 9 Jun 2026). California's SB 53, approved 29 September 2025, imposes on large frontier developers — those training above 10²⁶ operations with over $500m in revenue — a published frontier AI framework, critical-safety-incident reporting within 15 days or 24 hours where there is imminent risk of death or serious injury, anonymous internal whistleblower channels, and penalties up to $1,000,000 per violation enforceable by the Attorney General (California SB 53).

The pattern across all three: the obligations that survive political revision are the transparency, documentation and incident-reporting ones. Substantive duties — prevent discrimination, assess risk before deployment — are being deferred, narrowed or repealed. Procedural duties are not. A governance programme built around evidence production and disclosure is therefore the one most likely to remain useful across jurisdictions and revisions, which is a genuinely different design brief from the one most consultancies sold in 2024.

The voluntary baseline: NIST and ISO/IEC 42001

The NIST AI Risk Management Framework 1.0 was released 26 January 2023 and organizes practice into four functions — Govern, Map, Measure, Manage. Its Generative AI Profile (NIST AI 600-1), released 26 July 2024, extends it to generative systems, and NIST released a preliminary draft Cyber AI Profile (NIST IR 8596) in December 2025 bridging AI risk management to Cybersecurity Framework 2.0, and a concept note on 7 April 2026 for a further profile on trustworthy AI in critical infrastructure; the core framework itself remains at 1.0 and is under revision as part of the White House AI Action Plan, with no 1.1 or 2.0 finalized (NIST AI RMF). The shape of the activity says something on its own: NIST is shipping profiles — domain-specific overlays — rather than revising the core, which is what a standards body does when the underlying framework is sound and the application contexts are multiplying faster than the framework can absorb them. NIST's other relevant instrument is the adversarial-machine-learning taxonomy, NIST AI 100-2 E2025 (March 2025), which extends the taxonomy to supply-chain attacks, direct and indirect prompt injection, misuse and agent security, and which pairs each attack category with mitigations and an explicit assessment of those mitigations' limitations.

ISO/IEC 42001 is the certifiable one, and certification is what makes it commercially interesting: it produces an artefact a customer's procurement team can accept. The standard requires an AI management system with 38 controls across 9 control objectives in Annex A, with organizations selecting applicable controls in a Statement of Applicability and justifying exclusions. Certification runs as a two-stage audit — a Stage 1 documentation review of one to two days, then a Stage 2 implementation audit of one to three weeks assessing operational effectiveness through interviews and process observation — with surveillance audits at twelve-month intervals in years two and three and full recertification in year four (Cloud Security Alliance, 8 May 2025).

Two observations about this baseline. Adoption remains thin relative to the attention. ISO publishes no certificate registry, so there is no number to check — but the order of magnitude can be bracketed from a primary: Boston Consulting Group announced in January 2026 that it was "among the first 100 organizations globally" to be certified (BCG, 27 Jan 2026), and by mid-2026 newly certified firms were describing themselves as among the first several hundred. For a standard published in December 2023 that is slow, and the constraint appears to be accredited auditor capacity as much as demand. The commonly quoted "350+" is an aggregation of certification-body announcements rather than a registry count; it is flagged accordingly and no argument here rests on the level. The second is more substantive. Both NIST's framework and ISO's standard are management-system instruments: they specify that you shall identify risks, document decisions, assign accountability and monitor. Neither tells you whether a prompt-injection defence is adequate, whether a hallucination rate is acceptable for a given use, or what a defensible evaluation looks like. They are necessary, they are auditable, and they are silent on the technical questions in sections 2.3 and 2.4. An organization can be ISO/IEC 42001 certified and still ship an agent with the lethal trifecta intact.

Liability: the settled question and the open one

One question is largely settled and gets asked far too often; another is genuinely open and gets asked far too rarely.

The settled one is whether a company is responsible for what its automated system tells a customer. In Moffatt v Air Canada, 2024 BCCRT 149, the British Columbia Civil Resolution Tribunal rejected the airline's argument that its chatbot was a separate entity, holding that "it makes no difference whether the information comes from a static page or a chatbot" and that "it should be obvious to Air Canada that it is responsible for all the information on its website," with the applicable standard of care requiring "a company to take reasonable care to ensure their representations are accurate and not misleading" (McCarthy Tétrault on Moffatt, decision 19 Feb 2024). The sum at stake was a few hundred dollars. The principle is worth considerably more: output is attributable to the deployer, and the probabilistic nature of the generator is not a defence.

The open one is what happens when an autonomous agent acts — transacts, commits, deletes — on a company's behalf, and the act was neither instructed by a human nor anticipated by the deployer. Be precise about which part is actually open: the question is usually asked too broadly, and the broad version has a settled answer. Attribution is not the open part. The federal E-SIGN Act and the state UETA provisions on electronic agents predate all of this and already provide that a contract is not denied effect merely because an electronic agent formed it, so long as the agent's action is legally attributable to the person to be bound; ordinary agency doctrine — actual authority, apparent authorityThe authority an outsider reasonably believes a representative has, because of how the principal has presented them, even where no such authority was in fact granted. It is the doctrine that stops a company disowning a deal its own agent appeared entitled to strike., ratification when the principal keeps the benefit — supplies the attribution. A California statute enacted in 2025 goes further and forecloses a defendant from arguing that the AI autonomously caused the harm (Baker McKenzie, 1 Jul 2026). "The AI did it" has no more force than "my employee did it."

What is open is narrower and harder: where apparent authority ends when a counterparty deals with an agent the deployer never intended to grant that authority; how negligence and product-liability standards apply to an autonomous system's conduct; and how loss is allocated between deployer, model vendor and tool provider when three parties' design decisions combine into one bad action. No court has squarely resolved those, and this primer will not pretend otherwise. But the practical consequence of getting the settled part right is immediate: a company cannot plan on the theory that agent actions are legally orphaned, because they are not. The response available today is contractual and architectural rather than litigated: cap what agents can do, require human approval on irreversible actions (OWASP LLM06's central recommendation), maintain audit trails that reconstruct the decision chain, and negotiate the liability allocation in the vendor contract before deployment rather than after an incident.

Copyright: the training-side question is narrowing; the output-side one is contractual

The training-side exposure has moved fast in eighteen months, and mostly toward clarity. In the United States, the Copyright Office's Part 3 report on generative AI training — pre-publication version released 9 May 2025 — worked the four fair-use factors and declined to endorse compulsory licensing, preferring voluntary markets to develop; on the fourth factor it flagged the potential for AI-generated content to substitute for original works, while recognizing that highly transformative uses weigh in favour of fair use and that copying entire works "often weighs against" it (Jones Day on the Part 3 report, May 2025).

Two subsequent events matter more to a deployer than the report does. First, Bartz v Anthropic: on 20 July 2026 Judge Araceli Martínez-Olguín of the Northern District of California granted final approval to a $1.5 billion class settlement — roughly $3,000 per work, described as four times the statutory minimum for ordinary infringement — resolving claims over the acquisition and copying of works through 25 August 2025, with nearly 595,000 potential class members notified. Critically for anyone reading it as a precedent, output-based claims were not released, nor were claims for conduct after the cutoff, nor pending cases against other AI companies (Authors Guild, 20 Jul 2026). This was a settlement about how the training corpus was obtained, not a ruling that training is or is not fair use.

Second, in the UK, Getty Images (US) Inc & Ors v Stability AI Limited [2025] EWHC 2863 (Ch), handed down 4 November 2025 by Mrs Justice Smith. Getty abandoned its primary copyright, database-right and passing-off claims before trial, having accepted there was no evidence the training took place in the UK. On the surviving secondary-infringement claim the court held that "the Model itself does not store any of those Copyright Works; the model weights are not themselves an infringing copy and they do not store an infringing copy" — model weights are learned patterns, not stored reproductions. Getty succeeded only on limited trade-mark findings relating to generated watermarks on early Stable Diffusion versions (Latham & Watkins on Getty v Stability).

For an enterprise buyer the practical residue of all this is small and specific. Training-corpus liability sits with the model developer, and the mechanism by which it reaches a customer is price and availability, not a claim. Output-side exposure sits with the deployer, and the instrument that shifts it is the vendor's indemnity — which is conditional. Microsoft's Copilot Copyright Commitment, launched in September 2023, undertakes to defend paid commercial Copilot customers against intellectual-property claims and cover related legal costs — subject to four conditions: the customer must not disable or circumvent the built-in content filters and safety systems, must not knowingly use or distribute output in ways that infringe third-party rights, must hold sufficient rights to all input including data used for customization, and must accept that trademark claims are excluded. Free Copilot services and custom-built services such as Copilot Studio are outside the commitment (Microsoft's own announcement, 7 Sep 2023; conditions as summarized by TechTarget, 23 Jul 2024). The practical reading is that the indemnity is real and it is a compliance obligation in disguise: a team that turns off a content filter to reduce false positives may have turned off the indemnity with it, and almost nobody documents that decision as a legal one.

The disclosure record: what companies now tell their own investors

There is a useful independent check on whether this is a real risk class or a consultant's construct: what public companies say under securities law, where overstating and understating both carry consequences. The Conference Board's analysis of S&P 500 Form 10-K risk-factor sections found that 72% disclosed at least one material AI risk in 2025, up from 12% in 2023 — with reputational risk the largest cluster at 38% of disclosures, cybersecurity at 20%, and 41 companies citing evolving regulation, 24 citing intellectual property and 13 citing privacy. Over a quarter of the index still makes no explicit reference to AI at all (Harvard Law School Forum on Corporate Governance, 15 Oct 2025).

The specificity is rising too. MercadoLibre's fiscal-2025 Form 10-K, filed 25 February 2026, lists among its principal risk factors that "any use of AI/ML technologies in our operations may present additional labor, legal, regulatory, and social risks, which could lead to additional costs and impact our competitive position" — a general-purpose adopter, not an AI vendor, placing AI risk in its summary of principal risks alongside credit and logistics (MercadoLibre Inc., Form 10-K for FY2025, Item 1A, filed 25 Feb 2026). A full-text search of 10-K filings since mid-2025 returns exactly six using the specific term "shadow AI" — a corporate travel platform, a collaboration-governance vendor, a data-security vendor (Tenable Holdings, Form 10-K FY2025, filed 27 Feb 2026), a biodiesel producer, an online education provider and MercadoLibre. Six is a small number, and it should be read that way: the phrase has begun crossing from security marketing into the language of material risk, but it has not yet arrived. The broader S&P 500 signal above is the one carrying weight; this is a leading indicator of vocabulary, not of prevalence.

Sector overlays: where AI meets an existing regime

In US banking, model risk management has governed statistical models for over a decade under the interagency guidance issued as SR 11-7 and OCC Bulletin 2011-12, and supervisors extended its principles — validation, ongoing monitoring, governance, effective challengeA term of art in banking supervision: someone competent, genuinely independent of the model's builders, and senior enough to be listened to actually pushes back on its design, assumptions and results. The test is not whether a review happened but whether it could have changed anything. — to machine-learning models. That perimeter was redrawn in April 2026, and how it was redrawn matters more than that it was. On 17 April 2026 the federal banking agencies replaced SR 11-7, OCC Bulletin 2011-12 and FIL-22-2017 with new risk-based interagency guidance — issued as OCC Bulletin 2026-13, Federal Reserve SR 26-2 and FDIC FIL-15-2026 — and drew an explicit boundary around the new technology rather than extending the old framework over it: "Generative AI and agentic AI models are novel and rapidly evolving. As such, they are not within the scope of this guidance." The agencies added that they "plan to issue in the near future a request for information that addresses model risk management generally and considers, in particular, banks' use of AI, including generative AI and agentic AI and AI-based models" (OCC Bulletin 2026-13, 17 Apr 2026).

That is not a gap the supervisors overlooked; it is a gap they drew. And it corroborates from the regulator's side exactly what section 2.4 argued from the engineering side. A generative assistant is not a model in the sense SR 11-7 contemplated — it has no stable input-output mapping, no reproducible validation run, and no bounded use. A model-risk framework built on reproducibility cannot validate a system whose vendor controls the batching, and the agencies have now declined to pretend otherwise. The consequence for a bank is uncomfortable rather than liberating: a generative deployment sits outside the one supervisory framework its model-risk function knows how to run, in a period where an RFI is open and expectations are being formed. The defensible posture is to apply the rescinded guidance's principles — validation, ongoing monitoring, governance, effective challenge — voluntarily and to document why, because the alternative is to have no articulable framework at all when the RFI turns into guidance.

In US healthcare, the position is contractual rather than novel, and the gap is in the contract rather than the technology. A vendor that creates, receives, maintains or transmits protected health information on a covered entity's behalf is a business associate, and a business associate agreement is a legal prerequisite. The AI-specific terms that a standard BAA template does not address are the ones that matter: whether the vendor may use PHI only to provide the service or also for model training and product improvement, how the minimum-necessary standard is met when "large amounts of PHI must be ingested by AI technology to train it," and how de-identified derivatives are segregated to prevent re-identification. HHS's Office for Civil Rights proposed the first major update to the HIPAA Security Rule in twenty years in March 2025 — removing the distinction between required and addressable safeguards, which bears directly on AI deployments — but no finalized AI-specific OCR guidance is available (HIPAA Journal, 2 May 2025).

Large versus small. The compliance burden here scales super-linearly with a large firm's jurisdictional footprint: a multinational faces the EU AI Act, a shifting patchwork of US state laws, sector supervisors, and customer-imposed contractual regimes, and reconciling them is a standing function rather than a project. It has the resources to run that function, and — the underrated advantage — the leverage to make vendors sign bespoke terms. The small firm faces fewer regimes but has no ability to negotiate any of them and, crucially, faces the same absolute obligations: Article 4 AI literacy does not have a headcount threshold. What the EU does give SMEs is the lower penalty cap and, under the omnibus, simplified quality-management and documentation forms. The realistic small-company posture is to inherit compliance from platform choice — buy from vendors who publish their own conformity documentation and certifications, and let procurement do the work that a compliance department would otherwise do.

2.6The organization: pilot purgatory, ownership, and the talent gap

Organizations start far more than they run. That gap is the single best-measured phenomenon in this whole subject, and it is measured by several independent parties who agree on the shape while disagreeing on the numbers — which is exactly what you want from evidence.

"Pilot purgatory" is the trade phrase. The mechanism underneath it is specific. A pilot succeeds by demonstrating that a model can do a task. A production deployment must additionally establish who owns it, how its output is checked, what happens when it is wrong, how its cost is attributed, what data it may touch, how it is versioned, and who is called at 3am. None of those are model problems. All of them are organizational problems, and the pilot deliberately deferred every one of them, because deferring them is what made the pilot fast.

Gartner's forecasts describe the consequence rather than the cause. Its 2026 Hype Cycle assessment holds that at least half of generative-AI projects will overrun budgets due to poor architectural choices and lack of operational know-how, that most organizations attempting custom models will abandon them, and that none of the thirty AI technologies examined has reached the plateau of productivity, with GenAI-enabled applications the only technology ascending the slope of enlightenment. Its earlier prediction that over 40% of agentic AI projects will be cancelled by end-2027 — on escalating costs, unclear business value and inadequate risk controls — is the same argument applied to the next wave.

Ownership: the turf problem is real and now has a nominal answer

AI strategy sits awkwardly in a modern org chart because it is simultaneously a technology decision (CTO), an operational-systems decision (CIO), a risk decision (CISO), a data decision (CDO) and a business-model decision (the P&L owner). The market's answer has been a new title: IBM's Institute for Business Value work, as reported in the executive-search trade press, puts the share of surveyed organizations with a Chief AI Officer at 76% in 2026 against 26% in 2025, with the role typically reporting to the CEO (Hunt Scanlon, 21 Jul 2026; IBM's own report page returned an access block, so this figure rests on secondary reporting and the underlying sample size is not disclosed in it — treat the level as indicative and the direction as solid). That is a striking jump. A title created in a year is a signal of executive attention, not of resolved accountability, and appointing a Chief AI Officer without also settling whether they hold budget, veto rights over deployments, or only convening authority reproduces the turf problem one level up.

The structural pattern that practitioners converge on is hub-and-spoke, and it is measured rather than merely advocated. McKinsey's global AI survey found organizations most often running a hybrid or partly centralized model for tech talent and for the adoption of AI solutions — some resources held centrally, the rest distributed across functions and business units — while keeping risk and compliance and data governance in a fully centralized model such as a centre of excellence (McKinsey, 12 Mar 2025; fielded 16–31 Jul 2024, 1,491 respondents in 101 nations). The split is not arbitrary. The hub keeps what consistency makes cheaper — one gateway, one evaluation harness, one set of vendor terms — and the spokes keep what domain knowledge decides, which is mostly the question of which problems are worth solving at all. How large the hub should be is a different matter, and nothing published measures it. The headcounts in circulation come from firms selling the design and they do not agree with one another; a specific number here is a sales figure, not a finding. This is the same shape that data platforms and cloud centres of excellence converged on a decade earlier, for the same reason, and organizations that have been through that transition once tend to reach it faster.

The centralize-versus-federate debate is genuinely unsettled at the margins, and both sides have real arguments. Centralization delivers enforceable controls, negotiated vendor terms, one auditable trail and no duplicated platform spend; its failure mode is a bottleneck that business units route around, which recreates shadow AI with better intentions. Federation delivers speed and domain fit; its failure mode is exactly what IBM's breach data measures — 68% of breached organizations without governance to detect shadow AI. The evidence does not resolve the debate, but it does constrain it: the observed failures of federation are security and cost failures, and the observed failures of centralization are speed failures. An organization should pick according to which failure it can survive, and should be explicit that it is making that trade.

Talent and workforce: a demand signal, not a headcount plan

The reskilling-versus-hiring question resolves differently by layer. The scarce, expensive and genuinely external skills are platform and evaluation engineering — the people who build the gateway, the eval harness and the retrieval layer. The abundant and internally developable skill is domain-expert use: the underwriter, the paralegal, the support lead who knows what "good" looks like and can write the acceptance criteria an evaluation encodes. Organizations that hire only the first and neglect the second end up with excellent infrastructure serving use cases nobody can validate.

There is also a compliance dimension that most workforce plans miss entirely: Article 4 of the AI Act makes AI literacy a legal obligation for deployers in the EU, not merely a good idea. The July 2026 omnibus softened the standard from ensuring a sufficient level of literacy to taking measures that support its development, and said in terms that no specific level need be guaranteed for any individual — but it left the obligation binding on every deployer at every risk tier. The practical effect is to make the evidence the deliverable rather than the outcome, which is if anything a stronger reason to build the programme deliberately. A training programme built for regulatory evidence looks different from one built for enthusiasm — it is documented, role-differentiated by technical knowledge and deployment context, and retained. Organizations building one anyway should build it once, to that bar.

Large versus small. The divergence on this layer is not about resources. It is about where the decision physically sits. In a large enterprise the AI decision is distributed by construction — the hybrid, partly centralized model is the norm in McKinsey's survey — so the Chief AI Officer, the hub, and the whole centralize-versus-federate argument are all the work of reassembling an authority the org chart already split five ways. Smaller organizations do the opposite: those under $500m in annual revenue are likelier than others to report fully centralizing both tech talent and AI adoption (McKinsey, 12 Mar 2025), which is a polite way of saying one person decides. That is a real advantage, and it is the one that actually kills pilot purgatory: no turf, no hub to staff, no chargeback to negotiate, and a pilot can become production in the same week because whoever ran it also owns the budget and carries the risk. It is also the sharpest single-point failure anywhere in this report. That person is doing three other jobs, has no bench behind them, and everything they have learned about which outputs to trust walks out with them. The aggregate cost shows in the Census data: 37% of firms with 250 or more employees use AI against under 20% of firms with fewer than twenty, and over the survey period use rose among firms with at least twenty employees while not changing significantly among the smallest. The large firm's problem is that too many people have a say. The small firm's problem is that one person has all of it, and no successor.

2.7Vendor strategy: lock-in, the multi-model hedge, and proving ROI

Enterprise LLM spend is concentrated and has been reordering fast. Menlo Ventures' December 2025 estimate — from a survey of roughly 495 US enterprise decision-makers fielded 7–25 November 2025, combined with bottoms-up market analysis — puts Anthropic at 40% of enterprise LLM spend, up from 24% in 2024 and 12% in 2023; OpenAI at 27%, down from 50% in 2023; Google at 21%, up from 7%; and all others at 12%. Open-weight models account for roughly 11% of enterprise LLM usage, down from 19% the prior year, with Chinese open-source models around 1%.

Two facts from the same research complicate the standard lock-in narrative in a useful way. First, buyers do not actually switch much: Menlo's mid-year 2025 data found 66% of developers upgraded within their existing provider, 23% made no change, and only 11% switched vendors — and that performance, not price, drove the decision, with builders adopting frontier models "despite prior versions dropping 10x in price." Second, open-weight share fell rather than rose. Both cut against the intuition that commoditization plus price competition produces vendor mobility.

Why the gap between the strategic case for a hedge and the observed behaviour? Because lock-in in this market is not where the procurement conversation looks for it. Swapping an API endpoint is trivial. What does not swap is everything built around it: prompts tuned to one model's quirks, evaluation suites calibrated to its outputs, agent scaffolding relying on its specific tool-calling behaviour, fine-tuned artefacts, and — increasingly — the vendor's proprietary agent, memory and tool infrastructure. Anthropic's parameter-deprecation note is a small, precise illustration: temperature, top_p and top_k now return a 400 error on Claude Opus 4.7 and later when set to non-default values, with the recommendation to "omit and use prompting to guide model behavior." A team that had encoded sampling parameters into its control logic had that lever removed by a vendor decision. Multiply that by tool schemas, caching semantics, thinking-block handling and context management, and the switching cost is real without any of it appearing in a contract.

The open-weight route is the structural alternative and it is less open than the word implies. The Llama 3.1 Community Licence — released 23 July 2024 — is not an OSI-approved licence. It requires prominent "Built with Llama" attribution, requires derivative model names to begin with "Llama," requires a specific notice file in redistributions, and, under Additional Commercial Terms, provides that a licensee whose products exceed 700 million monthly active users on the release date "must request a license from Meta" (Llama 3.1 Community License). For the overwhelming majority of enterprises the user threshold is irrelevant. The naming and attribution requirements are not — they reach into product branding — and the fact that a licence can condition commercial use on the licensor's discretion at all is the point. Open weights buy control over hosting, residency and model lifetime. They do not necessarily buy licence freedom, and they transfer the entire operational burden — serving, scaling, patching, evaluating — to the adopter, which is a substantial part of why enterprise open-weight share fell.

Proving ROI to a board is the layer's hardest problem and the evidence says nobody has solved it. PwC's 2026 Global CEO Survey — 4,454 chief executives, released 20 January 2026 — found 56% reporting neither increased revenue nor decreased costs from AI over the prior twelve months, 26% reporting cost reductions, and only 12% achieving both; PwC's account of what distinguishes that minority is not model choice but "enterprise-wide deployments consistent with business strategy" underpinned by technology infrastructure, a defined roadmap, formal risk processes and a supportive culture (The Register on the PwC 2026 CEO Survey, 20 Jan 2026). Only 15% of finance leaders in the DoiT survey could calculate AI ROI without significant obstacles; only 9% of executives in the WitnessAI survey said more than three-quarters of their AI initiatives delivered measurable financial returns. The mechanical difficulties are three and they compound. Attribution: productivity gains diffuse across many people doing many tasks slightly faster, which does not aggregate into a line on a P&L. Counterfactual: the comparison case is "what we would have done otherwise," which nobody measured. And the moving baseline: the capability being evaluated improves during the evaluation, so a negative result at month three may be a positive result at month nine on the same code. The response pattern observed at organizations that report progress is to abandon enterprise-level ROI as the primary metric in favour of instrumented per-use-case unit economics: cost per resolved ticket, cost per document processed, cost per line of reviewed code, measured against the pre-AI baseline for that specific process. It is narrower than what boards ask for. It is the thing that can actually be measured.

Large versus small. Large enterprises get negotiated rates, private offers, roadmap access and the leverage to demand contractual terms — and pay for it with integration debt: legacy systems, data spread across incompatible stores, and change-management costs that dwarf the licence. Small firms have almost no leverage and near-total dependence on vendor decisions made without them in mind: a price change, a deprecation, a terms revision arrives as a fact. Their compensating advantage is genuine and easy to underrate — no integration debt, no legacy estate, and the ability to switch a whole workflow in a week because there is only one of it. On this layer alone, small size is closer to an advantage than a handicap.

2.8Company-size divergence, laid out in one place

The base-rate difference is measurable. The US Census Bureau's Business Trends and Outlook Survey, covering 14 December 2025 to 3 May 2026, put national AI use at 19.8% as of 3 May 2026 — the period's readings ranged between 17% and 20%, so this is the latest point, not an average — with 37% of firms with 250 or more employees and 32% of firms with 100–249 employees using AI, against under 20% of firms with fewer than 20 employees — and while use rose among firms with at least 20 employees over that period, it did not change significantly among the smallest. By sector the spread is wider still: 39.7% in Information and 33.9% in Finance and Insurance against 14% in Retail Trade (US Census Bureau, 26 May 2026).

But the interesting divergence is not the adoption rate. It is that the same problem has a different shape, and in three of the eight areas the small firm is genuinely better placed.

Two consequences follow. The first is that governance advice does not transfer down-market. A control framework built for an organization with a CISO, a DLP deployment and a platform team is not a scaled-down version of what a 200-person firm needs; it is a different thing. The small firm's realistic control set is short and blunt: one sanctioned provider on commercial terms, one hard spend cap, one written rule about what may never be pasted anywhere, and a standing refusal to deploy an agent that touches sensitive data, reads untrusted content and can act outward. Four controls, all enforceable by one person, covering the majority of the documented loss modes.

The second is subtler and cuts the other way. The small firm's dependence on vendor decisions is not merely a commercial disadvantage; it is a correlated one. When a vendor changes retention terms, deprecates a model, or reprices, every small customer is affected simultaneously and none has recourse. Where a large enterprise's AI risk is idiosyncratic — its own deployments, its own permission drift — a small company's is substantially systemic and shared with everyone else on the same tier. That is not a reason for despair; it is a reason to treat vendor-terms monitoring as a real, scheduled activity rather than something the legal team does at renewal.

§The problem–response matrix

Everything in Part II, in one place, with the response pattern named and its maturity judged. Two rows have no reliable response pattern at all, and saying so is more useful than filling the cell.

Everything in Part II describes a chat-era baseline: a model that reads and writes, connected to some documents. The 2026 direction of travel is agents that act — and each of the seven layers changes in a specific, predictable way when they do. This part sets out that escalation, the regulatory clock that will run alongside it, where vendor economics are actually heading, which operating models are working, and the four questions a Head of AI Strategy has to answer without a settled playbook.

3.1Agentic AI: the same risks, with the safety catch removed

The July 2026 Hugging Face intrusion is the clearest published illustration of what changes, and the shift it demonstrates is one of tempo rather than kind. Disclosed on 16 July 2026, the intrusion was orchestrated end-to-end by an autonomous AI agent: after an initial code-execution foothold, the attacker "escalated from worker-level code execution to node-level access, harvesting cloud and cluster credentials," then moved laterallyUsing access won on one machine or account to reach others inside the same network. It is the step that turns a single foothold into a breach, and it is what stolen credentials are for. using them. Forensics recorded roughly 17,000 logged attacker actions across multiple short-lived sandboxAn isolated, disposable environment where untrusted code is allowed to run without touching anything that matters. Short-lived ones are created and destroyed per task — which is why the actions here were logged at all. environments over a single weekend — a volume and velocity that, in the Cloud Security Alliance's assessment, saw "an agent, rather than a person, [execute] the escalation and lateral-movement phases at a volume and pace no human operator could sustain manually," with self-migrating command-and-controlThe attacker's own servers, which the code they have planted phones home to for orders. "Self-migrating" means it relocated itself when a host was taken away from it, so cutting off one address did not end the session. infrastructure and autonomous privilege escalationTurning limited access into greater access — from what one process is allowed to do into what the whole machine, or the whole cluster, is allowed to do. Done well it leaves the attacker indistinguishable from an administrator.. Hugging Face confirmed the intrusion did not reach public-facing models, datasets or Spaces (CSA research note, 20 Jul 2026).

Read the numbers, not the drama. Seventeen thousand actions in a weekend is roughly one action every ten seconds, sustained for seventy-two hours. Every detection and response process in a normal enterprise is calibrated to human tempo: an analyst triages, a ticket is raised, someone is paged, a decision is made. Against machine-speed lateral movement, a control loop that takes an hour is not a slow control — it is not a control at all. The CSA's recommendations follow directly and are the right ones: continuous event-driven monitoring tuned for agent-speed anomalies, short-lived per-task credentials instead of long-lived service accounts, and runtime controls that intercept agent actions before execution against context-aware policy. Note that the second and third are architectural decisions made before deployment, not detections added after.

The organizational readiness data says most companies are nowhere near this. Gravitee's February 2026 survey of over 900 executives and practitioners found 88% reporting confirmed or suspected AI-agent security incidents in the prior year, only 14.4% saying all agents went live with full security or IT approval, an average of 47.1% of agents actively monitored, only 21.9% treating agents as independent identity-bearing entities, and 45.6% still relying on shared API keys for agent-to-agent authentication — while 82% of executives felt confident that existing policies protect them from unauthorized agent actions. Deloitte's survey of 3,235 leaders across 24 countries finds the same gap from the governance side: about 74% expect moderate-to-extensive AI-agent use by 2027, while only 21% report a mature agent-governance model, defined as clear boundaries on autonomous versus human-approved decisions, real-time behavioural monitoring, and audit trails capturing the full chain of agent actions (Deloitte, 24 Apr 2026).

The response pattern: a taxonomy exists, and it is nine months old

This layer is the one where a reader would most expect the answer to be "nothing reliable yet." It is not, quite. On 9 December 2025 OWASP published a separate Top 10 for Agentic Applications — ten categories, ASI01 to ASI10, scoped deliberately to systems that plan, hold memory, call tools and act with delegated authority, and designed to sit alongside rather than inside the LLM list (OWASP GenAI Security Project, 9 Dec 2025). The categories are goal hijack (ASI01), tool misuse and exploitation (ASI02), identity and privilege abuse (ASI03), agentic supply-chain vulnerabilities (ASI04), unexpected code execution (ASI05), memory and context poisoning (ASI06), insecure inter-agent communication (ASI07), cascading failures (ASI08), human–agent trust exploitation (ASI09) and rogue agents (ASI10).

Two things follow, and the second is the useful one. First, the division of labour is now explicit: the 2026 LLM list covers the model as a component that accepts input and produces output, and the moment the model becomes an actor with tools it can call, the risk moves to the agentic list. A security programme that maps only to the LLM Top 10 has, by OWASP's own construction, mapped half its agent estate. Second — and this is what makes the taxonomy worth adopting rather than merely citing — every finding in this section lands on one of the ten. Gravitee's 45.6% still using shared API keys for agent-to-agent authentication is ASI03 and ASI07 simultaneously. The MCP transport problem in 2.3 is ASI04. The Hugging Face intrusion's escalation and lateral movement is ASI03 into ASI05. The approval-fatigue failure that defeats human-in-the-loop at volume is ASI09. Nothing here is a novel risk class that the industry has yet to name; what is missing is that most organizations have not mapped their agents onto a list that has existed since December.

The limit is the same one that applies to NIST and ISO in 2.5: a taxonomy tells you what to look for, not whether your controls are adequate. ASI01 does not tell a deployer what fraction of goal-hijack attempts its scaffolding stops, because — as 2.3 established — nobody can tell them that. The taxonomy's value is coverage, not assurance, and a board briefing that presents a completed ASI mapping as evidence of security has made the same category error as one that presents an ISO/IEC 42001 certificate.

One second-order consequence will land on the CISO before anyone has budgeted for it. If agents need identity, credentials, entitlements, joiners-movers-leaversThe identity housekeeping cycle: grant access when someone joins, change it when they move roles, remove it when they leave. Every organization has one for people. An agent is a new kind of joiner that is created and destroyed at software speed. processes and audit trails, then the identity and access managementThe systems of record for who — or what — may sign in to which resource, and the processes that grant, change and revoke that access. Historically sized for a population of employees, and priced that way too. estate — sized for a human population — has to absorb a non-human population that may be an order of magnitude larger and that is created and destroyed at software speed. Gravitee's finding that only 21.9% of teams treat agents as independent identity-bearing entities and 45.6% use shared API keys is the measurement of how far from that most organizations are. Shared credentials across agents mean no attribution, and no attribution means no forensics — which is exactly why the Hugging Face timeline is reconstructable only because the actions were logged in sandboxes.

3.2The regulatory trajectory: what actually becomes enforceable, and when

The forward calendar matters more than the commentary, in three respects.

First, the direction of the last twelve months has been deferral of substantive duties and retention of procedural ones, in Brussels and in US states alike. The EU pushed Annex III high-risk obligations from August 2026 to December 2027; Colorado repealed its algorithmic-discrimination duties and replaced them with transparency, notification and record-keeping effective January 2027; California's SB 53 imposes framework publication, incident reporting and whistleblower channels rather than substantive safety requirements. If a strategy leader has to bet on which obligations survive the next revision cycle, the evidence says: the ones that ask you to write things down and tell someone.

Second, the enforcement machinery arrives after the obligation, and the lag is deliberate. GPAI obligations applied from 2 August 2025 but the GPAI-specific fining power under Article 101 became available only a year later. A gap between duty and penalty is not the same as a gap between duty and liability — private claims, contractual warranties and customer questionnaires do not wait for the regulator — but it does mean that the first year of any new obligation is a documentation year rather than an enforcement year, and organizations should use it as one.

Third, the US federal-preemption question is genuinely unresolved and a plan should not depend on it resolving either way. The White House issued a national AI legislative framework in March 2026 signalling a push to replace the state patchwork; no statutory preemption has been enacted. A multinational's realistic planning assumption is continued divergence, which argues for building to the strictest applicable regime and documenting once — not for waiting.

Evidence: Regulation (EU) 2026/1744 (OJ 24 Jul 2026); AI Act Articles 4, 50 and 99; Skadden analysis of Colorado SB 26-189 (9 Jun 2026); California SB 53 text. Method: Dated obligations only; items whose timing depends on pending rulemaking are marked as contingent. Excludes Member-State implementing measures and sector supervisors. Synthesis: The next eighteen months are a documentation window, not an enforcement one — which is precisely when the documentation is cheapest to produce.

3.3Vendor economics: two curves that keep getting confused

This is the debate a strategy leader is most likely to get wrong, because both sides are quoting true numbers about different things.

The falling-cost side is right and the magnitude is remarkable. Epoch AI's analysis of state-of-the-art models on six benchmarks over three years found that the price of reaching a fixed performance level falls at rates spanning 9× to 900× per year depending on the task, with the price of matching GPT-4's performance on PhD-level science questions falling about 40× per year — while cautioning that the fastest declines happened most recently and "it's less clear that those will persist" (Epoch AI, LLM inference price trends). MIT and collaborators put the frontier figure more conservatively at "around 5× to 10× per year" for a given benchmark level across knowledge, reasoning, maths and software-engineering tasks, and — importantly — decompose it: after controlling for hardware price improvements and competition effects, they estimate algorithmic efficiency progress at around 3× per year, meaning a substantial part of the headline decline is competitive pricing and hardware, not algorithms (Gundlach, Lynch, Mertens & Thompson, arXiv:2511.23455, v2 23 Mar 2026).

And then the same paper delivers the other curve, in one sentence: "the price of running frontier models is rising between 3× to 18× per year due to bigger models and larger reasoning demands."

That is the whole debate resolved, and it resolves in a way that neither camp will love. If a company fixes its capability requirement — "this classifier must be as good as it was last year" — its unit cost collapses at 5–10× a year, and the vendor optimists are right. If a company chases the frontier — always the newest model, always more reasoning tokens — its unit cost rises, and the sceptics are right. Most enterprises do the second while budgeting as though they were doing the first, which is a fair description of how 79% of them came to overspend.

The practical consequence is a portfolio decision that almost nobody makes explicitly. A workload should be assigned to one of two regimes: capability-fixed, where the job is to hold quality constant and ride the cost curve down by migrating to cheaper models as they clear the bar, or capability-seeking, where the job is to buy the best available answer and the cost is expected to rise. Both are legitimate. Running them under one budget line, with one forecast, is not. And the migration discipline the first regime requires — an evaluation suite that can certify a cheaper model against the same bar — is the same asset that section 2.4 required for deprecation. That is the single highest-leverage piece of platform engineering in this entire subject: one artefact that simultaneously solves forced migration, cost optimization and quality regression.

Two further vendor-economics questions bear on the medium term. The first is whether today's prices are subsidized. Nobody outside the vendors knows, and this primer will not pretend to. What can be observed is that pricing moves in both directions on short notice, that competitive share shifted violently between 2023 and 2025, and that a market where the leader changed twice in three years is not one in which a buyer should assume price stability in either direction. The second is what happens to the buyer's leverage as the market consolidates or fragments — and here the evidence points in opposite directions simultaneously. Concentration rose (three vendors hold 88% of enterprise LLM spend), which argues for less buyer power; but open-weight models exist and are technically substitutable, which caps how far pricing can move before self-hosting becomes rational. That the enterprise open-weight share fell from 19% to 11% suggests the cap is currently well above where prices sit.

3.4Which operating models are working, and which are not

The pattern in the evidence is consistent enough to state as a working model, with the caveat that the evidence base is survey data and self-report rather than controlled comparison.

Working: the thin platform hub. A small central team owning the gateway, the evaluation harness, the model registry, vendor terms and the governance record — with delivery pushed to business units. It works because it centralizes exactly the things where a single instance creates leverage and decentralizes the things where domain knowledge dominates. Its diagnostic is whether business units route around it; if they do, it has become a bottleneck rather than a platform.

Working: use-case unit economics. Organizations that report progress on value are measuring cost and outcome per process, not per enterprise. This is less satisfying than an enterprise ROI number and it is the thing that can actually be computed.

Working: provisioning as a security control. The best-evidenced intervention in this primer is Netskope's observation that personal-account use among workplace gen-AI users fell from 78% to 47% in a single year while total users tripled — the exposure channel shrank in share against a rising base. It is one vendor's telemetry over one year, with overlapping categories, and it is a correlation rather than a controlled result; it is still the strongest evidence any response pattern here has. Give people the governed version of the tool they already want, fast, and the channel narrows without an enforcement fight.

Failing: policy without telemetry. IBM's finding that 68% of breached organizations had no governance to detect shadow AI — up from 63% — while incidence doubled is the clearest measurement of this failure mode. Most of those organizations had an AI policy. What they did not have was any mechanism that could observe a violation.

Failing: the title without the mandate. A Chief AI Officer population that grew from roughly a quarter to roughly three-quarters of surveyed organizations in a year has, by construction, been filled largely by people appointed into an undefined role. Where the appointment came with budget, veto rights over production deployment and ownership of the platform, it consolidates accountability. Where it came with a convening remit, it adds a fifth party to a four-way turf conflict.

Failing: treating agents as features. Deloitte's 21%-mature-governance figure against ~74% expected agent use by 2027, and Gravitee's 14.4% full-approval figure, describe the same thing from two angles: agents are being shipped through product processes rather than through the identity, security and change-management processes that a new class of actor requires.

3.5Four debates a strategy leader must take a position on

These are unresolved. Each is argued here on its merits, with what the evidence does and does not settle, and a stated view where one is defensible.

Centralize or federate AI governance?

The centralizing case: controls only work at a choke point, vendor terms only improve with aggregated volume, and the audit trail only exists if there is one system of record. The federating case: proximity to the problem determines whether a use case is worth building, and central teams reliably become queues. What the evidence settles is narrow but real — the documented failures of federation are cost and security failures with measurable price tags (43% shadow-AI incidence, $5.39m average, 79% overrun), while the documented failures of centralization are speed failures that show up as forgone benefit and are therefore invisible in incident data. That asymmetry in measurability is itself a bias that a strategy leader should correct for rather than be persuaded by. The defensible position: centralize the control plane (gateway, identity, evaluation, terms), federate the use cases, and set an explicit service-level commitment on the central team's turnaround — because the moment it slips, the shadow channel reopens.

Single vendor or multi-model hedge?

The hedge case rests on concentration risk and negotiating leverage, and is strengthened by how violently share moved between 2023 and 2025. The single-vendor case rests on the observed fact that 89% of buyers did not switch vendors in a year — Menlo's mid-2025 split of 66% upgrading within their existing provider plus 23% making no change, leaving 11% who switched — and that switching cost lives in scaffolding rather than endpoints — prompts, evals, agent frameworks, tool schemas, memory. The genuinely useful distinction is between portability and plurality. Building portability — a gateway abstraction, model-agnostic prompts, an evaluation suite that runs against any model — is cheap, reusable and pays for itself against deprecation regardless of whether you ever switch. Running plurality — two vendors live in production — costs double the operational surface for benefit that only materializes in a discontinuity. The defensible position: buy portability always, run plurality only where an outage or a terms change would be existential.

Build or buy?

Gartner's read — that most organizations attempting custom models will abandon them to cost, complexity and technical debt — is a strong prior against building models. It is not an argument against building systems, and the two get conflated constantly. The evidence supports a fairly clean split: do not build models; do build the control plane. The gateway, the evaluation harness, the retrieval layer with permission inheritance and the model registry are the assets that survive vendor change, and every one of them is a modest engineering project rather than a research programme. The organizations that report the least pain are the ones that treated those four as products with owners.

Are inference costs falling faster than usage grows?

Section 3.3 argues this is a false dichotomy that dissolves once you separate capability-fixed from capability-seeking workloads. The residual genuine uncertainty is what happens at the frontier if the 3–18× annual rise in the cost of running frontier models continues while the 5–10× decline for fixed capability decelerates — which Epoch explicitly warns may happen, since the fastest declines are the most recent. In that world, the gap between "good enough, very cheap" and "best available, very expensive" widens, and the strategic question becomes which of a company's workloads genuinely need the frontier. Most do not, and most organizations have never asked.

§Five things the vendor pitch leaves out

Each of these contradicts a specific consensus view, rests on disclosed evidence, and can be falsified.

One. Falling token prices and rising task costs are both true, and the industry quotes whichever suits. Consensus: "inference is getting exponentially cheaper, so cost is a transitional problem." The same 2026 analysis that measures a 5–10× annual decline in the price of a fixed benchmark level also finds "the price of running frontier models is rising between 3× to 18× per year." Add the vendor-disclosed mechanics — a tokenizer producing ~30% more tokens for the same text, thinking tokens billed as output and then re-billed as input, a 1.1× residency multiplier — and a headline price cut can arrive as a bill increase. Falsifier: published per-task cost benchmarks showing enterprise cost per completed task falling at the same rate as cost per token.

Two. The standard cure for unpredictable AI spend buys a price, not a supply. Consensus: "reserve capacity and the cost problem is solved." Microsoft's own documentation states that PTU quota "doesn't guarantee that capacity is available," that availability changes through the day with demand, and that "reservations don't guarantee capacity" — advising customers to deploy first and reserve second. The buyer converts price risk into availability risk and keeps it, on a risk register that has no row for it. Falsifier: a hyperscaler offering AI capacity reservations with a contractual availability SLA.

Three. "We don't train on your data" is the weakest of the three guarantees that matter, and the strongest one is being outrun by the roadmap. Consensus: the training question is the data question. In practice the training guarantee is nearly universal on commercial tiers and nearly irrelevant to the observed loss events; the operative controls are retention and compulsion. Zero data retention is the real instrument — and its coverage is endpoint-scoped, excluding stateful features, while the industry moves toward stateful agents. Meanwhile a court order in litigation between two other parties swept in API customers who lacked ZDR. Falsifier: vendors extending zero-retention terms to stateful agent, batch and file endpoints.

Four. Security here is bought with utility, and the price is now published. Consensus: guardrails are a feature you add. The strongest published prompt-injection defence delivers 77% of task success against 84% undefended, at roughly 2.7× the tokens, and every control in Figure 4 that measurably works does so by removing capability. OWASP states that fool-proof prevention is unclear; NIST publishes mitigation limitations alongside mitigations; OpenAI runs an automated adversarial attacker against its own browser agent. Falsifier: a defence achieving undefended-parity task success with negligible overhead on an independent benchmark.

Five. AI deployment is an information-governance audit that arrives whether or not you commissioned one. Consensus: assistants create a new data-leakage risk. The mechanism is the reverse. Microsoft's remediation list for Copilot oversharing is auto-labelling, access review, restricted content discovery and three-year retention deletion — none of which are AI controls. The assistant did not create the over-permissioned finance folder; it made twenty years of permission drift searchable in plain English. The cost of AI adoption therefore includes a records-management backlog most organizations have been deferring since the file server, and that cost belongs in the business case rather than in a surprise remediation budget. Falsifier: evidence that organizations deploying enterprise assistants incur no measurable increase in information-governance remediation work.

§The strongest case against everything above

Put fairly, the opposing view goes like this, and it deserves to be taken seriously because it has been right before.

Every one of these problems is the ordinary friction of an early platform shift, and the historical record on such frictions is that they resolve — not because anyone solved them heroically, but because tooling, contracts and professional norms caught up. Cloud computing in 2010 looked exactly like this. Costs were unpredictable and the discipline to control them did not exist; FinOps was invented, and now 1,192 practitioners manage $83bn. Security teams said the shared-responsibility modelThe cloud contract's division of labour: the provider secures the underlying infrastructure, the customer secures whatever it builds on top. The arguments are always about where exactly the line sits. was unworkable; CSPMCloud security posture management — tooling that continuously scans a cloud estate for misconfigurations and over-broad permissions, and reports what to go and close. tooling arrived. Compliance said data residency was impossible; regions and sovereign clouds were built. On this reading, the AI-specific pathologies catalogued above are a snapshot of a two-year-old operating discipline, and treating a two-year-old discipline's gaps as structural is the classic error of the sceptic who mistakes immaturity for impossibility. The supporting evidence is real: personal-account use fell from 78% to 47% in one year; managed-account use rose from 25% to 62%; the share of businesses without responsible-AI policies fell from 24% to 11%; AI-specific governance roles grew 17% in 2025; determinism was demonstrated to be achievable within eighteen months of the problem being properly diagnosed. Everything is improving quickly. Give it three years.

That argument is substantially correct about trajectory and it is wrong about two things, and the two things are the ones that matter.

The first is the control plane. In the cloud analogy, the buyer eventually gained control: you choose your instance types, your regions, your upgrade windows, your log retention. Every one of those became a customer-configurable property, which is exactly why the tooling could catch up. In enterprise AI the control plane is a third party's model, and the properties that matter — what version runs, how it is batched, how text is tokenized, when it retires — are not customer-configurable and show no sign of becoming so. Microsoft will let a customer set NoAutoUpgrade; the documented consequence is that the deployment stops working. That is not a control, it is a choice of failure mode. FinOps could mature because the underlying resource was observable and schedulable. Reproducible inference is now achievable — vLLM ships batch invariance — but only by taking the workload off the hosted API and running it yourself, which is precisely the move the cloud analogy says nobody makes at scale.

The second is that the two hardest problems here are not engineering-immature, they are structurally constrained. Prompt injection is not an unpatched bug; it is a consequence of a single-channel architecture, and the best published defence pays for security in utility and tokens. Enterprise ROI attribution is not a missing dashboard; it is a counterfactual problem that no instrumentation resolves. Cloud had no equivalent of either. A virtual machine is deterministic and a cloud migration's savings are measurable against a decommissioned data centre.

Where the optimist is right, and this primer should concede it plainly: most of the cost, data-governance and organizational problems in Part II are maturity problems, and they will look considerably smaller in 2029. The claim being defended is narrower than "AI adoption is structurally hard." It is that the seam between pilot and governed production is where value is currently lost, that the seam is made of cost accounting, data lineage and ownership rather than capability, and that two specific items — reproducible inference as a purchasable property of a hosted service, and enterprise-level ROI attribution — have no reliable response pattern today. On the first, the open serving stacks have moved faster than the commercial ones, which is itself a finding: the constraint is contractual, not technical. On the second, PwC's 56%-of-CEOs figure suggests the problem is not getting smaller with time.

§The limits of current understanding

Six places where the evidence base is genuinely thin, and where a reader should hold conclusions loosely.

The incident record is short and selected. Almost everything known about AI-specific security failures comes from disclosed research findings, a handful of published breaches and vendor telemetry. The AI Incident Database logged 362 incidents in 2025 against 233 in 2024 — a count of documented incidents, which rises with attention as well as with incidence. There is no equivalent of decades of breach-notification data, no actuarial base rate, and consequently no defensible frequency estimate for any AI-specific loss event.

The adoption and value surveys are self-report, and they disagree. Stanford, McKinsey, BCG, Deloitte and the Census Bureau all measure something called AI adoption and get answers spanning 19.8% to 88% because they survey different populations with different definitions. This primer has tried to keep those populations visible at every point of comparison. No reader should treat any single figure as the adoption rate.

Cost data is mostly vendor-side or survey-side, rarely instrumented. The overrun figures come from finance leaders reporting perceptions of their own budgets. The unit-cost figures in Figure 1 are computed from published rates and are exact — but they describe a hypothetical workload, not a measured production system. Nobody publishes audited enterprise AI unit economics, and the organizations that could are not incentivized to.

Response-pattern effectiveness is almost entirely unmeasured. This is the largest gap. There is good evidence that gateways, DSPM, evaluation suites and hub-and-spoke structures are used. There is very little evidence about how much they reduce loss. The single best-evidenced intervention in this primer — provisioning governed accounts, with the associated fall in personal-account use from 78% to 47% — is a correlation in one vendor's telemetry over one year, not a controlled result. Figure 11's maturity column is a judgement, and it is labelled as one.

The agentic evidence base is roughly twelve months old. Everything in Part III rests on 2026 survey data and a small number of disclosed incidents. The Hugging Face intrusion is one event. It is a genuinely instructive event, and it is one event.

Several primaries could not be reached. OpenAI's enterprise-privacy and prompt-injection pages, BCG's AI Radar 2026 publication, PwC's CEO-survey press page, ISO's standard page, Gartner's press releases, IBM's CEO study, the WEF Future of Jobs 2026 page and the CanLII text of Moffatt all returned access blocks to automated retrieval, and the EUR-Lex page for Regulation (EU) 2026/1744 returned an empty response — its dates and amendments are carried here on three independent legal analyses that agree on every one of them. Where a figure mattered, it has been sourced to the best available secondary account and flagged in place; where it could not be, it appears below rather than in the argument. Two consequences follow: BCG's and WEF's survey data are absent from this primer entirely rather than paraphrased from memory, and several figures that would ordinarily be Tier 1 are carried at Tier 2 because they reach the reader through a trade report rather than the issuing firm's own page. Where an issuer's report PDF blocks retrieval, its own press release often does not: IBM's headline breach figures and the operative US bank model-risk bulletin are both cited here from the issuing body's own page for exactly that reason. A block on one document is not the same as an unavailable fact.

§Not verified — carries no conclusion in this primer

§Evidence register

Every source below was opened and read in preparing this primer. Tier 1 = primary vendor, regulatory or standards documents, statutory text, SEC filings, and named-firm survey data with disclosed methodology. Tier 2 = peer-reviewed or security research and reputable trade/business/legal press. Tier 3 = single-source, aggregated or estimated, used only where flagged. Dates are publication or last-updated dates as shown on the source; access date for all is 16 August 2026.

Tier 1 — vendor, regulatory, standards and filing primaries

Tier 2 — peer-reviewed, security research, and reputable trade, business and legal press

Tier 3 — single-source or aggregated, used only where flagged in the text