Several board directors have asked me the same question lately: we are accountable for what our vendors do, more of it runs on AI, and none of us can see inside it. This is what I tell them: what regulators expect of the firm that buys, what two recent breaches say about where the risk sits, and the four questions a board can put to any vendor's AI without opening it.
Several board directors have come to me in the past few weeks with a version of the same question. Their firm is accountable for what its vendors do. More of what those vendors do now runs on AI. And nobody around the table can see inside it. They feel the responsibility clearly and the black box not at all, which is an uncomfortable place for people whose names are on the register. The honest answer is that you cannot see inside, and you should stop trying. The weights are not yours. The training data is not yours. What you tested at onboarding has shifted by the next release. What you can see is whether anyone is governing it: who inside the vendor owns the model, who inside your firm signs for it, what it was tested on, and what happens when it is wrong. You are not trusting the AI. You are trusting the governance around it. Last week I wrote this from the vendor's chair. This week is for the board that signs.
Most regulated firms never decided to adopt AI. It arrived inside services they already paid for: the administrator's reconciliation engine, the data vendor's summaries, the note-taker in the CRM, the research tool that now drafts the memo. Release by release, often switched on by default, rarely with a letter. Accountability did not move with it. The Central Bank of Ireland's 2026 Regulatory and Supervisory Outlook says that "widespread adoption of third-party AI tools changes the risk landscape and calls for stronger model governance", and its outsourcing guidance has put the board, not the provider, on the hook since 2021. In the UK, HM Treasury designated the first four critical third parties on 10 July: Amazon Web Services, Google Cloud, Microsoft and Oracle. The FCA's statement that day is worth reading twice. The regime "complements, but does not replace, existing outsourcing and operational resilience rules", and regulated firms "remain responsible for managing their own third-party arrangements". DORA, in force since January 2025, requires a register of every ICT provider and the subcontractors beneath them. And on 7 October the Monetary Authority of Singapore published binding guidelines on AI risk management with the sentence every other regulator will eventually write: financial institutions "remain accountable for AI used in the services they deliver, including AI developed, operated or provided by third parties", and where assurance cannot be obtained they "should consider limiting, suspending or replacing the use of the third-party AI service". None of this transfers responsibility. All of it adds a duty to know.
The regulation tells you who is accountable. It does not tell you how to look at something you cannot open. That gap is where most oversight frameworks in fund management now sit: written for a vendor estate that ran on rules, applied to one that increasingly runs on models.
In August last year attackers stole the OAuth tokens that connected Drift, a third-party AI chat agent for sales teams, to its customers' Salesforce systems. Over ten days they used those tokens to pull data from what Google's Threat Intelligence Group called "over 700 potentially impacted organizations". The campaign was "broad and opportunistic", taking "advantage of any organization using the Salesloft Drift integration with Salesforce". Nobody breached those 700 firms directly. Their vendor's AI agent held a key to the building and the key was copied.
In May this year a US regional bank filed an 8-K with the SEC to report that customer names, dates of birth and Social Security numbers had been put into what it could only describe as an "unauthorized AI-based software application". It disclosed because of "the volume and sensitive nature of the non-public information", and at the time of filing it was still "evaluating the customer data that was affected". An AI tool nobody had approved touched the data, and the bank had to tell its regulators before it could tell them how much. The AI tool is rarely the thing that fails. It is the identity with standing access through which something else fails, or the unapproved one through which data walks out.
If you want the cost of unwatched model access even when the model is your own, the SEC's Two Sigma order from January 2025 supplies it: a known control weakness flagged in March 2019, one employee able to change live trading models without approval for nearly two years, $90 million in penalties and $165 million repaid to the funds that lost out. A vendor's model is one step further from your sight than Two Sigma's were from theirs.
The structural point is this. Treat every agent, yours or your vendor's, as a privileged identity, with an inventory, defined access and authority, monitoring and a means of containment. Recognise the supply-chain pivot, where a compromised agent inside a supplier is turned against the supplier's clients. Plan for containment at machine speed, revoking access faster than a human can read the alert, while knowing that a wrong shutdown has its own cost. Vendor AI is a cyber question and an accountability question at once, and the board owns both.
Neither the FCA nor the Central Bank has issued an AI framework and I do not expect one. They judge outcomes. So firms borrow, and the one I use for the vendor estate is NIST's AI Risk Management Framework: American, voluntary, released in January 2023 with a generative AI profile in July 2024. It has four functions, Govern, Map, Measure and Manage, written for firms building their own models. It transfers to models you merely consume if you point each function outward, at the vendor rather than at your own data scientists. That is the spine I use.
Govern asks who inside your firm signs for the AI inside each vendor's service. Not the vendor relationship, which has an owner already, but the model within it, which usually does not. Name the person and put the vendor's AI on the same register as the agents you built yourself. The board's AI inventory should cover what the firm bought, not only what it made, and it should hold a record of the decisions taken about each one: who approved it, on what evidence, and when it was last looked at. When the regulator asks, that record is the governance.
Map asks where AI already sits in the estate and where it lands next. Your DORA register tells you who your providers are. It does not tell you which of them switched on a model since the last review, what standing access that model holds into your data, or whose subcontractor is running it. It is usually the document that surprises the board.
Measure asks what tests you can run on something you cannot open. Start with the vendor's own evidence: the model governance summary, bias testing and the population it was run against, the security posture including the last incident, the subprocessor list. Then add your own: a sample of outputs replayed against onboarding to see how far the behaviour has moved, and a check of what the entitlements actually permit, which is the Drift lesson. Last week's trust pack is this column read from the other side of the table. A vendor who has one makes your measurement short. A vendor who has not has told you something.
The test almost nobody runs: when the vendor announces a new model, treat it as a challenger, not an upgrade. Ask for it to run beside the incumbent on your own sample for a defined period, scored on one sheet: accuracy against your ground truth, consistency across repeated runs, latency, and cost per task. That last column is where the surprises live. A newer model that reasons at length can burn several times the tokens of the one it replaces to produce a similar answer, and if your pricing is usage-based that is a cost increase dressed as a feature release. Promotion from challenger to champion should be your decision on evidence, not the vendor's release note, and the right to run the comparison belongs in the contract.
Manage is what you ask for at renewal and rehearse before you need it. Notice of a material model change before it ships. The right to run the challenger before it goes live. A switch you can operate yourself, with access revoked in minutes rather than days, because the Drift tokens were in use for ten days. Disclosure of sub-outsourcing. Exit with your data and its lineage. Containment is a drill, not a clause.
| Function | The question the board asks | What good looks like |
|---|---|---|
| Govern | Who in our firm signs, by name, for the AI inside each vendor's service? | A named owner per vendor AI, on the same register as the agents you built; an inventory that covers bought AI as well as made; a record of the decisions taken about each one and by whom |
| Map | Which vendors switched on AI since their last review, what access does it hold, and whose subcontractors run it? | An estate map by service, with standing access and the sub-outsourcing chain, refreshed every release cycle |
| Measure | What was it tested on, how far has it moved since onboarding, and what happened in the last incident? | The vendor's evidence (governance summary, bias results, security posture, subprocessors), your own replayed sample, and a champion and challenger run before any new model is promoted |
| Manage | Can we switch it off ourselves, how fast, and can we leave with our data? | Notice of material model change, right to run the challenger, revocation in minutes, exit with data and lineage, containment rehearsed |
NIST's four functions pointed at the vendor estate, with the questions a board can put to any vendor's AI without seeing inside it.
Stop asking to see inside. Ask who owns it, where it is, what it was tested on and how you switch it off. If the vendor can answer in a pack your board can read, you have your diligence. If they cannot, you have your answer as well. And write the process down, because the day something goes wrong the regulator will not ask whether you understood the model. They will ask whether you governed the people who did.
If this is the question at your board, the Vendor AI Oversight Review is how I answer it. If you are the vendor on the other side of the table, last week's piece is yours.
Sources: Central Bank of Ireland Regulatory and Supervisory Outlook 2026; FCA statement on critical third parties, 10 July 2026; CyberScoop and Google Threat Intelligence Group on the Salesloft Drift campaign; The Register on the Community Bank 8-K filing; Monetary Authority of Singapore media release, 7 October 2026; SEC order against Two Sigma via Investment Executive; NIST AI Risk Management Framework. All linked inline.