If you sell software to fund managers, banks or insurers and have put an agent into your product that acts on its own, most of your clients, while aware of accountability, still file it under software upgrades. A named executive inside their firm is accountable for it all the same, and more so once it is probabilistic, because its behaviour can change without anyone changing it. What the first failures looked like, why probabilistic software breaks their due diligence, where the EU insists on a human, and what the person who signs will ask you once they notice.
If you run a fintech or RegTech selling into regulated firms, this column has spent three weeks looking at your business from the outside: the market, the revenue engine, the voices that speak for you. This week I want to turn the camera round, to the person whose name goes on the line when your software does something it should not.
In the sales piece I called that person the economic buyer. In regulated financial services they are a senior manager or a pre-approved controlled function holder, personally answerable to a regulator for the part of the business your product touches. They know that. The accountability is not news to them. What changes is what it covers, and it changes the day you deploy an agent that acts without a human, and again if that agent is probabilistic rather than rule-based. Not every product has crossed that line. Where yours has, most of your clients have not noticed. They bought an agent and booked it as a software upgrade, and the accountability they already carry now covers something they did not know they had taken on.
When a fund manager hires an operations analyst, the analyst gets a reporting line, a job description, access granted system by system, and a probation period in which someone watches the work. That is how their manager can say, truthfully, that the business is under control. When your release notes announce an agent that reconciles breaks or drafts onboarding decisions, it goes live inside the same firm doing work a person used to do, with none of that. Nobody is named as its manager. Its limits live in your documentation, not theirs. Its permissions are whatever the integration was granted on the day of install.
| The apparatus | What a new hire gets | What an agent usually arrives with | What the person who signs needs |
|---|---|---|---|
| Reporting line | A named manager who answers for their work | Nobody; it arrived in the vendor's last release | A named accountable human for every agent |
| Job description | A defined role, with limits | A capability list in the product documentation | A written mandate: what it may decide, and what it must escalate |
| Permissions | Access granted role by role, removed on exit | Whatever the integration was given at install | Access scoped to the mandate, reviewed like a joiner's |
| Review | Probation, appraisal, conduct monitoring | A vendor dashboard, if anyone opens it | A record of what it did, and why, that the accountable person can sign |
A person joins a regulated firm inside an accountability structure. An agent usually does not, and the person who signs for it will notice, probably after the first mistake or the first unannounced model update.
The first failures have already happened, not yet in a fund. Air Canada's chatbot invented a bereavement fare policy, and when the customer sued, the airline argued that the chatbot was "a separate legal entity that is responsible for its own actions". The tribunal disagreed: Air Canada "still bore responsibility for all the information on its website, whether it came from a static page or a chatbot."
Last year a Replit coding agent deleted a company's production database during a declared code freeze, then told its owner a rollback was impossible, which turned out to be untrue. Deloitte refunded part of its fee on a report for the Australian government after the citations and a quoted court judgment were found to have been invented by the model. One pattern: a firm put its name to work that software did, and found out what it had signed for afterwards. The vendor who explains that to your client first is the one they keep.
Your buyer's due diligence was built for deterministic software: same input, same output, every time. That software can fail spectacularly. Knight Capital's router, running old code moved out of sequence, lost the firm more than $460 million in forty-five minutes. But it failed predictably, doing the same wrong thing every time, and a test could have caught it.
An agent built on a language model is probabilistic. The same input can produce a different answer tomorrow, so the buyer cannot test the path, only the distribution of outcomes. And the distribution moves: the foundation model underneath gets updated on the provider's schedule, not yours or theirs, so an agent that passed acceptance testing in March can behave differently in June with nobody having changed a line of the client's configuration. When Goldman Sachs put hundreds of Devin coding agents alongside its engineers, QA Financial noted that an agent which writes its own tests means "a high test-pass rate is not necessarily evidence that the code is fit for production". Goldman has the engineers to build that assurance itself. A fifty-person ManCo does not, so the assurance has to come from the vendor, which means you.
Two more things the signer's regulator will ask, and so the signer will ask you. First, can they explain what the agent decided? Apple Card is instructive because the model was clean: New York's Department of Financial Services found that women and men with equivalent credit characteristics had similar outcomes. What it did find was that applicants could not get an explanation of their credit limits, and a six-month wait to appeal "frustrated applicants". A defensible decision the firm could not explain still became a regulatory investigation and a public bias story. The second is bias itself: whether an onboarding or suitability agent treats one client population differently from another, and whether anyone has tested it against a client base like theirs.
The EU has already written both into law for the highest-stakes uses. Under the AI Act, systems that evaluate "the creditworthiness of natural persons" or price life and health insurance are high-risk, and from December 2027 Article 14 requires that the client's own staff can "disregard, override or reverse the output" and "interrupt the system through a 'stop' button or a similar procedure". When a client insists on a human in the loop, they are describing the statute they will be held to. Sarah Breeden of the Bank of England put the counterweight at Sintra in June: our frameworks "were not built to contemplate autonomous agents, and relying on a human in the loop for all agent actions is unlikely to be realistic." Read the two together and the design question is not whether a human is in the loop, but which decisions get one, and whether your product lets the client draw that line.
Neither the FCA nor the Central Bank of Ireland has written an agent rulebook, and neither needs to. The UK's Senior Managers Regime already requires a senior manager to take reasonable steps to prevent a breach in their area, AI tools included, and the Treasury Committee has asked the FCA to spell that out by the end of this year. The Central Bank of Ireland puts it plainer: firms "remain fully accountable for outcomes generated by AI systems, including where solutions are developed, procured or operated by third party providers", and its conduct standards already require PCF holders in fund management companies to take reasonable steps to keep the business they run controlled effectively. "Procured or operated by third party providers" means you.
I have sat in that chair. At HSBC Alternative Investments, a fund administrator and a transfer agent did work every day that I was accountable for. I never ran their systems. I could carry it because I could see the work and knew who to call when it went wrong. An agent is the same arrangement with the visibility removed, and one that can change under them. Once your client sees that, the questions come quickly. Who, by role, inside their firm owns this agent? What may it decide alone, and what must it hand back? What can it reach? Can they explain one of its decisions to a client, or to their regulator? Has anyone tested how it treats different clients? And when the regulator asks what it did last quarter, is there a record they could sign, or a log only your engineers can read?
When I wrote my book, accountability was something people carried and machines supported. It still is. What has changed is how much of the work those people answer for is done by software you shipped, and how quietly that software can change. Send each agent into a client's firm the way they would take on a new hire, with its file, and you have told them what their accountability now covers before their regulator does. Leave it to them, and the most capable product in your category becomes the one they cannot sign for. The moat I described at the start of this series, distribution into buyers who already trust you, is built or spent right here.
Next week, the other side of that table: what the vendors who keep winning put in front of the buyer, and why trust is now the product spec.
Sources: American Bar Association (Moffatt v Air Canada); The Register; Fortune; SEC; QA Financial; New York DFS; EU AI Act; Bank of England; Treasury Committee via Regulation Tomorrow; Central Bank of Ireland via Pinsent Masons. All linked inline.
For the COO or risk lead who has to answer for the AI their vendors bring in, One Blackwater runs a three-week Vendor AI Oversight Review. Detail on the Services page.