What happens when AI gives the wrong financial advice?

Back to Insights

Ask Claude, one of the leading generative AI models, to recommend how to go rock climbing in the Cotswolds and you will receive a confident, well-structured, entirely wrong answer.

The Cotswolds is a designated area of natural beauty just north of Oxford. It comprises gentle hillsides and picturesque villages. There are no mountains. There are no rock faces. There is nowhere to go rock climbing. The AI model did not know this, but it did not know that it did not know. It generated a plausible answer because plausible answers are what it is designed to produce.

In travel planning, this is an amusing anecdote. In wealth management, the same behaviour applied to investment recommendations, risk assessments, or suitability decisions has consequences that are financial, regulatory, and reputational. A model that confidently recommends an inappropriate investment product to a client, or generates a risk assessment that fails to account for a relevant factor, or produces compliance documentation that looks authoritative but contains material errors, creates liability that sits with the firm and with named individuals under the Senior Managers and Certification Regime.

The question is not whether AI should be used in wealth management. It clearly should, and increasingly it is. The question is what ethical architecture needs to be in place before the confidence of the model meets the complexity of the client relationship.

The principles that matter

Having spent 18 years building operational infrastructure at HSBC Alternative Investments, where every process, every data flow, and every client-facing output required governance that could survive regulatory scrutiny, I am familiar with what happens when controls are treated as afterthoughts. They fail under pressure and the firm pays the price. The same principle applies to AI in wealth management, and the ethical framework needs to be specific rather than aspirational.

Data privacy and security is the foundation. Client information must be protected not just from unauthorised access and breaches but from subtler forms of exploitation: emotional profiling, behavioural tracking beyond the scope of the advisory relationship, and classification systems that score and rank investors without their knowledge or consent. The data that makes AI useful in wealth management is the same data that creates harm if it is misused.

Bias mitigation requires continuous effort, not a one-time audit. AI algorithms trained on historical data will reproduce the biases embedded in that data. In wealth management, that means investment advice, risk assessments, and client interactions can systematically disadvantage certain groups unless the firm actively audits and refines its models. Interaction bias, latent bias, and selection bias all need to be identified and addressed. The broader societal impact matters too: AI that escalates economic inequality or contributes to systemic financial risk is not serving the interests of the clients it was built to help.

Transparency is where most firms will be tested hardest. AI applications need to provide understandable explanations of their risk assessments and advice logic so that investors can make genuinely informed decisions. JPMorgan Chase has established an Explainable AI Centre of Excellence focused specifically on developing techniques and frameworks for model transparency and fairness. That level of institutional commitment is the benchmark, not the exception. However, transparency must be balanced against the protection of proprietary intellectual property. The GDPR Trade Secrets Directive provides a framework for that balance, but navigating it in practice requires careful design.

Investor autonomy is a principle that sounds obvious until you look at how recommendation systems actually work. AI can guide and advise, but the investor's right to make their own decisions must be preserved. Nudging techniques, where the system steers behaviour without the user recognising it, are an ethical line that wealth managers should not cross. The investor must always know when they are receiving AI-generated advice and must always retain the ability to override it.

Informed consent follows directly. Investors need to understand the extent to which AI is involved in managing their wealth, the limitations of AI-generated recommendations, and the fact that outputs are based on historical data and algorithmic patterns rather than certainty about future performance. Consent that is buried in terms and conditions is not informed consent.

Accountability: where the Cotswolds problem gets serious

The rock climbing recommendation was generated by a model with no accountability structure around it. Nobody was responsible for verifying it. Nobody was liable for the consequences of a traveller acting on it. In wealth management, that situation is unacceptable. Investment professionals must maintain oversight and control over AI systems. Errors made by AI are errors made by the firm. The remediation, the client communication, and the regulatory reporting all sit with named individuals, not with the algorithm.

The FCA's Consumer Duty, introduced in July 2023, raises the standard of care that wealth managers owe their clients. Enhanced client focus, increased transparency, demonstrable diligence, and proof that recommended products meet the needs of target clients are all now regulatory expectations. AI systems that generate recommendations without being able to explain why, or that optimise for metrics the client has not agreed to, sit in direct tension with these obligations.

The EU's AI Act, which will set a comprehensive regulatory framework for the development and use of AI across the European Union, adds a further layer. Financial advice sits within the category of high-risk AI applications that will face the most stringent governance requirements. Firms that build their ethical architecture now, before the regulation requires it, will find compliance significantly easier than those that retrofit governance onto systems already in production.

A conformity assessment procedure such as capAI, developed by Oxford Said Business School's Professor Holweg, offers a practical framework for assessing whether AI applications meet these emerging regulatory standards. It provides a structured approach to evaluating compliance, identifying gaps, and documenting the governance controls that regulators will increasingly expect to see.

The centaur model

The successful financial adviser of the next decade will operate as what Kenneth Cukier, Dr Frey, and Professor Osborne have described as a centaur: half human, half machine, combining the analytical power of AI with the emotional intelligence, judgement, and accountability that only a human adviser can provide. The AI handles the data processing, the pattern recognition, and the portfolio optimisation. The human handles the relationship, the context, the ethics, and the regulatory accountability.

That model only works if the ethical principles are built into the AI layer from the start. An AI system without transparency, without bias controls, without explainability, and without clear accountability is not a tool that augments a human adviser. It is a liability that the adviser's name is attached to.

The Cotswolds do not have rock faces. Claude did not know that. In wealth management, not knowing what you do not know is not an amusing anecdote. It is a regulatory event. The firms that build the ethical architecture before the event are the ones that will still be in the market afterwards.