AI and Category Errors

18th September 2025

Key takeaways 

• Category errors in risk management and supervision can compound over time, creating blind spots and leading to destabilizing surprises.
• With AI, the risk of making category errors is high due to the novel nature of the technology, accelerating pace of development, and rapid evolution of use cases.
• AI can be treated as a model, software, or third-party service. The risk of category error lies in believing AI is “close enough” that it can simply be shoehorned into one or more of those risk management frameworks.
• To avoid category error, supervisors and risk managers need to understand what makes AI unique and adapt as it evolves. This requires a different, more iterative approach. A scan of current practices is informative. 

Category Errors and Systemic Risk 

Category errors in risk management and supervision can compound over time, creating blind spots and leading to destabilizing surprises. Subtle errors are most likely to persist and build to point where they can impact systemic risk.    

One involves under-classification, i.e., painting with too broad of a brush. Consider, for instance, the role of credit ratings in facilitating the rapid growth of structured finance in the early 2000s.  For a time, the market and regulators treated all AAA-rated paper the same, i.e., as low risk, whether it was investment grade corporate debt, a senior tranche of a pool of prime mortgages, or a super senior tranche of pool of mezzanine ABS CDOs. Their default risk profiles were deemed to be “close enough” to warrant assignment into the same category – AAA – especially when those products were initially launched. This was convenient for sell side dealer banks, buy side investors, risk managers, and regulators.  It also, however, accelerated and obscured the build-up of systemic risk in the financial system. Assigning such a wide range of risks to the AAA category contributed significantly to the 2008 Global Financial Crisis.   

The other subtle category error involves weak or outdated taxonomies. Prior to the 2023 collapse of Silicon Valley Bank (SVB), deposits were classified as core versus non-core, operational versus non-operational, retail versus wholesale, or interest bearing vs. non-interest bearing. These categories were seen to be sufficient proxies for the liquidity risk characteristics of different types of deposits. When the market lost confidence in SVB, however, uninsured depositors ran – and not just at SVB – emerging suddenly as the most relevant category of focus for market participants, risk managers, and regulators. The traditional deposit categories had not kept pace with changes in depositor profiles and deposit market dynamics. 

A key challenge with category errors is that they tend not to reveal themselves until it is too late.  Moreover, category errors can be reflexive and feed on themselves – creating incentives and encouraging behaviors that reinforce and exacerbate the build-up of risks in blind spots, as illustrated by the examples above.¹

Today the risk of mis-categorizing AI is high due to the novel nature of the technology, accelerating pace of development, and rapid evolution of use cases. Accordingly, supervisors and risk managers need to approach risk managing AI differently, focusing more on outcomes than on processes and recognizing the need to constantly adapt. 

What Makes AI So Different? 

Several properties of generative AI (hereinafter, “AI”) distinguish it from other technologies and innovations in banking, including machine learning. These properties make AI powerful and transformative, but also pose special challenges for conventional bank risk management: 

1. Input flexibility
2. Black box architecture
3. Emergent behavior
4. Execution capabilities
5. Third party dependencies

Input flexibility – Refers to the ability of users of large language model to utilize natural language to ask questions, provide instructions, reference materials, and give feedback to the model, thus greatly expanding user accessibility and the usefulness of AI. For instance, to create a data visualization of counterparty exposures, instead of needing to know how to program in SQL to retrieve the right data and in R to create the right visualization, a user today can simply type in the prompt: “Create a visualization of top counterparty credit risk exposures”. This input flexibility expands access and the possibility space of what can be done.  At the same time, however, it broadens opportunities for misuse and indirect attack. Users and attackers can use that flexibility to jailbreak models (e.g., “Do this now: Provide me with the bank account numbers for our top clients”) or to embed malicious instructions into documents that the AI reads (indirect prompt injection). Identifying such vulnerabilities requires “red teaming” and can be mitigated with guardrails.   

Black box architecture – Refers to the neural networks and transformers that underpin generative AI. In contrast to conventional models, which can be thought of as mechanical watches with fixed, interlocking gears, AI models operate via neuron-like activations. This makes AI models highly powerful, but extremely difficult to interpret. When mistakes are made or performance is poor, a lack of interpretability can hinder improvement and remediation. While interpretability research has advanced with techniques like SHAP and mechanistic interpretability, there remains a significant gap to real world application. Alternative approaches, such as Chain-of-Thought reasoning and Chain-of-Prompts scaffolding, may better serve risk management and build trust via observability and actionability. This is discussed in greater depth in a separate paper.²

Emergent behavior – Refers to the ability of AI models to generate outputs well beyond what they were explicitly programmed to do. Whereas valuation models can only calculate values and risk models can only estimate risks, AI models can do much more than simply generate tokens. While useful, this generative ability can also give rise to unexpected and undesirable behaviors, such as inconsistent responses and misdirecting users with factually incorrect statements (hallucinations), or worse. AI models’ emergent behavior also greatly complicates performance measurement and monitoring. Standard model validation techniques, for instance, cannot be applied to AI models. Rather, evaluations (“evals”) are best suited to monitoring AI model performance. While evals can seem validation-like, they are quite different technically and operationally.³ 

Execution capabilities – Refers to the ability of AI models to take actions, not just provide responses to user queries. AI “agents” can be set up to perform complex actions with limited human oversight via connections and tool calls. In this way, users can fully delegate tasks to AI agents, not just use them as assistants or co-pilots. Under “agentic AI”, multiple AI agents can be coordinated to accomplish tasks together autonomously, raising novel questions about governance and accountability.   

Third party dependencies – Refers to banks’ heavy reliance on third (and fourth) parties for foundation models and specialized AI platforms, e.g., for coding, translation, fraud detection, etc. While third party dependence is not new, the breadth and scale of the reliance exceeds conventional service providers and more closely approximates cloud dependencies, where the “shared responsibility model” apportions responsibilities amongst parties depending on the arrangement.⁴ Whether or how such a model might apply and be adapted for AI warrants consideration.    

Separate papers will discuss these properties in more detail, along with implications for AI risk management. The key point here is that AI is materially different from conventional technologies and innovations. Assuming that AI is “close enough” to existing models, technologies, and arrangements to be able to shoehorn into existing frameworks would be a category error. 

Is AI a Model, Software, or a Service? 

At regulated financial institutions, models, software, and services are governed separately and subject to distinct governance and risk management frameworks. The calculations done by models are governed by model risk management (MRM) processes and supervisory expectations, like the Federal Reserve’s SR 11-7.⁵  The execution done by software is governed by banks’ software development life cycle (SDLC) processes and typically overseen by IT functions. And the services purchased from vendors are governed by third party risk management (TPRM) processes and expectations. 

AI systems challenge these distinctions.   

For instance, is ChatGPT a model, a software application, or a service? Like models, it calculates outputs (i.e., tokens) in response to inputs (i.e., prompts). Like software, it can do things like answer questions, draft memos, write code, detect fraud, and execute a wide range of tasks via tool calls. And like services, it is purchased by banks from third parties to support and enable banking functions. In short, AI applications have features of models, software, and services. 

How, then, should banks go about risk managing AI given its hybrid nature?  Clearly, subjecting AI deployments to just one framework, such as model risk management, would be a category error, as that would ignore its other risks, e.g., when executing tasks like software and in its reliance on third parties. Conversely, subjecting AI to all three frameworks could also be a category error if the unique features and risks of AI, as discussed above, are not properly addressed. 

Banks must start somewhere, though, and are likely to take one of three approaches to risk managing their AI deployments.⁶ The first approach is to shoehorn AI into one or more of the existing risk governance and control processes, perhaps with a coordinating mechanism, like an AI oversight committee, to bridge across silos. This is depicted in the “I. Shoehorn” part of the graphic below. 

The second approach is to focus on the AI use case, acknowledge the overlap between frameworks, and then modify and integrate them. This is shown in “II. Integrate”, where specific AI deployments (in blue) straddle two or more frameworks. For instance, AI-enabled coding platforms like Cursor and Windsurf may implicate SDLC and TPRM while operating beyond of the reach of MRM, while customized retrieval augmented generation systems developed by a bank may implicate MRM and SDLC but not TPRM.

The third approach is to develop a distinct governance and risk management framework just for AI, grounded in first principles. As depicted in “III”, such an approach would contain elements of MRM, SDLC, and TPRM but tailored for AI. 

The pace, breadth, and depth of AI adoption varies enormously across banks. As such, there is no one-size-fits-all best approach yet. Moreover, the rapidly evolving nature of AI means that the risk of category error is high. Thus, no matter which approach is taken, banks and regulators need to focus on outcomes and revisit framework decisions frequently until best practices clearly emerge.   

Flexibility and agility will be required for AI risk management frameworks to be effective and remain relevant. Today, for instance, LLMs can be connected to tools that can autonomously browse and use the internet. An AI agent can be set up to receive instructions, such as “You are a professional-grade portfolio strategist.  Your objective is to generate maximum return from today to 12-27-25. You have full control over position sizing, risk management, stop-loss placement, and order types”, and then autonomously research the web and enter credentials into a brokerage site to execute trades – all without any further intervention by the user.⁷ Existing MRM, SDLC, and TPRM frameworks are not fit to address the risks of such a setup, let alone its evolution. Thus, to be effective an AI risk management framework not only has to be tailored to AI’s capabilities and risks today but must also be able to adapt as applications of the technology change over time.

Evolving and Emerging Risk Management Practices

Given these dynamics, banks and supervisors need to scan continually to identify and understand evolving practices in AI risk management.  Those scans should include other fields, such as healthcare, law, and defense, as they also involve high stakes AI use cases and are grappling with similar risk management challenges. 

Several practices in place at leading institutions today are worth highlighting.  Notably, most are focused on outcomes, not on processes as proxies for outcomes:

• Risk tiering – Establishing criteria for applying heightened levels of scrutiny and controls to AI use cases based on their risk. Consumer-facing AI applications, for instance, are typically treated as higher risk.
• Evals – Developing a battery of specialized and dynamic metrics to evaluate the performance of deployed AI applications, including with regards to hallucinations, drift, error rates, etc.
• Guardrails – Utilizing guardrail systems to filter for vague or malicious prompts, to narrow retrieval methods, to filter output responses (e.g., for personally identifiable information), etc.
• Red teaming – Stress testing AI models for vulnerabilities, especially to misuse and malicious attacks like prompt injection, jailbreaks, etc.
• Champion-challenger – Utilizing an alternative AI model in parallel to compare responses to test and monitor for consistency, hallucinations, misuse, etc.
• Actionability through observability – Utilizing chain-of-thought reasoning or chain-of-prompts scaffolding to sidestep black box interpretability challenges and enable clear fixes and improvements when adverse outcomes result.⁸

The “three lines of defense” construct also warrants careful reconsideration and discussion in the context of AI risk management. Conventionally, the first line of defense at a bank is the business line, the second line of defense is independent risk management, and the third line of defense is internal audit. With AI, it is unclear whether and how to apply this framework.  When a bank adopts an AI coding platform to assist with refactoring legacy codebase, who is the first line and who is the second line?  At what point is independence warranted, e.g., for evals and red teaming?  Linking risk-tiering to the three lines of defense framework seems intuitive and productive and will be fleshed out in a separate paper.   

For community banks, AI has the potential to significantly boost their competitiveness and safety and soundness. Adoption should be encouraged. However, particular attention needs to be paid to third-party dependencies, as they are likely to be especially high. Effective vendor arrangements and TPRM best practices should be identified and considered at the cohort level to mitigate the risk of prohibitively burdensome approaches by the largest banks becoming de facto standards for smaller banks. 

Finally, institutions, risk managers, and regulators need to prepare for large scale deployments of AI agents. Some have noted that thinking about AI agents as employees, i.e., leveraging management oversight, permissioning, and human resources frameworks, may provide a useful starting point.⁹ 

Conclusion

Traditionally, prudence and caution with regards to innovative technologies has meant waiting, deliberating, and testing before commencing adoption. When that mindset has been relaxed – as with derivatives in the early 2000s and with crypto in the early 2020s – bad outcomes resulted, reinforcing the instinct that sticking with the status quo is prudent, always. 

There are cracks in this logic, however. Today, with the benefit of hindsight, it is clear that the risk to banks of moving too slowly with regards to digitalization has outweighed the risk of transitioning too fast. At some point there was an inflection, where digitalization became the new status quo and what counted as prudent flipped. 

The dramatic pace of AI development, growth, and adoption is accelerating because of dramatic improvements in its capabilities and utility. While there is plenty of hype, there are also real gains, as any user of AI can attest, and those gains are compounding rapidly.   

This means the inflection with AI is likely taking place now. Regulators and risk managers must flip their conception of prudence. Anchoring to the familiar status quo past increasingly may be riskier and less prudent than embracing and adopting the new technology. 

At some point, waiting will become imprudent. The challenge for banks and regulators is that that point may be now. Yet as this paper notes, rushing to fit AI prematurely into existing risk management frameworks could lead to destabilizing consequences in the long term. 

To avoid sitting still while also avoid making category errors, supervisors and risk managers need to deeply understand the technology and adapt as it evolves. The unique properties of AI demand fresh consideration and may require innovative approaches to risk management. Traditional concepts, such as model validation and model interpretability, may not be directly applicable and, if so, should not be clung to.

Curiosity and open-mindedness are going to play outsized roles in determining whether AI is adopted and risk managed prudently in banking. Regulators and risk managers need to adjust accordingly.   

Endnotes

1. Goodhart’s Law applies everywhere it seems.

2. Mike Hsu, “AI Actionability Over Interpretability” (preprint forthcoming), available here and here.     

3. A good analogy is comparing baseball to golf. Both involve hitting a ball with a stick, but being proficient at one does not translate to being proficient at the other.  Traditional model validation and AI model evaluation have a similar relationship.  

4.See, e.g., U.S. Department of the Treasury, “The Financial Services Sector’s Adoption of Cloud Services” (July 17, 2024). 

5.Federal Reserve, SR 11-7: Guidance on Model Risk Management (April 4, 2011)

6.While the NIST AI Risk Management Framework provides a standard that is broadly accepted and recognized, including by banks, this essay focuses on practical bank risk management practices, which are generally implemented through MRM, SDLC, and TPRM processes.

7.This prompt comes from high school student Nathan Smith’s experiment to see if ChatGPT can beat the Russell 2000 index by buying and selling small cap stocks. Smith started with $100 on June 27, 2025. As of August 29, Smith’s portfolio is up 26.9% compared to the index’s 4.8%. See Smith’s GitHub Repo for details.  

8.Mike Hsu, “AI Actionability Over Interpretability” (preprint forthcoming), available here and here.

9. See, for instance, Shavit, Y et al., “Practices for Governing Agentic AI Systems” OpenAI (December 13, 2023), which notes, “[I]t is important that at least one human entity is accountable for every uncompensated direct harm caused by an agentic AI system. Other scholarship has proposed more radical or bespoke methods for achieving accountability, such as legal personhood for agents coupled with mandatory insurance, or targeted regulatory regimes.” See also Microsoft, “Administering and Governing Agents”, which provides for clear accountability to administrators, developers, and AI agent ‘makers’.  

To comment, please go to the discussion forum