Generative AI governance: A tale of two models

AI Governance & Assurance
Principles & Frameworks

With the increase in public awareness of Generative AI, caused by the release of tools such as OpenAI’s ChatGPT, there has been a wave of fear and paranoia around AI, specifically how it should be regulated and governed. The purpose of this article is to ease the nerves of the public and business executives, putting the technology into context as well as providing actionable steps towards governance, by learning from history.

What is Generative AI?

First, what is AI?

Artificial Intelligence is a broad and often disputed discipline focused on automating computer processes to mimic human cognition. AI essentially consists of a type of model (or models) that abstracts reality and is built and optimized for a specific purpose (or in the world of 'general' intelligence, abstract reasoning).

A major area of confusion and contention is the relationship between AI and modeling. In our opinion, general AI or Terminator-type Artificial Intelligence does not currently exist outside of theory, so the distinction is not important. Some of the more advanced ‘AI’ that we are seeing, including ChatGPT, are Large Language Deep Learning models (LLMs) trained on massive amounts of data to make predictions about the next word. These models are not 'general intelligence' but models optimized to sound human. Note: Generative AI, unlike other forms of machine learning and statistical modeling, is not trained to be correct, just to sound correct.

To learn more on this topic, listen to the AI Fundamentalists Episode 2

Risks and best practices

Now that we have established what Generative AI is and hopefully removed some of the myths, let's discuss how we can tactically operationalize responsible use of Generative AI from risk management perspective1. There is a current trend by journalists, academics, and politicians to position AI as a risk and danger type with no comparative reference point or foundation for risk management. This article explores two existing paradigms for Generative AI governance. We should be focusing the discourse on how to apply and enhance these existing proven best practices instead of starting from scratch with normative philosophy. Statistical sciences and risk management have a marketing problem, but both have a wealth of knowledge on relevant disciplines and best practices that apply to the 'age of generative AI'.

A tale of two models

Governing LLMs - a decision tree

Model Risk Management

For models built to automate critical business processes and decisions, such as in fields like finance, aerospace, and statistical modeling, there are existing best practices for managing and governing these applications2.

Systems engineering is an interdisciplinary field that originated from Bell Labs in the 1940s as a means of identifying, designing, integrating, and managing complex engineering processes. It is heavily used throughout the engineering disciplines and was successfully leveraged during the Apollo program[2]. Systems Engineering unified the approach for holistically evaluating a complex system and incorporated risk management as an integral part of engineering. Systems engineering is not a secret but is rarely implemented outside of major mission-critical projects, and even then, its implementation is spotty.

After the 2008 Financial Crisis, the OCC and the Federal Reserve issued OCC 2011/12 - SR 11-7, a model risk management framework was inspired by systems engineering to mitigate the risk of models within the US banking sector3. OCC 2011/12 was subsequently updated in 2021 to encompass “AI” models[4]. OCC 2011/12 is now considered by many to be the gold standard in model risk management, providing a thorough approach for identifying, managing, and mitigating model risk at an individual and enterprise-wide level.

The difference and cause of concern around AI is not generally because of the type of algorithms that are being used, after they are broken down to first principles, but rather how they are deployed increasingly in processes where end-users' lives are materially impacted. To mitigate risk, levels of data preparation to ensure bias isn't present, along with enhanced validation, robust stress testing, and thorough ongoing monitoring, are needed. The paradigm for how the modeling system is being used is much more important than the specific algorithm.

IT Vendor Risk Management

The other take on Generative AI is that of a productivity enhancer. For example, before Generative AI was the hot new thing, it was already in your mobile phone via autocompletion, among many other products. Tools such as next-word suggestion, GitHub Co-Pilot, and other tools where generative AI is used to enhance productivity are used with humans in the loop who are ultimately responsible for determining if the suggestion is helpful. These decisions are not mission-critical, made autonomously without a human in the loop, or material to the business. We argue that outside of specific optimized use cases for large language models, Generative AI will manifest itself for the majority of businesses as a productivity enhancer.

For most productivity tools used with humans in the loop, there are existing paradigms for governance: IT Vendor Risk and the SOC 2, type 2 report. Companies currently decide if the use of Dropbox is warranted by evaluating the data being shared, the safeguards in place at the SaaS provider level, and the corporate data criticality level. If a company is concerned about their employees putting critical data into ChatGPT or equivalent and this data being used for training purposes, they should block it and buy a tool that provides the protection they need.

As with the the example of the financial crisis and OCC model risk management framework, there is also historical precedent for how to manage the proliferation of productivity tools, Shadow IT. Shadow IT, when individuals or departments deploy IT systems without the knowledge and support of the IT department, has been an issue for over a decade. With the advent of Sarbanes Oxley and other major regulatory initiatives, shadow IT became a major issue for many corporations. The more restrictive official systems and processes were the greater the drive for separate systems. With the rise of SaaS companies, such as Dropbox, shadow IT proliferated. The cybersecurity industry has developed many technological solutions to the problem, along with IT management's focus on the cultural changes of providing the technology and products that its customers need to be productive, a holistic playbook exists4. Now best practices are using tools for data loss prevention, white-listing endpoints, etc., and relying on IT vendor risk management to determine which pieces of software to purchase to meet end-user needs as well as meet the enterprise's security posture.


Generative AI is a development in productivity, but for the vast majority of organizations, leveraging third-party productivity tools provides a higher likelihood of success than building your models. For some organizations, generative AI is worth the hard yards if the proper foundation of data and modeling expertise is in place. In either case, evaluating this new technology behooves us all to understand how we got here and what we can glean from history and related fields. As the philosopher George Santayana is attributed to writing: "Those who cannot remember the past are condemned to repeat it [5].


[1]: K. J. Schlager, "Systems engineering-key to modern development," in IRE Transactions on Engineering Management, vol. EM-3, no. 3, pp. 64-66, July 1956, doi: 10.1109/IRET-EM.1956.5007383.

[2]: Shea, Garrett. “NASA Systems Engineering Handbook Revision 2.” NASA, 20 June 2017,

[3]: “The Fed - Supervisory Letter SR 11-7 on Guidance on Model Risk Management -- April 4, 2011.” Accessed August 9, 2023.

[4]: “Model Risk Management: New Comptroller’s Handbook Booklet,” August 18, 2021.

[5]: “The Project Gutenberg eBook of The Life of Reason, by George Santayana.” Accessed August 9, 2023.


  1. For this post we are focusing specifically on Generative AI but these two paradigms apply for all forms of modeling systems.
  2. As historical context, WW2 caused a flurry of innovation in navigation systems, along with the Electronic Numerical Integrator and Computer, one of the first general computers, which gave rise to automation of models. Many academic and scientific disciplines took to modeling in tandem with the computer revolution.
  3. Due to faulty modeling assumptions and lack of rigor leading up to the financial crisis. Causes out of scope for this article
  4. The audit industry also contributed to the fix by the AICPA launching the SOC 2 audit process for internal control testing of service organizations, making it easier for IT departments to rely on using third-party SaaS companies for productivity tools, such as Slack.