With the increase in public awareness of Generative AI, caused by the release of tools such as OpenAI’s ChatGPT, there has been a wave of fear and paranoia around AI, specifically how it should be regulated and governed. The purpose of this article is to ease the nerves of the public and business executives, putting the technology into context as well as providing actionable steps towards governance, by learning from history.
Artificial Intelligence is a broad and often disputed discipline focused on automating computer processes to mimic human cognition. AI essentially consists of a type of model (or models) that abstracts reality and is built and optimized for a specific purpose (or in the world of 'general' intelligence, abstract reasoning).
A major area of confusion and contention is the relationship between AI and modeling. In our opinion, general AI or Terminator-type Artificial Intelligence does not currently exist outside of theory, so the distinction is not important. Some of the more advanced ‘AI’ that we are seeing, including ChatGPT, are Large Language Deep Learning models (LLMs) trained on massive amounts of data to make predictions about the next word. These models are not 'general intelligence' but models optimized to sound human. Note: Generative AI, unlike other forms of machine learning and statistical modeling, is not trained to be correct, just to sound correct.
To learn more on this topic, listen to the AI Fundamentalists Episode 2
Now that we have established what Generative AI is and hopefully removed some of the myths, let's discuss how we can tactically operationalize responsible use of Generative AI from risk management perspective1. There is a current trend by journalists, academics, and politicians to position AI as a risk and danger type with no comparative reference point or foundation for risk management. This article explores two existing paradigms for Generative AI governance. We should be focusing the discourse on how to apply and enhance these existing proven best practices instead of starting from scratch with normative philosophy. Statistical sciences and risk management have a marketing problem, but both have a wealth of knowledge on relevant disciplines and best practices that apply to the 'age of generative AI'.
For models built to automate critical business processes and decisions, such as in fields like finance, aerospace, and statistical modeling, there are existing best practices for managing and governing these applications2.
Systems engineering is an interdisciplinary field that originated from Bell Labs in the 1940s as a means of identifying, designing, integrating, and managing complex engineering processes. It is heavily used throughout the engineering disciplines and was successfully leveraged during the Apollo program[2]. Systems Engineering unified the approach for holistically evaluating a complex system and incorporated risk management as an integral part of engineering. Systems engineering is not a secret but is rarely implemented outside of major mission-critical projects, and even then, its implementation is spotty.
After the 2008 Financial Crisis, the OCC and the Federal Reserve issued OCC 2011/12 - SR 11-7, a model risk management framework was inspired by systems engineering to mitigate the risk of models within the US banking sector3. OCC 2011/12 was subsequently updated in 2021 to encompass “AI” models[4]. OCC 2011/12 is now considered by many to be the gold standard in model risk management, providing a thorough approach for identifying, managing, and mitigating model risk at an individual and enterprise-wide level.
The difference and cause of concern around AI is not generally because of the type of algorithms that are being used, after they are broken down to first principles, but rather how they are deployed increasingly in processes where end-users' lives are materially impacted. To mitigate risk, levels of data preparation to ensure bias isn't present, along with enhanced validation, robust stress testing, and thorough ongoing monitoring, are needed. The paradigm for how the modeling system is being used is much more important than the specific algorithm.
The other take on Generative AI is that of a productivity enhancer. For example, before Generative AI was the hot new thing, it was already in your mobile phone via autocompletion, among many other products. Tools such as next-word suggestion, GitHub Co-Pilot, and other tools where generative AI is used to enhance productivity are used with humans in the loop who are ultimately responsible for determining if the suggestion is helpful. These decisions are not mission-critical, made autonomously without a human in the loop, or material to the business. We argue that outside of specific optimized use cases for large language models, Generative AI will manifest itself for the majority of businesses as a productivity enhancer.
For most productivity tools used with humans in the loop, there are existing paradigms for governance: IT Vendor Risk and the SOC 2, type 2 report. Companies currently decide if the use of Dropbox is warranted by evaluating the data being shared, the safeguards in place at the SaaS provider level, and the corporate data criticality level. If a company is concerned about their employees putting critical data into ChatGPT or equivalent and this data being used for training purposes, they should block it and buy a tool that provides the protection they need.
As with the the example of the financial crisis and OCC model risk management framework, there is also historical precedent for how to manage the proliferation of productivity tools, Shadow IT. Shadow IT, when individuals or departments deploy IT systems without the knowledge and support of the IT department, has been an issue for over a decade. With the advent of Sarbanes Oxley and other major regulatory initiatives, shadow IT became a major issue for many corporations. The more restrictive official systems and processes were the greater the drive for separate systems. With the rise of SaaS companies, such as Dropbox, shadow IT proliferated. The cybersecurity industry has developed many technological solutions to the problem, along with IT management's focus on the cultural changes of providing the technology and products that its customers need to be productive, a holistic playbook exists4. Now best practices are using tools for data loss prevention, white-listing endpoints, etc., and relying on IT vendor risk management to determine which pieces of software to purchase to meet end-user needs as well as meet the enterprise's security posture.
Generative AI is a development in productivity, but for the vast majority of organizations, leveraging third-party productivity tools provides a higher likelihood of success than building your models. For some organizations, generative AI is worth the hard yards if the proper foundation of data and modeling expertise is in place. In either case, evaluating this new technology behooves us all to understand how we got here and what we can glean from history and related fields. As the philosopher George Santayana is attributed to writing: "Those who cannot remember the past are condemned to repeat it [5].
[1]: K. J. Schlager, "Systems engineering-key to modern development," in IRE Transactions on Engineering Management, vol. EM-3, no. 3, pp. 64-66, July 1956, doi: 10.1109/IRET-EM.1956.5007383.
[2]: Shea, Garrett. “NASA Systems Engineering Handbook Revision 2.” NASA, 20 June 2017, http://www.nasa.gov/connect/ebooks/nasa-systems-engineering-handbook.
[3]: “The Fed - Supervisory Letter SR 11-7 on Guidance on Model Risk Management -- April 4, 2011.” Accessed August 9, 2023. https://www.federalreserve.gov/supervisionreg/srletters/sr1107.htm.
[4]: OCC.gov. “Model Risk Management: New Comptroller’s Handbook Booklet,” August 18, 2021. https://occ.gov/news-issuances/bulletins/2021/bulletin-2021-39.html.
[5]: “The Project Gutenberg eBook of The Life of Reason, by George Santayana.” Accessed August 9, 2023. https://www.gutenberg.org/files/15000/15000-h/15000-h.htm.
