Scale model documentation the right way for AI governance

Avoid common mistakes when documenting complex modeling systems

Documentation is considered the bane of corporate existence by some. For others, a great documentation system makes their organized minds super happy. Sometimes missing documentation can be a small problem. But in other cases, missing a digital paper trail or knowledge base can have bad economic effects on the business or customer experience. This is why many business leaders are asking today: how do we scale our model documentation methods and processes for AI and complex modeling systems?

A secondary question, that may or may not be top of mind - but should be: how do we scale documentation in a way that is enabling for teams across the enterprise?

As with most technology, the adoption of AI and machine learning has been much faster than the implementation of governing practices for modeling systems- including model documentation.

Many organizations today are using one of the “bandaid” solutions below for documenting models, controls, and other important information that supports their modeling systems. None of these solutions are sustainable for scaling enterprise AI.

Five places your model documentation probably lives now (but shouldn’t)

Documents (Word, Google Docs, etc.)
Using a digital word document tool usually leads to disjointed files, user access, and modification problems. It’s also not very enabling, as far as setting up (and following) best practices for model governance.
Spreadsheets
“I could use another spreadsheet in my life” said (almost) no business professional ever. Spreadsheets are great for some functions, but they’re not nearly enough for creating a true documentation system to cover the modeling lifecycle.
Email
There are many issues with using email to discuss and document important information related to complex modeling systems. The top one is security, but what about the fun of finding email trails from a person that is no longer at the company?
Chat tools (Slack, Teams, etc.)
Slack and other chat tools have been game-changers for remote team collaboration over the past couple of years. But we probably all agree chat tools are not appropriate for important approvals and documentation. It’s easy to get caught up in the busy day-to-day and just use what’s in front of us, but this is not a sustainable practice.‍
Notebooks (Jupyter, Colab, Zeppelin, etc.)
Model builders often place comments and key pieces of their model building process in notebooks and integrated development environments where the models are built. A fundamental problem with this practice is the notebook and relevant information doesn’t travel with the model to production. The context is lost, and business partners don't have awareness of this documentation, nor the ability to easily navigate or understand, even with access.

It might sound crazy, but going beyond the basics for your modeling system documentation can help you improve - while beating your competition in the meantime.

What are the benefits of good AI documentation?

Below are some of the benefits for using a scalable software solution built for documenting complex modeling systems.

An organized, clear system of record

The problem with using any of the five options above for documentation is that those tools are not enabling for users and stakeholders. Plus, they’re not a real system. Enterprises using disjointed documents, spreadsheets, one-off chats, email, or notebooks, are not only putting the business at risk. Undervaluing the importance of documentation also slows down teams and cuts productivity across the company (data science, risk management, IT).

Implementing a documentation system can help teams govern AI, statistical, and other high-impact models by enabling the following (at scale):

Create, organize, and share the key information business and risk partners want
Proactively share expectations as far as what documentation or governance practices need to be followed
Have a centralized, standardized process for good documentation
Have consistency in how documentation is created and maintained
Maintain tracking of user activity to support accountability

‍Tools for collaboration and automation

Using the five tools above for modeling system documentation and model governance usually means opting for a short-term convenience over a long-term strategy geared towards excellence. When doing so, teams are missing an opportunity to implement a more robust tool that provides:

Alerts and notifications, for example, when a model is due for review
Ability to create tasks and workflows
A system that is scalable with options for automation
Options to easily share documentation and evidence across teams, and with third parties

‍Integration with other business applications

One main benefit of integrating other applications with your modeling system documentation is the ability to provide (and easily access) key evidence related to models. For example, integrating tools for data governance, model monitoring, and model serving helps build important proof that best practices for industry standards and emerging technology frameworks are being followed.

Leveraging these integrations also enables automations and workflows, and decreases having to rely on manual input.

Using one of the five tools above prevents teams from creating the connective tissue between documentation and other key technologies that relate to and complete documentation for models.

‍A competitive edge

“Better visibility” is a phrase that’s probably used too much in the realm of enterprise software - but it’s mentioned so often because it’s a central business need. Leveraging a software system for model governance and documentation gives teams:

Clearer insights into modeling system performance
Faster production times, as a result of streamlined communication and governance documentation
Audit trails to easily share information cross-functionally and with key stakeholders

Building models is already hard enough - documentation and visibility into not just the systems, but the thought process and sets of decisions made when building the model are so important. Model building teams deserve purpose-built solutions to enable capturing these key pieces of the journey. Good AI deserves great governance tools.

Early adopters that get model governance right will enable faster processes and better visibility - and ultimately gain a leg up on their competition.

Getting in front of AI governance

Like most enterprise initiatives, the project of implementing good governance will be what your team makes of it. Taking the time to implement today’s top technology - for documentation and beyond - alongside AI strategy best practices will be game-changing for the companies that get it right. With the challenge of governance comes the opportunity to manage bias and risk, streamline processes in order to optimize for excellence, and set your team up to adopt AI and future complex modeling systems for sustainable business advantage.

‍