The following commentary was posted by our CEO Anthony Habayeb to share observations about the initial release of the model draft as it was shared in August 2023.
Over the past week, we have met with regulators, industry stakeholders, and customers to discuss the National Association of Insurance Commissioners (NAIC) Model Bulletin "Use of Algorithms, Predictive Models, and Artificial Intelligence Systems by Insurers."
This draft represents a significant collaboration between regulators and the industry. There will be broad feedback towards the final version as well as questions about the timeline and resources required to implement these concepts, but this first bulletin draft achieved a streamlined summary of model risk management and governance requirements.
- Many of the model governance and risk management requirements are fundamentally good modeling practices. Establishing controls and processes aligned with the bulletin would not only support regulatory compliance but also drive better AI projects and modeling systems. Examples of fundamental good practices from the bulletin include data lineage, minimization, suitability, benchmarking against alternative model types, drift monitoring, traceability, reproducibility, interpretability, and objective validation testing.
- Like the recent FTC inspection of OpenAI, the NAIC and insurance regulators are working to leverage existing laws and authority to improve model governance and minimize adverse outcomes from AI and advanced modeling systems.
- The draft emphasizes the modeling project and the term "AI System life cycle," rather than just the model itself. There is a healthy balance of expectations versus over-engineered requirements. The draft references NIST as a sample standard but intentionally does not dictate specific bias validations, other standards, or statistical tests that should be performed. Different models and use cases require different approaches.
- The draft intentionally calls out models that most would not define as "Artificial Intelligence," which will cause some discussion. For example, GLMs are in scope. Insurance companies and the industry will ultimately realize better business results and outcomes thanks to more robust model governance practices.
- The suggestion of a centralized or federated committee ("decisions by committee") is not scalable. Effective and scalable governance can and should leverage the proven distribution of responsibilities across three lines of defense and establish objective and appropriately incentivized model risk and governance stakeholders.
- There needs to be some reconciliation and consideration of the relationships and/or overlap between the ideas in this bulletin and requirements for existing model filings and exams. Robust model governance assurances consistent with this bulletin could support streamlined filing and exam processes.