Many financial institutions are rapidly developing and adopting AI models. They’re using the models to achieve new competitive advantages such as being able to make faster and more successful underwriting decisions. However, AI models introduce new risks.
In a previous post, I describe why AI models increase risk exposure compared to the more traditional, rule-based models that have been in use for decades. In short, if AI models have been trained on biased data, lack explainability, or perform inadequately, they can expose organizations to as much as seven-figure losses or fines. Financial institutions need a new approach for managing AI model risk, one that aligns with guidance such as the Federal Reserve Board’s SR-11-7 and helps ensure that AI models are fair, transparent and robust.
Another challenge for financial institutions is that developing and validating AI models from concept to deployment can take as long as a year. This must be reduced to speed AI ROI.
IBM Watson OpenScale, available in IBM Cloud Pak for Data and IBM Cloud, can help with these challenges. Let’s take a closer look.
New beta program for AI model risk management
Watson OpenScale automates active testing of AI models to support their validation and ongoing monitoring—for models built and running anywhere. This enables companies to enhance compliance with regulations and mitigate business risk.
To date, Watson OpenScale has been used mainly to monitor and test AI models already in production. Now, our open beta program enables users to take advantage of the following capabilities:
Validate AI models in pre-production with advanced, customizable tests
Models must be validated throughout their lifecycle and especially before they are deployed into production. Organizations can use Watson OpenScale to configure fully customizable model validation tests and set custom thresholds. If a chosen threshold is exceeded, Watson OpenScale documents results and sends a notification. Model validation tests include:
- Fairness/bias detection
- Quality checks such as accuracy, precision, and recall
- Drift detection
- Tests that explain individual outcomes
View a dashboard of outcomes and actionable reports
Watson OpenScale displays test results graphically in a user-friendly dashboard. The new dashboard includes information about the model, the datasets used for validation, and a summary of test results. Users can click on results to drill down to additional details.
Watson OpenScale can also export a factsheet of information about a model in a PDF format. It contains all relevant information, including training and validation data sets and test results. When models are in production, the factsheet includes production metrics.
The factsheet can be easily shared with other users or stored in an external system. It can save time when creating formal validation reports.
Compare model performance side-by-side
Some model validators build their own challenger models to test the quality of a new model. Others test the model against their organization’s benchmark models to compare performance. To serve both kinds of use cases, Watson OpenScale enables models and their test results to be compared side-by-side on a single screen, speeding the validation process.
Automatically transfer tests for a validated model to production and continue monitoring
When a model has been approved, Watson OpenScale will transfer its validation test configuration into production, then continue to monitor the model and run the tests periodically and automatically. This simplifies the work of validation teams—especially when organizations are running multiple models.
Synchronize results with governance, risk and compliance solutions
Good governance is critical in managing model risk. Many financial institutions use dedicated governance, risk and compliance (GRC) solutions to document model inventories and provide a workflow that guides users in managing a model through its life cycle.
GRC solutions typically focus mainly on documentation. They tend to have fewer capabilities that support the actual validation of AI models.
Watson OpenScale provides many needed AI model validation and management capabilities, and it is designed so that it will be able to integrate with GRC solutions and synchronize test results and reporting. No manual copying or data entry is required. This helps ensure that GRC solutions contain the latest information about a model.
The first GRC solution that can be integrated with Watson OpenScale out-of-the-box is IBM OpenPages Model Risk Governance.
Try Watson OpenScale and its MRM capabilities at no charge
See these short videos on how Watson OpenScale can help detect and correct bias, drift, and provide explainability for AI outcomes. Put your hands on the solution and try its model risk management capabilities at no charge here.