TOPICS DU JOUR IN

MODEL RISK MANAGEMENT (MRM)

Presentation by OCC, FDIC, and Federal Reserve staff

January 2020
 Agencies’ Approach to MRM
2

 Models and MRM continue to be items of interest going into 2020

 MRM and CECL
 MRM for AI/ML models
 Increased use of vendor models
 Role of models in Basel trading, counterparty credit & CVA rules
 In all of these efforts, agencies are working together to ensure
consistency in approaches
 Also conducting significant outreach to help clarify how MRM principles
are relevant for these different areas
 Tying our assessments of MRM to safety-and-soundness standards
 Calibrating supervisory approach while retaining focus on MRM principles
 Also, tailoring our MRM assessments by size & complexity, even among
large firms
 Continue to see benefits in having ongoing dialogue between supervisors
and industry
 Unfortunately, we still hear some “myths” that need debunking
 Agencies’ Approach to MRM (cont)
3

 Some parties have the mistaken impression that supervisors formally

approve use of all models
 Only in selected cases does that happen (e.g., Market Risk Rule)
 The real goal is to have firms set their own internal processes for
developing, reviewing, and approving models
 Of course includes strong effective challenge
 Good processes produce good outcomes – seeing sound processes
gives us confidence that there will be good outcomes in the future
 MRM is considered part of the broader governance and controls
framework
 The next slides discuss relevance of MRM to certain areas of current
focus
 Flexibility in Applying MRM Principles
4

 There is implicit flexibility (or in some cases explicit) in applying MRM

principles that firms might not fully recognize
 In some cases it is not inconsistent with MRM principles for a model to
be put into production before validation
 Perhaps makes sense for the firm to assume more model risk to reduce
risks in other areas
 General expectation is for such cases to be transparent
 And might involve the application of compensating controls
 Good for such cases to be treated as exceptions to policy
 Similar issues relate to model changes
 Some model changes might not warrant re-validation
 Materiality is a key aspect here, too
 Good practice is for firm’s MRM policy to define ex ante the range of
possible changes/conditions warranting a new review vs not
 Flexibility in Applying MRM (cont)
5

 Yet another case relates to assessing model performance

 Different models are assessed in different ways – not expecting the same
performance criteria or methods across all model types
 For some models, there is ample feedback on performance, while for
others less so (e.g., BAU vs stress models)
 Agencies not claiming that all models can be “backtested” but some type
of outcomes analysis should be conducted
 Test what is feasible, benchmark and evaluate conceptually – model may
still be used even if some testing is not possible or not conducted
 But clarify extent of performance testing conducted and remaining
uncertainty – and let users know that information
 General rules-of-thumb are to be transparent about what has
been done and not done, apply exception processes where
needed, and look at bigger risk picture
 Goal is to maintain overall safety and soundness
 Scalability and MRM
6

 The MRM guidance contains several references to incorporating

materiality into MRM
 This means that models of higher materiality/importance should
have MRM of higher intensity and frequency
 And that MRM can be of lower intensity and frequency for models
of lower materiality
 Again, all this is done in the interest of safety and soundness
 Practically, this means that MRM should be scaled within a firm,
based on risk
 Supervisors not interested in overkill or a checklist of tests – that is a
poor allocation of resources and not sensible from business side
 But is contingent on a firm properly tiering its models by
risk/materiality
 Scalability and MRM (cont)
7
 AI/ML and MRM
8

 Supervisors appreciate that AI/ML has potential to enhance

modeling practices
 Important for a firm’s governing bodies to think carefully about the
risks from AI/ML use and how to manage & control those
(responsible innovation)
 Appropriate controls over firms’ applications and technologies also
include how they are used
 This is especially true for any applications/technologies that are new
and not fully tested in a variety of conditions
 The lack of full explainability in AI/ML can translate into higher
level of uncertainty about an approach’s suitability
 Could warrant the use of compensating controls
 In many cases, firms are applying their existing MRM framework to
AI/ML approaches
 Some AI/ML might not be a “model” – so other controls can work
 AI/ML and MRM (cont)
9

 Not all potential consequences from AI/ML are knowable now

 We are in early days of adoption and there could be unintended
consequences
 Firms should be continually vigilant for new issues, including
application of existing products/tech to new areas
 New products/tech are a common area in which problems can arise
(many past examples outside the AI/ML space)
 Increased reliance on data for AI/ML can present challenges
 Should have staff with the right skill sets to review AI/ML
 Firms should not believe that AI/ML approaches are immune to
vulnerabilities or problems simply because they are purported to be
“intelligent” or able to “learn”
 Agencies are currently thinking about the suitability of existing
laws/regs/guidance to AI/ML
 Use of Third Parties in MRM
10

 Firms might choose to employ vendor models or third parties to

conduct some MRM work
 This does not require approval from supervisors
 Such arrangements can work well, provided there is sound risk mgmt
 For vendor models, important for validation to focus on the specific
use at that individual firm
 Outsourcing validation and audit for MRM can work; however, those
are not one-time events, but ongoing processes
 Risk and responsibility remain with the supervised firm
 Be wary of use of third parties just to achieve cost cutting
 Seeing greater use of third parties in the fintech space
 Firms should confirm they understand all the risks (including cyber and
consumer) and have proper controls to address them
 CECL and MRM
11

 As the agencies have noted, sound governance & controls cover all
aspects of CECL, such as
 Assumptions on data and modeling choices
 Qualitative adjustments that could be based on quantitative estimates
 Any remaining qualitative overlays also subject to controls
 This means that firms should apply MRM principles to models used
for CECL
 A meaningful characterization of model risk in CECL…
 Covers all model components in relation to CECL estimates
 Informs decisions on qualitative adjustments to CECL model estimates
 Important to understand impact on loss estimates of modeling
choices and sensitivities to assumptions
 CECL and MRM (cont)
12

 Adapting existing models for CECL involves MRM evaluation in

relation to CECL objectives including
 Data and sample design
 Performance testing – level, frequency, horizon and tolerance level
 Testing should be consistent with how models are applied in production
 Besides initial validation, CECL MRM should continue over time
 Useful to attribute changes in loss estimates to sources
 There is considerable flexibility in applying MRM to CECL models
 The frequency and rigor of MRM activities in CECL should vary
according to model materiality/risk
 What further support can agencies provide on CECL/MRM?
 Some Closing Words of Caution
13

 Significant progress in MRM over the past decade

 Appropriate opportunity now to normalize MRM
 But as supervisors we are trained to watch for downside risk
 Recently, noticed some deprioritization of MRM
 These include reduced MRM budgets, lower seniority of MRM staff,
and generally less attention to MRM in the firm
 These anecdotes might not be a pattern – but good to confirm that
they are not early risk indicators
 Remember that MRM is there to anticipate issues, not just react
 Protect especially against any weakening in effective challenge
 More decisions are being made by models today
 Will those models and associated MRM function well in a downturn?
 And model risk influences/affects other types of risk
 Good for everyone to guard against overconfidence