P2.T7.

Operational & Integrated Risk Management

Bionic Turtle FRM Practice Questions

“Supervisory Guidance on Model Risk Management,”

Federal Deposit Insurance Corporation, June 7, 2017.

By David Harper, CFA FRM CIPM

www.bionicturtle.com
“Supervisory Guidance on Model Risk Management”

P2.T7.20.3. MODEL RISK MANAGEMENT (ACCORDING TO FDIC) ................................................. 3

2
“Supervisory Guidance on Model Risk Management”
P2.T7.20.3. Model risk management (according to FDIC)
Learning Objectives: Describe model risk and explain how model risk can arise in the
implementation of a model. Describe elements of an effective process to manage model
risk. Explain best practices for the development and implementation of a model. Describe
elements of a strong model validation process and challenges to an effective validation
process.

20.3.1. Which of the following is the BEST guiding principle for managing model risk?

a) Effective challenge
b) Model development
c) Regulatory compliance
d) Latest and best technology

20.3.2. According to the FDIC, model risk management starts with robust model development,
implementation, and use (aka, the first element. The second element is a sound validation
process. The third element is governance). Which of the following is a TRUE statement about
this first element (development, implementation and use) of model risk management?

a) Model development should be a straightforward and routine technical process

b) To exercise conservatism, pick an extreme point on a given modeled distribution
c) Banks should never model output with judgmental or qualitative adjustments
d) Although the nature of testing varies by the context and type of model, testing is an
integral part of model development

20.3.3. According to the FDIC, an effective model (risk) validation framework should include
three core elements. What are the three core elements?

a) Implementation; ongoing error correction; and reporting

b) Evaluation of sound governance; back-testing; and vendor validation
c) Evaluation of conceptual soundness; ongoing monitoring; and outcomes analysis
d) Evaluation of simulated errors; expert screening; and policy/procedure validation

3
Answers:

20.3.1. A. Effective challenge

Explains the FDIC (emphasis ours), "A guiding principle for managing model risk is effective
challenge of models, that is, critical analysis by objective, informed parties who can identify
model limitations and assumptions and produce appropriate changes. Effective challenge
depends on a combination of incentives, competence, and influence. Incentives to provide
effective challenge to models are stronger when there is greater separation of that challenge
from the model development process and when challenge is supported by well-designed
compensation practices and corporate culture. Competence is a key to effectiveness since
technical knowledge and modeling skills are necessary to conduct appropriate analysis and
critique. Finally, challenge may fail to be effective without the influence to ensure that actions
are taken to address model issues. Such influence comes from a combination of explicit
authority, stature within the organization, and commitment and support from higher levels of
management.1"

In regard to (B), (C), and (D), each is an inferior GUIDING PRINCIPLE:

 Model development is merely one of the ELEMENTS of model risk management, not
itself a guiding principle
 Regulatory compliance will not ensure good or best practices
 The "latest and best technology" might support good model risk management, but recent
technologies can also be the "bleeding edge," such that technological sophistication
itself should not be the goal!

20.3.2. D. TRUE: Although the nature of testing varies by the context and type of model,
testing is an integral part of model development

In regard to (A), (B) and (C), each is FALSE.

 Model development is neither straightforward nor a routine technical process!
 To pick an extreme point on a given modeled distribution is not necessarily conservative
if the distribution is mis-specified or mis-estimated in the first place.
 Judgmental or qualitative adjustments are a KEY part of model
development/implementation/use: "In addition, a considerable amount of subjective
judgment is exercised at various stages of model development, implementation, use,
and validation. It is important for decision makers to recognize that this subjectivity
elevates the importance of sound and comprehensive model risk management
processes." ... "Banks should ensure that the development of the more judgmental and
qualitative aspects of their models is also sound. In some cases, banks may take
statistical output from a model and modify it with judgmental or qualitative adjustments
as part of model development.1"

1
“Supervisory Guidance on Model Risk Management,” Federal Deposit Insurance Corporation, June 7, 2017

4
20.3.3. C. TRUE: Evaluation of conceptual soundness; ongoing monitoring; and
outcomes analysis

According to the FDIC (pages 9 - 11 but highly extracted. Key Elements of Comprehensive
Validation):

"An effective validation framework should include three core elements:

 Evaluation of conceptual soundness, including developmental evidence;
 Ongoing monitoring, including process verification and benchmarking
 Outcomes analysis, including back-testing
1. Evaluation of Conceptual Soundness: ... This element involves assessing the quality of the
model design and construction. It entails review of documentation and empirical evidence
supporting the methods used and variables selected for the model ... Where appropriate to the
particular model, banks should employ sensitivity analysis in model development and validation
to check the impact of small changes in inputs and parameter values on model outputs to make
sure they fall within an expected range. Unexpectedly large changes in outputs in response to
small changes in inputs can indicate an unstable model. Varying several inputs simultaneously
as part of sensitivity analysis can provide evidence of unexpected interactions, particularly if the
interactions are complex and not intuitively clear.

2. Ongoing Monitoring: ... Ongoing monitoring is essential to evaluate whether changes in

products, exposures, activities, clients, or market conditions necessitate adjustment,
redevelopment, or replacement of the model and to verify that any extension of the model
beyond its original scope is valid.

3. Outcomes Analysis: ... Back-testing is one form of outcomes analysis; specifically, it involves
the comparison of actual outcomes with model forecasts during a sample time period not used
in model development and at an observation frequency that matches the forecast horizon or
performance window of the model.2"

Discuss in forum here: https://www.bionicturtle.com/forum/threads/p2-t7-20-3-model-risk-

management-according-to-fdic.23232/

2
“Supervisory Guidance on Model Risk Management,” Federal Deposit Insurance Corporation, June 7, 2017