Financial Management Phase III IGTC

Bangalore

Basel I II III

Report submitted
by
Sachin Todur (B0924)

Introduction
Financial Management Phase III IGTC
Bangalore

Basel I is the round of deliberations by central bankers from around the world, and
in 1988, the Basel Committee (BCBS) in Basel, Switzerland, published a set of
minimal capital requirements for banks. This is also known as the 1988 Basel
Accord, and was enforced by law in the Group of Ten (G-10) countries in 1992 .
Basel I is now widely viewed as outmoded. Indeed, the world has changed as
financial conglomerates, financial innovation and risk management have developed.
Therefore, a more comprehensive set of guidelines, known as Basel II are in the
process of implementation by several countries and new updates in response to the
financial crisis commonly described as Basel III.

Background

The Committee was formed in response to the messy liquidation of a Cologne-based

bank (Herstatt) in 1974. On 26 June 1974, a number of banks had released
Deutsche Mark (German Mark) to the Bank Herstatt in exchange for dollar payments
deliverable in New York. On account of differences in the time zones, there was a
lag in the dollar payment to the counter-party banks, and during this gap, and
before the dollar payments could be effected in New York, the Bank Herstatt was
liquidated by German regulators.

This incident prompted the G-10 nations to form towards the end of 1974, the Basel
Committee on Banking Supervision, under the auspices of the Bank of International
Settlements (BIS) located in Basel, Switzerland.

Basel I, that is, the 1988 Basel Accord, primarily focused on credit risk. Assets of
banks were classified and grouped in five categories according to credit risk,
carrying risk weights of zero (for example home country sovereign debt), ten,
twenty, fifty, and up to one hundred percent (this category has, as an example,
most corporate debt). Banks with international presence are required to hold capital
equal to 8 % of the risk-weighted assets.

Basel I, that is, the 1988 Basel Accord, primarily focused on credit risk. Assets of
banks were classified and grouped in five categories according to credit risk,
carrying risk weights of zero (for example home country sovereign debt), ten,
twenty, fifty, and up to one hundred percent (this category has, as an example,
most corporate debt). Banks with international presence are required to hold capital
equal to 8 % of the risk-weighted assets. However, large banks like JPMorgan Chase
found Basel I's 8% requirement to be unreasonable, and implemented credit default
swaps so that in reality they would have to hold capital equivalent to only 1.6% of
assets.

Since 1988, this framework has been progressively introduced in member countries
of G-10, currently comprising 13 countries, namely, Belgium, Canada, France,
Germany, Italy, Japan, Luxembourg, Netherlands, Spain, Sweden, Switzerland,
Financial Management Phase III IGTC
Bangalore
United Kingdom and the United States of America. Most other countries, currently
numbering over 100, have also adopted, at least in name, the principles prescribed
under Basel I. The efficiency with which they are enforced varies, even within
nations of the Group of Ten.

Basel II is the second of the Basel Accords, which are recommendations on banking
laws and regulations issued by the Basel Committee on Banking Supervision. The
purpose of Basel II, which was initially published in June 2004, is to create an
international standard that banking regulators can use when creating regulations
about how much capital banks need to put aside to guard against the types of
financial and operational risks banks face. Advocates of Basel II believe that such an
international standard can help protect the international financial system from the
types of problems that might arise should a major bank or a series of banks
collapse. In practice, Basel II attempts to accomplish this by setting up rigorous risk
and capital management requirements designed to ensure that a bank holds capital
reserves appropriate to the risk the bank exposes itself to through its lending and
investment practices. Generally speaking, these rules mean that the greater risk to
which the bank is exposed, the greater the amount of capital the bank needs to
hold to safeguard its solvency and overall economic stability.

Objective

The final version aims at:

Ensuring that capital allocation is more risk sensitive;

Separating operational risk from credit risk, and quantifying both;

Attempting to align economic and regulatory capital more closely to reduce

the scope for regulatory arbitrage.

While the final accord has largely addressed the regulatory arbitrage issue,
there are still areas where regulatory capital requirements will diverge from
the economic.

Basel II has largely left unchanged the question of how to actually define
bank capital, which diverges from accounting equity in important respects.
The Basel I definition, as modified up to the present, remains in place.

Basel II uses a "three pillars" concept – (1) minimum capital requirements

(addressing risk), (2) supervisory review and (3) market discipline – to promote
greater stability in the financial system.

The Basel I accord dealt with only parts of each of these pillars. For example: with
respect to the first Basel II pillar, only one risk, credit risk, was dealt with in a simple
Financial Management Phase III IGTC
Bangalore
manner while market risk was an afterthought; operational risk was not dealt with
at all.

The first pillar

The first pillar deals with maintenance of regulatory capital calculated for
three major components of risk that a bank faces: credit risk, operational
risk, and market risk. Other risks are not considered fully quantifiable at this
stage.

The credit risk component can be calculated in three different ways of

varying degree of sophistication, namely standardized approach, Foundation
IRB and Advanced IRB. IRB stands for "Internal Rating-Based Approach".

For operational risk, there are three different approaches - basic indicator
approach or BIA, standardized approach or TSA, and the internal
measurement approach (an advanced form of which is the advanced
measurement approach or AMA).

For market risk the preferred approach is VaR (value at risk).

As the Basel 2 recommendations are phased in by the banking industry it will

move from standardised requirements to more refined and specific
requirements that have been developed for each risk category by each
individual bank. The upside for banks that do develop their own bespoke risk
measurement systems is that they will be rewarded with potentially lower
risk capital requirements. In future there will be closer links between the
concepts of economic profit and regulatory capital.

Credit Risk can be calculated by using one of three approaches:

1. Standardised Approach

2. Foundation IRB (Internal Ratings Based) Approach

3. Advanced IRB Approach

The standardised approach sets out specific risk weights for certain types of
credit risk. The standard risk weight categories are used under Basel 1 and
are 0% for short term government bonds, 20% for exposures to OECD Banks,
50% for residential mortgages and 100% weighting on unsecured
commercial loans. A new 150% rating comes in for borrowers with poor
Financial Management Phase III IGTC
Bangalore

credit ratings. The minimum capital requirement (the percentage of risk

weighted assets to be held as capital) remains at 8%.

For those Banks that decide to adopt the standardised ratings approach they
will be forced to rely on the ratings generated by external agencies. Certain
Banks are developing the IRB approach as a result.

The second pillar

The second pillar deals with the regulatory response to the first pillar, giving
regulators much improved 'tools' over those available to them under Basel I.
It also provides a framework for dealing with all the other risks a bank may
face, such as systemic risk, pension risk, concentration risk, strategic risk,
reputational risk, liquidity risk and legal risk, which the accord combines
under the title of residual risk. It gives banks a power to review their risk
management system.

The Third pillar

Buyers and sellers in a market are said to be constrained by market discipline in

setting prices because they have strong incentives to generate revenues and avoid
bankruptcy. This means, in order to meet economic necessity, buyers must avoid
prices that will drive them into bankruptcy and sellers must find prices that will
generate revenue (or suffer the same fate).

Market discipline is a topic of particular concern because of banking deposit

insurance laws. Most governments offer deposit insurance for people making
deposits with banks. Normally, bank managers have strong incentives to
avoid risky loans and other investments. However, mandated deposit
insurance eliminates much of the risk to bankers. This constitutes a loss of
market discipline. In order to counteract this loss of market discipline,
governments introduce regulations aimed at preventing bank managers from
taking excessive risk. Today market discipline is introduced into the Basel II
Capital Accord as a pillar of prudential banking regulation.

Base II and the regulators

One of the most difficult aspects of implementing an international agreement

is the need to accommodate differing cultures, varying structural models,
and the complexities of public policy and existing regulation. Banks’ senior
management will determine corporate strategy, as well as the country in
Financial Management Phase III IGTC
Bangalore

which to base a particular type of business, based in part on how Basel II is

ultimately interpreted by various countries' legislatures and regulators.

To assist banks operating with multiple reporting requirements for different

regulators according to geographic location, there are several software
applications available. These include capital calculation engines and extend
to automated reporting solutions which include the reports required under
COREP/FINREP.

BASEL III (sometimes "Basel 3") refers to a new update to the Basel Accords
that is under development. The term appeared in the literature as early as
2005 and is now in common usage anticipating this next revision to the
Basel Accords. The Bank for International Settlements (BIS) itself began
referring to this new international regulatory framework for banks as "Basel
III" in September 2010.

• The draft Basel III regulations include:

o "tighter definitions of Tier 1 capital; banks must hold 4.5% by
January 2015, then a further 2.5%, totalling 7%.
o the introduction of a leverage ratio,
o a framework for counter-cyclical capital buffers,
o measures to limit counterparty credit risk,
o and short and medium-term quantitative liquidity ratios."

Professionals and officers with Basel II knowledge and experience will be

required to lead the new Basel III projects, and they have started studying
the differences from the Basel II framework.

Development of the New Basel III Standard

Summary of Changes Proposed in Basel III

• First, the quality, consistency, and transparency of the capital base will
be raised.
o Tier 1 capital: the predominant form of Tier 1 capital must be
common shares and retained earnings
o Tier 2 capital instruments will be harmonised
o Tier 3 capital will be eliminated.
Financial Management Phase III IGTC
Bangalore

• Second, the risk coverage of the capital framework will be

strengthened.
o Strengthen the capital requirements for counterparty credit
exposures arising from banks’ derivatives, repo and securities
financing transactions
o Raise the capital buffers backing these exposures
o Reduce procyclicality and
o Provide additional incentives to move OTC derivative contracts to
central counterparties (probably clearing houses)
o Provide incentives to strengthen the risk management of
counterparty credit exposures
• Third, the Committee will introduce a leverage ratio as a
supplementary measure to the Basel II risk-based framework.
o The Committee therefore is introducing a leverage ratio
requirement that is intended to achieve the following objectives:
 Put a floor under the build-up of leverage in the banking
sector
 Introduce additional safeguards against model risk and
measurement error by supplementing the risk based
measure with a simpler measure that is based on gross
exposures.
• Fourth, the Committee is introducing a series of measures to promote
the build up of capital buffers in good times that can be drawn upon in
periods of stress ("Reducing procyclicality and promoting
countercyclical buffers").
o The Committee is introducing a series of measures to address
procyclicality:
 Dampen any excess cyclicality of the minimum capital
requirement;
 Promote more forward looking provisions;
 Conserve capital to build buffers at individual banks and
the banking sector that can be used in stress; and
o Achieve the broader macroprudential goal of protecting the
banking sector from periods of excess credit growth.
 Requirement to use long term data horizons to estimate
probabilities of default,
 downturn loss-given-default estimates, recommended in
Basel II, to become mandatory
 Improved calibration of the risk functions, which convert
loss estimates into regulatory capital requirements.
 Banks must conduct stress tests that include widening
credit spreads in recessionary scenarios.
o Promoting stronger provisioning practices (forward looking
provisioning):
 Financial Management Phase III IGTC
Bangalore

 Advocating a change in the accounting standards towards

an expected loss (EL) approach (usually, EL amount :=
LGD*PD*EAD).
• Fifth, the Committee is introducing a global minimum liquidity standard
for internationally active banks that includes a 30-day liquidity
coverage ratio requirement underpinned by a longer-term structural
liquidity ratio.
• The Committee also is reviewing the need for additional capital,
liquidity or other supervisory measures to reduce the externalities
created by systemically important institutions.

Vendor models for credit risk

measurement and management

Vendor models for credit risk measurement and

management
Observations from a review of selected models

Motivation and scope of the review

A number of vendors produce and sell products that are designed to support credit-
risk measurement and management functions within financial institutions. Such
products may comprise risk measurement models, data, or systems developed by
external commercial entities.
Financial Management Phase III IGTC
Bangalore
Interest in such vendor products has grown recently, as financial institutions seek to
meet new requirements such as the internal ratings-based (IRB) approaches of the
Basel II framework. Indeed, the potential use of vendor models is explicitly
contemplated within Basel II.1 Banks may choose to use vendor-developed models
and datasets within the context of IRB processes to assign exposures to certain
rating grades or segments, or to estimate IRB risk parameters. In its Newsletter No.
8 (March 2006), the Basel Committee on Banking Supervision (Basel Committee)
provides guidance for the use of such products within the foundation or advanced
IRB approaches.2
Even though individual institutions have primary responsibility for assessing the
suitability of vendor products as part of their validation processes, supervisors have
a natural interest in understanding the details of these products as well. Supervisors
need to assess whether the use of the model, as opposed to the model itself, is in
accordance with regulatory expectations and requirements. A solid understanding
of a vendor model allows a supervisor to assess the conditions under which the
model can be used by a given institution as part of the IRB approach. These
conditions may include various ways of adjusting and complementing the model’s
output in order to eliminate any systematic biases, or to bring it in line with credit
risk characteristics that are specific to the institution’s portfolio. The supervisor
would need to confirm whether a product does what the institution or vendor claims
it does, and which aspects of the model meet (or do not meet) relevant regulatory
and supervisory requirements.
Against this backdrop, the Research Task Force (RTF) of the Basel Committee
initiated a review of selected vendor credit-risk products, focusing on models that
could be used to estimate probability of default (PD), loss-given-default (LGD), or
exposure at default (EAD), and models that could be used to assign ratings or
produce credit scores, for wholesale or retail credit exposures. The output of the
review project was designed to meet two general objectives. First, it systematically
described features of vendor products in a consistent way to facilitate comparisons
among them. Second, it identified aspects of vendor models that relate to key
supervisory issues.
For its review of vendor credit-risk models, the RTF consulted with members of the
validation subgroup of the Basel Committee Standards Implementation Group to
identify models that were in reasonably wide use within the banking industry, were
available as commerical off-the-shelf products, and were used by banks in more
than one country. The resulting selection of models was not comprehensive, but did
seem to be broadly representative of vendor models in current use in banking,
although not necessarily of those used in every country represented on the Basel
Committee.
Vendors supported the RTF work by providing access to confidential technical
documentation as well as to the models themselves. Overall, cooperation by
vendors was strong. Vendors seemed to acknowledge the likely benefits of
providing supervisors with a more robust understanding of their products, and they
consistently made themselves available for in-depth discussions of the models.
The rest of this paper provides a high-level discussion of certain observations from
the RTF review of vendor products for credit risk measurement and management.
Financial Management Phase III IGTC
Bangalore

Overview of vendor credit-risk models

Overall, the market for vendor models of credit risk is still developing. Descriptions
of it –including this one – must be considered specific to the point in time at which
they are prepared. The pace of change and innovation may be especially high due
to the rapid rate of change in the discipline of credit risk measurement and
management, as well as to recent regulatory initiatives like the Basel II framework.
Additional evolution is likely, as recent market turmoil may provide a challenging
testing ground for existing risk models. Even over the relatively brief course of the
RTF review, vendors changed methodology as new methods appeared to provide
superior performance. In addition, vendors continued to expand coverage for some
models, adding to existing coverage of geography, types of exposures or types of
borrowers.
Most of the models presently available focus on the probability of a “credit event”
such as delinquency or default, in which a borrower or counterparty fails to perform
as contractually expected. The predominance of such models among vendor
offerings likely reflects the historical development of quantitative credit-risk
modelling; in addition, the probability of delinquency or default by an obligor
generally is believed to be affected little if at all by practices of the lender, so that a
single vendor model may be used by many lenders with no more than minor
modifications. A small number of the models reviewed focused on conditional credit
losses, described in practice as “loss in the event of default” or, as in Basel II, “loss-
given-default” or LGD. These models clearly constitute a smaller segment of the
overall market for vendor credit-risk models. Part of the explanation may be that
relatively little data on recoveries (or default-related losses) is available publicly to
support research and model development, and data may be difficult or costly to
acquire. It is also possible that conditional credit losses depend heavily on firm-
specific workout or recovery practices, creating a significant challenge for
commercial off-the-shelf models, which by definition are based on data that are not
firm specific. No vendor models were identified that addressed estimation of EAD.

In some cases, vendors provide multiple models as part of a “suite” within a single
product offering. The alternative models may reflect differing measures of the credit
event (such as different measures of default or delinquency), or may be based on
different methods or different sets of variables. In other cases, modelling
alternatives are offered as distinct products from a given vendor, or are marketed
and supported by different organisational units within a single vendor firm. Banks
using vendor models thus must choose not only among competing vendors, but
among the models offered by a particular vendor. Since the most appropriate model
generally varies depending on the specific business application, having a choice of
models is valuable; however, the larger the number of choices, and the smaller the
differences between the various alternatives, the more complicated the decision
becomes for potential users.
The number of users varies widely between models. Some models had few current
clients, although this was unusual and may have simply reflected an early stage of
the product’s life cycle. A few models have been widely adopted across the financial
industry, although in some cases vendors appear to count firms as clients even if
Financial Management Phase III IGTC
Bangalore
the model was purchased only for testing purposes, or was being used in a limited
way by staff in one part of a large firm.
Most products provide clear and intuitive output to model users. The output of many
vendor products includes not only the essential modelling results, but also other
information to support interpretation and use, such as key factors that determine
the final result, benchmarks for comparison or mapping between different risk
scales (such as between ratings or scores and default probabilities). In a few cases,
ease of use appears to be one of the main “value propositions” of the vendor
model, particularly where the product reflects simple or well-known methods and
publicly available data.
One potential barrier to use of any vendor product is the need for a technological
platform, one that meets relevant technical requirements. For most vendor models,
technical infrastructure requirements on the user side are modest. This is
understandable, since placing significant hardware or other technical requirements
on users would tend to raise the cost for clients, limiting the potential market and
placing the vendor at a competitive disadvantage. Vendors generally find low-
burden ways to deliver and maintain products. Many models are web-based; this
form of delivery typically requires little or no specialised IT infrastructure on the part
of clients.

Key critical observations

Over the course of the RTF review, several tendencies or observations emerged that
may be of interest to users of vendor models, bank supervisors and model vendors.
Some suggest potential directions for improvement in common practice. Ultimately,
improvements in practices related to vendor models will help promote better credit
risk management.

Advanced methods used in some models may not be well documented

Many vendor models reflect sophisticated techniques. Computational and analytical
complexity typically is not evident from the user interface; vendors are able to
make very complicated models easy for users to apply. However, whether the
methods on which a model is based are simple or complex, it is important that
users be able to acquire a solid understanding of those methods in order to
evaluate the suitability of the model for specific business applications under
consideration. In this regard, clear documentation and description of the methods
are essential. However, the review of vendor models found that many vendor
products include important elements that are undocumented. In several cases,
crucial documentation of aspects of models was made available only to paying
clients, not to prospective clients or others. Although this is perhaps understandable
as an approach to protecting intellectual property and proprietary model elements,
such practices can make it difficult for prospective clients to properly evaluate the
model.
Financial Management Phase III IGTC
Bangalore
It can be difficult to distinguish facts from marketing in vendor
documentation
Although users should be expected to perform appropriate testing of a model before
deciding to use it, review of documentation provided by the vendor also typically
forms part of the decision process. Vendor documentation varies considerably in the
level of detail provided about key elements, such as the nature of the reference
data used for model development and the results of validation. Reasonably
comprehensive technical information was available for some models, although
certain details that could be vital for a complete understanding sometimes were not
disclosed. More generally, vendors are understandably reluctant to include in their
documentation any information that might reflect unfavourably on the product; as a
result, it is difficult to regard vendor documentation as unbiased. Vendor
documentation serves a dual purpose, and while one of those purposes is to inform,
another is to promote the product.
Third party publications in professional journals or other sources may be valuable
supplements to information available from the vendor. Vendors frequently cite
outside publications to support decisions made in the modelling process, as
evidence of model effectiveness, or for other reasons. However, in some cases the
authors of such publications actually are employed by or affiliated with the vendor,
and these connections are not always made clear either in vendor documentation
when the publications are cited or in the publications themselves. Since authors
connected with the vendor are less likely to be entirely neutral in their view of the
vendor product, credibility would be enhanced if relationships between authors and
vendors were clearly stated, to allow clients and others to evaluate the reliability of
such evidence.
Most models are based on data from a limited geography or a limited
population; vendors could provide guidance to users on types of obligors
or exposures to which models are (or are not) applicable
Many models are developed from data with limited geographic coverage. Perhaps
as a result of the location of the primary vendors or data availability, some models
are developed on data sets that primarily or exclusively reflect US data.
Developmental or reference samples also often are skewed toward larger obligors
or publicly traded obligors, either by design or because those are the only data
available to the vendor at reasonable cost. Certain types of obligors, such as
financial firms, are routinely excluded from reference data sets by many vendors.
However, credit-risk models are most reliable when applied to obligors or exposures
that are similar to those used in their development. Banks and other users should
proceed more cautiously when attempting to apply a model to obligors or exposures
that differ substantially from those in the reference data set for the model.
It is important that users of a model have a clear understanding of the nature of the
reference data set and its limitations. Some vendors provide greater levels of detail
regarding the composition of the sample along key dimensions. Vendors often have
the most comprehensive understanding of the limitations of the data, and therefore
are in the best position to assist clients in understanding the relative applicability of
the model to different exposures, obligor or lines of business. Vendors may be able
to provide guidance on whether there are some situations in which the model
should not be applied, or should be applied only with extreme caution.
Although definitions of default and loss vary widely, most generally are
portrayed as “consistent with Basel II”
Financial Management Phase III IGTC
Bangalore
The correspondence between Basel II and vendor definitions of default and loss was
an explicit component of this review. Almost all vendor credit-risk models included
in their
4 Vendor models for credit risk measurement and management
Financial Management Phase III IGTC
Bangalore
documentation a discussion of alignment with Basel II, and almost all portrayed the
model as “consistent” with the requirements of Basel II. At the same time, the
specific definitions of default and loss were observed to vary to a considerable
degree across the vendor products. Basel definitions do provide for a limited degree
of interpretive latitude in practice, so it is possible that implementations in vendor
models could all differ from one another and still be consistent with the Basel II
framework. However, it is more likely that many of them are actually at least
somewhat inconsistent with a strict reading of the Basel definitions.
A degree of inconsistency perhaps should be expected, since models must be
developed from data, and in many cases the available data were generated over a
period of time during which the Basel definition had not yet been determined.
Moreover, certain aspects of the Basel II definitions of default are difficult to capture
in a vendor dataset. For example, if an obligor is unlikely to repay obligations in full,
this obligor should be characterised as “in default” under Basel II; while lenders may
have relevant information to make a determination of “unlikely to pay”, vendors
may not.
Where definitions of default or loss inherent in reference data are different from
those in Basel II, models based on those data may still be useful, but effective use
likely requires some degree and type of adjustment, either through direct
modification of model results or through adjustment of other aspects of broader risk
measurement processes (eg adjusting the way the model results are used as an
input to other components of a risk measurement system). Determining the
appropriate adjustments requires a thorough understanding of the definitional
differences, the impact on the model or data, and the implications for model results.
This would be easier for model users if vendors were forthright about the
differences and their impact rather than minimising those differences, and if
vendors provided more support for users to make appropriate adjustments.
Comparison across similar models can be helpful, but is difficult to do
Comparison of alternative models – models based on different methods or different
reference data – can be a highly constructive and valuable component of model
evaluation. Comparisons can improve understanding of strengths and weaknesses
of modelling methods and specifications, and support better interpretation of model
performance. However, existing models can be difficult to compare due to
differences in datasets and validation methodologies, even when they are based on
similar conceptual foundations and use similar modelling processes. Performance
measures for particular models often are motivated by the underlying methodology,
or are tailored to reflect a vendor’s perception of the primary interests of users (eg
whether the focus is on rank ordering and relative risk, or on the absolute
magnitude of risk measures). Some vendors do provide their own comparisons to
other models, but these may be limited or may not encompass the most relevant
alternatives, and are not direct comparisons to other vendor products. Moreover,
vendors may not have the proper incentives to provide unbiased comparisons. Very
few third-party model comparisons exist in the public domain, although a few
published research papers provide comparisons between examples of models that,
even if not identical to vendor models, are similar enough to make the comparisons
instructive.
Some vendors provide informative validation results, although often these
are in-sample results only
Validation of models is crucial for determining whether a model is suitable for use.
Effective validation requires that appropriate data be available, and that staff
Financial Management Phase III IGTC
Bangalore
performing the validation are sufficiently competent and have a sufficiently detailed
understanding of the model so that they can identify and test all relevant aspects.
Because of the proprietary nature of the data
or methods in many vendor models, technically proficient staff employed by the
vendors often are in the best position to do thorough and effective validation. Of
course, there is some question of whether vendors’ incentives are sufficiently well
aligned with those of model users. Market forces may be of some help in this
regard, since a model that performs poorly is unlikely to be a commercial success in
the long run. However, banks and supervisors may be reluctant to rely on possibly
slow market forces to sort out good models from bad.
A review of available vendor validation results concluded that such results are
helpful as a way to enhance the understanding of any model. However, there was
clear variation across vendors in the quality and thoroughness of validation, with
some quite good, others less so. One concern was that much validation of models
was “in-sample”, with the same data used both for development and for validation.
Some vendors create and use holdout samples for validation, but there is a natural
tension inherent in holding out data during development that might improve the
power of the model, which is likely to be seen as a first-order concern, in order to
improve validation, which to some may appear to be second-order. The frequency
of validation varies; conducting regular, periodic revalidation would appear to be a
sound practice that some vendors follow, while for others revalidation is irregular
and infrequent.
Vendors could do more to encourage and support end-user validation,
consistent with supervisory expectations
Principles developed by the Basel Committee emphasise the importance of end-user
assessments of the appropriateness of any vendor model, and of the key role of
validation in such assessments. 3 Some vendors explicitly support end-user
validation of this sort, emphasising in their documentation and their communication
with clients the importance of validation, and providing instructions or training on
how validation can and should be performed. However, other vendors do little to
support or encourage end-user validation. In some cases, data or modelling details
that would seem to be essential for effective validation are not revealed by the
vendor. Although vendors naturally take steps to protect proprietary elements of
products, the variation in practice across vendors suggests that some vendors could
strike a better balance in this regard. Vendors should recognise that market
acceptance of the product over the longer term depends on the ability of users to
conduct validation as part of their own risk management and to meet supervisory
expectations for model use.
Vendors may have information that would help users determine if
contemplated modifications to models are reasonable
Many models allow at least some degree of customisation; some include certain
parameters that can be set or modified by users. The ability to change features of
vendor models raises two potential issues for users. One is simply the need to have
a comprehensive understanding of what the various potential changes are. Vendors
could help ensure that clients have a thorough understanding of the aspects of the
model that can be modified so that appropriate controls can be developed and put
in place by users.
The second issue is that model users need a good understanding of the impact of
any user modifications, and how that impact may depend, if at all, on the nature of
Financial Management Phase III IGTC
Bangalore
the obligor or exposure being considered. Vendors are likely to have insight into
such issues, and could
Financial Management Phase III IGTC
Bangalore
help clients achieve a solid understanding. They may also be able to advise users
on types of changes that might be considered. For example, where input values or
parameters are subject to user choice, certain values may be more reasonable than
others. Developers of a model likely have a better sense of this than users, and
could share this knowledge with their clients.
Vendor products vary in the ease with which they can be used for stress
testing
Supervisory guidance and risk management practices increasingly emphasise the
importance of rigorous stress testing to identify and assess risks in financial
institutions. Some vendor models explicitly allow for the creation and evaluation of
stress scenarios by users. However, others make stress testing difficult. In some
cases, difficulties arise because key inputs or relationships cannot be modified by
the user. In other cases, the model requires certain relationships to be maintained
between variables, and it is difficult or impossible for the user to assess whether
these relationships are being maintained as assumptions associated with a stress
scenario are applied to the model.
Stress testing using vendor models also confronts a trade-off related to model
complexity. Models that include more variables or more complex relationships may
lead to a richer and more realistic description of how risk is related to underlying
factors. However, such models may also make stress testing more difficult; a larger
number of variables requires a more elaborate specification of the stress scenario,
creating the challenge of ensuring that all key elements of the scenario are
internally consistent. Again, because of the necessarily proprietary nature of vendor
products, it is difficult for vendors to fully reveal all of the model details that users
might find useful for stress testing. However, vendors might further consider how
their models can be used in stress testing, and whether additional helpful disclosure
might be possible without jeopardising the competitive position and value of the
product.

Conclusions
Certain aspects of the observations described in this report suggest potential points
of future emphasis or improvement, particularly for vendors or for model users. In
many cases, there is an element of tension between the natural desire of vendors to
protect proprietary elements of models and the needs of model users to have
enough information about models to comply with regulatory expectations and
principles of sound model use. These tensions likely will always exist to some
extent, and the commercial interests of vendors cannot and should not be ignored.
However, there appear to be a number of areas in which a modest amount of
additional disclosure by vendors could significantly enhance the ability of
institutions to use credit risk models in ways that conform more closely to
supervisory expectations and sound management practices.

Points of potential improvement include:


Incomplete documentation or undocumented model features sometimes make it
difficult for model users to properly evaluate models, especially on a prospective
basis.
Financial Management Phase III IGTC
Bangalore

Vendor documentation can provide valuable information, but often reflects an
inherent bias, and publications by staff or affiliated researchers are sometimes cited
as if they are written by disinterested third parties.
Financial Management Phase III IGTC
Bangalore


Limitations of reference data sets create a need for caution when applying models
to obligors or exposures that differ markedly or systematically from observations in
development samples.


Models generally are suitable for use in some cases but not in others, and vendors
may be able to use their insight into the development and operation of models to
better indicate for clients any specific business uses for which the vendors’ products
are not suitable.


Model users may need to adjust model results (or their business use) for definitions
of default or loss that differ from those required under Basel II, and vendors could
provide valuable information to support appropriate adjustments.


Comparisons between competing models could be valuable, but are difficult to do
and in practice are rarely performed in an informative way.


Vendors often are in the best position to conduct certain aspects of model
validation, but may lack the proper incentives to conduct credible, unbiased testing.


Out-of-sample validation is valuable, but is not a common practice among vendors.


Regular, periodic re-validation is a sound practice for model vendors, but actual
approaches to the frequency of re-validation vary.


While some vendors provide explicit support for highly important end-user
validation of models, others provide little or no support.


Vendors can help clients understand which aspects of models can be modified, and
the implications of those modifications for model applicability and performance.


Certain elements of models or their documentation might be modified to better
support stress testing.

As the market for vendor credit-risk models develops and matures, vendors and end
users may encounter opportunities to consider the implications of some of the
points discussed in this report, and in particular to balance the legitimate
proprietary interests of model vendors against the benefits of greater transparency.
As noted above, improvements in practices related to vendor models, whether
Financial Management Phase III IGTC
Bangalore
driven by vendors or by their clients, will help promote better credit risk
management.