Welcome to Scribd. Sign in or start your free trial to enjoy unlimited e-books, audiobooks & documents.Find out more
Standard view
Full view
of .
Look up keyword
Like this
0 of .
Results for:
No results containing your search query
P. 1
Auditing Derivatives: Model Validation

Auditing Derivatives: Model Validation

Ratings: (0)|Views: 51|Likes:
Published by Jasvinder Josen
This article discusses the importance of the Model Validation group in an investment bank and how the auditor should go about auditing this function.
This article discusses the importance of the Model Validation group in an investment bank and how the auditor should go about auditing this function.

More info:

Published by: Jasvinder Josen on Dec 18, 2011
Copyright:Attribution Non-commercial


Read on Scribd mobile: iPhone, iPad and Android.
download as DOCX, PDF, TXT or read online from Scribd
See more
See less





This article appeared in The Malaysian Accountant journal,Sep-Oct 2011 issue 
Auditing Derivatives: Think of what can go wrong -Model Validation
By Jasvin Josen
Auditing derivatives, in all likelihood, is one of the most challenging areas for an auditor. Ifocus on thinking of what can go wrong rather than just a standard review, for, derivativesare very flexible products and thus have a great potential to flourish in a reckless way whenunguarded by proper supervision and regulation.In the last five articles, we have observed the different issues that can surface as a derivativeinstrument moves along the trading floor and the Controlling Group, under the watchfuleyes of Risk Management (with its Value-at-Risk model) and the Valuation team.I must introduce a last function called Model Validation. This function has long beenestablished in investment banks in the U.S. and Europe but possibly only now taking root inAsian investment banks.This article will discuss the Model Validation role and the key points that the auditor shouldbe mindful of, in his examination.
The Model Validation function
The main role of the Model Validation team is to ensure that the pricing models developed in-houseor purchased from external vendors are credible. They want to be certain that the model is stable inevery way, when giving out prices and risk measures. The team usually report to the RiskManagement Group or it forms part of the Risk Management team.
So, what can go wrong?The pressure
In order to be convinced that pricing models are stable and robust, the Model Validation teamdevelops independent models for comparison. If they are satisfied with the results, the secondphase is to perform vigorous tests such as back tests, stress tests and bug testing on the originalmodel to determine its points of failure, if any.There are several things that can go wrong. The more complicated the derivative, the morechallenging this task becomes. E very parameter has to be tested to see if the model will collapse
when the parameter goes out of range. But some parameters are “derived parameters”, i.e. these
parameters are calibrated from standard products in the market. The concern here is that not allmarket environments are considered when the calibration is performed.This team has a major responsibility in representing to its bank that a model can be safely used. The
team’s test
ing jobs can go on forever as they will always be something that can fall over in themodel. For this reason, they will find an effective way to detain the main risks in the most efficienttime. The concern here is that when the line is drawn, it is hoped that the procedure is able to catchall the significant and possible flaws.Time pressure is always a concern. Banks usually need the model to be approved before a new tradecan be priced or booked in the system. The Model Validation team will be under pressure to getback with the results fast so that the bank does not lose the potential client. The risk here isimportant insights can get lost when the team has a tight or sometimes unreasonable deadline.The auditor will want to look into the skill and capabilities of the team. He will also want to look intothe number of models tested within a time frame and compare it to previous times or other banks toget an idea if the team is being stressed. If there is indication of danger signs, the auditor would
want to review if “corners were cut” at all and in what way.
Compliance within the rest of the bank
For each model, the Model Validation team comes up with a Model Summary detailing thefollowing:
Description of model
Products that can be priced
Model restrictions
Stress testing results
Model stabilityThis document is very useful for the bank. However in my experience, not everyone refers to andadheres to its restrictions and guidance. Important advice can be missed when this happens.The auditor will want to review if there is a procedure in the bank to ensure adherence to thisimportant document.
Model Risk
According to an interesting paper published in the Monetary and Economic Studies, 2000
, “
ModelRisk and Its Control
” by T
oshiyasu Kato and Toshinao Yoshiba, m
odel risk is defined as “the risk
arising from the use of a model which cannot accurately evaluate market prices, or which is not a
mainstream model in the market.” In risk
measurement models,
model risk is defined as “the risk of 
not accurately estimating the probability of future losses
Sources of model risk in pricing models can include use of wrong assumptions and errors inestimations of parameters. Sources of model risk in risk measurement models consist of theauthenticity of the assumed distribution and errors in the logical framework of the model.It is a normal practice for banks to set up a reserve to allow for this risk to accommodate forpotential losses. Scenario analysis is undertaken for various fluctuation patterns of risk factors, toestablish position limits.The Model Validation team attempts to assess the model risk by conducting stress and back testing.The challenge in back testing is in obtaining the results from testing the pricing model in the past, i.e.generating data that was not actually available at that time. This task is inherently exposed to anyinherent model risk. Thus the accuracy and reliability of the back testing results would be hampered,and it will be difficult to quantify the results.The challenge in stress testing is including all the possible scenarios and adopting a realisticprobability distribution. Looking back at the credit crisis, we have to wonder about the results whenstress testing was done to the models for pricing and measuring the risk of credit debt obligations(CDOs).The auditor will want to review in detail stress testing and back testing results to observe if themethodology is consistent and logical. He would also be interested to bounce some ideas with therisk manager to gauge how comfortable he is with the model testing results. Further, the auditorwould also want to know the areas of improvement identified by the bank itself in reaction to thetesting results. This gives him an idea of the extent of the model risk involved.
Difficulty with purchased models
When it comes to models purchased from external vendors, difficulty lies in obtaining enough detailon the application such as calibration results and future calibration needs, bugging results, stresstesting and back testing results, if any were done.The auditor would want to ensure that these important tests are performed adequately by at leastone of the parties. If it is the task of the external vendor, the Model Validation team must obtainadequate proof in writing and must have avenues to perform continued testing in future. If the taskis with the Model Validation team, the vendor must provide enough information to perform testingso as not to lose any important shortcoming in the model.
It is important to perceive the Model Validation team as modeling for insights, not numbers. Theteam cannot afford to get lost in the model and testing results and lose the big picture. As for theauditor, it is perhaps more effective if he reviews the reports produced by this team and his
interviews with relevant parties to gain insight as to how much he can rely on the team’s opinion
and reports.As the Malaysian and Asian banks start introducing a wider spectrum of financial derivatives, I hope Ihave provided at least some of the basic tools you will need. Perhaps the points raised in this series

You're Reading a Free Preview

/*********** DO NOT ALTER ANYTHING BELOW THIS LINE ! ************/ var s_code=s.t();if(s_code)document.write(s_code)//-->