Journal of Risk (39-54) Volume 9/Number 2, Winter 2006/07 Quality control of risk measures: backtesting VAR models Victor H. de fa Pena* Department of Statistics, Columbia University, Room 1027, Mail Code 4690, 1255 Amsterdam / ‘Avenue, NewYork, NY 10027, USA Ricardo Rivera State of New York Banking Department and New York Universty, One State Street, New York, NY 10004,USA Jesus Ruiz-Mata Lehmann Brothers, 745 7th Avenue, Second Floar, New York, NY, 10019, USA This paper introduces a new slatistical approach to assessing the quality of riske sures: quality control of risk measures (QCRM). The approach is applied to the problem of backtesting value-at-risk (VAR) models. VAR models are ‘used to predict the maximum likely Tosses in a bank's portfolio at a specified confidence level and time horjgon. The widely accepted VAR backtesting ‘procedure outlined by the Basel Committee for Banking Supervision controls ‘the probability of rejecting the model when the model is correct. A drawback of the Basel approach is its limited power to conttol the probability of accept ‘ng an incorrect VAR model. By exploiting the binomial structure ofthe testing problem, QCRM provides a more balanced testing procedure, which results in a uniform reduction of the probability of accepting a wrong model QCRM consists of three elements: the first, is a new hypothesis-testing problem in which the null and alternative hypotheses are exchanged to control the probability of accepting an inaccurate model. The second element is a new approach for comparing the power of the QCRM and Basel tess in terms of the ie probability of rejecting correct and incorrect models. The third element involves : the use of the technique of pivoting the cumulative distribution function to ‘obtain one-sided confidence intervals for the probability of an exception, The use of these confidence intervals results in new scceptance/rejection regions for tests of the VAR model. We compace these to ones commonly used inthe financial literature. 1 Introduction Banking regulators and risk managers of financial institutions often wish to ] assess the adequacy of a model. They frequently test the null hypothesis that the model is correct against an alternative hypothesis that the model is incorrect. In this paper we present an approach to assist regulators and risk managers in this, *Corresponding author. ‘The first author acknowledges support from NSF grant DMS 5.24517, 3940 Victor H. de la Pena, Ricardo Rivera and Jesus Ruiz-Mata task. We illustrate the approach by presenting an application to the problem of backtesting value-at-risk (VAR) models. ‘The (1 ~§)100% VAR number is the (1—6) x 100 percentile of the distribu- tion of the portfolio losses for a specified time horizon. VAR backtesting is the process by which financial institutions periodically compare daily profits and losses with the model-generated risk measures to gauge the accuracy of their VAR models. In 1996 the Basel Committee for Banking Supervision (“Basel”) developed a framework for backtesting the internal models used to calculate regulatory capi- tal for market risk (see Basel (1996)). Alternative statistical evaluation method- ologies have been proposed. For example, the evaluation based on the binomial distribution is discussed by Kupiec (1995); the interval forecast method proposed by Christoffersen (1998); the distribution forecast method by Cmkovie and Drachman (1996); and the magnitude loss function method by Lopez (1996). We start with the following question: What are regulators and befik managers really doing when they apply the Basel VAR backtesting methodology? ‘The Basel backtesting procedure tests the null hypothesis that the bank’s VAR ‘model predicts losses accurately against the alternative hypothesis that the model predicts losses incorrectly. H§: VAR model is correct vs. H}: VAR model is incorrect a ‘The test statistic used is based on the number of exceptions generated by the VAR model. For a given trading day, an exception occurs when the loss exceeds the model-based VAR. ‘The test postulates that the probability of an exception, p, is equal to 0.01 and tests it against the alternative hypothesis that the probability of an exception is, greater than 0.01. The test is based on the number of exceptions in 250 trading days. The test rejects the VAR model if the number of exceptions is greater than or equal to 10 and accepts the model otherwise. When evaluating statistical tests, it is common practice to examine their type I and Il error rates. Under the Basel backtesting procedure, the type I error rate is the probability of rejecting the VAR model when the model is correct, while the type Il error rate is the probability of accepting the model when the model is incorrect. ‘The Basel backtesting procedure is designed for controlling the c level, the probability of rejecting the VAR model when the model is correct. For this test, the a level is the probability that number of exceptions, out of 250 daily obser- vations, is greater than or equal to 10, when the probability of an exception is p = 0.01. The a level of the test is 0.0003 (or 0.03%). Although the test establishes a very conservative threshold for controlling the type I error rate, it is not designed to control the type Il error rate. Therefore,Quality control of risk measures: backtesting VAR models it does not control the probability of accepting the VAR model when the model is incorrect. This drawback is pointed out on page 5 of Basel (1996): “The Committee of ‘course recognizes that tests of this type are limited in their power to distinguish ‘an accurate model from an inaccurate model”. ‘To address this issue we introduce an alternative approach to the Basel back- testing procedure. We call this methodology quality control of risk measures (QCRM). Its goal is to enhance the ability of the test to reject an incorrect model. It consists of three elements: first, the test introduces a new hypothesis testing problem in which the null and alternative hypotheses are exchanged. The goal is to control the probability of accepting a wrong model. The second element consists of a new definition of the power of the tests that allows the comparison ‘of QCRM and Basel backtesting procedures. The third element involves the use of a technique to obtain accurate estimates of the acceptance/rejection regions. ‘The new hypothesis testing problem HQ: VAR model is incorrect vs. H®: VAR model is correct @ Under QCRM, the acceptance of the null hypothesis is equivalent to the rejection of the VAR model. The model is accepted when the null hypothesis in (2) is rejected and, hence, when there is overwhelming evidence supporting the model. ‘A type I error for QCRM occurs when the VAR model is accepted but the model is incorrect, and a type II error happens when the model is rejected although the model is correct, Table 1 shows correct and incorrect decisions ‘when the null and the alternative hypotheses are true. ‘The following relationship is important for understanding the properties of the QCRM test: a Probability of rejecting the VAR model when the model is incorrect = 1 — Probability of accepting the VAR model when it is incorrect. Under QCRM the probability of accepting the VAR model when the model is incorrect is set al Jevel «x: this means thai the procedure controls the type 1 error rate. By setting the o: level to a small value (<1%), QCRM guarantees a high probability (299%) of rejecting a wrong model. Once this level is fixed, the use TABLE I Type I and Il errors for QCRM. Decision I: Decision 2: VAR model is rejected VAR model is accepted Model correct Type Hl error Correct assessment Model incorrect Correct assessment Type | error Volume 9/2 Winer 200/07 4a