#
A Holistic Model Validation Framework for Current Expected Credit Loss (CECL) Model Development and Implementation^{ †}

^{†}

## Abstract

**:**

## 1. Introduction and Discussion

## 2. Review of the Literature

## 3. Model Validation of CECL Model Development and Implementation

- Clear articulation of the business purpose and portfolio profile pertaining to the model;
- Comprehensive discussion of the model history;
- Rigorous review of academic and practitioner literature relevant to the model;
- Precise exposition of model mathematics and functional forms;
- Complete examination of model data sourcing, cleansing processes and data quality analysis;
- Thorough narrative on the segmentation and variable selection process, encompassing both statistical analyses and expert panel dialogue;
- Informative description of estimation results, including comparison to challenger and candidate models;
- Detailed analysis of all model testing that while supporting the choice of the champion model while also honestly representing potential weaknesses;
- Understandable detail around any model adjustments, overlays, and risk mitigation elements layered onto model output.

## 4. Example of a CECL Modeling Framework: Time-Series VAR Methodologies for Credit Model Estimation and Macroeconomic Scenario Generation

## 5. Example of CECL Model Development Validation: Analysis of Model Data, Estimation Results, Challengers, and Performance Testing

- Real gross domestic product growth (RGDP);
- Real gross domestic investment (RDIG);
- Consumer price index (CPI);
- Real disposable personal income (RDPI);
- Unemployment rate (UNEMP);
- Three-month treasury bill rate (MTBR);
- Ten-year treasury bond rate (10YTBR);
- BBB corporate bond rate (BBBCR);
- Dow Jones index (DJI);
- National house price index (HPI);
- Nominal disposable personal income growth (NDPIG);
- Mortgage rate (MR);
- CBOE’s equity market volatility index (VIX);
- Commercial real estate price index (CREPI).

- Transformations of chosen variables should indicate stationarity;
- Signs of coefficient estimates are economically intuitive;
- Probability values of coefficient estimates indicate statistical significance at conventional confidence levels;
- Residual diagnostics indicate white-noise behavior;
- Model performance metrics (goodness of fit, risk ranking, and cumulative error measures) are within industry-accepted thresholds of acceptability;
- Scenarios rank order intuitively (i.e., severely adverse scenario stress losses exceeding scenario base expected losses).

- UNEMP;
- BBBCY;
- CREPI;
- VIX;
- CORPSPR.

- Commercial and industrial loans to total assets (CILTA);
- Commercial and development loans growth rate (CDLGR);
- Trading account assets to total assets (TAATA);
- Other real estate owned to total assets (OROTA);
- Total unused commitments growth rate (TUCGR).

- Model 1: macroeconomic—UNEMP and BBBCY; idiosyncratic—none;
- Model 2: macroeconomic—UNEMP and BBBCY; idiosyncratic—CILTA and CDLGR;
- Model 3: macroeconomic—UNEMP and CREPI; idiosyncratic—none;
- Model 4: macroeconomic—UNEMP and CREPI; idiosyncratic—TAAA and CDLGR;
- Model 5: macroeconomic—UNEMP and CORPSPR; idiosyncratic—none;
- Model 6: macroeconomic—UNEMP and CORPSPR; idiosyncratic—OROTA;
- Model 7: macroeconomic—CREPI and VIX; idiosyncratic—none;
- Model 8: macroeconomic—CREPI and VIX; idiosyncratic—TAAA and OROTA;
- Model 9: macroeconomic—CORPSPR, UNEMP, and VIX; idiosyncratic—none;
- Model 10: macroeconomic—CORPSPR, UNEMP, and VIX; idiosyncratic—TUCGR;
- Model 11: macroeconomic—BBBCY, UNEMP, and CREPI; idiosyncratic—none;
- Model 12: macroeconomic—BBBCY, UNEMP, and CREPI; idiosyncratic—CDLGR;
- Model 13: macroeconomic—BBBCY, UNEMP, and CORPSPR; idiosyncratic—none;
- Model 14: macroeconomic—BBBCY, UNEMP, and CORPSPR; idiosyncratic—TAAA.

^{2}), ranges from 87% to 97% across models, which is good performance by industry standards and broadly comparable. The best-fitting model is number 2, the bivariate macroeconomic specification with UNEMP and BBBCY, with idiosyncratic variables CILA and CDLG, having an AR

^{2}of 97.7%. The worst fitting model is number 7, the bivariate macroeconomic specification with CREPI and VIX, with no idiosyncratic variables, having an AR

^{2}of 87.5%. The autoregressive coefficient estimates all show that the NCORs display significant mean reversion, having values ranging from −0.89 to −0.65 and averaging −0.77.

- Generalized Cross-Validation (GCV);
- Squared Correlation (SC);
- Root-Mean-Squared Error (RMSE);
- Cumulative Percentage Error (CPE);
- Akaike Information Criterion (AIC).

## 6. The Quantification of Model Risk According to the Principle of Relative Entropy

## 7. Conclusions and Future Directions

- More granular classes of credit risk models, such as ratings migration or PD/LGD scorecard/regression;
- Alternative datasets, for example, bank or loan level data;
- More general classes of regression model, such as logistic, semi-parametric, or machine learning/artificial intelligence techniques (Jacobs 2018);
- Applications related to stress testing, such as regulatory or economic capital.

## Funding

## Conflicts of Interest

## Appendix A. Details of VAR Model Specification, Estimation, and Markov Switching Extension

## References

- Basel Committee on Banking Supervision. 2006. International Convergence of Capital Measurement and Capital Standards: A Revised Framework. Basel: The Bank for International Settlements. [Google Scholar]
- Basel Committee on Banking Supervision. 2009. Principles for Sound Stress Testing Practices and Supervision. Consultative Paper No. 155. Basel: The Bank for International Settlements. [Google Scholar]
- Batchelor, Roy A., and Parni Dua. 1990. Product differentiation in the economic forecasting industry. International Journal of Forecasting 6: 311–16. [Google Scholar] [CrossRef]
- Baum, Leonard E., and Ted Petrie. 1966. Statistical Inference for Probabilistic Functions of Finite State Markov Chains. The Annals of Mathematical Statistics 37: 1554–63. [Google Scholar] [CrossRef]
- Baum, Leonard E., Ted Petrie, George Soules, and Norman Weiss. 1970. A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains. Annals of Mathematical Statistics 41: 164–71. [Google Scholar] [CrossRef]
- Bellotti, Anthony, and Jonathan Crook. 2013. Forecasting and stress testing credit card default using dynamic models. International Journal of Forecasting 4: 563–74. [Google Scholar] [CrossRef] [Green Version]
- Bernanke, Benjamin S., and Cara S. Lown. 1991. The credit crunch. Brookings Papers on Economic Activity 2: 205–47. [Google Scholar] [CrossRef] [Green Version]
- Berrospide, Jose M., and Rocheelle M. Edge. 2010. The effects of bank capital on lending: What do we know, what does it mean. International Journal of Central Banking 6: 5–55. [Google Scholar]
- Blackwell, David, and Lambert Koopmans. 1957. On the Identifiable Problem for Functions of Finite Markov Chains. The Annals of Mathematical Statistics 28: 1011–15. [Google Scholar] [CrossRef]
- Board of Governors of the Federal Reserve System. 2011. Paper presented at Supervisory Guidance on Model Risk Management, Supervisory Letter 11-7, Washington, DC, USA, April 4.
- Box, George E. P., and Gwilym M. Jenkins. 1970. Times Series Analysis: Forecasting and Control. San Francisco: Holden-Day. [Google Scholar]
- Brockwell, Peter J., and Richard A. Davis. 1991. Time Series: Theory and Methods. New York: Springer. [Google Scholar]
- Carlson, Mark, Hui Shan, and Missaka Warusawitharana. 2013. Capital ratios and bank lending: A matched bank approach. Journal of Financial Intermediation 22: 663–87. [Google Scholar] [CrossRef] [Green Version]
- Chae, Sarah, Robert F. Sarama, Cindy M. Vocjtech, and James Z. Wang. 2018. The Impact of the Current Expected Credit Loss Standard (CECL) on the Timing and Comparability of Reserves. Finance and Economics Discussion Series. FEDS Working Paper No. 2018-020. Available online: https://ssrn.com/abstract=3192150 (accessed on 20 April 2019).
- Commandeur, Jacques J. F., and Siern J. Koopman. 2007. Introduction to State Space Time Series Analysis. New York: Oxford University Press. [Google Scholar]
- Cornett, Marcia M., Jamie J. McNutt, Philip E. Strahan, and Hassan Tehranian. 2011. Liquidity risk management and credit supply in the financial crisis. Journal of Financial Economics 101: 297–312. [Google Scholar] [CrossRef]
- Financial Accounting Standards Board. 2016. Accounting Standards Update No. 2016-13, Financial Instruments—Credit Losses (Topic 326): Measurement of Credit Losses on Financial Instruments. June. Available online: https://www.fasb.org/jsp/FASB/Document_C/DocumentPage?cid=1176168232528 (accessed on 1 May 2020).
- Francis, William Barry, and Matthew Osborne. 2009. Bank Regulation, Capital and Credit Supply: Measuring the Impact Prudential Standards. Occasional Paper 36. London: Financial Services Authority. [Google Scholar]
- Glasserman, Paul, and Xingbo Xu. 2013. Robust risk measurement and model risk. Quantitative Finance 14: 29–58. [Google Scholar] [CrossRef]
- Goldfeld, Stephen M., and Richard E. Quandt. 1973. The estimation of structural shifts by switching regressions. Annal of Economic and Social Measurement 2: 475–85. [Google Scholar]
- Hamilton, James D. 1988. A rational expectation analysis of changes in regime: An investigation of the term structure of interest rates. Journal of Economic Dynamics and Control 12: 385–423. [Google Scholar] [CrossRef]
- Hansen, Lars P., and Thomas J. Sargent. 2007. Robustness. Princeton: Princeton University Press. [Google Scholar]
- Heller, Alex. 1965. On Stochastic Processes Derived from Markov Chains. The Annals of Mathematical Statistics 36: 1286–91. [Google Scholar] [CrossRef]
- International Accounting Standards Board. 2014. International Reporting for Financial Statement Number 9. International Accounting Standards Board. Available online: https://www.ifrs.org/-/media/project/financial-instruments/project-summaries/ifrs-9-project-summary-july-2014.pdf (accessed on 1 May 2020).
- Jacobs, Michael, Jr. 2015. The quantification and aggregation of model risk: Perspectives on potential approaches. The Journal of Financial Engineering and Risk Management 2: 124–54. [Google Scholar]
- Jacobs, Michael, Jr. 2017. A mixture of distributions model for the term structure of interest rates with an application to risk management. American Research Journal of Business and Management 3: 1–17. [Google Scholar]
- Jacobs, Michael, Jr. 2018. The validation of machine learning models for the stress testing of credit risk. The Journal of Risk Management in Financial Institutions 11: 1–26. [Google Scholar]
- Jacobs, Michael, Jr. 2019a. An analysis of the impact of modeling assumptions in the Current Expected Credit Loss (CECL) framework on the provisioning for credit loss. The Journal of Risk and Control 6: 65–113. [Google Scholar]
- Jacobs, Michael, Jr. 2019b. The accuracy of alternative supervisory methodologies for the stress testing of credit risk. The Journal of Financial Engineering and Risk Management. forthcoming. [Google Scholar]
- Jacobs, Michael, Jr., Luther Klein, and Akber Merchant. 2015a. Emerging Trends in Model Risk Management. Accenture Consulting. Available online: https://www.accenture.com/_acnmedia/pdf-46/accenture-emerging-trends-model-risk-management.pdf (accessed on 1 May 2020).
- Jacobs, Michael, Jr., Ahmet K. Karagozoglu, and Frank J. Sensenbrenner. 2015b. Stress testing and model validation: Application of the Bayesian approach to a credit risk portfolio. The Journal of Risk Model Validation 9: 41–70. [Google Scholar] [CrossRef]
- Kishan, Ruby P., and Timothy Opiela. 2000. Bank size, bank capital, and the bank lending channel. Journal of Money, Credit, and Banking 32: 121–41. [Google Scholar] [CrossRef]
- Loregian, Attilio, and Angela Meucci. 2016. Neither “Normal” nor “Lognormal”: Modeling Interest Rates across All Regimes. Financial Analysts Journal 72: 68–82. [Google Scholar]
- Pearson, Karl. 1894. Contribution to the Mathematical Theory of Evolution. Philosophical Transactions of the Royal Society 185: 71–110. [Google Scholar]
- R Development Core Team. 2019. R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing, ISBN 3-900051-07-0. [Google Scholar]
- Sims, Christopher A. 1980. Macroeconomics and reality. Econometrica 48: 1–48. [Google Scholar] [CrossRef] [Green Version]
- Skoglund, James. 2018. Quantification of Model Risk in Stress Testing and Scenario Analysis. Cary: SAS Institute. [Google Scholar]
- Stock, James H., and Mark W. Watson. 2001. Vector autoregressions. Journal of Economic Perspectives 15: 101–15. [Google Scholar] [CrossRef] [Green Version]
- Tjostheim, D. 1986. Non-linear time series: A selective view. Scandinavian Journal of Statistics 21: 97–130. [Google Scholar]

1 | |

2 | These are given by FASB (1975) and FASB (1993) for performing and impaired instruments, respectively. |

3 | This is also called the R&S period. |

4 | Several challenges are associated with macroeconomic forecasting, related to changes in the structure of the economy, measurement errors in data, and behavioral biases (Batchelor and Dua 1990). |

5 | Dodd-Frank Act Stress Testing—“DFAST”. |

6 | We note that, in some cases, such as lines of credit, the error is measured relative to an interpretation. |

7 | We note that while this may be straightforward for term loans, there would be great challenges doing this for lines of credit, and in that context this may be feasible only for model elements such as PD forecasts. |

8 | Refer to Stock and Watson (2001) for a discussion of the basic aspects of macroeconomic forecasting (i.e., characterization, forecasting, inferences, and policy advice regarding macroeconomic time series and the structure of the economy). |

9 | In fact, the exogenous variables $\left\{{X}_{t}\right\}$ can represent both stochastic and non-stochastic (deterministic) variables, examples being sinusoidal seasonal (periodic) functions of time, used to represent the seasonal fluctuations in the output process $\left\{{Y}_{t}\right\}$, or intervention analysis modelling in which a simple step (or pulse indicator) function taking the values of 0 or 1 indicates the effect of output due to unusual intervention events in the system. |

10 | The full model development code package, an R project containing all data and code, is available upon request. |

11 | These are available at https://www5.fdic.gov/sdi/main.asp?formname=standard. |

12 | We perform this model selection in an R script designed for this purpose, using the libraries dse and tse to estimate and evaluate VAR and MS-VAR models (R Development Core Team 2019). |

13 | We leave out the last two years of available data, 1Q16–4Q17, in order to have a holdout sample for testing the accuracy of the models. We also choose to start our sample in 2001, as we believe that the earlier period would reflect economic conditions not relevant for the last decade and also because, in the financial industry, this is a standard starting point for CCAR and DFAST stress-testing models. |

14 | This is similar to the findings of Loregian and Meucci (2016) and Jacobs (2017) in the context of modeling US Treasury yields. We observe that this mixture adequately characterizes the empirical distributions of the data in this paper. |

15 | Estimation results for the VAR and MS-VAR model are available upon request. The models are all convergent and goodness-of-fit metrics fall within industry standards. Signs of coefficient estimates are in line with economic intuition and estimates are all significant at conventional levels. We use the dse, tseries, and MSBVAR libraries in R in order to perform the estimations (R Development Core Team 2019). |

16 | We note that this observation is consistent with findings in the literature that employ ensemble techniques, such as Bayesian averaging or random forest models, where collections of more parsimonious models properly weighted are likely to out-perform more highly parameterized models on an out-of-sample basis. |

**Figure 1.**Net Charge-off Rates (NCORs), Provision for Loan and Lease Losses (PLLL), and the Allowance for Loan and Lease Losses (ALLL) as a percentage of total assets—all insured depository institutions in the United States (US) (Federal Deposit Insurance Corporation Statistics on Depository Institutions Report—Schedule FR Y-9C).

**Figure 2.**Overview of a general framework for the validation of model development—model data, estimation, conceptual review, and model testing.

**Figure 3.**Challenges in the validation of Current Expected Credit Loss (CECL) model development—benchmarking and backtesing.

**Figure 4.**Framework for process verification and assessment of CECL model implementation and execution.

**Figure 10.**Kullback–Leibler relative entropy worst-case loss for model risk quantification plot—unemployment rate, CBOE equity volatility index, BBB corporate—five-year treasury bond spread, and total uncommitted loan growth (FDIC SDI Report, Federal Reserve Board 4Q91–4Q15 and Jacobs (2018) models).

**Table 1.**Summary statistics of historical Y9 credit loss rates, banking system idiosyncratic variables, and macroeconomic variables (FDIC SDI Report and Federal Reserve Board 4Q91–4Q15).

Input Variables—Idiosyncratic Variables | Input Variables—Macroeconomic Variables | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|

NCOR | CIL | CDL | TAA | OREO | TTUC | UNEMP | BBBCY | CREPI | VIX | CORPSPR | |

Count | 65 | 65 | 64 | 65 | 65 | 64 | 65 | 65 | 65 | 65 | 65 |

Mean | 0.35% | 0.10 | 0.01 | 0.05 | 0.001 | 0.01 | 6.31 | 5.54 | 201.12 | 26.56 | 2.99 |

StdDev. | 0.23% | 0.01 | 0.05 | 0.01 | 0.0010 | 0.03 | 1.73 | 1.29 | 40.64 | 12.09 | 1.15 |

Min. | 0.13% | 0.09 | −0.09 | 0.03 | 0.001 | −0.09 | 4.10 | 3.70 | 138.70 | 12.70 | 1.40 |

25 Prc. | 0.17% | 0.10 | −0.04 | 0.04 | 0.001 | −0.001 | 5.00 | 4.50 | 167.30 | 18.90 | 2.30 |

Med. | 0.26% | 0.1054 | 0.02 | 0.06 | 0.0009 | 0.013 | 5.70 | 5.50 | 198.50 | 22.70 | 3.00 |

75 Prc. | 0.44% | 0.11 | 0.04 | 0.05 | 0.002 | 0.02 | 7.70 | 6.40 | 234.0 | 30.80 | 3.50 |

Max. | 0.93% | 0.13 | 0.16 | 0.08 | 0.004 | 0.06 | 9.90 | 9.40 | 278.70 | 80.90 | 7.20 |

CV | 0.67 | 0.09 | 7.21 | 0.20 | 0.73 | 4.53 | 0.27 | 0.23 | 0.20 | 0.46 | 0.38 |

Skew | 1.32 | 0.1477 | −0.21 | 0.46 | 0.91 | −1.25 | 0.80 | 0.77 | 0.21 | 1.96 | 1.50 |

Kurt. | 0.67 | −0.24 | −0.06 | 0.16 | −0.59 | 2.05 | −0.72 | 0.34 | −1.12 | 5.51 | 4.13 |

**Table 2.**Correlations of historical Y9 credit loss rates, banking system idiosyncratic variables, and macroeconomic variables (FDIC SDI Report and Federal Reserve Board 4Q91–4Q15).

NCOR | CIL | CDL | TAA | OREO | TTUC | UN-EMP | BBBCY | CREPI | VIX | CORP-SPR | |
---|---|---|---|---|---|---|---|---|---|---|---|

NCOR | |||||||||||

CIL | −0.59 | ||||||||||

CDL | −0.82 | 0.39 | |||||||||

TAA | 0.63 | −0.43 | −0.34 | ||||||||

OREO | 0.69 | −0.59 | −0.84 | 0.06 | |||||||

TUC | −0.64 | 0.13 | 0.61 | −0.07 | −0.59 | ||||||

UNEMP | 0.91 | −0.62 | −0.80 | 0.30 | 0.94 | −0.41 | |||||

BBBCY | 0.52 | −0.25 | −0.31 | 0.48 | 0.13 | −0.17 | 0.07 | ||||

CREPI | −0. 80 | 0.56 | 0.58 | −0.13 | −0.66 | 0.25 | −0.54 | −0.41 | |||

VIX | 0.64 | −0.22 | −0.52 | 0.05 | 0.45 | −0.27 | 0.38 | 0.60 | −0.22 | ||

CORPSPR | 0.69 | −0.45 | −0.55 | 0.08 | 0.45 | −0.21 | 0.61 | 0.47 | −0.25 | 0.81 |

**Table 3.**Vector autoregressive CECL model estimation results compared—historical Y9 credit loss rates, banking system idiosyncratic, and macroeconomic variables (FDIC SDI Report and Federal Reserve Board 4Q91–4Q15).

Statistics | NCOR Lag | Int. | UN-EMP | BBBCY | CREPI | BBBCY | VIX | CILTA | CD-LGR | TAATA | OREO-TA | TUCG |
---|---|---|---|---|---|---|---|---|---|---|---|---|

Esti. | −0.69 | −0.31% | 0.16% | 0.04% | ||||||||

PV | 1.0 × 10^{−10} | 1.0 × 10^{−10} | 1.6 × 10^{−3} | 3.1 × 10^{−4} | ||||||||

AR2 | 92.62% | |||||||||||

Esti. | −0.67 | −0.04% | 0.01% | 0.03% | −0.9% | −0.78% | ||||||

PV | 1.0 × 10^{−10} | 5.2 × 10^{−3} | 1.9 × 10^{−3} | 1.0 × 10^{−3} | 1.0 × 10^{−9} | 1.0 × 10^{−9} | ||||||

AR2 | 96.69% | |||||||||||

Esti. | −0.71 | −0.03% | 0.23% | 0.02% | ||||||||

PV | 1.0 × 10^{−10} | 3.1 × 10^{−2} | 2.0 × 10^{−3} | 1.6 × 10^{−2} | ||||||||

AR2 | 89.69% | |||||||||||

Est. | −0.70 | −0.03% | 0.01% | −0.43% | −0.66% | 4.75% | ||||||

PV | 1.0 × 10^{−10} | 6.2 × 10^{−2} | 2.6 × 10^{−2} | 2.0 × 10^{−13} | 1.6 × 10^{−7} | 1.0 × 10^{−10} | ||||||

AR2 | 95.55% | |||||||||||

Est. | −0.70 | −0.05% | 0.01% | 0.01% | ||||||||

PV | 1.0 × 10^{−10} | 3.5 × 10^{−3} | 3.0 × 10^{−2} | 2.8 × 10^{−3} | ||||||||

AR2 | 87.45% | |||||||||||

Est. | −0.85 | 0.01% | 0.05% | 0.01% | ||||||||

PV | 1.0 × 10^{−10} | 2.8 × 10^{−2} | 3.5 × 10^{−3} | 1.2 × 10^{−2} | ||||||||

AR2 | 94.88% | |||||||||||

Est. | −0.82 | −0.12% | −0.20% | 0.06% | ||||||||

PV | 1.0 × 10^{−10} | 1.1 × 10^{−11} | 6.7 × 10^{−3} | 8.9 × 10^{−7} | ||||||||

AR2 | 87.45% | |||||||||||

Est. | −0.79 | −0.29% | −0.30% | 0.04% | 5.0% | 9.22% | ||||||

PV | 1.0 × 10^{−10} | 1.0 × 10^{−10} | 4.2 × 10^{−7} | 2.5 × 10^{−4} | 1.0 × 10^{−10} | 3.5 × 10^{−3} | ||||||

AR2 | 95.18% | |||||||||||

Est. | −0.89 | 0.003% | 0.00% | 0.06% | ||||||||

PV | 1.0 × 10^{−10} | 4.0 × 10^{−2} | 4.9 × 10^{−3} | 2.3 × 10^{−7} | ||||||||

AR2 | 92.82% | |||||||||||

Est. | −0.88 | −0.18% | 0.001% | 0.01% | −0.61% | |||||||

PV | 1.0 × 10^{−10} | 1.0 × 10^{−10} | 4.9 × 10^{−3} | 2.2 × 10^{−2} | 2.3 × 10^{−2} | |||||||

AR2 | 92.88% | |||||||||||

Est. | −0.79 | −0.13% | 0.01% | 0.02% | −0.22% | |||||||

PV | 1.0 × 10^{−10} | 5.0 × 10^{−15} | 1.2 × 10^{−2} | 1.5 × 10^{−2} | 8.4 × 10^{−4} | |||||||

AR2 | 91.97% | |||||||||||

Est. | −0.85 | 0.03% | 0.01% | 0.02% | −0.25% | −0.54% | ||||||

PV | 1.0 × 10^{−10} | 1.3 × 10^{−2} | 1.5 × 10^{−3} | 4.6 × 10^{−2} | 2.6 × 10^{−7} | 1.2 × 10^{−6} | ||||||

AR2 | 96.44% | |||||||||||

Est. | −0.65 | −0.35% | 0.05% | 0.04% | 0.02% | |||||||

PV | 0 | 0 | 5.0 × 10^{−6} | 1.1 × 10^{−4} | 5.6 × 10^{−4} | |||||||

AR2 | 93.12% | |||||||||||

Esti. | −0.85 | −0.31% | 0.001% | 0.03% | 0.01% | 3.6% | ||||||

PV | 1.0 × 10^{−10} | 1.0 × 10^{−10} | 4.6 × 10^{−4} | 7.9 × 10^{−3} | 2.8 × 10^{−2} | 1.0 × 10^{−10} | ||||||

AR2 | 94.71% |

Time Period | Statistic | Model | |||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|

UNEMP & CORP-SPR | UNEMP, CORP-SPR, TAATA & CDLG | UNEMP & CREPI | UNEMP, CREPI, TAATA & CDLG | UNEMP & CORP-SPR | UNEMP, CORP-SPR & OREOTA | CREPI & VIX | CREPI, VIX, OREOTA & TAATA | VIX, UNEMP & CORP-SPR | VIX, UNEMP, CORPSPR & TUCLG | BBBCY, UNEMP & CREPI | BBBCY, UNEMP, CREPI & CDLG | BBBCY, UNEMP & CORPSPR | BBBCY, UNEMP. CORPSPR & TAATA | ||

Development Sample | GCV | 1.2 × 10^{−10} | 1.3 × 10^{−10} | 1.7 × 10^{−10} | 1.0 × 10^{−11} | 2.0 × 10^{−10} | 1.0 × 10^{−10} | 1.0 × 10^{−10} | 1.0 × 10^{−11} | 1.0 × 10^{−10} | 1.0 × 10^{−10} | 1.0 × 10^{−10} | 6.0 × 10^{−11} | 1.1 × 10^{−10} | 1.1 × 10^{−10} |

SC | 93.1% | 97.1% | 90.1% | 95.9% | 88.9% | 95.2% | 90.0% | 95.7% | 93.4% | 93.7% | 92.8% | 96.8% | 93.7% | 95.3% | |

RMSE | 6.4 × 10^{−4} | 6.6 × 10^{−4} | 7.7 × 10^{−4} | 5.3 × 10^{−4} | 8.2 × 10^{−4} | 5.6 × 10^{−4} | 7.8 × 10^{−4} | 5.5 × 10^{−4} | 6.6 × 10^{−4} | 6.5 × 10^{−4} | 6.7 × 10^{−4} | 4.5 × 10^{−4} | 6.2 × 10^{−4} | 5.7 × 10^{−4} | |

CPE | −0.02% | 0.02% | 0.02% | 0.01% | −0.01% | −0.10% | −0.01% | 0.09% | 0.10% | 0.14% | 0.04% | −0.02% | 0.21% | 0.04% | |

AIC | −603.0 | −589.5 | −591.6 | −583.1 | −584.9 | −589.2 | −590.8 | −579.0 | −561.0 | −561.4 | −598.5 | −631.7 | −606.6 | −576.3 | |

Perfect Forecast Prediction | GCV | 4.8 × 10^{−8} | 2.0 × 10^{−8} | 1.6 × 10^{−9} | 1.6 × 10^{−9} | 3.0 × 10^{−10} | 1.1 × 10^{−8} | 1.5 × 10^{−8} | 2.9 × 10^{−8} | 71.0 × 10^{−9} | 1.1 × 10^{−8} | 8.1 × 10^{−9} | 2.7 × 10^{−10} | 7.1 × 10^{−8} | 4.3 × 10^{−8} |

SC | 0.40% | 3.8% | 5.2% | 7.9% | 50.4% | 0.10% | 10.1% | 42.1% | 30.3% | 19.7% | 1.2% | 68.4% | 2.1% | 6.6% | |

RMSE | 0.0020 | 0.0020 | 0.0004 | 0.0010 | 0.0002 | 0.0010 | 0.0010 | 0.0020 | 0.0010 | 0.0020 | 0.0010 | 0.0002 | 0.0030 | 0.0030 | |

CPE | −86.0% | −50.0% | 5.5% | −30.0% | 4.9% | −40.0% | 38.1% | −67.1% | 29.2% | 35.6% | −32.8% | 5.1% | −103.8% | −67.5% | |

AIC | −79.0 | −82.3 | −106.5 | −89.0 | −119.0 | −88.5 | −88.6 | −79.2 | −92.5 | −86.8 | −91.3 | −116.5 | −74.0 | −76.0 | |

Fed Base Prediction | GCV | 4.0 × 10^{−8} | 2.2 × 10^{−8} | 6.0 × 10^{−10} | 170 × 10^{−8} | 1.1 × 10^{−9} | 2.6 × 10^{−8} | 6.5 × 10^{−3} | 1.9 × 10^{−7} | 7.8 × 10^{−8} | 1.3 × 10^{−7} | 1.8 × 10^{−8} | 4.7 × 10^{−9} | 2.0 × 10^{−8} | 9.1 × 10^{−8} |

SC | 80.2% | 87.3% | 21.8% | 29.0% | 4.9% | 4.5% | 0.7% | 0.2% | 0.8% | 0.44% | 81.8% | 2.0% | 1.4% | 9.7% | |

RMSE | 0.0020 | 0.0020 | 0.0002 | 0.0020 | 0.0003 | 0.0010 | 0.0020 | 0.0100 | 0.0030 | 0.0050 | 0.0020 | 0.0010 | 0.0020 | 0.0040 | |

CPE | −80.1% | −58.9% | −7.4% | −49.0% | 7.4% | −61.0% | −715.0% | −152.0 | −80.0% | −100.6% | −50.6% | 6.1% | 48.9% | −35.4% | |

AIC | −80.5 | −81.2 | −113.9 | −83.2 | −109.2 | −82.0 | −77.8 | −64.3 | −73.2 | −67.2 | −85.4 | −93.7 | −83.9 | −70.0 | |

VAR Model Base Prediction | GCV | 6.7 × 10^{−8} | 4.2 × 10^{−8} | 4.7 × 10^{−10} | 1.7 × 10^{−8} | 1.2 × 10^{−9} | 1.1 × 10^{−8} | 6.3 × 10^{−8} | 2.1 × 10^{−7} | 8.6 × 10^{−8} | 1.4 × 10^{−7} | 2.7 × 10^{−8} | 2.5 × 10^{−9} | 4.6 × 10^{−9} | 1.3 × 10^{−7} |

SC | 23.7% | 27.8% | 13.8% | 28.7% | 9.58% | 4.9% | 0.60% | 0.10% | 0.74% | 0.34% | 45.7% | 9.9% | 12.5% | 7.3% | |

RMSE | 0.0030 | 0.0030 | 0.0002 | 0.0020 | 0.0003 | 0.0010 | 0.0300 | 0.0100 | 0.0030 | 0.0050 | 0.0020 | 0.0010 | 0.0010 | 0.005 | |

CPE | −102.0% | −80.2% | −2.6% | −47.2% | 9.0% | −40.4% | −75.0% | −158.0% | −83.4% | −104.8% | −64.7% | −8.7% | 25.8% | −59.0% | |

AIC | −76.4 | −76.1 | −116.1 | −83.4 | −108.7 | −88.5 | −76.9 | −63.4 | −72.4 | −66.6 | −81.6 | −98.8 | −95.9 | −66.8 | |

MS-VAR Model Base Prediction | GCV | 2.0 × 10^{−8} | 1.5 × 10^{−8} | 320 × 10^{−9} | 1.2 × 10^{−8} | 3.1 × 10^{−9} | 6.3 × 10^{−9} | 4.6 × 10^{−8} | 1.4 × 10^{−7} | 8.1 × 10^{−8} | 1.4 × 10^{−7} | 3.9 × 10^{−9} | 1.8 × 10^{−8} | 3.4 × 10^{−8} | 7.6 × 10^{−8} |

SC | 46.2% | 42.97% | 0.63% | 7.71% | 4.62% | 0.40% | 0.30% | 2.1% | 0.80% | 0.44% | 0.42% | 1.1% | 85.1% | 12.3% | |

RMSE | 0.0010 | 0.0020 | 0.0010 | 0.0020 | 0.0010 | 0.001 | 0.0020 | 0.0100 | 0.00300 | 0.0050 | 0.0010 | 0.0020 | 0.0020 | 0.0040 | |

CPE | −53.9% | −45.6% | 13.0% | −19.0% | 16.9% | −30.5% | −65.0% | −136.0% | −80.3% | −105.9% | −10.6% | 37.4% | −73.8% | −26.9% | |

AIC | −86.0 | −84.37 | −100.6 | −86.4 | −101.0 | −93.3 | −79.3 | −66.5 | −72.9 | −66.4 | −97.2 | −85.1 | −79.7 | −73.4 |

Macroeconomic | Idiosyncratic | Fed | VAR | MS-VAR | Min | Avg | Stdev | Max | |
---|---|---|---|---|---|---|---|---|---|

UNEMP & BBBCY | None | Base | 0.0338% | 0.0448% | 0.0781% | 23.82% | 26.82% | 3.6% | 30.5% |

Worse | 0.0617% | 0.0519% | 0.1146% | ||||||

RMRE | 30.49% | 26.14% | 23.82% | ||||||

UNEMP & BBBCY | CILA & CDLG | Base | 0.0696% | 0.0344% | 0.0920% | 25.61% | 30.17% | 5.6% | 36.5% |

Worse | 0.0956% | 0.0713% | 0.1237% | ||||||

RMRE | 36.47% | 28.43% | 25.61% | ||||||

UNEMP & CREPI | None | Base | 0.1567% | 0.1648% | 0.1913% | 19.87% | 24.65% | 4.2% | 27.7% |

Worse | 0.1642% | 0.1777% | 0.2300% | ||||||

RMRE | 26.20% | 27.88% | 19.87% | ||||||

UNEMP & CREPI | TAATA & CDLG | Base | 0.0868% | 0.0895% | 0.1366% | 25.15% | 31.25% | 5.3% | 34.7% |

Worse | 0.1296% | 0.1364% | 0.2115% | ||||||

RMRE | 33.92% | 34.69% | 25.15% | ||||||

UNEMP & CORPSPR | None | Base | 0.1817% | 0.1845% | 0.1979% | 15.36% | 21.64% | 5.6% | 25.9% |

Worse | 0.2059% | 0.2063% | 0.2289% | ||||||

RMRE | 25.98% | 23.57% | 15.36% | ||||||

UNEMP & CORPSPR | OREOTA | Base | 0.0661% | 0.1008% | 0.1177% | 21.58% | 26.11% | 3.9% | 28.6% |

Worse | 0.1085% | 0.1305% | 0.1329% | ||||||

RMRE | 28.14% | 28.62% | 21.58% | ||||||

CREPI & VIX | None | Base | 0.1129% | 0.1166% | 0.1021% | 17.22% | 25.18% | 6.9% | 29.4% |

Worse | 0.1711% | 0.1700% | 0.1614% | ||||||

RMRE | 29.39% | 28.93% | 17.22% | ||||||

CREPI & VIX | OREOTA & TAATA | Base | 0.1458% | 0.1564% | 0.1187% | 30.92% | 39.35% | 7.4% | 44.6% |

Worse | 0.1026% | 0.1101% | 0.0714% | ||||||

RMRE | 42.60% | 44.53% | 30.92% | ||||||

VIX, UNEMP & CORPSPR | None | Base | 0.1331% | 0.1381% | 0.1361% | 29.65% | 36.78% | 6.3% | 41.7% |

Worse | 0.1398% | 0.1436% | 0.1221% | ||||||

RMRE | 39.00% | 41.69% | 29.65% | ||||||

VIX, UNEMP & CORPSPR | TUCLG | Base | 0.1658% | 0.1718% | 0.1741% | 36.39% | 43.61% | 6.5% | 49.0% |

Worse | 0.2162% | 0.2157% | 0.2091% | ||||||

RMRE | 48.98% | 45.47% | 36.39% | ||||||

BBBCY, UNEMP & CCREP | None | Base | 0.0836% | 0.0598% | 0.1513% | 33.63% | 39.64% | 5.2% | 42.7% |

Worse | 0.1089% | 0.0972% | 0.2006% | ||||||

RMRE | 42.70% | 42.57% | 33.63% | ||||||

BBBCY, UNEMP & CCREP | CDLG | Base | 0.1796% | 0.1545% | 0.2326% | 41.44% | 47.41% | 5.4% | 51.9% |

Worse | 0.2527% | 0.1990% | 0.3305% | ||||||

RMRE | 51.91% | 48.89% | 41.44% | ||||||

BBBCY, UNEMP & CORPSPR | None | Base | 0.2520% | 0.2128% | 0.0444% | 26.26% | 36.68% | 9.0% | 42.4% |

Worse | 0.3330% | 0.2400% | 0.0723% | ||||||

RMRE | 42.36% | 41.42% | 26.26% | ||||||

BBBCY, UNEMP & CORPSPR | TAATA | Base | 0.1766% | 0.1981% | 0.1746% | 42.93% | 46.81% | 3.6% | 50.0% |

Worse | 0.3231% | 0.3283% | 0.3194% | ||||||

RMRE | 47.49% | 50.02% | 42.93% | ||||||

Relative Model Risk Error Summary Statistics—Macroeconomic Scenario Generation Models | Min | 25.98% | 23.57% | 15.36% | |||||

Avg | 37.54% | 36.63% | 27.84% | ||||||

StdDev | 8.75% | 9.22% | 8.48% | ||||||

Max | 51.91% | 50.02% | 42.93% | ||||||

Relative Model Risk Error Summary Statistics—Total Macroeconomic Scenario Generation & Credit Loss Models | Min | 15.36% | |||||||

Avg | 34.01% | ||||||||

StdDev | 9.68% | ||||||||

Max | 51.91% |

© 2020 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Jacobs, M., Jr.
A Holistic Model Validation Framework for Current Expected Credit Loss (CECL) Model Development and Implementation. *Int. J. Financial Stud.* **2020**, *8*, 27.
https://doi.org/10.3390/ijfs8020027

**AMA Style**

Jacobs M Jr.
A Holistic Model Validation Framework for Current Expected Credit Loss (CECL) Model Development and Implementation. *International Journal of Financial Studies*. 2020; 8(2):27.
https://doi.org/10.3390/ijfs8020027

**Chicago/Turabian Style**

Jacobs, Michael, Jr.
2020. "A Holistic Model Validation Framework for Current Expected Credit Loss (CECL) Model Development and Implementation" *International Journal of Financial Studies* 8, no. 2: 27.
https://doi.org/10.3390/ijfs8020027