Validation Guidelines

Accurate scenario analysis requires layering validation from deterministic mathematics through empirical comparisons. This document summarises the validation regime currently supported by the simulator and highlights planned extensions.

Tiered Validation Approach

Mathematical Validation

  • Deterministic baselines – For deals with fixed inputs, verify simulated amortisation schedules, cash flows, NPV, and IRR against independent spreadsheets or analytic formulas. Report any variance outside numerical tolerances.

  • DSCR checks – Confirm debt-service coverage ratio uses the intended income measure divided by total debt service each period. Cross-check against manual calculations for representative deals.

  • IRR confirmation – For each scenario, evaluate the NPV at the solved IRR and ensure it is approximately zero to validate the root-finding routine.

  • Distribution sampling – Run statistical tests (e.g. KS test) on sampled distributions. When truncation is configured, compute the conditional moments of the truncated distribution and compare to empirical sample statistics. Document deviations and adjust inputs or implementations as needed.

  • Exposure construction – Independently calculate exposure schedules that combine collateral, insurance, and revenue offsets for deterministic deals. Ensure simulator outputs match manual schedules period by period, including negative contributions when collateral is over-funded.

Statistical Validation of the Math Package

  • Parameter regression tests – Exercise each DistributionSpec mode (direct parameters, quantile fits, histograms, empirical samples) via unit and stochastic tests to confirm the generated random variables honour configured truncation and quantisation policies.

  • Fit verification – For fitted distributions (normal, lognormal, truncated normal, beta), compare quantiles and moments against ground-truth references to ensure the fitting routines recover parameters within tolerance.

  • Sampling diagnostics – Use large-sample draws to verify sample means/variances stay within statistical error bands derived from the theoretical distribution, accounting for quantisation rounding error.

  • Truncation integrity – Validate that conditional truncation and clipping respect bounds across PDFs, CDFs, inverse-CDF sampling, and support calculations.

  • Utility coverage – Keep regression tests around helper utilities (e.g. hazard-rate curves, prepayment-rate curves) so scenario engines receive well-behaved probability paths when stitching the math package into simulations.

Scenario Validation

  • Edge cases – Stress extreme inputs (very high rates, zero residual value, immediate default) to confirm graceful handling of boundary conditions.

  • Sensitivity sweeps – Vary one assumption at a time (interest rate, default probability, etc.) and confirm resulting metrics move in the expected direction (e.g. higher rates reduce NPV).

  • Correlation and prepayment – When correlation matrices are provided, ensure empirical correlations of simulated defaults or prepayments align with configured values; verify independence when no matrix is supplied. For full prepayments, test both constant-rate and PSA models to confirm timing distributions and outstanding-balance payouts. For partial prepayments, confirm balances step down by the specified fractions and macro factors shift event frequencies according to the logistic formulation.

  • Residual value & depreciation – Compare simulated residual paths against analytic depreciation curves (straight-line, declining balance, etc.). When macro or idiosyncratic factors are supplied, ensure cross-deal correlations match expectations. Validate obsolescence shocks trigger with the correct timing and magnitude.

  • Planned validation extensions for upcoming metrics, operational drivers, and other roadmap items are tracked in TODO.md.

  • Temporal dynamics – Test staggered origination dates and multiple time axes to ensure cash flows start on the correct dates, hazards apply on the proper clocks (deal-age, portfolio-age, calendar time), and equivalent relative timing produces identical outputs regardless of start date selection.

  • Exposure & insurance – Construct scenarios with defaults, liquidation shortfalls, insured values, collateral schedules, and revenue/fee curves. Confirm insurance claims, collateral contributions, and fee offsets reduce exposure as intended and that negative contributions return cash when collateral accounts are over-funded.

Empirical Validation

  • Align default and loss-given-default distributions, residual values, and other stochastic drivers to observed data or published studies when available. Calibrate parameters so simulated loss distributions replicate historical experience, documenting assumptions and any gaps caused by limited data.

Cross-Validation with Other Models

  • Compare simulator outputs to spreadsheet implementations or commercial tools for identical deals. Investigate unexplained differences and attribute them to stochastic features or modelling choices.

  • For correlation structures, cross-check portfolio loss distributions against analytic frameworks (e.g. Gaussian copula approximations) to ensure dependence assumptions are implemented correctly.

Extended Validation Plan

Roadmap items that expand the scope of validation (e.g. deterministic vs stochastic comparisons, distribution monitoring, fairness audits, backtesting, report verification) are documented in TODO.md and implemented as the corresponding product capabilities land.

Operating the Regime

  • Maintain regression tests across deterministic and stochastic suites, expanding coverage as new validation capabilities ship.

  • Keep notebooks and scripts for sensitivity, correlation, and empirical checks under version control so reviewers can reproduce validation steps.

  • Document outstanding gaps and roadmap items (e.g. additional loss metrics, operational risk modelling) in TODO.md to preserve traceability between planned features and validation coverage.