openquant.evaluation

Pure Python. Signatures are as written in the source; descriptions are the docstrings. All Python modules.

Research evaluation: is a backtest's Sharpe ratio evidence of skill?

A thin layer over the compiled backtest_stats and strategy_risk modules that works from a returns series instead of from pre-computed moments:

• probabilistic_sharpe_ratio, deflated_sharpe_ratio, minimum_track_record_length (Bailey and López de Prado 2012, 2014; AFML Chapter 14, §14.7);
• meta_label_metrics: precision, recall and F1 of a meta-labeling overlay against the primary model it filters (AFML Chapter 3, §3.6);
• strategy_failure_probability (AFML Chapter 15, §15.4);
• TrialRegistry: a JSON file that records every configuration tried, across runs, so the deflated Sharpe ratio deflates by the number of trials actually run.

Conventions, shared with backtest_stats: every Sharpe ratio here is **per period** (mean over sample standard deviation, not annualised), skewness and kurtosis are the population moment ratios, and kurtosis is **raw** (3 for a normal distribution).

Classes

ReturnMoments

class ReturnMoments

The sample statistics PSR, DSR and MinTRL are computed from.

Trial

class Trial

One configuration's record in a TrialRegistry.

TrialRegistry

class TrialRegistry

Every configuration tried, persisted to a JSON file so the count survives restarts.

Record each backtest with record(config, returns). The file is rewritten atomically (a temporary file in the same directory, then os.replace) and re-read before every write, so separate processes or notebook sessions pointed at the same path share one count. Recording a configuration that is already registered replaces its entry: re-running the same notebook does not inflate the trial count. Two processes recording at the same instant can still lose one write; the registry is not a database.

TrialRegistry.__init__

__init__(path: str | os.PathLike[str]) -> None

TrialRegistry.path

path() -> Path

TrialRegistry.trials

trials() -> list[Trial]

The registered trials, oldest first.

TrialRegistry.n_trials

n_trials() -> int

TrialRegistry.reload

reload() -> None

Re-read the file, picking up trials other processes have recorded.

TrialRegistry.record

record(config: Any, returns: Iterable[float]) -> Trial

Register one configuration's backtest returns and persist the registry.

Constant returns (for example all zeros from a configuration that took no bet) have no Sharpe ratio, but the configuration was still tried, so it is recorded rather than rejected: with Sharpe 0, skewness 0 and kurtosis 3 (the normal values). Sharpe 0 is what a strategy with no excess return earns. The trial counts towards n_trials and its 0 enters sharpe_std(), so deflated_sharpe_ratio deflates by it like any other trial. Dropping it instead would undercount the trials and deflate too weakly.

TrialRegistry.sharpe_std

sharpe_std() -> float

Standard deviation (population) of the registered per-period Sharpe ratios.

TrialRegistry.expected_max_sharpe

expected_max_sharpe() -> float

SR0 for this registry's trial count and Sharpe-ratio dispersion.

TrialRegistry.deflated_sharpe_ratio

deflated_sharpe_ratio(returns: Iterable[float]) -> float

DSR of returns, deflated by every trial in the registry.

Record the evaluated configuration too: it is one of the trials.

Functions

config_hash

config_hash(config: Any) -> str

SHA-256 of the configuration's canonical JSON (sorted keys, no whitespace).

Two configurations hash equal exactly when they serialise to the same JSON, so key order does not matter but 1 and 1.0 differ. Raises TypeError for a configuration JSON cannot represent.

deflated_sharpe_ratio

deflated_sharpe_ratio(
    returns: Iterable[float],
    trial_sharpes: Sequence[float] | None = None,
    *,
    n_trials: int | None = None,
    sharpe_std: float | None = None,
) -> float

DSR: the PSR measured against SR0, the best Sharpe ratio luck alone would produce.

Give the trials either as trial_sharpes — every configuration's per-period Sharpe ratio, including this one — or as n_trials and sharpe_std, the count and the standard deviation of their Sharpe ratios. TrialRegistry.deflated_sharpe_ratio takes both from a registry.

expected_max_sharpe

expected_max_sharpe(n_trials: int, sharpe_std: float) -> float

SR0: the expected maximum per-period Sharpe ratio of n_trials skill-less trials.

SR0 = σ_SR·[(1 - γ)·Z⁻¹(1 - 1/N) + γ·Z⁻¹(1 - 1/(N·e))], γ the Euler–Mascheroni constant.

meta_label_metrics

meta_label_metrics(
    meta_labels: Sequence[int],
    meta_predictions: Sequence[float],
    threshold: float = 0.5,
) -> dict[str, float | int]

Precision, recall and F1 of a meta-labeling overlay, and of the primary model alone.

meta_labels[i] is 1 when the primary model's bet i was profitable, else 0. meta_predictions[i] is the secondary model's probability (or 0/1 decision) that it was; the overlay takes the bet when it is at least threshold. The primary model alone takes every bet, so its precision is the share of profitable bets and its recall is 1. A ratio with a zero denominator is reported as 0.0.

minimum_track_record_length

minimum_track_record_length(
    returns: Iterable[float],
    benchmark_sr: float = 0.0,
    alpha: float = 0.05,
) -> float

MinTRL: observations needed for PSR(benchmark_sr) to reach 1 - alpha.

MinTRL = 1 + (1 - γ3·SR + (γ4 - 1)/4·SR²)·(Z(1 - α) / (SR - SR*))².

Returns math.inf when the observed Sharpe ratio is at or below the benchmark: no track record of this quality is long enough.

probabilistic_sharpe_ratio

probabilistic_sharpe_ratio(returns: Iterable[float], benchmark_sr: float = 0.0) -> float

PSR: the probability that the true per-period Sharpe ratio exceeds benchmark_sr.

PSR(SR*) = Z[(SR - SR*)·sqrt(T - 1) / sqrt(1 - γ3·SR + (γ4 - 1)/4·SR²)].

return_moments

return_moments(returns: Iterable[float]) -> ReturnMoments

Per-period Sharpe ratio, skewness and raw kurtosis of a returns series.

std is the sample standard deviation (n - 1); skewness and kurtosis are the population ratios m3/m2^1.5 and m4/m2^2, as in Bailey and López de Prado.

strategy_failure_probability

strategy_failure_probability(
    bet_outcomes: Iterable[float],
    years_elapsed: float,
    target_sharpe: float,
    investor_horizon_years: float,
    *,
    bootstrap_iterations: int = 1000,
    seed: int = 42,
    kde_bandwidth: float | None = None,
) -> dict[str, Any]

Probability that the strategy misses an annualised target_sharpe (AFML §15.4).

Calls strategy_risk.estimate_strategy_failure_probability on the per-bet outcomes and returns its report; failure_probability is its empirical_failure_probability, the share of bootstrapped precisions at or below the precision the target requires.