Function mean_decrease_accuracy 

pub fn mean_decrease_accuracy<C: SimpleClassifier>(
    model: &mut C,
    x: &[Vec<f64>],
    y: &[f64],
    feature_names: &[String],
    splits: &[(Vec<usize>, Vec<usize>)],
    sample_weight: Option<&[f64]>,
    scoring: Scoring,
    seed: u64,
) -> Result<BTreeMap<String, ImportanceStats>, FeatureImportanceError>

Mean decrease accuracy (AFML Snippet 8.3): for each split, fit on the train rows, score the test rows, then score them again with one feature column shuffled; importance is the relative loss of score. Shuffles draw from a StdRng seeded with seed, so a given seed always gives the same result.

For each split in splits ((train_indices, test_indices) into the rows of x), model is fitted on the training rows (with their sample_weight), the test rows are scored, and then each feature column is shuffled in turn within the test rows and scored again. The per-fold importance is (base - perm) / (0 - perm) for Scoring::NegLogLoss and (base - perm) / (1 - perm) for Scoring::Accuracy and Scoring::F1; it is 0 when the denominator is 0 or the ratio is not finite. Test-fold scores are weighted by sample_weight. Accuracy and F1 use SimpleClassifier::predict (so an override is honoured), negative log loss uses SimpleClassifier::predict_proba. The result is the mean over folds and its standard error (sample deviation, ddof 1, over sqrt(n_folds)).

1 means damaging the feature destroyed everything the model had, 0 that the model did not need it, negative that it did better without it. Because the shuffle stays within each test fold, a very persistent feature is somewhat understated.

Errors

• FeatureImportanceError::EmptyXy if x or y is empty.
• FeatureImportanceError::XyLengthMismatch if x.len() != y.len().
• FeatureImportanceError::LengthMismatch ("feature_names") if the first row of x does not have one entry per feature name.
• FeatureImportanceError::RaggedX if the rows of x differ in length.
• FeatureImportanceError::CrossValidation wrapping CrossValidationError::SplitIndexOutOfRange if a split index is not a row of x, CrossValidationError::LengthMismatch if sample_weight does not have one entry per row, or CrossValidationError::PredictionCountMismatch if model does not return one prediction per test row.

use openquant::cross_validation::{Scoring, SimpleClassifier};

/// A fixed rule: P(y = 1) is the first column. `fit` learns nothing.
struct FirstColumn;
impl SimpleClassifier for FirstColumn {
    fn fit(&mut self, _x: &[Vec<f64>], _y: &[f64], _w: Option<&[f64]>) {}
    fn predict_proba(&self, x: &[Vec<f64>]) -> Vec<f64> {
        x.iter().map(|r| r[0]).collect()
    }
}

let y = [0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0];
let (a, b): (Vec<usize>, Vec<usize>) = ((0..8).collect(), (8..16).collect());
let splits = vec![(b.clone(), a.clone()), (a, b)];
let names: Vec<String> = ["signal", "noise"].map(String::from).to_vec();

let x: Vec<Vec<f64>> = y.iter().enumerate().map(|(i, v)| vec![*v, i as f64 / 16.0]).collect();
let mda = openquant::feature_importance::mean_decrease_accuracy(
    &mut FirstColumn, &x, &y, &names, &splits, None, Scoring::Accuracy, 7,
)?;
// The model never reads the second column, so shuffling it changes nothing.
assert_eq!(mda["noise"].mean, 0.0);
assert_eq!(mda["noise"].std, 0.0);
// Shuffling the first column costs every correct prediction it moves: (1 - s) / (1 - s) = 1
// in each fold. (A shuffle can leave a fold's four 0s and four 1s where they were, one
// chance in 70; that fold would score 0, the value given to a zero denominator.)
assert_eq!(mda["signal"].mean, 1.0);