chemicalchecker.tool.targetmate.nonconformist.evaluation.run_experiment

run_experiment(models, csv_files, iterations=10, folds=10, fit_params=None, scoring_funcs=None, significance_levels=None, normalize=False, verbose=False, header=0)

Performs a cross-validation evaluation of one or several conformal predictors on a collection of data sets in csv format.

Parameters

modelsobject or iterable

Conformal predictor(s) to evaluate.

csv_filesiterable

List of file names (with absolute paths) containing csv-data, used to evaluate the conformal predictor.

iterationsint

Number of iterations to use for evaluation. The data set is randomly shuffled before each iteration.

foldsint

Number of folds to use for evaluation.

fit_paramsdictionary

Parameters to supply to the conformal prediction object on training.

scoring_funcsiterable

List of evaluation functions to apply to the conformal predictor in each fold. Each evaluation function should have a signature scorer(prediction, y, significance).

significance_levelsiterable

List of significance levels at which to evaluate the conformal predictor.

verboseboolean

Indicates whether to output progress information during evaluation.

Returns

scorespandas DataFrame

Tabulated results for each data set, iteration, fold and evaluation function.