Metrics API

RunMetadata

h2ml.evaluation.metrics.RunMetadata dataclass

Domain-specific context attached to results after CV. Keeps metadata out of the CV engine.

Attributes:

Name	Type	Description
schema	Optional[str]	Top-level folder / feature schema / experiment name.
target	Optional[str]	Target column name.
stage	str	Pipeline stage that produced the row, set automatically: - "default": step 1 (all features, default params) - "reduced": step 3 (reduced features, default params) - "optimized": step 4 (reduced or full features, optimized params)
batch	Optional[str]	Optional batch/run identifier (run ID, date, or any grouping label).
y_transform	Optional[str]	Winning y-transform name, set by the transform sweep. None otherwise.
notes	Optional[str]	Free text — useful for an experiments log.

to_dict()

Non-None fields as a dict with capitalised keys, for DataFrame columns.

compute_metrics_all

h2ml.evaluation.metrics.compute_metrics_all(cv_results, metadata=None)

Compute per-fold metrics for a list of CVResults (all models).

Parameters:

Name	Type	Description	Default
cv_results	list[CVResult]	Output of CrossValidator.run_all().	required
metadata	Optional[RunMetadata]	Optional RunMetadata to attach to every row.	None

Returns:

Type	Description
DataFrame	DataFrame with one row per fold per model.

Raises:

Type	Description
RuntimeError	If every model has empty folds (all CV runs failed).

aggregate_metrics

h2ml.evaluation.metrics.aggregate_metrics(fold_df, group_by_stage=True)

Aggregate per-fold results into mean ± std per model.

Parameters:

Name	Type	Description	Default
fold_df	DataFrame	Output of compute_metrics() or compute_metrics_all().	required
group_by_stage	bool	If True (default), group by Stage alongside Model and other metadata columns so each (model, stage) pair gets its own row. Set False to collapse all stages per model — used in step 4 when the optimized run's folds should be averaged without a Stage breakdown.	True

Returns:

Type	Description
DataFrame	DataFrame with one row per model, columns suffixed _Mean and _Std.

select_best

h2ml.evaluation.metrics.select_best(agg_df, metric='AUC_Test_Mean', minimize=False, n_folds=None)

Select the best model from an aggregated metrics DataFrame.

When n_folds is provided, ranking uses a lower/upper confidence bound (LCB for maximize metrics, UCB for minimize metrics):

maximize:  score = mean - std / sqrt(n_folds)   → idxmax
minimize:  score = mean + std / sqrt(n_folds)   → idxmin

This prefers models whose conservative-bound performance is highest, penalising fold variance in proportion to 1/sqrt(n_folds). When n_folds is None the raw mean is used (original behaviour).

Parameters:

Name	Type	Description	Default
agg_df	DataFrame	Output of aggregate_metrics().	required
metric	str	Column name to rank by (must end in _Mean, e.g. AUC_Test_Mean).	'AUC_Test_Mean'
minimize	bool	If True, select the row with the lowest value (e.g. RMSE, MAE).	False
n_folds	Optional[int]	Number of CV folds used to produce agg_df. When provided, selection uses confidence-bound scoring instead of raw mean.	None

Returns:

Type	Description
dict	Dict with keys model_name, metric, value, row.

Raises:

Type	Description
ValueError	If metric is not a column of agg_df.
RuntimeError	If all values in the metric column are NaN (every model failed all CV folds).

compare_results

h2ml.evaluation.compare.compare_results(results, labels=None, metric=None, n_folds=None)

Summarise multiple PipelineResult objects as a single comparison DataFrame.

One row per result. Columns:

Run                — label (from labels, or Run_0 / Run_1 / …)
Metric             — short metric name used for Score_Mean (e.g. "AUC", "R2")
Best_Model         — winning model class name
Best_Stage         — default | reduced | optimized
Y_Transform        — winning y-transform, or None
Score_Mean         — mean CV score for the selected metric
Score_Std          — fold std for the selected metric
Conservative_Bound — variance-penalised score used for ranking. Direction is
                     derived automatically from the metric name via METRIC_MINIMIZE:
                       higher-is-better: Score_Mean - Score_Std / sqrt(n_folds)
                       lower-is-better:  Score_Mean + Score_Std / sqrt(n_folds)
                     A model with high mean but high variance is pulled toward
                     a worse value, so stable models are preferred over
                     optimistically-estimated ones.
Brier_Mean         — Brier_Test_Mean from the winning stage's agg_df
                     (classification only; NaN otherwise)
OOF_Brier          — Brier score computed on the assembled OOF predictions
                     (more robust than the fold-averaged Brier_Mean;
                     classification only; NaN otherwise)
N_Features         — feature count in the winning feature stage
Completed_Steps    — list of pipeline steps that finished

Parameters:

Name	Type	Description	Default
results	list['PipelineResult']	List of PipelineResult objects to compare.	required
labels	Optional[list[str]]	Optional run labels (same length as results). Defaults to "Run_0", "Run_1", …	None
metric	Optional[str]	Short metric name to use as the common comparison column (e.g. "AUC", "R2", "RMSE"). When provided, Score_Mean and Score_Std are read from the <metric>_Test_Mean/Std columns of each result's winning-stage agg DataFrame, making cross-run scores directly comparable even when runs were optimised on different metrics. When omitted, Score_Mean falls back to each result's best_model_value. Minimisation direction and sort order are derived automatically from METRIC_MINIMIZE — no need to pass ascending manually.	None
n_folds	Optional[int]	Number of CV folds used in all runs. When provided it overrides automatic inference from the fold DataFrames, which can fail if results were loaded from disk or the fold DataFrames are absent. Conservative_Bound cannot be computed without a fold count and will fall back to Score_Mean if both this argument and inference fail.	None

Returns:

Type	Description
DataFrame	DataFrame sorted by Conservative_Bound (falls back to Score_Mean when
DataFrame	fold count cannot be inferred), one row per result.

Raises:

Type	Description
ValueError	If labels length does not match results, or if metric is None while the results were scored on more than one metric.

Example

r1 = pipeline_a.run(store) r2 = pipeline_b.run(store) compare_results([r1, r2], labels=["baseline", "spatial_cv"], metric="AUC")