micm_nlp.evals.eval

Metrics: what is computed after each evaluation, and on what.

get_compute_metrics builds the compute_metrics callable handed to the HF Trainer. Between raw model output and a number there are several configurable steps, all driven by task.preproc_rules: decode or flatten predictions, drop padded positions, convert label ids to names or back, verify predictions and labels line up, group by task, and finally compute whatever task.metric_groups names.

get_preprocess_logits_for_metrics returns the companion hook that runs before logits leave the GPU. Taking the argmax there is what keeps an evaluation from accumulating full logits over the whole split. It is also where preproc_rules.label_restricted_likelihood restricts the answer-slot argmax to the candidate label tokens in ds.label.names, lm-eval-harness multiple_choice style, while returning the same prediction shape as the ordinary causal-LM path — so nothing downstream needs to branch.

ds_split may be passed as a callable thunk instead of a Dataset; it is resolved once at call time, which is how the split gets reordered to match a batch sampler that does not yield in dataset order.

Attributes

labels_k
predictions_k

Functions

add_prefix_to_metrics(results, prefix)	Prefix every metric name, e.g. accuracy -> eval_accuracy.
compute_metrics_by_metric_groups(predictions, labels, ...)	Compute every metric group named in task.metric_groups.
convert_label_names_to_floats(predictions, labels)	Parse decoded label strings as floats, for regression-style tasks.
convert_label_names_to_ids(predictions, labels)	Map decoded label strings to class ids.
count_decoded_unknown_label_predictions(predictions, ...)	In case of decoded predictions and labels
get_compute_metrics(config, label_pad_id, ...)	Build the compute_metrics callable the HuggingFace Trainer expects.
get_preprocess_logits_for_metrics(config[, ...])	Build the HF Trainer preprocess_logits_for_metrics hook.
group_preds_labels(predictions, labels, ds_split, group_by)	Split predictions and labels into per-group arrays for multi-task scoring.
postproc_metrics(results, config, add_prefix)	Make a metrics dict JSON-safe and prefix its keys.
preproc_preds_labels(predictions, labels, config, ...)	Run predictions and labels through task.preproc_rules before scoring.
verify_labels_match(ds_split, labels, config)	Verify that the labels match the dataset split labels

Module Contents

micm_nlp.evals.eval.add_prefix_to_metrics(results, prefix)

Prefix every metric name, e.g. accuracy -> eval_accuracy.

micm_nlp.evals.eval.compute_metrics_by_metric_groups(predictions, labels, config)

Compute every metric group named in task.metric_groups.

Each group names its metrics by string and they are loaded through evaluate.combine; with eval.per_task set, a group scores only its own task’s rows and its metric names are prefixed with the task name. Groups whose rows are absent are skipped with a message.

Raises:

ValueError – if no group produced anything – an empty metric dict is almost always a misconfiguration rather than a real result.

micm_nlp.evals.eval.convert_label_names_to_floats(predictions, labels)

Parse decoded label strings as floats, for regression-style tasks.

Note the nested string_to_float helper is defined but not used: the conversion below calls float() directly, so a prediction that does not parse raises rather than falling back to -1.0.

micm_nlp.evals.eval.convert_label_names_to_ids(predictions, labels)

Map decoded label strings to class ids.

The class list is derived from the labels, sorted – so the id space comes from the gold data, and a prediction outside it maps to -1 rather than inventing a class. How often that happens is printed, since a high unknown rate means the model is not producing label-shaped text at all.

micm_nlp.evals.eval.count_decoded_unknown_label_predictions(predictions, label_classes)

In case of decoded predictions and labels

micm_nlp.evals.eval.get_compute_metrics(config, label_pad_id, metric_prefix, eval_path, tokenizer, ds_split)

Build the compute_metrics callable the HuggingFace Trainer expects.

A closure rather than a method, because Trainer calls it with only (predictions, labels) – everything else it needs has to be captured here.

Parameters:

• config – the run config; supplies the metric groups and preprocessing rules.

• label_pad_id – label padding id, excluded from scoring.

• metric_prefix – prefix for the returned metric names (eval_, test_).

• eval_path – directory for artefacts such as the confusion matrix.

• tokenizer – needed when the rules ask for decoding.

• ds_split – the split being scored. May be a thunk, so ordering decisions can be deferred until the dataloader exists – a length-sorted batch sampler yields rows in a different order than the dataset. It is resolved once, so verification and preprocessing see one snapshot.

Returns:

the compute_metrics function.

micm_nlp.evals.eval.get_preprocess_logits_for_metrics(config, num_virtual_tokens=None, tokenizer=None)

Build the HF Trainer preprocess_logits_for_metrics hook.

Reduces raw logits to the prediction shape get_compute_metrics consumes, BEFORE they are cached for the eval loop. The two form a pair: whatever shape this emits is exactly what compute_metrics expects to receive.

micm_nlp.evals.eval.group_preds_labels(predictions, labels, ds_split, group_by)

Split predictions and labels into per-group arrays for multi-task scoring.

Parameters:

group_by – dataset column holding the group id (usually the task id).

Returns:

(grouped_preds, grouped_labels), each keyed by group id plus an 'all' key holding the ungrouped arrays, so overall and per-task metrics come from one pass.

micm_nlp.evals.eval.postproc_metrics(results, config, add_prefix)

Make a metrics dict JSON-safe and prefix its keys.

numpy arrays become lists and numpy scalars become Python scalars, so the result survives being written to disk and logged.

micm_nlp.evals.eval.preproc_preds_labels(predictions, labels, config, label_pad_id, tokenizer, ds_split)

Run predictions and labels through task.preproc_rules before scoring.

The steps are applied in a fixed order: flatten, drop padded positions, decode ids to text, strip and lowercase, then convert label names to floats or ids. Each is off unless the rules turn it on, so a task that predicts ids directly passes through untouched.

Returns:

the processed (predictions, labels), grouped by task when per_task is set.

micm_nlp.evals.eval.verify_labels_match(ds_split, labels, config)

Verify that the labels match the dataset split labels

micm_nlp.evals.eval.labels_k = 'references'

micm_nlp.evals.eval.predictions_k = 'predictions'