micm_nlp.evals.metrics.log_likelihood

Log-likelihood ranking for multiple-choice QA.

Each candidate answer is a separate sample. predictions carries the sequence log-likelihood in column 0 and the sequence length in column 1. Candidates are grouped by task.metric_groups[0].args.group_by (e.g. a question id), the highest length-normalised likelihood in each group wins, and accuracy is the fraction of groups whose winner carries the configured correct_flag.

This is the ranking scheme. The alternative — restricting the answer-slot argmax to the candidate label tokens — lives in micm_nlp.evals.eval behind preproc_rules.label_restricted_likelihood.

Functions

compute_log_likelihood_accurac(predictions, labels, ...)

Score multiple-choice accuracy by comparing per-option log-likelihoods.

Module Contents

micm_nlp.evals.metrics.log_likelihood.compute_log_likelihood_accurac(predictions, labels, config, ds_split)

Score multiple-choice accuracy by comparing per-option log-likelihoods.

A generative model answers a multiple-choice question by being scored on every option: each option is one row, and rows belonging to the same question share a group id. The chosen answer is the highest-scoring row in the group, and the question is correct if that row is the one flagged correct.

Log-likelihood is length-normalised before comparing – dividing by the sequence length, floored at 1 – because otherwise the shortest option wins almost by construction.

group_by and correct_flag name the dataset columns, and come from task.metric_groups[0].args.

Parameters:

  • predictions – array with sequence log-likelihood in column 0 and sequence length in column 1.

  • labels – unused; present for the compute_metrics signature.

  • config – the run config, read for the metric-group args.

  • ds_split – the split being scored, read for the group and correctness columns.

Returns:

accuracy, mean_ll and mean_normalized_ll.