micm_nlp.evals.metrics.multirc

MultiRC scoring through the official SuperGLUE scorer.

evaluate.load('super_glue', 'multirc') will not accept a flat prediction array: each prediction must be tagged with its (paragraph, question, answer) index triple. This module rebuilds those triples from the idx/paragraph, idx/question and idx/answer columns of ds_split, coercing predictions and labels to 0/1 along the way.

Functions

compute_multirc(predictions, labels, ds_split)

Score MultiRC with the official SuperGLUE metric.

Module Contents

micm_nlp.evals.metrics.multirc.compute_multirc(predictions, labels, ds_split)

Score MultiRC with the official SuperGLUE metric.

evaluate.load('super_glue', 'multirc') will not take a flat prediction array: MultiRC scores per question, so each prediction has to carry the paragraph/question/answer indices it belongs to. This rebuilds that structure from ds_split’s idx/* columns and coerces predictions and labels to 0/1, accepting either booleans or the strings 'true'/'1'.

Parameters:

  • predictions – one prediction per answer row.

  • labels – gold labels, same order.

  • ds_split – the split being scored; supplies idx/paragraph, idx/question and idx/answer.

Returns:

the official scorer’s output (exact match and F1a).

Raises:

Exception – re-raised after printing the offending sample, if a row is missing its index columns.