micm_nlp.tokenizers.decoding

Label-aware decoding helpers.

decode and batch_decode drop every position equal to label_pad_id before handing token ids to the tokenizer, so the ignore index used in label tensors (-100 by convention) never reaches it and never shows up as text.

Functions

batch_decode(texts, tokenizer, label_pad_id[, ...])	Batched decode(): one string per row, each stripped.
decode(texts, tokenizer, label_pad_id[, ...])	Decode one label sequence, dropping the padding the loss ignored.

Module Contents

micm_nlp.tokenizers.decoding.batch_decode(texts, tokenizer, label_pad_id, skip_special_tokens=True)

Batched decode(): one string per row, each stripped.

Parameters:

• texts – 2-D tensor of token ids, one label sequence per row.

• tokenizer – tokenizer to decode with.

• label_pad_id – the id used as label padding.

• skip_special_tokens – passed through to the tokenizer.

Returns:

one decoded string per row.

micm_nlp.tokenizers.decoding.decode(texts, tokenizer, label_pad_id, skip_special_tokens=True)

Decode one label sequence, dropping the padding the loss ignored.

Label tensors are padded with label_pad_id (-100 by convention) rather than the tokenizer’s pad id, so they cannot be handed to tokenizer.decode directly – the padding is not a valid token id. This masks it out first.

Parameters:

• texts – a single 1-D tensor of token ids.

• tokenizer – tokenizer to decode with.

• label_pad_id – the id used as label padding.

• skip_special_tokens – passed through to the tokenizer.

Returns:

the decoded string, stripped.