micm_nlp.tokenizers.architectures

Tokenizer classes that HuggingFace does not ship.

Two tokenizers that exist because a published experiment needed them, both built by wrapping a stock HuggingFace tokenizer rather than by training a new one:

BertByT5Tokenizer

A byte-level ByT5 vocabulary wearing BERT’s special tokens, so byte-level segmentation can drive an encoder-only BERT-style model. From the Georgian tokenization comparison (ICNLSP 2024), where byte-level segmentation was one of the methods evaluated.

CustomXlmRoberta

XLM-R’s multilingual SentencePiece vocabulary re-dressed with the special tokens and post-processor of a different target architecture.

This module mirrors micm_nlp.models.architectures and stays small for the same reason: a tokenizer that can be selected by name through tokenizer.cls needs no code here.

Classes

BertByT5Tokenizer	A byte-level ByT5 vocabulary wearing BERT's special tokens.
CustomXlmRoberta	XLM-R's multilingual vocabulary, re-dressed for another architecture.

Module Contents

class micm_nlp.tokenizers.architectures.BertByT5Tokenizer(byt5_name='google/byt5-small', *args, **kwargs)

Bases: transformers.ByT5Tokenizer

A byte-level ByT5 vocabulary wearing BERT’s special tokens.

ByT5 tokenizes to raw UTF-8 bytes, which sidesteps the vocabulary question entirely – attractive for a language whose script is poorly served by subword vocabularies. But ByT5 is an encoder-decoder tokenizer and does not build the [CLS] ... [SEP] input an encoder-only model expects. This class keeps ByT5’s byte vocabulary and borrows BERT’s special tokens and its build_inputs_with_special_tokens, so byte-level segmentation can drive a BERT-style model.

Note the composition is by delegation, not inheritance of behaviour: the instance copies the configured ByT5 tokenizer’s __dict__ over its own.

Parameters:

• byt5_name – ByT5 checkpoint to take the byte vocabulary from.

• args – forwarded to ByT5Tokenizer.

• kwargs – forwarded to ByT5Tokenizer.

bert

bert_tok_name = 'bert-base-uncased'

byt5

class micm_nlp.tokenizers.architectures.CustomXlmRoberta(model_arch=ModelArchSE.BERT)

XLM-R’s multilingual vocabulary, re-dressed for another architecture.

XLM-R’s SentencePiece vocabulary is the reason to reach for it – 100 languages, trained on CC100. Its input format is often not what you want: a BERT-style encoder expects different special tokens and a different post-processor. This loads xlm-roberta-base’s tokenizer and applies the special tokens and post-processor of model_arch instead.

Like BertByT5Tokenizer, composition is by copying the wrapped tokenizer’s __dict__; this class does not subclass a tokenizer.

Apply model_arch’s special tokens and post-processor to XLM-R’s tokenizer.

Parameters:

model_arch – a ModelArchSE member.

hf_name = 'xlm-roberta-base'

xlmr