micm_nlp.tokenizers.tokenizer¶

Tokenizer factory, special-token plumbing, and tokenizer training.

load(config) is what the pipeline calls. It resolves a path from tokenizer.source / name / type / algorithm, hands it to AutoTokenizer.from_pretrained, and passes tokenizer.args through verbatim.

add_special_tokens, add_post_processor and replace_unk_token_manually adapt a tokenizer to a target architecture’s input format, driven by the per-family token tables in micm_nlp.enums. They are what allow a vocabulary trained for one model family to feed a model from another.

TokenizerTrainer trains a tokenizer from a corpus — HuggingFace WordPiece, byte-level BPE, or native SentencePiece — and saves an LM-adapted copy alongside it. Together with tokenize_sentences/tokenize_words this is the machinery behind the Georgian tokenization comparison (ICNLSP 2024); the fine-tuning path needs only load.

Attributes¶

`fasttokenizer_pref`
`post_processors_map`
`spacy`
`special_tokens_map`
`tokenizers_args_map`
`tokenizers_map`
`whitespace_tokenize`

Classes¶

TokenizerTrainer

Train a new tokenizer from a corpus, rather than load an existing one.

Functions¶

`add_post_processor`(tokenizer, lm_arch)	Add post processing to tokenizer
`add_special_tokens`(tokenizer, lm_arch)	Add special tokens to tokenizer
`gen_path`(source, name, type[, algorithm])	Get Tokenizer Path According Type, Algorithm and Name
`load`(config)	Load Pretrained AutoTokenizer According Language Models Configuration.
`replace_unk_token_manually`(tokenizer, lm_arch)	Replace unk_token with new_unk_token
`to_lm_fasttok`(tokenizer, lm_arch)	Save tokenizer as fast tokenizer
`tokenize_sentences`(text[, method])	Split text into sentences with the chosen splitter.
`tokenize_words`(text[, method])	Split text into words -- NLTK's punctuation-aware tokenizer, or whitespace.
`vocab_size_by_lm_size`(lm_param_size, lm_embedding_size)	Calculate vocabulary size according to language model size

Module Contents¶

class micm_nlp.tokenizers.tokenizer.TokenizerTrainer(config)¶

Train a new tokenizer from a corpus, rather than load an existing one.

The counterpart of the load path in this module: given corpus files and a vocabulary size, it trains SentencePiece (natively), or a HuggingFace WordPiece or byte-level BPE model, and writes it under artefacts/tokenizers. Which one is decided by model.type – see TokTypeSE.

Initiate Tokenizer For Training New Tokenizer Model

save_lm_adapted_hf_tokenizers(tokenizer)¶: Save WordPiece as fast tokenizers For BERT and ELECTRA language model architecture

save_lm_adapted_native_sentpieces()¶: Convert Sentencepiece to fast tokenizers And save for each language model architectures (BERT, ELECTRA, ROBERTA, XLMR, XLNET)

train()¶

Train the tokenizer named by model.type and save it.

Raises:: ValueError – on a type this trainer cannot train. Only native SentencePiece, WordPiece and byte-level BPE are implemented – the others in TokTypeSE are recognised by the config layer but have no trainer here.

train_hf_byte_level_bpe()¶: Train ByteLevelBPETokenizer

train_hf_wordpiece()¶: Train BertWordPieceTokenizer

train_native_sentpiece()¶: Train SentencePiece tokenizer

min_frequency = 2¶

sentpiece_pref = 'sp'¶

type¶

unigram_pref = 'unigram'¶

vocab_size¶

micm_nlp.tokenizers.tokenizer.add_post_processor(tokenizer, lm_arch)¶: Add post processing to tokenizer tokenizer._tokenizer is the underlying Tokenizer (from tokenizers library)

micm_nlp.tokenizers.tokenizer.add_special_tokens(tokenizer, lm_arch)¶: Add special tokens to tokenizer

micm_nlp.tokenizers.tokenizer.gen_path(source, name, type, algorithm=None)¶: Get Tokenizer Path According Type, Algorithm and Name

micm_nlp.tokenizers.tokenizer.load(config)¶

Load Pretrained AutoTokenizer According Language Models Configuration.

Relevant config keys:

tokenizer:
    source:       local       | huggingface
    name:         local_dir   | hf_name
    type:         wordpiece   | sentpiece
    parralelism:  'false'     | 'true'

micm_nlp.tokenizers.tokenizer.replace_unk_token_manually(tokenizer, lm_arch)¶: Replace unk_token with new_unk_token

micm_nlp.tokenizers.tokenizer.to_lm_fasttok(tokenizer, lm_arch)¶: Save tokenizer as fast tokenizer

micm_nlp.tokenizers.tokenizer.tokenize_sentences(text, method=SentTokTypeSE.NLTK)¶

Split text into sentences with the chosen splitter.

KA selects the Georgian splitter, which is imported lazily and cached – constructing it reads two data files and builds a Punkt model, wasted work for the other methods.

Parameters:

text – text to split.
method – a SentTokTypeSE member.

Raises:

ValueError – on an unknown method.

micm_nlp.tokenizers.tokenizer.tokenize_words(text, method=WordTokTypeSE.NLTK_PUNCT)¶

Split text into words – NLTK’s punctuation-aware tokenizer, or whitespace.

Parameters:

text – text to split.
method – a WordTokTypeSE member.

Raises:

ValueError – on an unknown method.

micm_nlp.tokenizers.tokenizer.vocab_size_by_lm_size(lm_param_size, lm_embedding_size, ratio=0.2)¶: Calculate vocabulary size according to language model size

micm_nlp.tokenizers.tokenizer.fasttokenizer_pref = 'tokenizer'¶

micm_nlp.tokenizers.tokenizer.post_processors_map¶

micm_nlp.tokenizers.tokenizer.spacy¶

micm_nlp.tokenizers.tokenizer.special_tokens_map¶

micm_nlp.tokenizers.tokenizer.tokenizers_args_map¶

micm_nlp.tokenizers.tokenizer.tokenizers_map¶

micm_nlp.tokenizers.tokenizer.whitespace_tokenize¶