micm_nlp.tokenizers

Tokenizer loading, architecture adaptation, and sentence splitting.

• tokenizer — load(), the AutoTokenizer factory; special tokens and post-processors per architecture; tokenize_sentences; tokenizer training.

• decoding — label-aware decode / batch_decode.

• architectures — the two tokenizer classes HuggingFace does not ship, BertByT5Tokenizer and CustomXlmRoberta.

• ka_sen_tok — KaSenTok, a Georgian sentence splitter. Not a subword tokenizer, and not reachable through AutoTokenizer.

Adding a backbone normally needs nothing here: tokenizer.cls in the YAML resolves a class by name at runtime.

Submodules

• tokenizer
• decoding
• architectures
• ka_sen_tok