micm_nlp.tokenizers.ka_sen_tok

Georgian sentence tokenizer.

Wraps NLTK’s PunktSentenceTokenizer with a Georgian abbreviation list, then post-processes the result: a sentence ending in a period whose final token is a known abbreviation ending is re-joined with the sentence that follows it. Newlines and tabs are treated as sentence boundaries before splitting.

Two data files ship with the package, both extracted from Georgian Wikipedia:

data/wiki.abbrs.txt

885 abbreviations, one per line, each written with its trailing period. They are handed to Punkt as abbrev_types (which expects them without the period), so Punkt does not treat those periods as sentence ends.

data/abbr.ends.txt

379 tokens that commonly end an abbreviated form. Punkt splits on some of these anyway; the post-processing pass in KaSenTok.tokenize() glues those sentences back together.

From A Comparison of Different Tokenization Methods for the Georgian Language (ICNLSP 2024). This is a sentence splitter, not a subword tokenizer: it has no place in AutoTokenizer and is reached through micm_nlp.tokenizers.tokenizer.tokenize_sentences().

Classes

KaSenTok

Sentence-split Georgian text.

Module Contents

class micm_nlp.tokenizers.ka_sen_tok.KaSenTok(abbreviations_path=None, abbr_ends_path=None)

Sentence-split Georgian text.

Parameters:

• abbreviations_path – override for data/wiki.abbrs.txt.

• abbr_ends_path – override for data/abbr.ends.txt.

Both default to the copies shipped with the package.

Load the abbreviation lists and build the Punkt tokenizer.

Downloads NLTK’s punkt models only if they are missing, so constructing this on a machine that already has them does not touch the network.

Parameters:

• abbreviations_path – override for the shipped wiki.abbrs.txt.

• abbr_ends_path – override for the shipped abbr.ends.txt.

tokenize(text: str) → list[str]

Split text into sentences.

abbr_ends

abbreviations

custom_sent_ends_tt

remove_digits_tt

tokenizer