micm_nlp.enums

Every categorical choice the configuration layer recognises.

String enums, so a YAML value compares equal to its member without conversion: config.mode == ModeSE.FINETUNE works on the raw string 'finetune'.

Roughly three families live here — pipeline vocabulary (ModeSE, DeviceSE), dataset and task vocabulary (DsCatSE, DsTypeSE, DsSplitSE, TaskCatSE, TaskNameSE, EvalTypeSE), and per-architecture special-token tables (BertTokenSE, RobertaTokenSE, XLMRobertaTokenSE, T5TokenSE, …) that the tokenizer factory consults when adding special tokens and post-processors.

Note model.architecture is deliberately not validated against ModelArchSE: it is a free-form string used for run-directory naming.

Classes

BertTokenSE	BERT's special tokens.
DeviceSE	Where a run executes. GPU and CUDA are both accepted spellings.
DsCatSE	Top level of the artefacts/datasets tree: raw text, corpora,
DsSplitSE	Dataset split names. NONE is the empty string, for a dataset held as a
DsStateSE	How far a dataset has been processed, and which variant is on disk --
DsTypeSE	How a dataset is loaded. HUGGINGFACE pulls from the Hub;
ElectraTokenSE	ELECTRA's special tokens -- identical to BERT's.
EvalTypeSE	What kind of evaluate object to load. A metric scores predictions
ModeSE	What a run does: train from scratch, finetune a checkpoint, evaluate,
ModelArchSE	Architecture families the tokenizer factory knows how to dress -- it
PretSourceSE	Where a pretrained model or tokenizer comes from: the Hub, or a local path.
RobertaTokenSE	RoBERTa's special tokens.
SaveDatasetAsSE	On-disk format when writing a dataset: CSV, or HuggingFace's own
SentTokTypeSE	Sentence splitter to use. KA selects
StrEnum	Stand-in for enum.StrEnum on Python 3.10, which does not have it.
T5TokenSE	T5's special tokens.
TaskCatSE	Task family, which decides the head and the loss.
TaskNameSE	The concrete task: masked and permutation language modelling, sentiment
TokAlgSE	Training algorithm for a SentencePiece tokenizer: BPE or unigram.
TokTypeSE	Subword algorithm to train or load. BYTE_LEVEL is the ByT5-style
WordTokTypeSE	Word tokenizer to use when a step needs words rather than subwords --
XLMRobertaTokenSE	XLM-R's special tokens -- identical to RoBERTa's.
XLNetTokenSE	XLNet's special tokens, including the end-of-paragraph and

Module Contents

class micm_nlp.enums.BertTokenSE

Bases: enum.StrEnum

BERT’s special tokens.

One member per role, so the tokenizer factory can ask any of these tables for BOS/EOS/PAD/… without knowing which architecture it is holding. additional() returns the tokens beyond that common set.

Initialize self. See help(type(self)) for accurate signature.

classmethod additional()

Special tokens beyond the common BOS/EOS/SEP/CLS/PAD/UNK/MASK set.

BOS = '[CLS]'

CLS = '[CLS]'

EOS = '[SEP]'

MASK = '[MASK]'

PAD = '[PAD]'

SEP = '[SEP]'

UNK = '[UNK]'

class micm_nlp.enums.DeviceSE

Bases: enum.StrEnum

Where a run executes. GPU and CUDA are both accepted spellings.

Initialize self. See help(type(self)) for accurate signature.

CPU = 'cpu'

CUDA = 'cuda'

GPU = 'gpu'

class micm_nlp.enums.DsCatSE

Bases: enum.StrEnum

Top level of the artefacts/datasets tree: raw text, corpora, benchmarks, or collections.

Initialize self. See help(type(self)) for accurate signature.

BENCHMARKS = 'benchmarks'

COLLECTIOS = 'collections'

CORPORA = 'corpora'

RAW = 'raw'

class micm_nlp.enums.DsSplitSE

Bases: enum.StrEnum

Dataset split names. NONE is the empty string, for a dataset held as a single unsplit table.

Initialize self. See help(type(self)) for accurate signature.

NONE = ''

TEST = 'test'

TRAIN = 'train'

VALIDATION = 'validation'

class micm_nlp.enums.DsStateSE

Bases: enum.StrEnum

How far a dataset has been processed, and which variant is on disk – tokenized, split, subsetted, or filtered by length.

Initialize self. See help(type(self)) for accurate signature.

LONG = 'long'

SHORT = 'short'

SPLITS = 'splits'

SUBSET = 'subset'

TOKENIZED = 'tokenized'

class micm_nlp.enums.DsTypeSE

Bases: enum.StrEnum

How a dataset is loaded. HUGGINGFACE pulls from the Hub; HUGGINGFACE_SAVED reads a local save_to_disk directory.

Initialize self. See help(type(self)) for accurate signature.

CSV = 'csv'

HUGGINGFACE = 'huggingface'

HUGGINGFACE_SAVED = 'huggingface_saved'

JSON = 'json'

TEXT = 'text'

class micm_nlp.enums.ElectraTokenSE

Bases: enum.StrEnum

ELECTRA’s special tokens – identical to BERT’s.

One member per role, so the tokenizer factory can ask any of these tables for BOS/EOS/PAD/… without knowing which architecture it is holding. additional() returns the tokens beyond that common set.

Initialize self. See help(type(self)) for accurate signature.

classmethod additional()

Special tokens beyond the common BOS/EOS/SEP/CLS/PAD/UNK/MASK set.

BOS = '[CLS]'

CLS = '[CLS]'

EOS = '[SEP]'

MASK = '[MASK]'

PAD = '[PAD]'

SEP = '[SEP]'

UNK = '[UNK]'

class micm_nlp.enums.EvalTypeSE

Bases: enum.StrEnum

What kind of evaluate object to load. A metric scores predictions against labels, a comparison scores two models against each other, and a measurement describes a dataset rather than a model.

Initialize self. See help(type(self)) for accurate signature.

COMPARISON = 'comparison'

MEASUREMENT = 'measurement'

METRIC = 'metric'

class micm_nlp.enums.ModeSE

Bases: enum.StrEnum

What a run does: train from scratch, finetune a checkpoint, evaluate, test, clean a workspace, or preprocess a dataset without training.

Initialize self. See help(type(self)) for accurate signature.

CLEAN = 'clean'

EVALUATE = 'evaluate'

FINETUNE = 'finetune'

PREPROCESS = 'preprocess'

TEST = 'test'

TRAIN = 'train'

class micm_nlp.enums.ModelArchSE

Bases: enum.StrEnum

Architecture families the tokenizer factory knows how to dress – it selects the special-token table and post-processor.

Note this is not what model.architecture in a config is validated against; that field is free-form and used for run-directory naming.

Initialize self. See help(type(self)) for accurate signature.

AYA = 'aya'

BERT = 'bert'

ELECTRA = 'electra'

ROBERTA = 'roberta'

T5 = 't5'

XGLM = 'xglm'

XLMR = 'xlmr'

XLNET = 'xlnet'

class micm_nlp.enums.PretSourceSE

Bases: enum.StrEnum

Where a pretrained model or tokenizer comes from: the Hub, or a local path.

Initialize self. See help(type(self)) for accurate signature.

HUGGINGFACE = 'huggingface'

LOCAL = 'local'

class micm_nlp.enums.RobertaTokenSE

Bases: enum.StrEnum

RoBERTa’s special tokens.

One member per role, so the tokenizer factory can ask any of these tables for BOS/EOS/PAD/… without knowing which architecture it is holding. additional() returns the tokens beyond that common set.

Initialize self. See help(type(self)) for accurate signature.

classmethod additional()

Special tokens beyond the common BOS/EOS/SEP/CLS/PAD/UNK/MASK set.

BOS = '<s>'

CLS = '<s>'

EOS = '</s>'

MASK = '<mask>'

PAD = '<pad>'

SEP = '</s>'

UNK = '<unk>'

class micm_nlp.enums.SaveDatasetAsSE

Bases: enum.StrEnum

On-disk format when writing a dataset: CSV, or HuggingFace’s own save_to_disk layout.

Initialize self. See help(type(self)) for accurate signature.

CSV = 'csv'

HUGGINGFACE = 'huggingface'

class micm_nlp.enums.SentTokTypeSE

Bases: enum.StrEnum

Sentence splitter to use. KA selects KaSenTok, the Georgian splitter; the others are NLTK’s and spaCy’s.

Initialize self. See help(type(self)) for accurate signature.

KA = 'kast'

NLTK = 'nltkst'

SPACY = 'spacyst'

class micm_nlp.enums.StrEnum

Bases: str, enum.Enum

Stand-in for enum.StrEnum on Python 3.10, which does not have it.

Subclassing str gives the same property the real thing does, and the one this module depends on: a member compares equal to its value, so a YAML string matches without conversion.

Initialize self. See help(type(self)) for accurate signature.

class micm_nlp.enums.T5TokenSE

Bases: enum.StrEnum

T5’s special tokens.

One member per role, so the tokenizer factory can ask any of these tables for BOS/EOS/PAD/… without knowing which architecture it is holding. additional() returns the tokens beyond that common set.

Initialize self. See help(type(self)) for accurate signature.

classmethod additional()

Special tokens beyond the common BOS/EOS/SEP/CLS/PAD/UNK/MASK set.

BOS = '<s>'

CLS = '<s>'

EOS = '</s>'

MASK = '<mask>'

PAD = '<pad>'

SEP = '</s>'

UNK = '<unk>'

class micm_nlp.enums.TaskCatSE

Bases: enum.StrEnum

Task family, which decides the head and the loss. TEXT_GENERATION is for decoder-only models, TEXT_TO_TEXT for encoder-decoder ones.

Initialize self. See help(type(self)) for accurate signature.

LANGUAGE_MODELING = 'language_modeling'

STRUCTURAL_ANALYSIS = 'structural_analysis'

TEXT_CLASSIFICATION = 'text_classification'

TEXT_GENERATION = 'text_generation'

TEXT_PAIR_CLASSIFICATION = 'text_pair_classification'

TEXT_SIMILARITIY = 'text_similarity'

TEXT_TO_TEXT = 'text_to_text'

TOKEN_CLASSIFICATION = 'token_classification'

class micm_nlp.enums.TaskNameSE

Bases: enum.StrEnum

The concrete task: masked and permutation language modelling, sentiment analysis, NER, POS tagging, topic detection.

Initialize self. See help(type(self)) for accurate signature.

DMLM = 'dmlm'

MLM = 'mlm'

NER = 'ner'

PLM = 'plm'

POS = 'pos'

SA = 'sa'

TOPIC = 'topic'

class micm_nlp.enums.TokAlgSE

Bases: enum.StrEnum

Training algorithm for a SentencePiece tokenizer: BPE or unigram.

Initialize self. See help(type(self)) for accurate signature.

BPE = 'bpe'

UNIGRAM = 'unigram'

class micm_nlp.enums.TokTypeSE

Bases: enum.StrEnum

Subword algorithm to train or load. BYTE_LEVEL is the ByT5-style byte vocabulary; the SentencePiece entries distinguish the native trainer from HuggingFace’s.

Initialize self. See help(type(self)) for accurate signature.

BPE = 'bpe'

BYTE_LEVEL = 'byte_level'

BYTE_LEVEL_BPE = 'byte_level_bpe'

HUGGINGFACE_SENTPIECE = 'huggingface_sentpiece'

NATIVE_SENTPIECE = 'native_sentpiece'

WORDPIECE = 'wordpiece'

class micm_nlp.enums.WordTokTypeSE

Bases: enum.StrEnum

Word tokenizer to use when a step needs words rather than subwords – NLTK’s whitespace or punctuation tokenizer.

Initialize self. See help(type(self)) for accurate signature.

NLTK_PUNCT = 'nltk_punct'

NLTK_WHITESPACE = 'nltk_whitespace'

class micm_nlp.enums.XLMRobertaTokenSE

Bases: enum.StrEnum

XLM-R’s special tokens – identical to RoBERTa’s.

One member per role, so the tokenizer factory can ask any of these tables for BOS/EOS/PAD/… without knowing which architecture it is holding. additional() returns the tokens beyond that common set.

Initialize self. See help(type(self)) for accurate signature.

classmethod additional()

Special tokens beyond the common BOS/EOS/SEP/CLS/PAD/UNK/MASK set.

BOS = '<s>'

CLS = '<s>'

EOS = '</s>'

MASK = '<mask>'

PAD = '<pad>'

SEP = '</s>'

UNK = '<unk>'

class micm_nlp.enums.XLNetTokenSE

Bases: enum.StrEnum

XLNet’s special tokens, including the end-of-paragraph and end-of-document markers it adds beyond the usual set.

One member per role, so the tokenizer factory can ask any of these tables for BOS/EOS/PAD/… without knowing which architecture it is holding. additional() returns the tokens beyond that common set.

Initialize self. See help(type(self)) for accurate signature.

classmethod additional()

Special tokens beyond the common BOS/EOS/SEP/CLS/PAD/UNK/MASK set.

BOS = '<s>'

CLS = '<cls>'

EOD = '<eod>'

EOP = '<eop>'

EOS = '</s>'

MASK = '<mask>'

PAD = '<pad>'

SEP = '<sep>'

UNK = '<unk>'