micm_nlp.pipeline

Pipeline — high-level wiring of the core components.

CONFIG → TOKENIZER → DATASET → MODEL → TRAINER

Usage:

from micm_nlp.pipeline import load_dataset, load_model, run

Functions

load_dataset(config)	CONFIG → DATASET
load_model(config)	CONFIG → MODEL
preprocess_dataset(config[, tokenizer])	CONFIG → TOKENIZER → DATASET
run(config)	CONFIG → TOKENIZER → DATASET → MODEL → TRAINER → results

Module Contents

micm_nlp.pipeline.load_dataset(config)

CONFIG → DATASET

micm_nlp.pipeline.load_model(config)

CONFIG → MODEL

micm_nlp.pipeline.preprocess_dataset(config, tokenizer=None)

CONFIG → TOKENIZER → DATASET

micm_nlp.pipeline.run(config)

CONFIG → TOKENIZER → DATASET → MODEL → TRAINER → results