micm_nlp.datasets.dataset

DATASET — load, standardise, preprocess and tokenize the data.

One class covers the whole data path. Construction loads from the HuggingFace Hub, a saved HF dataset on disk, or CSV/TXT/JSON files, and renames the configured input and label keys to the canonical names in DATASET.keys, so everything downstream sees the same column names whatever the source called them.

preprocess(tokenizer) then runs the ds.preproc_rules pipeline: optional subsetting, splitting (by ratio or by token length), and tokenization in three stages — pre_rules (sentence splitting, EOS appending, text-to-text reframing), rules (handed to the tokenizer verbatim), and post_rules (EOS-aware truncation, sample concatenation, length sorting).

The remaining methods are inspection and assembly: length statistics and percentiles used to size truncation and batches, per-split previews, and get_concatenated_dataset for stitching several datasets into one — how consumer repos build multilingual source sets from one config instantiated per language.

Classes

DATASET

The whole data path: load, standardise, preprocess, tokenize, split, save.

Module Contents

class micm_nlp.datasets.dataset.DATASET(config, hf_datasets=None)

The whole data path: load, standardise, preprocess, tokenize, split, save.

Loading happens in __init__, driven by ds.type – a local CSV/TXT/JSON file, a Hub dataset, or a save_to_disk directory. Several datasets can be loaded and concatenated, which is how a multilingual source set is assembled from one per-language template path.

keys fixes the column names the rest of the package expects (input_ids, labels, task_ids, …); ds.input.standardize_key and its siblings rename the dataset’s own columns onto them, so downstream code never has to know what a particular corpus called its text column.

The HuggingFace object is reachable as hf, and the splits as train / test / validation.

Load the dataset described by the config, then print a summary.

Parameters:

  • config – the run config; the ds block decides what is loaded.

  • hf_datasets – already-loaded HuggingFace datasets to wrap instead of reading from disk or the Hub.

analyze_lengths()

Computes and returns a dictionary of input/label length stats for easy CSV logging or pretty printing.

get_concatenated_dataset(datasets)

Concatenate several datasets split by split.

A split is only present in the result if at least one input has it, so concatenating datasets with differing splits does not fail – it yields the union.

Parameters:

datasets – datasets to merge, each a split-keyed mapping.

Returns:

one dataset with each split concatenated in the given order.

get_first_available_split()

Return the first present split, preferring test, then validation, then train.

For steps that need some data to inspect – length statistics, a sanity print – and should not care which split provides it. The order prefers held-out data, so an inspection does not silently look at training rows.

Returns:

(dataset, split_name).

Raises:

ValueError – if the dataset has no splits at all.

static get_name(config)

Resolve a dataset’s display name, preferring descriptive_name.

Also writes the result back to config.descriptive_name, so callers that read the config later see the resolved value – this is not a pure getter.

input_len_longer_than(perc=0.999)

Returns the length at the given percentile among all inputs (default 90%).

label_len_longer_than(perc=0.999)

Returns the length at the given percentile among all labels (default 90%).

longest_input_len()

Returns the max length for inputs across all splits.

longest_label_len()

Returns the max length for labels across all splits.

preprocess(tokenizer)

Preprocess Dataset

print_examples_from_splits(num_examples=1)

Print a few examples from each dataset split.

save(save_as, dirs, train_split=None, test_split=None, validation_split=None)

Write the dataset to disk as CSV or in HuggingFace’s own format.

Parameters:

  • save_as – a SaveDatasetAsSE member.

  • dirs – destination path under artefacts/datasets.

  • train_split – split to write instead of this object’s train.

  • test_split – as above, for test.

  • validation_split – as above, for validation.

FILE_EXT_MAP: ClassVar[dict]
property hf

The underlying HuggingFace DatasetDict.

keys
seed = 42