micm_nlp.path

Filesystem layout: the workspace root and the artefacts/ tree beneath it.

Two kinds of path live here. PACKAGE_DIR points inside the installed package and is read-only. Everything else hangs off the workspace — the user’s project directory — which must be set once via set_root() before any accessor is called; they raise otherwise. micm_nlp.init() does that for you:

workspace()/artefacts/{models,datasets,tokenizers,evals}

Attributes

PACKAGE_DIR

Functions

artefacts_dir(→ pathlib.Path)	<workspace>/artefacts — the root of everything this package writes.
datasets_dir(→ pathlib.Path)	artefacts/datasets — raw, preprocessed and tokenized datasets.
evals_dir(→ pathlib.Path)	artefacts/evals — evaluation runs: metrics, predictions, plots.
find_dirs_by_prefix(root_dir, dir_prefix)	Recursively find every directory under root_dir whose name starts with
get_dir_items(dir_path[, only_dirs, only_files])	List the immediate contents of a directory, non-recursively.
models_dir(→ pathlib.Path)	artefacts/models — saved checkpoints and PEFT adapters.
set_root(workspace)	Set the workspace root. Call once at startup.
tokenizers_dir(→ pathlib.Path)	artefacts/tokenizers — tokenizers trained by this package.
wandb_dir(→ pathlib.Path)	Parent directory for wandb's own wandb/ folder.
workspace(→ pathlib.Path)	Return the workspace root.

Module Contents

micm_nlp.path.artefacts_dir() → pathlib.Path

<workspace>/artefacts — the root of everything this package writes.

micm_nlp.path.datasets_dir() → pathlib.Path

artefacts/datasets — raw, preprocessed and tokenized datasets.

micm_nlp.path.evals_dir() → pathlib.Path

artefacts/evals — evaluation runs: metrics, predictions, plots.

micm_nlp.path.find_dirs_by_prefix(root_dir, dir_prefix)

Recursively find every directory under root_dir whose name starts with dir_prefix.

Walks the whole tree, so cost grows with the size of root_dir — the progress bar is there because an artefacts/ tree can be large.

Parameters:

• root_dir – directory to search under.

• dir_prefix – prefix a directory’s name must start with.

Returns:

absolute paths as strings, in walk order.

micm_nlp.path.get_dir_items(dir_path, only_dirs=False, only_files=False)

List the immediate contents of a directory, non-recursively.

Parameters:

• dir_path – directory to list.

• only_dirs – return only subdirectories.

• only_files – return only files.

Returns:

names, with a trailing / on directories. A missing dir_path yields [] rather than raising, so callers can probe a path that may not exist yet.

micm_nlp.path.models_dir() → pathlib.Path

artefacts/models — saved checkpoints and PEFT adapters.

micm_nlp.path.set_root(workspace: str | pathlib.Path)

Set the workspace root. Call once at startup.

micm_nlp.path.tokenizers_dir() → pathlib.Path

artefacts/tokenizers — tokenizers trained by this package.

micm_nlp.path.wandb_dir() → pathlib.Path

Parent directory for wandb’s own wandb/ folder.

Returns artefacts/ itself, not a subdirectory: wandb appends wandb/ to whatever it is given, so the run tree lands at artefacts/wandb/.

micm_nlp.path.workspace() → pathlib.Path

Return the workspace root.

Raises:

RuntimeError – if set_root() has not been called. Every accessor below goes through here, so an unset root fails loudly at the first path request rather than silently writing into the current directory.

micm_nlp.path.PACKAGE_DIR