micm_nlp.utils¶

General-purpose helpers shared across the package.

Four loose groups:

Class resolution — resolve_cls turns the class names that appear in YAML (model.pretrained.cls, trainer.cls, data_collator.cls) into real classes by looking them up across a list of modules.
Serialisation — JSON, YAML and pickle round-trips, a numpy-aware JSON encoder, and conversions between dicts and simple namespaces.
Introspection — object and file sizes, signature-filtered kwargs, dict diffs, UUID validation, global monkey-patching.
Script plumbing — argument parsing and run identifiers (get_time_id).

Nothing here is specific to a model, dataset or task.

Classes¶

NumpyEncoder

JSON encoder that turns np.ndarray into a list.

Functions¶

`copy_simple_nsp`(simple_nsp)	Deep-copy a namespace tree by round-tripping it through JSON.
`dict_diff`(d1, d2[, print_diff])	Compare two dictionaries and show the differences between their keys and values.
`dict_to_json_file`(dict, json_file_path)	Write a dict to a JSON file, overwriting it.
`dict_to_simple_nsp`([dictionary])	Convert a dict to a nested `SimpleNamespace`.
`dict_to_yaml_file`(dict, yaml_file_path)	Write a dict to a YAML file in block style, overwriting it.
`file_len`(path[, print_lines])	Count the lines in a file by reading it, with a progress bar.
`filter_kwargs_by_method_signature`(method, kwargs)	Filter kwargs to only include valid parameters for the given method.
`format_seconds`(n)	Format a duration in seconds as `H:MM:SS` (`datetime.timedelta`).
`format_size`(num[, suffix])	Format a byte count with a binary unit prefix (KiB, MiB, ...).
`get_placeholder`(name)	Build a `<PLACEHOLDER:name>` marker for prompt templating.
`get_script_param`(len, number[, default, p])	Read a positional `sys.argv` entry, with a default and its type.
`get_time_id`()	A sortable `YYYYmmdd_HHMMSS` stamp, used to name run directories.
`info`([file, name, package])	Print the calling module's file, module and package names.
`is_sublist`(smaller_list, larger_list)	Check if smaller_list is a sublist of larger_list.
`is_valid_uuid`(uuid_to_test[, version])	Whether a string is a UUID of the given version, in canonical form.
`json_dumps`(object, **kwargs)	`json.dumps` with this package's defaults: indented, key order preserved,
`json_dumps_numpy`(object, **kwargs)	`json_dumps()` with `NumpyEncoder`, so arrays serialise as lists.
`json_dumps_simple_nsp`(simple_nsp)	Serialise a `SimpleNamespace` tree to JSON, via `safe_json_default()`.
`json_file_to_dict`(json_file_path)	Read a JSON file into a dict.
`json_file_to_simple_nsp`(json_file_path)	Read a JSON file into a nested `SimpleNamespace`.
`json_load_simple_nsp`(json_string)	Parse JSON into nested `SimpleNamespace` objects instead of dicts.
`monkey_patch_globally`(name, new_obj[, verbose])	Rebind `name` to `new_obj` in every already-imported module that has it.
`p`(*objects[, end, sep])	Pretty prints multiple objects in a readable format using appropriate dump functions.
`parse_config_name`()	Parse a single positional config name from the command line.
`parse_script_args`([ap])	Extention Example:
`pickle_load`(path)	Unpickle the object stored at `path`.
`pickle_save`(object, path)	Pickle an object to `path`.
`print_list`(list)	Print each element of an iterable on its own line.
`print_stack`()	Print the current call stack as `file:line - function` lines.
`print_traceback`([show_locals, width, extra_lines])	Install Rich tracebacks and immediately raise, to see a formatted stack.
`read_jsons_in_folder`(folder_path)	loads all .json files inside given folder and returns their list
`resolve_cls`(cls_name, modules[, yaml_path])	Resolve a bare class name by importing it from one of modules (str or
`safe_json_default`(o)	A `json.dumps(default=...)` hook that never raises.
`scientific_notation_to_float`(item)	Recursively turn scientific-notation strings into floats.
`simple_nsp_to_dict`(simple_nsp)	Convert a nested `SimpleNamespace` back to plain dicts.
`simple_nsp_to_json_file`(object, json_file_path)	Write a `SimpleNamespace` tree to a JSON file, overwriting it.
`simple_nsp_to_yaml_file`(object, yaml_file_path)	Write a `SimpleNamespace` tree to a YAML file, overwriting it.
`simple_nsps_to_params`(*simple_nsps)	Flatten several namespaces into one kwargs dict.
`sizeof_file`(file)	Human-readable size of a file, or `0` if it does not exist.
`sizeof_object`(object)	Human-readable `sys.getsizeof` -- shallow, so containers understate.
`tik`(tok, key, callback[, params])	Call `callback`, recording how long it took into `tok[key]`.
`to_utf8_if_binary`(text)	Decode UTF-8 bytes to `str`, for a single value or a list of them.
`try_set_add`(set, element)	Attempts to add an element to the set.
`update_simple_nsp`(simple_nsps, updates)	Set attributes on a namespace in place, from a dict or mapping.
`yaml_file_to_dict`(yaml_file_path)	Read a YAML file into a dict, using `yaml.safe_load`.
`yaml_file_to_simple_nsp`(yaml_file_path)	Read a YAML file into a nested `SimpleNamespace`.

Module Contents¶

class micm_nlp.utils.NumpyEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)¶

Bases: json.JSONEncoder

JSON encoder that turns np.ndarray into a list.

Handles arrays only; for scalars and arbitrary objects use safe_json_default().

Constructor for JSONEncoder, with sensible defaults.

If skipkeys is false, then it is a TypeError to attempt encoding of keys that are not str, int, float, bool or None. If skipkeys is True, such items are simply skipped.

If ensure_ascii is true, the output is guaranteed to be str objects with all incoming non-ASCII characters escaped. If ensure_ascii is false, the output can contain non-ASCII characters.

If check_circular is true, then lists, dicts, and custom encoded objects will be checked for circular references during encoding to prevent an infinite recursion (which would cause an RecursionError). Otherwise, no such check takes place.

If allow_nan is true, then NaN, Infinity, and -Infinity will be encoded as such. This behavior is not JSON specification compliant, but is consistent with most JavaScript based encoders and decoders. Otherwise, it will be a ValueError to encode such floats.

If sort_keys is true, then the output of dictionaries will be sorted by key; this is useful for regression tests to ensure that JSON serializations can be compared on a day-to-day basis.

If indent is a non-negative integer, then JSON array elements and object members will be pretty-printed with that indent level. An indent level of 0 will only insert newlines. None is the most compact representation.

If specified, separators should be an (item_separator, key_separator) tuple. The default is (’, ‘, ‘: ‘) if indent is None and (‘,’, ‘: ‘) otherwise. To get the most compact JSON representation, you should specify (‘,’, ‘:’) to eliminate whitespace.

If specified, default is a function that gets called for objects that can’t otherwise be serialized. It should return a JSON encodable version of the object or raise a TypeError.

default(obj)¶: Serialise np.ndarray as a list; defer everything else to the base.

micm_nlp.utils.copy_simple_nsp(simple_nsp)¶

Deep-copy a namespace tree by round-tripping it through JSON.

Only what JSON can represent survives; see safe_json_default() for how the rest degrades.

micm_nlp.utils.dict_diff(d1, d2, print_diff=True)¶

Compare two dictionaries and show the differences between their keys and values.

Parameters:

d1 – First dictionary
d2 – Second dictionary
print_diff – If True, print the differences (default: True)

Returns:

Dictionary containing the differences

micm_nlp.utils.dict_to_json_file(dict, json_file_path)¶: Write a dict to a JSON file, overwriting it.

micm_nlp.utils.dict_to_simple_nsp(dictionary=None)¶

Convert a dict to a nested SimpleNamespace.

Runs scientific_notation_to_float() on the way in, so a YAML learning rate written 5e-5 – which YAML hands over as a string – arrives as a float.

micm_nlp.utils.dict_to_yaml_file(dict, yaml_file_path)¶: Write a dict to a YAML file in block style, overwriting it.

micm_nlp.utils.file_len(path, print_lines=False)¶

Count the lines in a file by reading it, with a progress bar.

Parameters:

path – file to count.
print_lines – also print the count.

Returns:

number of lines.

micm_nlp.utils.filter_kwargs_by_method_signature(method, kwargs)¶: Filter kwargs to only include valid parameters for the given method.

micm_nlp.utils.format_seconds(n)¶: Format a duration in seconds as H:MM:SS (datetime.timedelta).

micm_nlp.utils.format_size(num, suffix='B')¶: Format a byte count with a binary unit prefix (KiB, MiB, …).

micm_nlp.utils.get_placeholder(name)¶: Build a <PLACEHOLDER:name> marker for prompt templating.

micm_nlp.utils.get_script_param(len, number, default=None, p=False)¶

Read a positional sys.argv entry, with a default and its type.

When default is given, the argument is coerced to type(default).

Parameters:

len – len(sys.argv), passed in by the caller.
number – index into sys.argv.
default – value when the argument is absent; also fixes the type.
p – print the resolved value.

micm_nlp.utils.get_time_id()¶: A sortable YYYYmmdd_HHMMSS stamp, used to name run directories.

micm_nlp.utils.info(file=__file__, name=__name__, package=__package__)¶

Print the calling module’s file, module and package names.

The defaults bind to this module, so a caller wanting its own identity has to pass __file__, __name__ and __package__ explicitly.

micm_nlp.utils.is_sublist(smaller_list, larger_list)¶

Check if smaller_list is a sublist of larger_list.

Args: - smaller_list (list or np.array): The list to be checked as a sublist. - larger_list (list or np.array): The list in which to search for the sublist.

Returns: - bool: True if smaller_list is a sublist of larger_list, False otherwise.

micm_nlp.utils.is_valid_uuid(uuid_to_test, version=4)¶

Whether a string is a UUID of the given version, in canonical form.

Stricter than UUID() alone: the round-tripped string must equal the input, so a UUID written without hyphens is rejected. Used to tell a run directory named by UUID from one named otherwise.

micm_nlp.utils.json_dumps(object, **kwargs)¶: json.dumps with this package’s defaults: indented, key order preserved, non-ASCII left as-is so Georgian and other non-Latin text stays readable.

micm_nlp.utils.json_dumps_numpy(object, **kwargs)¶: json_dumps() with NumpyEncoder, so arrays serialise as lists.

micm_nlp.utils.json_dumps_simple_nsp(simple_nsp)¶: Serialise a SimpleNamespace tree to JSON, via safe_json_default().

micm_nlp.utils.json_file_to_dict(json_file_path)¶: Read a JSON file into a dict.

micm_nlp.utils.json_file_to_simple_nsp(json_file_path)¶: Read a JSON file into a nested SimpleNamespace.

micm_nlp.utils.json_load_simple_nsp(json_string)¶

Parse JSON into nested SimpleNamespace objects instead of dicts.

This is what makes config access attribute-style (config.model.pretrained) all the way down.

micm_nlp.utils.monkey_patch_globally(name: str, new_obj, verbose=False)¶

Rebind name to new_obj in every already-imported module that has it.

A blunt instrument for patching a symbol that other modules imported by value (from x import y), where patching the defining module alone would not be seen. Modules imported after this call keep the original.

Returns:: how many modules were patched.

micm_nlp.utils.p(*objects, end='\n', sep=' ')¶

Pretty prints multiple objects in a readable format using appropriate dump functions.

Parameters:

*objects – One or more objects of any type to be printed
end – String to append after the last value (default: newline)
sep – Separator between objects (default: space)

micm_nlp.utils.parse_config_name()¶

Parse a single positional config name from the command line.

The positional counterpart of parse_script_args(), which expects --config.

micm_nlp.utils.parse_script_args(ap=None)¶: Extention Example: import argparse ap = argparse.ArgumentParser() ap.add_argument(’–arg_2’, type=str, help=’Named Flag Argument’) args, config_name = parse_script_args(ap)

micm_nlp.utils.pickle_load(path)¶: Unpickle the object stored at path.

micm_nlp.utils.pickle_save(object, path)¶: Pickle an object to path.

micm_nlp.utils.print_list(list)¶: Print each element of an iterable on its own line.

micm_nlp.utils.print_stack()¶: Print the current call stack as file:line - function lines.

micm_nlp.utils.print_traceback(show_locals=False, width=120, extra_lines=1)¶

Install Rich tracebacks and immediately raise, to see a formatted stack.

A debugging aid: it always raises Exception('Ephemeral Exception'). There is no way to call it without an exception escaping.

micm_nlp.utils.read_jsons_in_folder(folder_path)¶: loads all .json files inside given folder and returns their list

micm_nlp.utils.resolve_cls(cls_name, modules, yaml_path=None)¶: Resolve a bare class name by importing it from one of modules (str or list, tried in order). Used to load HF / custom classes named in YAML without maintaining a registry. Raises ValueError on missing/unknown name.

micm_nlp.utils.safe_json_default(o)¶

A json.dumps(default=...) hook that never raises.

numpy scalars become Python scalars, arrays become lists, objects with a __dict__ become that dict, and anything else falls back to str(o) – which is what makes enums serialisable.

micm_nlp.utils.scientific_notation_to_float(item)¶

Recursively turn scientific-notation strings into floats.

YAML gives 5e-5 back as a string, not a float, so a learning rate written that way would reach the optimizer as text. This walks dicts and lists and converts any string containing e that parses as a float; everything else is returned untouched.

micm_nlp.utils.simple_nsp_to_dict(simple_nsp)¶: Convert a nested SimpleNamespace back to plain dicts.

micm_nlp.utils.simple_nsp_to_json_file(object, json_file_path)¶: Write a SimpleNamespace tree to a JSON file, overwriting it.

micm_nlp.utils.simple_nsp_to_yaml_file(object, yaml_file_path)¶: Write a SimpleNamespace tree to a YAML file, overwriting it.

micm_nlp.utils.simple_nsps_to_params(*simple_nsps)¶

Flatten several namespaces into one kwargs dict.

Later namespaces overwrite earlier ones on key collisions. Only the top level is merged; nested namespaces are copied by reference.

micm_nlp.utils.sizeof_file(file)¶: Human-readable size of a file, or 0 if it does not exist.

micm_nlp.utils.sizeof_object(object)¶: Human-readable sys.getsizeof – shallow, so containers understate.

micm_nlp.utils.tik(tok, key, callback, params=())¶

Call callback, recording how long it took into tok[key].

Parameters:

tok – dict to write the formatted duration into.
key – key to write it under.
callback – the callable to time.
params – positional arguments for callback.

Returns:

whatever callback returned.

micm_nlp.utils.to_utf8_if_binary(text)¶

Decode UTF-8 bytes to str, for a single value or a list of them.

A list is judged by its first element. Text that is already str passes through.

micm_nlp.utils.try_set_add(set, element)¶: Attempts to add an element to the set. Returns True if the element was added (it did not exist in the set). Returns False if the element was not added (it already existed in the set).

micm_nlp.utils.update_simple_nsp(simple_nsps, updates)¶: Set attributes on a namespace in place, from a dict or mapping.

micm_nlp.utils.yaml_file_to_dict(yaml_file_path)¶: Read a YAML file into a dict, using yaml.safe_load.

micm_nlp.utils.yaml_file_to_simple_nsp(yaml_file_path)¶

Read a YAML file into a nested SimpleNamespace.

This is the path a config takes on its way to CONFIG.