beamds.beam.utils package

Submodules

beamds.beam.utils.utils_all module

class beamds.beam.utils.utils_all.BeamDict(initial_data=None, **kwargs)

Bases: dict, Namespace

items()

Return a set-like object providing a view on the dict’s items.

keys()

Return a set-like object providing a view on the dict’s keys.

pop(k[, d]) → v, remove specified key and return the corresponding value.

If the key is not found, return the default if given; otherwise, raise a KeyError.

values()

Return an object providing a view on the dict’s values.

class beamds.beam.utils.utils_all.BeamJsonEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)

Bases: JSONEncoder

default(obj)

Implement this method in a subclass such that it returns a serializable object for o, or calls the base implementation (to raise a TypeError).

For example, to support arbitrary iterators, you could implement default like this:

def default(self, o):
    try:
        iterable = iter(o)
    except TypeError:
        pass
    else:
        return list(iterable)
    # Let the base class default method raise the TypeError
    return super().default(o)

exception beamds.beam.utils.utils_all.CachedAttributeException

Bases: Exception

Custom exception to be raised instead of AttributeError in cached properties.

class beamds.beam.utils.utils_all.DataBatch(index, label, data)

Bases: tuple

data

Alias for field number 2

index

Alias for field number 0

label

Alias for field number 1

class beamds.beam.utils.utils_all.DataObject(data, data_type=None)

Bases: object

property data_type

class beamds.beam.utils.utils_all.LimitedSizeDict(size_limit=None, on_removal=None)

Bases: OrderedDict

class beamds.beam.utils.utils_all.LimitedSizeDictFactory(size_limit=None, on_removal=None)

Bases: object

class beamds.beam.utils.utils_all.NoneClass(*args, **kwargs)

Bases: object

class beamds.beam.utils.utils_all.NullClass

Bases: object

class beamds.beam.utils.utils_all.Slicer(x, x_type=None, wrap_object=False)

Bases: object

class beamds.beam.utils.utils_all.ThreadSafeDict(*args, **kwargs)

Bases: dict

clear()

Remove all items from the dict.

copy()

Return a shallow copy of the dict.

items()

Return a set-like object providing a view on the dict’s items.

keys()

Return a set-like object providing a view on the dict’s keys.

pop(k[, d]) → v, remove specified key and return the corresponding value.

If the key is not found, return the default if given; otherwise, raise a KeyError.

popitem()

Remove and return a (key, value) pair as a 2-tuple.

Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty.

setdefault(key, default=None)

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

update([E, ]**F) → None.  Update D from mapping/iterable E and F.

If E is present and has a .keys() method, then does: for k in E.keys(): D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

values()

Return an object providing a view on the dict’s values.

class beamds.beam.utils.utils_all.Timer(logger, name='', silent=False, timeout=None, task=None, task_args=None, task_kwargs=None, graceful=False)

Bases: object

property elapsed

pause()

reset()

run(*args, **kwargs)

beamds.beam.utils.utils_all.beam_base_port()

beamds.beam.utils.utils_all.beam_service_port(service)

beamds.beam.utils.utils_all.beam_traceback(exc_type=None, exc_value=None, tb=None, context=3)

beamds.beam.utils.utils_all.build_container_from_tupled_keys(keys, values, sorted_keys=None)

class beamds.beam.utils.utils_all.cached_property(func)

Bases: cached_property

beamds.beam.utils.utils_all.check_if_port_is_available(port)

beamds.beam.utils.utils_all.deserialize_annotation(annotation_str, global_ns=None)

Convert serialized annotation back to its original format.

beamds.beam.utils.utils_all.dict_to_list(x, convert_str=True)

beamds.beam.utils.utils_all.dict_to_signature(d, global_ns=None)

Convert a dictionary representation back to a Signature object.

beamds.beam.utils.utils_all.dictionary_iterator(d)

beamds.beam.utils.utils_all.filter_dict(d, keys)

beamds.beam.utils.utils_all.find_free_port()

beamds.beam.utils.utils_all.find_port(port=None, get_port_from_beam_port_range=True, application='none', blacklist=None, whitelist=None)

beamds.beam.utils.utils_all.get_cached_properties(obj)

beamds.beam.utils.utils_all.get_class_properties(cls)

beamds.beam.utils.utils_all.get_closest_item_with_tuple_key(x, key)

beamds.beam.utils.utils_all.get_edit_distance(s1, s2)

beamds.beam.utils.utils_all.get_edit_ratio(s1, s2)

beamds.beam.utils.utils_all.get_item_with_tuple_key(x, key)

beamds.beam.utils.utils_all.get_notebook_name()

Execute JS code to save Jupyter notebook name to variable notebook_name

beamds.beam.utils.utils_all.get_number_of_cores()

beamds.beam.utils.utils_all.get_public_ip()

beamds.beam.utils.utils_all.getmembers(object, predicate=None)

Return all members of an object as (name, value) pairs sorted by name. Optionally, only return members that satisfy a given predicate.

beamds.beam.utils.utils_all.has_kwargs(func)

beamds.beam.utils.utils_all.identity_function(x, **kwargs)

beamds.beam.utils.utils_all.include_patterns(*patterns)

Factory function that can be used with copytree() ignore parameter. Arguments define a sequence of glob-style patterns that are used to specify what files to NOT ignore. Creates and returns a function that determines this for each directory in the file hierarchy rooted at the source directory when used with shutil.copytree().

beamds.beam.utils.utils_all.insert_tupled_key(x, k, v, default=None, keys=None)

beamds.beam.utils.utils_all.int_or_float(x)

beamds.beam.utils.utils_all.is_arange(x, convert_str=True, sort=True)

beamds.beam.utils.utils_all.is_boolean(x)

beamds.beam.utils.utils_all.is_notebook() → bool

beamds.beam.utils.utils_all.jupyter_like_traceback(exc_type=None, exc_value=None, tb=None, context=3)

beamds.beam.utils.utils_all.lazy_property(fn)

beamds.beam.utils.utils_all.mixin_dictionaries(*dicts)

class beamds.beam.utils.utils_all.nested_defaultdict(default_factory=None, **kwargs)

Bases: defaultdict

static default_factory_dict()

static default_factory_list()

beamds.beam.utils.utils_all.new_container(k)

beamds.beam.utils.utils_all.none_function(*args, **kwargs)

beamds.beam.utils.utils_all.parse_string_number(x, time_units=None, unit_prefixes=None, timedelta_format=True, return_units=False)

beamds.beam.utils.utils_all.parse_text_to_protocol(text, protocol='json')

beamds.beam.utils.utils_all.pprint(obj, logger=None, level='info')

beamds.beam.utils.utils_all.pretty_format_number(x, short=False)

beamds.beam.utils.utils_all.pretty_print_dict(d, name=None, dense=True)

beamds.beam.utils.utils_all.pretty_print_timedelta(seconds)

beamds.beam.utils.utils_all.rate_string_format(n, t)

beamds.beam.utils.utils_all.recursive_func(x, func, *args, _ignore_none=False, **kwargs)

beamds.beam.utils.utils_all.retrieve_name(var)

beamds.beam.utils.utils_all.retry(func=None, retries=3, logger=None, name=None, verbose=False, sleep=1, timeout=None)

beamds.beam.utils.utils_all.return_constant(a)

beamds.beam.utils.utils_all.run_forever(func=None, *args, sleep=1, name=None, logger=None, **kwargs)

beamds.beam.utils.utils_all.running_platform() → str

beamds.beam.utils.utils_all.safe_getmembers(obj, predicate=None)

beamds.beam.utils.utils_all.serialize_annotation(annotation)

Convert annotation to a serializable format.

beamds.beam.utils.utils_all.set_item_with_tuple_key(x, key, value)

beamds.beam.utils.utils_all.signature_to_dict(signature)

Convert a Signature object to a dictionary representation.

beamds.beam.utils.utils_all.slice_array(x, index, x_type=None, indices_type=None, wrap_object=False)

beamds.beam.utils.utils_all.squeeze_scalar(x, x_type=None)

beamds.beam.utils.utils_all.strip_prefix(text, prefix)

beamds.beam.utils.utils_all.strip_suffix(text, suffix)

beamds.beam.utils.utils_all.tqdm_beam(x, *args, threshold=10, stats_period=1, message_func=None, enable=None, notebook=True, **argv)

Beam’s wrapper for the tqdm progress bar. It features a universal interface for both jupyter notebooks and .py files. In addition, it provides a “lazy progress bar initialization”. The progress bar is initialized only if its estimated duration is longer than a threshold.

Parameters:

• x

• threshold (float) – The smallest expected duration (in Seconds) to generate a progress bar. This feature is used only if enable is set to None.

• stats_period (float) – The initial time period (in seconds) to calculate the ineration statistics (iters/sec). This statistics is used to estimate the expected duction of the entire iteration.

• message_func (func) – A dynamic message to add to the progress bar. For example, this message can plot the instantaneous loss.

• enable (boolean/None) – Whether to enable the progress bar, disable it or when set to None, use lazy progress bar.

• notebook (boolean) – A boolean that overrides the internal calculation of is_notebook. Set to False when you want to avoid printing notebook styled tqdm bars (for example, due to multiprocessing).

beamds.beam.utils.utils_ds module

class beamds.beam.utils.utils_ds.GPUManager

Bases: object

static logical_devices(physical_devices)

static physical_devices(logical_devices=None)

beamds.beam.utils.utils_ds.as_dataframe(x, target='pandas', **kwargs)

beamds.beam.utils.utils_ds.as_list(x, length=None, **kwargs)

beamds.beam.utils.utils_ds.as_numpy(x, **kwargs)

beamds.beam.utils.utils_ds.as_scipy_coo(x)

beamds.beam.utils.utils_ds.as_scipy_csr(x)

beamds.beam.utils.utils_ds.as_something_recursively(as_something_func)

beamds.beam.utils.utils_ds.as_tensor(x, **kwargs)

beamds.beam.utils.utils_ds.batch_augmentation(augmentations)

beamds.beam.utils.utils_ds.batch_augmentation_(x, augmentations)

beamds.beam.utils.utils_ds.beam_device(device)

beamds.beam.utils.utils_ds.beam_hash(x, bytes_threshold=1000000, fast=True)

beamds.beam.utils.utils_ds.big_array_representation(x)

beamds.beam.utils.utils_ds.check_nvlink()

beamds.beam.utils.utils_ds.collate_chunks(*xs, keys=None, dim=0, on='index', how='outer', method='tree', squeeze=True, logger=None)

beamds.beam.utils.utils_ds.concat_polars_horizontally(data, **kwargs)

beamds.beam.utils.utils_ds.container_len(x)

beamds.beam.utils.utils_ds.divide_chunks(x, chunksize=None, n_chunks=None, partition=None, squeeze=False, dim=0, x_type=None, chunksize_policy='round')

beamds.beam.utils.utils_ds.empty_element(x, x_type=None)

beamds.beam.utils.utils_ds.get_chunks(x, chunksize=None, n_chunks=None, partition=None, dim=0)

beamds.beam.utils.utils_ds.hash_tensor(x, fast=False, coarse=False)

This function returns a deterministic hash of the tensor content @param x: the tensor to hash @param fast: whether to consider only the first and last elements of the tensor for hashing @param coarse: whether to apply coarse hashing where the tensor is quantized into low resolution (16bit) tensor @return: an integer representing the hash value

beamds.beam.utils.utils_ds.is_chunk(path, chunk_pattern='_chunk')

beamds.beam.utils.utils_ds.is_empty(x)

beamds.beam.utils.utils_ds.iter_container(x)

beamds.beam.utils.utils_ds.object_size(x, x_type=None)

beamds.beam.utils.utils_ds.recursive(func)

beamds.beam.utils.utils_ds.recursive_batch(x, *args, in_place=False, **kwargs)

beamds.beam.utils.utils_ds.recursive_chunks(x, chunksize=None, n_chunks=None, partition=None, squeeze=False, dim=0, x_type=None, chunksize_policy='round')

beamds.beam.utils.utils_ds.recursive_clone(x, *args, in_place=False, **kwargs)

beamds.beam.utils.utils_ds.recursive_collate_chunks(*xs, dim=0, on='index', how='outer', method='tree')

beamds.beam.utils.utils_ds.recursive_concatenate(data, dim=0)

beamds.beam.utils.utils_ds.recursive_detach(x, *args, in_place=False, **kwargs)

beamds.beam.utils.utils_ds.recursive_device(x)

beamds.beam.utils.utils_ds.recursive_devices(x, *args, in_place=False, **kwargs)

beamds.beam.utils.utils_ds.recursive_elementwise(func)

beamds.beam.utils.utils_ds.recursive_empty_elements(x, *args, in_place=False, **kwargs)

beamds.beam.utils.utils_ds.recursive_flat_array(x, x_type=None, tolist=True)

beamds.beam.utils.utils_ds.recursive_flatten(x, flat_array=False, x_type=None, tolist=True, _root=True, depth=-1)

beamds.beam.utils.utils_ds.recursive_flatten_with_keys(x)

beamds.beam.utils.utils_ds.recursive_hierarchical_keys(x)

beamds.beam.utils.utils_ds.recursive_items(x, level=1)

beamds.beam.utils.utils_ds.recursive_keys(x, level=1)

beamds.beam.utils.utils_ds.recursive_len(x, *args, in_place=False, **kwargs)

beamds.beam.utils.utils_ds.recursive_merge(dfs, method='tree', **kwargs)

beamds.beam.utils.utils_ds.recursive_same_device(x)

beamds.beam.utils.utils_ds.recursive_shape(x, *args, in_place=False, **kwargs)

beamds.beam.utils.utils_ds.recursive_size(x)

beamds.beam.utils.utils_ds.recursive_size_summary(x, mode='sum')

beamds.beam.utils.utils_ds.recursive_slice(x, *args, in_place=False, **kwargs)

beamds.beam.utils.utils_ds.recursive_slice_columns(x, *args, in_place=False, **kwargs)

beamds.beam.utils.utils_ds.recursive_squeeze(x, *args, in_place=False, **kwargs)

beamds.beam.utils.utils_ds.recursive_to_cpu(x, *args, in_place=False, **kwargs)

beamds.beam.utils.utils_ds.recursive_types(x, *args, in_place=False, **kwargs)

beamds.beam.utils.utils_ds.recursive_values(x, level=1)

beamds.beam.utils.utils_ds.recursive_yield(func, keys=True, values=True, level=-1)

beamds.beam.utils.utils_ds.set_seed(seed=-1, constant=0, increment=False, deterministic=False)

Parameters:

• seed – set -1 to avoid change, set 0 to randomly select seed, set [1, 2**31) to get new seed

• constant – a constant to be added to the seed

• increment – whether to generate incremental seeds

• deterministic – whether to set torch to be deterministic

Returns:

None

beamds.beam.utils.utils_ds.slice_to_index(s, l=None, arr_type='tensor', sliced=None)

beamds.beam.utils.utils_ds.to_device(data, device='cuda', half=False, dtype=None, brain=False)

Module contents