dasf.utils.funcs

Generic and regular functions.

Attributes

`GPU_SUPPORTED`
`JAX_SUPPORTED`
`KVIKIO_SUPPORTED`
`NV_COMP_BATCH_CODEC_SUPPORTED`

Classes

NotebookProgressBar

ProgressBar representation for ipython notebooks only.

Functions

`human_readable_size`(size[, decimal])	converts data size into the proper measurement
`get_worker_info`(client)	Returns a list of workers (sorted), and the DNS name for the master host
`sync_future_loop`(futures)	Synchronize all futures submitted to workers.
`download_file`(url[, filename, directory])	Download a generic file and save it.
`download_file_from_gdrive`(file_id[, filename, directory])	Download a file from Google Drive using gdrive file id.
`get_machine_memory_avail`()	Return free memory available from a single machine.
`set_executor_default`()	Return executor as a CPU (default) instance.
`set_executor_cpu`()	Return executor as a CPU instance.
`set_executor_gpu`()	Return executor as a GPU instance.
`set_executor_multi_cpu`()	Return executor as a Multi CPU instance.
`set_executor_multi_gpu`()	Return executor as a GPU instance.
`is_executor_single`(dtype)	Return if the executor is a single machine instance.
`is_executor_cluster`(dtype)	Return if the executor is a cluster instance.
`is_executor_cpu`(dtype)	Return if the executor is a CPU instance.
`is_executor_gpu`(dtype)	Return if the executor is a GPU instance.
`executor_to_string`(dtype)	Return the executor type as a string.
`is_gpu_supported`()	Return if GPU is supported.
`is_kvikio_supported`()	Return if kvikio is supported (installed).
`is_gds_supported`()	Return if GPU Direct Store is supported.
`is_kvikio_compat_mode`()	Return if Kvikio is running in compatibility mode.
`is_nvcomp_codec_supported`()	Return if NVidia Compressor Codecs are supported.
`is_jax_supported`()	Return if JAX is supported.
`is_dask_local_supported`()	Return if Dask is supported locally by the executor.
`get_dask_running_client`()	Get Dask runner stanza.
`get_backend_supported`(func)	Get backend support.
`is_dask_supported`()	Return if Dask is supported by the executor.
`is_dask_gpu_supported`()	Return if any node supports GPU.
`get_gpu_from_workers`()	Return if any worker has a GPU available.
`get_gpu_count`()	Get single node GPU count.
`get_dask_gpu_count`([fetch])	Get how many GPUs are available in each worker.
`get_dask_gpu_names`([fetch])	Get all GPU names of each worker.
`block_chunk_reduce`(dask_data, output_chunk)	Reduce the chunk according the new output size.
`trim_chunk_location`(block_info, depth[, index])	Trim an overlapped chunk to the exact size of the chunk.
`get_dask_mem_usage`(profiler)	Get Dask memory usage profile.
`is_notebook`()	Return if the code is being executed in a IPyNotebook.
`weight_gaussian`(shape)	Produces a NDArray for a given shape with a Gaussian Distribution in all directions starting from the center
`weight_radial`(shape)	Produces a NDArray for a given shape with a decreasing rate starting from the center

Module Contents

dasf.utils.funcs.GPU_SUPPORTED

dasf.utils.funcs.JAX_SUPPORTED

dasf.utils.funcs.KVIKIO_SUPPORTED = True

dasf.utils.funcs.NV_COMP_BATCH_CODEC_SUPPORTED = True

dasf.utils.funcs.human_readable_size(size, decimal=3)[source]

converts data size into the proper measurement

Return type:: str

dasf.utils.funcs.get_worker_info(client)[source]

Returns a list of workers (sorted), and the DNS name for the master host The master is the 0th worker’s host

Return type:: list

dasf.utils.funcs.sync_future_loop(futures)[source]: Synchronize all futures submitted to workers.

class dasf.utils.funcs.NotebookProgressBar[source]

Bases: threading.Thread

ProgressBar representation for ipython notebooks only.

Constructor of the Progress Bar

MIN_CUR

MIN_TOTAL

bar = None

percentage = None

data = None

__lock

__current

__total

__error = False

show()[source]: Return the HTML representation of the ProgressBar.

set_current(current, total)[source]: Set current value of the bar progress

set_error(error)[source]: Set an error if it exists.

run()[source]: Thread main loop that updates the bar progress.

dasf.utils.funcs.download_file(url, filename=None, directory=None)[source]: Download a generic file and save it.

dasf.utils.funcs.download_file_from_gdrive(file_id, filename=None, directory=None)[source]: Download a file from Google Drive using gdrive file id.

dasf.utils.funcs.get_machine_memory_avail()[source]: Return free memory available from a single machine.

dasf.utils.funcs.set_executor_default()[source]: Return executor as a CPU (default) instance.

dasf.utils.funcs.set_executor_cpu()[source]: Return executor as a CPU instance.

dasf.utils.funcs.set_executor_gpu()[source]: Return executor as a GPU instance.

dasf.utils.funcs.set_executor_multi_cpu()[source]: Return executor as a Multi CPU instance.

dasf.utils.funcs.set_executor_multi_gpu()[source]: Return executor as a GPU instance.

dasf.utils.funcs.is_executor_single(dtype)[source]

Return if the executor is a single machine instance.

Return type:: bool

dasf.utils.funcs.is_executor_cluster(dtype)[source]

Return if the executor is a cluster instance.

Return type:: bool

dasf.utils.funcs.is_executor_cpu(dtype)[source]

Return if the executor is a CPU instance.

Return type:: bool

dasf.utils.funcs.is_executor_gpu(dtype)[source]

Return if the executor is a GPU instance.

Return type:: bool

dasf.utils.funcs.executor_to_string(dtype)[source]

Return the executor type as a string.

Return type:: str

dasf.utils.funcs.is_gpu_supported()[source]

Return if GPU is supported.

Return type:: bool

dasf.utils.funcs.is_kvikio_supported()[source]

Return if kvikio is supported (installed).

Return type:: bool

dasf.utils.funcs.is_gds_supported()[source]

Return if GPU Direct Store is supported.

Return type:: bool

dasf.utils.funcs.is_kvikio_compat_mode()[source]

Return if Kvikio is running in compatibility mode.

Return type:: bool

dasf.utils.funcs.is_nvcomp_codec_supported()[source]

Return if NVidia Compressor Codecs are supported.

Return type:: bool

dasf.utils.funcs.is_jax_supported()[source]

Return if JAX is supported.

Return type:: bool

dasf.utils.funcs.is_dask_local_supported()[source]

Return if Dask is supported locally by the executor.

Return type:: bool

dasf.utils.funcs.get_dask_running_client()[source]: Get Dask runner stanza.

dasf.utils.funcs.get_backend_supported(func)[source]: Get backend support.

dasf.utils.funcs.is_dask_supported()[source]

Return if Dask is supported by the executor.

Return type:: bool

dasf.utils.funcs.is_dask_gpu_supported()[source]

Return if any node supports GPU.

Return type:: bool

dasf.utils.funcs.get_gpu_from_workers()[source]

Return if any worker has a GPU available.

Return type:: bool

dasf.utils.funcs.get_gpu_count()[source]

Get single node GPU count.

Return type:: int

dasf.utils.funcs.get_dask_gpu_count(fetch=True)[source]

Get how many GPUs are available in each worker.

Return type:: int

dasf.utils.funcs.get_dask_gpu_names(fetch=True)[source]

Get all GPU names of each worker.

Return type:: list

dasf.utils.funcs.block_chunk_reduce(dask_data, output_chunk)[source]: Reduce the chunk according the new output size.

dasf.utils.funcs.trim_chunk_location(block_info, depth, index=0)[source]: Trim an overlapped chunk to the exact size of the chunk.

dasf.utils.funcs.get_dask_mem_usage(profiler)[source]: Get Dask memory usage profile.

dasf.utils.funcs.is_notebook()[source]

Return if the code is being executed in a IPyNotebook.

Return type:: bool

dasf.utils.funcs.weight_gaussian(shape)[source]: Produces a NDArray for a given shape with a Gaussian Distribution in all directions starting from the center

dasf.utils.funcs.weight_radial(shape)[source]: Produces a NDArray for a given shape with a decreasing rate starting from the center