dasf.pipeline.executors.dask

Dask executor module.

Classes

`DaskPipelineExecutor`	A pipeline engine based on dask data flow.
`DaskTasksPipelineExecutor`	A not centric execution engine based on dask.
`DaskPBSPipelineExecutor`

Functions

setup_dask_protocol([protocol])

Module Contents

dasf.pipeline.executors.dask.setup_dask_protocol(protocol=None)[source]

class dasf.pipeline.executors.dask.DaskPipelineExecutor(address=None, port=8786, local=False, use_gpu=False, profiler=None, protocol=None, gpu_allocator='cupy', cluster_kwargs=None, client_kwargs=None)[source]

Bases: dasf.pipeline.executors.base.Executor

A pipeline engine based on dask data flow.

Keyword arguments: address – address of the Dask scheduler (default None). port – port of the Dask scheduler (default 8786). local – kicks off a new local Dask cluster (default False). use_gpu – in conjunction with local, it kicks off a local CUDA Dask

cluster (default False).

profiler – sets a Dask profiler. protocol – sets the Dask protocol (default TCP) gpu_allocator – sets which is the memory allocator for GPU (default cupy). cluster_kwargs – extra Dask parameters like memory, processes, etc. client_kwargs – extra Client parameters.

address

port

local

property ngpus: int

Return type:: int

property is_connected: bool

Return type:: bool

property info: str

Return type:: str

execute(fn, *args, **kwargs)[source]

register_plugin(plugin)[source]

Parameters:: plugin (Union[distributed.diagnostics.plugin.WorkerPlugin, distributed.diagnostics.plugin.NannyPlugin])

register_dataset(**kwargs)[source]

has_dataset(key)[source]

get_dataset(key)[source]

shutdown(gracefully=True)[source]

close()[source]

class dasf.pipeline.executors.dask.DaskTasksPipelineExecutor(address=None, port=8786, local=False, use_gpu=True, profiler=None, protocol=None, gpu_allocator='cupy', cluster_kwargs=None, client_kwargs=None)[source]

Bases: DaskPipelineExecutor

A not centric execution engine based on dask.

Keyword arguments: address – address of the Dask scheduler (default None). port – port of the Dask scheduler (default 8786). local – kicks off a new local Dask cluster (default False). use_gpu – in conjunction with local, it kicks off a local CUDA Dask

cluster (default False).

profiler – sets a Dask profiler. gpu_allocator – sets which is the memory allocator for GPU (default cupy). cluster_kwargs – extra Dask parameters like memory, processes, etc. client_kwargs – extra Client parameters.

_tasks_map

pre_run(pipeline)[source]

post_run(pipeline)[source]

execute(fn, *args, **kwargs)[source]

register_dataset(**kwargs)[source]

has_dataset(key)[source]

get_dataset(key)[source]

shutdown(gracefully=True)[source]

close()[source]

class dasf.pipeline.executors.dask.DaskPBSPipelineExecutor(**kwargs)[source]

Bases: dasf.pipeline.executors.base.Executor

client