dasf.pipeline.executors

Submodules

Classes

Executor

DaskPBSPipelineExecutor

DaskPipelineExecutor

A pipeline engine based on dask data flow.

DaskTasksPipelineExecutor

A not centric execution engine based on dask.

LocalExecutor

Package Contents

class dasf.pipeline.executors.Executor[source]
property is_connected: bool
Return type:

bool

property info: str
Return type:

str

has_dataset(key)[source]
Return type:

bool

register_dataset(**kwargs)[source]
abstract get_dataset(key)[source]
register_plugin(plugin)[source]
pre_run(pipeline)[source]
post_run(pipeline)[source]
execute(fn, *args, **kwargs)[source]
shutdown()[source]
class dasf.pipeline.executors.DaskPBSPipelineExecutor(**kwargs)[source]

Bases: dasf.pipeline.executors.base.Executor

client
class dasf.pipeline.executors.DaskPipelineExecutor(address=None, port=8786, local=False, use_gpu=False, profiler=None, protocol=None, gpu_allocator='cupy', cluster_kwargs=None, client_kwargs=None)[source]

Bases: dasf.pipeline.executors.base.Executor

A pipeline engine based on dask data flow.

Keyword arguments: address – address of the Dask scheduler (default None). port – port of the Dask scheduler (default 8786). local – kicks off a new local Dask cluster (default False). use_gpu – in conjunction with local, it kicks off a local CUDA Dask

cluster (default False).

profiler – sets a Dask profiler. protocol – sets the Dask protocol (default TCP) gpu_allocator – sets which is the memory allocator for GPU (default cupy). cluster_kwargs – extra Dask parameters like memory, processes, etc. client_kwargs – extra Client parameters.

address
port
local
property ngpus: int
Return type:

int

property is_connected: bool
Return type:

bool

property info: str
Return type:

str

execute(fn, *args, **kwargs)[source]
register_plugin(plugin)[source]
Parameters:

plugin (Union[distributed.diagnostics.plugin.WorkerPlugin, distributed.diagnostics.plugin.NannyPlugin])

register_dataset(**kwargs)[source]
has_dataset(key)[source]
get_dataset(key)[source]
shutdown(gracefully=True)[source]
close()[source]
class dasf.pipeline.executors.DaskTasksPipelineExecutor(address=None, port=8786, local=False, use_gpu=True, profiler=None, protocol=None, gpu_allocator='cupy', cluster_kwargs=None, client_kwargs=None)[source]

Bases: DaskPipelineExecutor

A not centric execution engine based on dask.

Keyword arguments: address – address of the Dask scheduler (default None). port – port of the Dask scheduler (default 8786). local – kicks off a new local Dask cluster (default False). use_gpu – in conjunction with local, it kicks off a local CUDA Dask

cluster (default False).

profiler – sets a Dask profiler. gpu_allocator – sets which is the memory allocator for GPU (default cupy). cluster_kwargs – extra Dask parameters like memory, processes, etc. client_kwargs – extra Client parameters.

_tasks_map
pre_run(pipeline)[source]
post_run(pipeline)[source]
execute(fn, *args, **kwargs)[source]
register_dataset(**kwargs)[source]
has_dataset(key)[source]
get_dataset(key)[source]
shutdown(gracefully=True)[source]
close()[source]
class dasf.pipeline.executors.LocalExecutor(use_gpu=None, backend='numpy', gpu_allocator='cupy')[source]
backend
property ngpus: int
Return type:

int

property is_connected: bool
Return type:

bool

pre_run(pipeline)[source]
post_run(pipeline)[source]
get_backend()[source]
execute(fn, *args, **kwargs)[source]