dasf.pipeline.executors
=======================

.. py:module:: dasf.pipeline.executors


Submodules
----------

.. toctree::
   :maxdepth: 1

   /autoapi/dasf/pipeline/executors/base/index
   /autoapi/dasf/pipeline/executors/dask/index
   /autoapi/dasf/pipeline/executors/ray/index
   /autoapi/dasf/pipeline/executors/wrapper/index


Classes
-------

.. autoapisummary::

   dasf.pipeline.executors.Executor
   dasf.pipeline.executors.DaskPBSPipelineExecutor
   dasf.pipeline.executors.DaskPipelineExecutor
   dasf.pipeline.executors.DaskTasksPipelineExecutor
   dasf.pipeline.executors.LocalExecutor


Package Contents
----------------

.. py:class:: Executor

   .. py:property:: is_connected
      :type: bool



   .. py:property:: info
      :type: str



   .. py:method:: has_dataset(key)


   .. py:method:: register_dataset(**kwargs)


   .. py:method:: get_dataset(key)
      :abstractmethod:



   .. py:method:: register_plugin(plugin)


   .. py:method:: pre_run(pipeline)


   .. py:method:: post_run(pipeline)


   .. py:method:: execute(fn, *args, **kwargs)


   .. py:method:: shutdown()


.. py:class:: DaskPBSPipelineExecutor(**kwargs)

   Bases: :py:obj:`dasf.pipeline.executors.base.Executor`


   .. py:attribute:: client


.. py:class:: DaskPipelineExecutor(address=None, port=8786, local=False, use_gpu=False, profiler=None, protocol=None, gpu_allocator='cupy', cluster_kwargs=None, client_kwargs=None)

   Bases: :py:obj:`dasf.pipeline.executors.base.Executor`


   A pipeline engine based on dask data flow.

   Keyword arguments:
   address -- address of the Dask scheduler (default None).
   port -- port of the Dask scheduler (default 8786).
   local -- kicks off a new local Dask cluster (default False).
   use_gpu -- in conjunction with `local`, it kicks off a local CUDA Dask
               cluster (default False).
   profiler -- sets a Dask profiler.
   protocol -- sets the Dask protocol (default TCP)
   gpu_allocator -- sets which is the memory allocator for GPU (default cupy).
   cluster_kwargs -- extra Dask parameters like memory, processes, etc.
   client_kwargs -- extra Client parameters.


   .. py:attribute:: address


   .. py:attribute:: port


   .. py:attribute:: local


   .. py:property:: ngpus
      :type: int



   .. py:property:: is_connected
      :type: bool



   .. py:property:: info
      :type: str



   .. py:method:: execute(fn, *args, **kwargs)


   .. py:method:: register_plugin(plugin)


   .. py:method:: register_dataset(**kwargs)


   .. py:method:: has_dataset(key)


   .. py:method:: get_dataset(key)


   .. py:method:: shutdown(gracefully=True)


   .. py:method:: close()


.. py:class:: DaskTasksPipelineExecutor(address=None, port=8786, local=False, use_gpu=True, profiler=None, protocol=None, gpu_allocator='cupy', cluster_kwargs=None, client_kwargs=None)

   Bases: :py:obj:`DaskPipelineExecutor`


   A not centric execution engine based on dask.

   Keyword arguments:
   address -- address of the Dask scheduler (default None).
   port -- port of the Dask scheduler (default 8786).
   local -- kicks off a new local Dask cluster (default False).
   use_gpu -- in conjunction with `local`, it kicks off a local CUDA Dask
               cluster (default False).
   profiler -- sets a Dask profiler.
   gpu_allocator -- sets which is the memory allocator for GPU (default cupy).
   cluster_kwargs -- extra Dask parameters like memory, processes, etc.
   client_kwargs -- extra Client parameters.


   .. py:attribute:: _tasks_map


   .. py:method:: pre_run(pipeline)


   .. py:method:: post_run(pipeline)


   .. py:method:: execute(fn, *args, **kwargs)


   .. py:method:: register_dataset(**kwargs)


   .. py:method:: has_dataset(key)


   .. py:method:: get_dataset(key)


   .. py:method:: shutdown(gracefully=True)


   .. py:method:: close()


.. py:class:: LocalExecutor(use_gpu=None, backend='numpy', gpu_allocator='cupy')

   .. py:attribute:: backend


   .. py:property:: ngpus
      :type: int



   .. py:property:: is_connected
      :type: bool



   .. py:method:: pre_run(pipeline)


   .. py:method:: post_run(pipeline)


   .. py:method:: get_backend()


   .. py:method:: execute(fn, *args, **kwargs)