dasf.transforms

Init module for all transformation structures.

Submodules

• dasf.transforms.base
• dasf.transforms.memory
• dasf.transforms.operations
• dasf.transforms.transforms

Classes

MappedTransform	Class representing a MappedTransform based on Transform
ReductionTransform	Class representing a Reduction based on Transform
TargeteredTransform	Class representing a Targetered Transform operation of the pipeline.
Fit	Class representing a Fit operation of the pipeline.
FitPredict	Class representing a Fit with Predict operation of the pipeline.
FitTransform	Class representing a Fit with Transform operation of the pipeline.
GetParams	Class representing a Get Parameters operation of the pipeline.
Predict	Class representing a Predict operation of the pipeline.
ComputeDaskData	Allow persisting a dask array to memory. It will gather the data blocks
PersistDaskData	Allow persisting a dask array to memory and return a copy of the object.
Reshape	Get a slice of a cube. An inline slice is a section over the x-axis.
SliceArray	Class representing a Transform operation of the pipeline.
SliceArrayByPercent	Class representing a Transform operation of the pipeline.
ArraysToDataFrame	Class representing a Transform operation of the pipeline.
ArrayToHDF5	Class representing a Transform operation of the pipeline.
ArrayToZarr	Class representing a Transform operation of the pipeline.
ExtractData	Extract data from Dataset object
Normalize	Normalize data object
ZarrToArray	Class representing a Transform operation of the pipeline.

Package Contents

class dasf.transforms.MappedTransform(function, depth=None, boundary=None, trim=True, output_chunk=None, drop_axis=None, new_axis=None)

Bases: Transform

Class representing a MappedTransform based on Transform object.

This object refers to any operation that can be done in blocks. In special, for Dask chunks. There are several ways of doing that. This class tries to simplify how the functions are applied into a block.

Parameters

functionCallable

A function that will be applied in a block.

depthtuple

The value of the boundary elements per axis (the default is None).

boundarystr

The type of the boundary. See Dask boundaries for more examples (the default is None).

trimbool

Option to trim the data after an overlap (the default is True).

output_chunktuple

New shape of the output after computing the function (the default is None).

drop_axistuple

Which axis should be deleted after computing the function (the default is None).

new_axistuple

Which axis represent a new axis after computing the function (the default is None).

function

depth

boundary

trim

output_chunk

drop_axis

new_axis

__lazy_transform_generic(X, xp, **kwargs)

_lazy_transform_cpu(X, **kwargs)

Respective lazy transform mocked function for CPUs.

_lazy_transform_gpu(X, **kwargs)

Respective lazy transform mocked function for GPUs.

_transform_cpu(X, **kwargs)

Respective immediate transform mocked function for local CPU(s).

_transform_gpu(X, **kwargs)

Respective immediate transform mocked function for local GPU(s).

transform(X, **kwargs)

Generic transform funtion according executor.

class dasf.transforms.ReductionTransform(output_size, func_aggregate, func_chunk, func_combine=None)

Bases: Transform

Class representing a Reduction based on Transform object.

This is a simple MapReduction operation using Dask.

Parameters

output_sizetuple

The size of the new output.

func_aggregateCallable

The function called to aggregate the result of each chunk.

func_chunkCallable

The function applied in each chunk.

func_combineCallable

The function to combine each reduction of aggregate (the default is None).

output_size

func_aggregate

func_chunk

func_combine

_operation_aggregate_cpu(block, axis=None, keepdims=False)

_operation_aggregate_gpu(block, axis=None, keepdims=False)

_operation_combine_cpu(block, axis=None, keepdims=False)

_operation_combine_gpu(block, axis=None, keepdims=False)

_operation_chunk_cpu(block, axis=None, keepdims=False)

_operation_chunk_gpu(block, axis=None, keepdims=False)

_lazy_transform_cpu(X, *args, **kwargs)

Respective lazy transform mocked function for CPUs.

_lazy_transform_gpu(X, *args, **kwargs)

Respective lazy transform mocked function for GPUs.

_transform_cpu(X, *args, **kwargs)

Respective immediate transform mocked function for local CPU(s).

_transform_gpu(X, *args, **kwargs)

Respective immediate transform mocked function for local GPU(s).

transform(X, *args, **kwargs)

Generic transform funtion according executor.

class dasf.transforms.TargeteredTransform(run_local=None, run_gpu=None)

Bases: Transform

Class representing a Targetered Transform operation of the pipeline.

This specific transform operates according the parameters of the constructor.

Parameters

run_localbool

Define that the operator will run locally and not distributed.

run_gpubool

Define if the operator will use GPU(s) or not.

Constructor of the class TargeteredTransform.

_run_local

_run_gpu

class dasf.transforms.Fit

Bases: Operator

Class representing a Fit operation of the pipeline.

abstract _lazy_fit_cpu(X, y=None, **kwargs)

Respective lazy fit mocked function for CPUs.

abstract _lazy_fit_gpu(X, y=None, **kwargs)

Respective lazy fit mocked function for GPUs.

abstract _fit_cpu(X, y=None, **kwargs)

Respective immediate fit mocked function for local CPU(s).

abstract _fit_gpu(X, y=None, **kwargs)

Respective immediate fit mocked function for local GPU(s).

fit(X, y, sample_weight=None, **kwargs)

Generic fit funtion according executor.

static fit_from_model(model, X, y, sample_weight=None, **kwargs)

Return the model of a previous created object.

class dasf.transforms.FitPredict

Bases: Operator

Class representing a Fit with Predict operation of the pipeline.

abstract _lazy_fit_predict_cpu(X, y=None, **kwargs)

Respective lazy fit with predict mocked function for CPUs.

abstract _lazy_fit_predict_gpu(X, y=None, **kwargs)

Respective lazy fit with predict mocked function for GPUs.

abstract _fit_predict_cpu(X, y=None, **kwargs)

Respective immediate fit with predict mocked function for local CPU(s).

abstract _fit_predict_gpu(X, y=None, **kwargs)

Respective immediate fit with predict mocked function for local GPU(s).

fit_predict(X, y=None, **kwargs)

Generic fit with predict funtion according executor.

static fit_predict_from_model(model, X, y, sample_weight=None, **kwargs)

Return the model of a previous created object.

class dasf.transforms.FitTransform

Bases: Operator

Class representing a Fit with Transform operation of the pipeline.

abstract _lazy_fit_transform_cpu(X, y=None, **kwargs)

Respective lazy fit with transform mocked function for CPUs.

abstract _lazy_fit_transform_gpu(X, y=None, **kwargs)

Respective lazy fit with transform mocked function for GPUs.

abstract _fit_transform_cpu(X, y=None, **kwargs)

Respective immediate fit with transform mocked function for local CPU(s).

abstract _fit_transform_gpu(X, y=None, **kwargs)

Respective immediate fit with transform mocked function for local GPU(s).

fit_transform(X, y=None, **kwargs)

Generic fit with transform funtion according executor.

static fit_transform_from_model(model, X, y, sample_weight=None, **kwargs)

Return the model of a previous created object.

class dasf.transforms.GetParams

Bases: Operator

Class representing a Get Parameters operation of the pipeline.

abstract _lazy_get_params_cpu(deep=True, **kwargs)

Respective lazy get_params mocked function for CPUs.

abstract _lazy_get_params_gpu(deep=True, **kwargs)

Respective lazy get_params mocked function for GPUs.

abstract _get_params_cpu(deep=True, **kwargs)

Respective immediate get_params mocked function for local CPU(s).

abstract _get_params_gpu(deep=True, **kwargs)

Respective immediate get_params mocked function for local GPU(s).

get_params(deep=True, **kwargs)

Generic get_params funtion according executor.

class dasf.transforms.Predict

Bases: Operator

Class representing a Predict operation of the pipeline.

abstract _lazy_predict_cpu(X, sample_weight=None, **kwargs)

Respective lazy predict mocked function for CPUs.

abstract _lazy_predict_gpu(X, sample_weight=None, **kwargs)

Respective lazy predict mocked function for GPUs.

abstract _predict_cpu(X, sample_weight=None, **kwargs)

Respective immediate predict mocked function for local CPU(s).

abstract _predict_gpu(X, sample_weight=None, **kwargs)

Respective immediate predict mocked function for local GPU(s).

predict(X, sample_weight=None, **kwargs)

Generic predict funtion according executor.

static predict_from_model(model, X, sample_weight=None, **kwargs)

Return the model of a previous created object.

class dasf.transforms.ComputeDaskData

Bases: dasf.transforms.base.Transform

Allow persisting a dask array to memory. It will gather the data blocks from all workers and resembles locally.

__lazy_transform_generic(X)

_lazy_transform_cpu(X)

Respective lazy transform mocked function for CPUs.

_lazy_transform_gpu(X)

Respective lazy transform mocked function for GPUs.

_transform_cpu(X)

Respective immediate transform mocked function for local CPU(s).

_transform_gpu(X)

Respective immediate transform mocked function for local GPU(s).

class dasf.transforms.PersistDaskData

Bases: dasf.transforms.base.Transform

Allow persisting a dask array to memory and return a copy of the object. It will gather the data blocks from all workers and resembles locally.

__lazy_transform_generic(X)

_lazy_transform_cpu(X)

Respective lazy transform mocked function for CPUs.

_lazy_transform_gpu(X)

Respective lazy transform mocked function for GPUs.

_transform_cpu(X)

Respective immediate transform mocked function for local CPU(s).

_transform_gpu(X)

Respective immediate transform mocked function for local GPU(s).

class dasf.transforms.Reshape(shape=None)

Bases: dasf.transforms.base.Fit

Get a slice of a cube. An inline slice is a section over the x-axis.

Parameters

iline_indexint

The index of the inline to get.

shape

fit(X, y=None)

Generic fit funtion according executor.

Parameters:

shape (tuple)

class dasf.transforms.SliceArray(output_size)

Bases: dasf.transforms.base.Transform

Class representing a Transform operation of the pipeline.

x

transform(X)

Generic transform funtion according executor.

class dasf.transforms.SliceArrayByPercent(x=100.0, y=100.0, z=100.0)

Bases: dasf.transforms.base.Transform

Class representing a Transform operation of the pipeline.

x

y

z

transform(X)

Generic transform funtion according executor.

class dasf.transforms.ArraysToDataFrame

Bases: dasf.transforms.base.Transform

Class representing a Transform operation of the pipeline.

_build_dataframe(data, columns, xp, df)

_lazy_transform(xp, df, **kwargs)

_lazy_transform_cpu(X=None, **kwargs)

Respective lazy transform mocked function for CPUs.

_lazy_transform_gpu(X=None, **kwargs)

Respective lazy transform mocked function for GPUs.

_transform(xp, df, **kwargs)

_transform_cpu(X=None, **kwargs)

Respective immediate transform mocked function for local CPU(s).

_transform_gpu(X=None, **kwargs)

Respective immediate transform mocked function for local GPU(s).

class dasf.transforms.ArrayToHDF5(dataset_path, chunks=None, save=True, filename=None)

Bases: dasf.transforms.base.Transform

Class representing a Transform operation of the pipeline.

dataset_path

chunks

save = True

filename

static _convert_filename(url)

_lazy_transform_generic_all(data)

_transform_generic_all(data)

_lazy_transform_generic(X, **kwargs)

_transform_generic(X, **kwargs)

_lazy_transform_gpu(X, **kwargs)

Respective lazy transform mocked function for GPUs.

_lazy_transform_cpu(X, **kwargs)

Respective lazy transform mocked function for CPUs.

_transform_gpu(X, **kwargs)

Respective immediate transform mocked function for local GPU(s).

_transform_cpu(X, **kwargs)

Respective immediate transform mocked function for local CPU(s).

class dasf.transforms.ArrayToZarr(chunks=None, save=True, filename=None)

Bases: dasf.transforms.base.Transform

Class representing a Transform operation of the pipeline.

chunks

save = True

filename

static _convert_filename(url)

_lazy_transform_generic_all(data)

_transform_generic_all(data, chunks, **kwargs)

_lazy_transform_generic(X, **kwargs)

_transform_generic(X, **kwargs)

_lazy_transform_gpu(X, **kwargs)

Respective lazy transform mocked function for GPUs.

_lazy_transform_cpu(X, **kwargs)

Respective lazy transform mocked function for CPUs.

_transform_gpu(X, **kwargs)

Respective immediate transform mocked function for local GPU(s).

_transform_cpu(X, **kwargs)

Respective immediate transform mocked function for local CPU(s).

class dasf.transforms.ExtractData

Bases: dasf.transforms.base.Transform

Extract data from Dataset object

transform(X)

Extract data from datasets that contains internal data.

Parameters

XDataset-like

A dataset object that could be anything that contains an internal structure representing the raw data.

Returns

dataAny

Any representation of the internal Dataset data.

class dasf.transforms.Normalize

Bases: dasf.transforms.base.Transform

Normalize data object

transform(X)

Normalize the input data based on mean() and std().

Parameters

XAny

Any data that could be normalized based on mean and standard deviation.

Returns

dataAny

Normalized data

class dasf.transforms.ZarrToArray(chunks=None, save=True, filename=None)

Bases: dasf.transforms.base.Transform

Class representing a Transform operation of the pipeline.

chunks

save

filename

static _convert_filename(url)

transform(X)

Generic transform funtion according executor.