dasf.ml.preprocessing

Init module for Pre Processing ML algorithms.

Submodules

dasf.ml.preprocessing.standardscaler

Classes

StandardScaler

Standardize features by removing the mean and scaling to unit variance.

Package Contents

class dasf.ml.preprocessing.StandardScaler(copy=True, with_mean=True, with_std=True, **kwargs)[source]

Bases: dasf.transforms.base.Fit, dasf.transforms.base.FitTransform, dasf.transforms.base.TargeteredTransform

Standardize features by removing the mean and scaling to unit variance.

The standard score of a sample x is calculated as:

z = (x - u) / s

where u is the mean of the training samples or zero if with_mean=False, and s is the standard deviation of the training samples or one if with_std=False.

Centering and scaling happen independently on each feature by computing the relevant statistics on the samples in the training set. Mean and standard deviation are then stored to be used on later data using transform().

Standardization of a dataset is a common requirement for many machine learning estimators: they might behave badly if the individual features do not more or less look like standard normally distributed data (e.g. Gaussian with 0 mean and unit variance).

For instance many elements used in the objective function of a learning algorithm (such as the RBF kernel of Support Vector Machines or the L1 and L2 regularizers of linear models) assume that all features are centered around 0 and have variance in the same order. If a feature has a variance that is orders of magnitude larger than others, it might dominate the objective function and make the estimator unable to learn from other features correctly as expected.

StandardScaler is sensitive to outliers, and the features may scale differently from each other in the presence of outliers. For an example visualization, refer to Compare StandardScaler with other scalers.

This scaler can also be applied to sparse CSR or CSC matrices by passing with_mean=False to avoid breaking the sparsity structure of the data.

Parameters

copybool, default=True: If False, try to avoid a copy and do inplace scaling instead. This is not guaranteed to always work inplace; e.g. if the data is not a NumPy array or scipy.sparse CSR matrix, a copy may still be returned.
with_meanbool, default=True: If True, center the data before scaling. This does not work (and will raise an exception) when attempted on sparse matrices, because centering them entails building a dense matrix which in common use cases is likely to be too large to fit in memory.
with_stdbool, default=True: If True, scale the data to unit variance (or equivalently, unit standard deviation).

Attributes

scale_ndarray of shape (n_features,) or None: Per feature relative scaling of the data to achieve zero mean and unit variance. Generally this is calculated using np.sqrt(var_). If a variance is zero, we can’t achieve unit variance, and the data is left as-is, giving a scaling factor of 1. scale_ is equal to None when with_std=False.

Added in version 0.17: scale_
mean_ndarray of shape (n_features,) or None: The mean value for each feature in the training set. Equal to None when with_mean=False and with_std=False.
var_ndarray of shape (n_features,) or None: The variance for each feature in the training set. Used to compute scale_. Equal to None when with_mean=False and with_std=False.
n_features_in_int: Number of features seen during fit.

Added in version 0.24.
feature_names_in_ndarray of shape (n_features_in_,): Names of features seen during fit. Defined only when X has feature names that are all strings.

Added in version 1.0.
n_samples_seen_int or ndarray of shape (n_features,): The number of samples processed by the estimator for each feature. If there are no missing samples, the n_samples_seen will be an integer, otherwise it will be an array of dtype int. If sample_weights are used it will be a float (if no missing data) or an array of dtype float that sums the weights seen so far. Will be reset on new calls to fit, but increments across partial_fit calls.

Constructor of the class StandardScaler.

_lazy_fit_cpu(X, y=None, sample_weight=None)[source]

Compute the mean and std to be used for later scaling using Dask with CPU only.

Parameters

X{array-like, sparse matrix} of shape (n_samples, n_features): The data used to compute the mean and standard deviation used for later scaling along the features axis.
yNone: Ignored.
sample_weightarray-like of shape (n_samples,), default=None: Individual weights for each sample.

Returns

selfobject: Fitted scaler.

_lazy_fit_gpu(X, y=None, sample_weight=None)[source]

Compute the mean and std to be used for later scaling using Dask with GPU only.

Parameters

X{array-like, sparse matrix} of shape (n_samples, n_features): The data used to compute the mean and standard deviation used for later scaling along the features axis.
yNone: Ignored.
sample_weightarray-like of shape (n_samples,), default=None: Individual weights for each sample.

Returns

selfobject: Fitted scaler.

_fit_cpu(X, y=None, sample_weight=None)[source]

Compute the mean and std to be used for later scaling using CPU only.

Parameters

X{array-like, sparse matrix} of shape (n_samples, n_features): The data used to compute the mean and standard deviation used for later scaling along the features axis.
yNone: Ignored.
sample_weightarray-like of shape (n_samples,), default=None: Individual weights for each sample.

Added in version 0.24: parameter sample_weight support to StandardScaler.

Returns

selfobject: Fitted scaler.

_fit_gpu(X, y=None, sample_weight=None)[source]

Compute the mean and std to be used for later scaling using CPU only.

Parameters

X{array-like, sparse matrix} of shape (n_samples, n_features): The data used to compute the mean and standard deviation used for later scaling along the features axis.
yNone: Ignored.
sample_weightarray-like of shape (n_samples,), default=None: Individual weights for each sample.

Returns

selfobject: Fitted scaler.

_lazy_fit_transform_cpu(X, y=None)[source]

Fit to data, then transform it using Dask with CPUs only.

Fits transformer to X and y with optional parameters fit_params and returns a transformed version of X.

Parameters

Xarray-like of shape (n_samples, n_features): Input samples.
yarray-like of shape (n_samples,) or (n_samples, n_outputs), default=None: Target values (None for unsupervised transformations).
**fit_paramsdict: Additional fit parameters.

Returns

X_newndarray array of shape (n_samples, n_features_new): Transformed array.

_lazy_fit_transform_gpu(X, y=None)[source]

Fit to data, then transform it using Dask with GPUs only.

Fits transformer to X and y with optional parameters fit_params and returns a transformed version of X.

Parameters

Xarray-like of shape (n_samples, n_features): Input samples.
yarray-like of shape (n_samples,) or (n_samples, n_outputs), default=None: Target values (None for unsupervised transformations).
**fit_paramsdict: Additional fit parameters.

Returns

X_newndarray array of shape (n_samples, n_features_new): Transformed array.

_fit_transform_cpu(X, y=None)[source]

Fit to data, then transform it using CPU only.

Fits transformer to X and y with optional parameters fit_params and returns a transformed version of X.

Parameters

Xarray-like of shape (n_samples, n_features): Input samples.
yarray-like of shape (n_samples,) or (n_samples, n_outputs), default=None: Target values (None for unsupervised transformations).
**fit_paramsdict: Additional fit parameters.

Returns

X_newndarray array of shape (n_samples, n_features_new): Transformed array.

_fit_transform_gpu(X, y=None)[source]

Fit to data, then transform it using GPU only.

Fits transformer to X and y with optional parameters fit_params and returns a transformed version of X.

Parameters

Xarray-like of shape (n_samples, n_features): Input samples.
yarray-like of shape (n_samples,) or (n_samples, n_outputs), default=None: Target values (None for unsupervised transformations).
**fit_paramsdict: Additional fit parameters.

Returns

X_newndarray array of shape (n_samples, n_features_new): Transformed array.

_lazy_partial_fit_cpu(X, y=None, sample_weight=None)[source]

Online computation of mean and std on X for later scaling using Dask with CPUs only.

All of X is processed as a single batch. This is intended for cases when fit() is not feasible due to very large number of n_samples or because X is read from a continuous stream.

The algorithm for incremental mean and std is given in Equation 1.5a,b in Chan, Tony F., Gene H. Golub, and Randall J. LeVeque. “Algorithms for computing the sample variance: Analysis and recommendations.” The American Statistician 37.3 (1983): 242-247:

Parameters

X{array-like, sparse matrix} of shape (n_samples, n_features): The data used to compute the mean and standard deviation used for later scaling along the features axis.
yNone: Ignored.
sample_weightarray-like of shape (n_samples,), default=None: Individual weights for each sample.

Returns

selfobject: Fitted scaler.

_lazy_partial_fit_gpu(X, y=None, sample_weight=None)[source]

Online computation of mean and std on X for later scaling using Dask with GPUs only.

All of X is processed as a single batch. This is intended for cases when fit() is not feasible due to very large number of n_samples or because X is read from a continuous stream.

The algorithm for incremental mean and std is given in Equation 1.5a,b in Chan, Tony F., Gene H. Golub, and Randall J. LeVeque. “Algorithms for computing the sample variance: Analysis and recommendations.” The American Statistician 37.3 (1983): 242-247:

Parameters

X{array-like, sparse matrix} of shape (n_samples, n_features): The data used to compute the mean and standard deviation used for later scaling along the features axis.
yNone: Ignored.
sample_weightarray-like of shape (n_samples,), default=None: Individual weights for each sample.

Returns

selfobject: Fitted scaler.

_partial_fit_cpu(X, y=None, sample_weight=None)[source]

Online computation of mean and std on X for later scaling CPU only.

All of X is processed as a single batch. This is intended for cases when fit() is not feasible due to very large number of n_samples or because X is read from a continuous stream.

The algorithm for incremental mean and std is given in Equation 1.5a,b in Chan, Tony F., Gene H. Golub, and Randall J. LeVeque. “Algorithms for computing the sample variance: Analysis and recommendations.” The American Statistician 37.3 (1983): 242-247:

Parameters

X{array-like, sparse matrix} of shape (n_samples, n_features): The data used to compute the mean and standard deviation used for later scaling along the features axis.
yNone: Ignored.
sample_weightarray-like of shape (n_samples,), default=None: Individual weights for each sample.

Returns

selfobject: Fitted scaler.

_partial_fit_gpu(X, y=None, sample_weight=None)[source]

Online computation of mean and std on X for later scaling using GPU only.

All of X is processed as a single batch. This is intended for cases when fit() is not feasible due to very large number of n_samples or because X is read from a continuous stream.

The algorithm for incremental mean and std is given in Equation 1.5a,b in Chan, Tony F., Gene H. Golub, and Randall J. LeVeque. “Algorithms for computing the sample variance: Analysis and recommendations.” The American Statistician 37.3 (1983): 242-247:

Parameters

X{array-like, sparse matrix} of shape (n_samples, n_features): The data used to compute the mean and standard deviation used for later scaling along the features axis.
yNone: Ignored.
sample_weightarray-like of shape (n_samples,), default=None: Individual weights for each sample.

Returns

selfobject: Fitted scaler.

_lazy_transform_cpu(X, copy=None)[source]

Perform standardization by centering and scaling using Dask with CPUs only.

Parameters

X{array-like, sparse matrix of shape (n_samples, n_features): The data used to scale along the features axis.
copybool, default=None: Copy the input X or not.

Returns

X_tr{ndarray, sparse matrix} of shape (n_samples, n_features): Transformed array.

_lazy_transform_gpu(X, copy=None)[source]

Perform standardization by centering and scaling using Dask with GPUs only.

Parameters

X{array-like, sparse matrix of shape (n_samples, n_features): The data used to scale along the features axis.
copybool, default=None: Copy the input X or not.

Returns

X_tr{ndarray, sparse matrix} of shape (n_samples, n_features): Transformed array.

_transform_cpu(X, copy=None)[source]

Perform standardization by centering and scaling using CPU only.

Parameters

X{array-like, sparse matrix of shape (n_samples, n_features): The data used to scale along the features axis.
copybool, default=None: Copy the input X or not.

Returns

X_tr{ndarray, sparse matrix} of shape (n_samples, n_features): Transformed array.

_transform_gpu(X, copy=None)[source]

Perform standardization by centering and scaling using GPU only.

Parameters

X{array-like, sparse matrix of shape (n_samples, n_features): The data used to scale along the features axis.
copybool, default=None: Copy the input X or not.

Returns

X_tr{ndarray, sparse matrix} of shape (n_samples, n_features): Transformed array.

_lazy_inverse_transform_cpu(X, copy=None)[source]

Undo the scaling of X according to feature_range using Dask with CPUs only.

Parameters

Xarray-like of shape (n_samples, n_features): Input data that will be transformed. It cannot be sparse.

Returns

Xtndarray of shape (n_samples, n_features): Transformed data.

_lazy_inverse_transform_gpu(X, copy=None)[source]

Undo the scaling of X according to feature_range using Dask with GPUs only.

Parameters

Xarray-like of shape (n_samples, n_features): Input data that will be transformed. It cannot be sparse.

Returns

Xtndarray of shape (n_samples, n_features): Transformed data.

_inverse_transform_cpu(X, copy=None)[source]

Undo the scaling of X according to feature_range using CPU only.

Parameters

Xarray-like of shape (n_samples, n_features): Input data that will be transformed. It cannot be sparse.

Returns

Xtndarray of shape (n_samples, n_features): Transformed data.

_inverse_transform_gpu(X, copy=None)[source]

Undo the scaling of X according to feature_range using GPU only.

Parameters

Xarray-like of shape (n_samples, n_features): Input data that will be transformed. It cannot be sparse.

Returns

Xtndarray of shape (n_samples, n_features): Transformed data.