minerva.data.readers

Submodules

• minerva.data.readers.base_file_iterator
• minerva.data.readers.csv_reader
• minerva.data.readers.index_reader
• minerva.data.readers.mdio_reader
• minerva.data.readers.multi_reader
• minerva.data.readers.patched_array_reader
• minerva.data.readers.png_reader
• minerva.data.readers.reader
• minerva.data.readers.tabular_reader
• minerva.data.readers.tiff_reader
• minerva.data.readers.zarr_reader

Classes

LazyPaddedPatchedArrayReader	Reads patches from a NumPy array.
LazyPaddedPatchedMDIOReader	Reads patches from a MDIO array. This class is a subclass of
LazyPaddedPatchedZarrReader	Reads patches from a Zarr array. This class is a subclass of
NumpyArrayReader	This class is used to read data from a NumPy array. It is designed to generate
PNGReader	A base class for iterating over files in a directory in a custom sorted
PatchedArrayReader	This class is used to read data from a NumPy array. It is designed to generate
PatchedMDIOReader	Reads patches from a MDIO array. This class is a subclass of
PatchedZarrReader	Reads patches from a Zarr array. This class is a subclass of
TiffReader	A base class for iterating over files in a directory in a custom sorted
_Reader	Base class for readers. Readers define an ordered collection of data and

Package Contents

class minerva.data.readers.LazyPaddedPatchedArrayReader(data, data_shape, stride=None, pad_width=None, pad_mode='constant', pad_kwargs=None)

Bases: PatchedArrayReader

Reads patches from a NumPy array. This class is a subclass of PatchedArrayReader and is designed to perform padding only when the patch consumed by __get_item__ is in a region that uses the padding (boundary regions). If no padding is necessary, use PatchedArrayReader.

Reads data from a NumPy array and generates patches from it.

Parameters

dataArrayLike

The input array from which patches are generated.

data_shapeTuple[int, …]

The shape of the patches to be extracted. This will be the shape of the subarray that is returned when a patch is accessed using __getitem__.

strideTuple[int, …], optional

The stride between consecutive patches. If None, the stide will be the same as data_shape. By default None

pad_widthTuple[Tuple[int, int], …], optional

The width of padding to be applied to the data array. By default None, that is, no padding is applied. Check the documentation of numpy.pad for more information.

pad_modestr, optional

The padding mode, by default “constant”. Check the documentation of numpy.pad for more information.

pad_kwargsdict, optional

Additional keyword arguments for padding, by default None

Examples

```python >>> import numpy as np >>> # Generate a 10x10 array >>> data = np.arange(100).reshape(10, 10) >>> # Create a reader that generates 5x5 patches with a stride of 2 in the >>> # first dimension and 5 in the second dimension. >>> reader = PatchedArrayReader( >>> data, >>> data_shape=(5, 5), >>> stride=(2, 5), >>> ) >>> # Printing the number of patches that can be extracted from the data >>> print(len(reader)) 6 >>> # Printing the indices of the patches >>> print(reader.indices) [(0, 0), (0, 5), (2, 0), (2, 5), (4, 0), (4, 5)] >>> # Fetch the first patch and print its shape >>> print(reader[0].shape) (5, 5) >>> # Fetch the third patch and print its content >>> print(reader[2]) [[20 21 22 23 24]

[30 31 32 33 34] [40 41 42 43 44] [50 51 52 53 54] [60 61 62 63 64]]

```

__getitem__(idx)

Fetch a patch from the data array.

Parameters

idxint

The index of the patch to be fetched.

Returns

np.ndarray

The patch that was fetched from the data array with shape data_shape

Parameters:

idx (int)

Return type:

numpy.ndarray

_get_patches()

Compute the left upper corner indices of the patches that will be extracted from the data array. The patches are extracted with a stride between them. A list of indices is returned, where each index is a tuple of integers representing the coordinates of the left upper corner of the patches.

Returns

List[Tuple[int, …]]

A list of indices (coordinates) representing the left upper corner of the patches.

Return type:

List[Tuple[str, Tuple[int, Ellipsis]]]

_pad_data(pad_width, mode='constant', **kwargs)

Apply padding to the data array.

Parameters

pad_widthTuple[Tuple[int, int], …]

The width of padding to be applied to the data array

modestr, optional

The padding mode, by default “constant”

Parameters:

pad_width (Tuple[Tuple[int, int], Ellipsis])

Parameters:

• data (numpy.typing.ArrayLike)

• data_shape (Tuple[int, Ellipsis])

• stride (Optional[Tuple[int, Ellipsis]])

• pad_width (Optional[Tuple[Tuple[int, int], Ellipsis]])

• pad_mode (str)

• pad_kwargs (Optional[Dict])

class minerva.data.readers.LazyPaddedPatchedMDIOReader(*args, path, mdio_data='data/chunked_012', **kwargs)

Bases: minerva.data.readers.zarr_reader.LazyPaddedPatchedZarrReader

Reads patches from a MDIO array. This class is a subclass of LazyPaddedPatchedZarrReader and is designed to read patches from the the data Zarr array inside thre MDIO array subdirectory, performing padding in a lazy manner (padding is done in __getitem__ call). If no padding is necessary, use PatchedMDIOReader.

Reads patches from a MDIO array. This class is a subclass of LazyPaddedPatchedZarrReader. All other parameters are the same as parent class. Please refer to the documentation of LazyPaddedPatchedZarrReader for more information.

Parameters

pathPathLike

Path to the MDIO array.

mdio_data: str

Name of data array inside MDIO subdirectory. Defaults to ‘data/chunked_012’

Notes

1. The MDIO array is expected to be stored on disk. If the array is not stored on disk, it is recommended to use the LazyPaddedPatchedArrayReader class instead.

2. Padding is computed when necessary during __getitem__ calls.

Examples

```python >>> from pathlib import Path >>> data_path = Path(“data.mdio”) >>> reader = LazyPaddedPatchedMDIOReader( … path=data_path, … data_shape=(5, 5), … stride=(2, 5), … pad_width=((1,1), (0,2)), … ) >>> print(len(reader)) >>> print(reader[0])

mdio_path

Parameters:

• path (minerva.utils.typing.PathLike)

• mdio_data (str)

class minerva.data.readers.LazyPaddedPatchedZarrReader(*args, path, **kwargs)

Bases: minerva.data.readers.patched_array_reader.LazyPaddedPatchedArrayReader

Reads patches from a Zarr array. This class is a subclass of LazyPaddedPatchedArrayReader and is designed to read patches from a Zarr array, performing padding in a lazy manner (padding is done in __getitem__ call). If no padding is necessary, use PatchedZarrReader.

Reads patches from a Zarr array. This class is a subclass of LazyPaddedPatchedArrayReader. All other parameters are the same as parent class. Please refer to the documentation of LazyPaddedPatchedArrayReader for more information. This class can be used when padding is necessary and the whole dataset doesn’t fit into memory.

Parameters

pathPathLike

Path to the Zarr array.

Notes

1. The Zarr array is expected to be stored on disk. If the array is not stored on disk, it is recommended to use the LazyPaddedPatchedArrayReader class instead.

2. Padding is computed when necessary during __getitem__ calls.

Examples

```python >>> from pathlib import Path >>> data_path = Path(“data.zarr”) >>> reader = LazyPaddedPatchedZarrReader( … path=data_path, … data_shape=(5, 5), … stride=(2, 5), … pad_width=((1,1), (0,2)), … ) >>> print(len(reader)) >>> print(reader[0])

path

Parameters:

path (minerva.utils.typing.PathLike)

class minerva.data.readers.NumpyArrayReader(data, data_shape, stride=None, pad_width=None, pad_mode='constant', pad_kwargs=None, allow_pickle=True, npz_key=None)

Bases: PatchedArrayReader

This class is used to read data from a NumPy array. It is designed to generate patches from the data and provides sequential access to them. This class can serve as a base class for other readers.

Assumptions: - The input data is expected to be a NumPy-like array, that is, it should

support NumPy-like indexing.

• Patches are fixed-size subarrays of the data.

• Patches can have overlap between them.

Reads data from a NumPy array and generates patches from it.

Parameters

dataArrayLike

The input array from which patches are generated.

data_shapeTuple[int, …]

The shape of the patches to be extracted. This will be the shape of the subarray that is returned when a patch is accessed using __getitem__.

strideTuple[int, …], optional

The stride between consecutive patches. If None, the stide will be the same as data_shape. By default None

pad_widthTuple[Tuple[int, int], …], optional

The width of padding to be applied to the data array. By default None, that is, no padding is applied. Check the documentation of numpy.pad for more information.

pad_modestr, optional

The padding mode, by default “constant”. Check the documentation of numpy.pad for more information.

pad_kwargsdict, optional

Additional keyword arguments for padding, by default None

Examples

```python >>> import numpy as np >>> # Generate a 10x10 array >>> data = np.arange(100).reshape(10, 10) >>> # Create a reader that generates 5x5 patches with a stride of 2 in the >>> # first dimension and 5 in the second dimension. >>> reader = PatchedArrayReader( >>> data, >>> data_shape=(5, 5), >>> stride=(2, 5), >>> ) >>> # Printing the number of patches that can be extracted from the data >>> print(len(reader)) 6 >>> # Printing the indices of the patches >>> print(reader.indices) [(0, 0), (0, 5), (2, 0), (2, 5), (4, 0), (4, 5)] >>> # Fetch the first patch and print its shape >>> print(reader[0].shape) (5, 5) >>> # Fetch the third patch and print its content >>> print(reader[2]) [[20 21 22 23 24]

[30 31 32 33 34] [40 41 42 43 44] [50 51 52 53 54] [60 61 62 63 64]]

```

Parameters:

• data (Union[numpy.typing.ArrayLike, minerva.utils.typing.PathLike])

• data_shape (Tuple[int, Ellipsis])

• stride (Optional[Tuple[int, Ellipsis]])

• pad_width (Optional[Tuple[Tuple[int, int], Ellipsis]])

• pad_mode (str)

• pad_kwargs (Optional[Dict])

• allow_pickle (bool)

• npz_key (Optional[str])

class minerva.data.readers.PNGReader(path, sort_method=None, delimiter=None, key_index=0, reverse=False)

Bases: minerva.data.readers.base_file_iterator.BaseFileIterator

A base class for iterating over files in a directory in a custom sorted order.

Load image files from a directory.

Parameters

pathUnion[Path, str]

The path to the directory containing the image files. Files will be searched recursively.

sort_methodOptional[List[str]], optional

A list specifying how to sort each part of the filename. Each element can be either “text” (lexicographical) or “numeric” (numerically). By default, None, which will use “numeric” if numeric parts are detected.

delimiterOptional[str], optional

The delimiter to split filenames into components, by default None.

key_indexUnion[int, List[int]], optional

The index (or list of indices) of the part(s) of the filename to use for sorting. If a list is provided, files will be sorted based on multiple parts in sequence. Thus, first by the part at index 0, then by the part at index 1, and so on. By default 0.

reversebool, optional

Whether to sort in reverse order, by default False.

Raises

NotADirectoryError

If the path is not a directory.

__getitem__(index)

Retrieve the PNG file at the specified index.

Parameters:

index (int)

Return type:

numpy.ndarray

__str__()

Return type:

str

root_dir

Parameters:

• path (minerva.utils.typing.PathLike)

• sort_method (Optional[List[str]])

• delimiter (Optional[str])

• key_index (Union[int, List[int]])

• reverse (bool)

class minerva.data.readers.PatchedArrayReader(data, data_shape, stride=None, pad_width=None, pad_mode='constant', pad_kwargs=None)

Bases: minerva.data.readers.reader._Reader

This class is used to read data from a NumPy array. It is designed to generate patches from the data and provides sequential access to them. This class can serve as a base class for other readers.

Assumptions: - The input data is expected to be a NumPy-like array, that is, it should

support NumPy-like indexing.

• Patches are fixed-size subarrays of the data.

• Patches can have overlap between them.

Reads data from a NumPy array and generates patches from it.

Parameters

dataArrayLike

The input array from which patches are generated.

data_shapeTuple[int, …]

The shape of the patches to be extracted. This will be the shape of the subarray that is returned when a patch is accessed using __getitem__.

strideTuple[int, …], optional

The stride between consecutive patches. If None, the stide will be the same as data_shape. By default None

pad_widthTuple[Tuple[int, int], …], optional

The width of padding to be applied to the data array. By default None, that is, no padding is applied. Check the documentation of numpy.pad for more information.

pad_modestr, optional

The padding mode, by default “constant”. Check the documentation of numpy.pad for more information.

pad_kwargsdict, optional

Additional keyword arguments for padding, by default None

Examples

```python >>> import numpy as np >>> # Generate a 10x10 array >>> data = np.arange(100).reshape(10, 10) >>> # Create a reader that generates 5x5 patches with a stride of 2 in the >>> # first dimension and 5 in the second dimension. >>> reader = PatchedArrayReader( >>> data, >>> data_shape=(5, 5), >>> stride=(2, 5), >>> ) >>> # Printing the number of patches that can be extracted from the data >>> print(len(reader)) 6 >>> # Printing the indices of the patches >>> print(reader.indices) [(0, 0), (0, 5), (2, 0), (2, 5), (4, 0), (4, 5)] >>> # Fetch the first patch and print its shape >>> print(reader[0].shape) (5, 5) >>> # Fetch the third patch and print its content >>> print(reader[2]) [[20 21 22 23 24]

[30 31 32 33 34] [40 41 42 43 44] [50 51 52 53 54] [60 61 62 63 64]]

```

__getitem__(idx)

Fetch a patch from the data array.

Parameters

idxint

The index of the patch to be fetched.

Returns

np.ndarray

The patch that was fetched from the data array with shape data_shape

Parameters:

idx (int)

Return type:

numpy.ndarray

__len__()

Return the number of patches that can be extracted from the data array.

Returns

int

The number of patches that can be extracted from the data array.

Return type:

int

__str__()

Return type:

str

_get_patches()

Compute the left upper corner indices of the patches that will be extracted from the data array. The patches are extracted with a stride between them. A list of indices is returned, where each index is a tuple of integers representing the coordinates of the left upper corner of the patches.

Returns

List[Tuple[int, …]]

A list of indices (coordinates) representing the left upper corner of the patches.

Return type:

List[Tuple[int, Ellipsis]]

_pad_data(pad_width, mode='constant', **kwargs)

Apply padding to the data array.

Parameters

pad_widthTuple[Tuple[int, int], …]

The width of padding to be applied to the data array

modestr, optional

The padding mode, by default “constant”

Parameters:

pad_width (Tuple[Tuple[int, int], Ellipsis])

data

data_shape

indices = []

pad_kwargs

pad_mode = 'constant'

pad_width = None

shape

stride

Parameters:

• data (numpy.typing.ArrayLike)

• data_shape (Tuple[int, Ellipsis])

• stride (Optional[Tuple[int, Ellipsis]])

• pad_width (Optional[Tuple[Tuple[int, int], Ellipsis]])

• pad_mode (str)

• pad_kwargs (Optional[Dict])

class minerva.data.readers.PatchedMDIOReader(*args, path, mdio_data='data/chunked_012', **kwargs)

Bases: minerva.data.readers.zarr_reader.PatchedZarrReader

Reads patches from a MDIO array. This class is a subclass of PatchedZarrReader and is designed to read patches from the the data Zarr array inside thre MDIO array subdirectory.

Reads patches from a MDIO array. This class is a subclass of PatchedZarrReader. All other parameters are the same as parent class. Please refer to the documentation of PatchedZarrReader for more information.

Parameters

pathPathLike

Path to the MDIO array.

mdio_data: str

Name of data array inside MDIO subdirectory. Defaults to ‘data/chunked_012’

Notes

1. The MDIO array is expected to be stored on disk. If the array is not stored on disk, it is recommended to use the PatchedArrayReader class instead.

2. When using padding, the padding is applied to the entire array. This will load the entire array into memory. If the array is too large to fit into memory, it is recommended to pad before. See LazyPaddedPatchedMDIOReader for an a lazy alternative.

Examples

```python >>> from pathlib import Path >>> data_path = Path(“data.mdio”) >>> reader = PatchedMDIOReader( … path=data_path, … data_shape=(5, 5), … stride=(2, 5), … ) >>> print(len(reader)) >>> print(reader[0])

mdio_path

Parameters:

• path (minerva.utils.typing.PathLike)

• mdio_data (str)

class minerva.data.readers.PatchedZarrReader(*args, path, **kwargs)

Bases: minerva.data.readers.patched_array_reader.PatchedArrayReader

Reads patches from a Zarr array. This class is a subclass of PatchedArrayReader and is designed to read patches from a Zarr array.

Reads patches from a Zarr array. This class is a subclass of PatchedArrayReader. All other parameters are the same as parent class. Please refer to the documentation of PatchedArrayReader for more information.

Parameters

pathPathLike

Path to the Zarr array.

Notes

1. The Zarr array is expected to be stored on disk. If the array is not stored on disk, it is recommended to use the PatchedArrayReader class instead.

2. When using padding, the padding is applied to the entire array. This will load the entire array into memory. If the array is too large to fit into memory, it is recommended to pad before. See LazyPaddedPatchedZarrReader for an a lazy alternative.

Examples

```python >>> from pathlib import Path >>> data_path = Path(“data.zarr”) >>> reader = PatchedZarrReader( … path=data_path, … data_shape=(5, 5), … stride=(2, 5), … ) >>> print(len(reader)) >>> print(reader[0])

path

Parameters:

path (minerva.utils.typing.PathLike)

class minerva.data.readers.TiffReader(path, sort_method=None, delimiter=None, key_index=0, reverse=False)

Bases: minerva.data.readers.base_file_iterator.BaseFileIterator

A base class for iterating over files in a directory in a custom sorted order.

Load image files from a directory.

Parameters

pathUnion[Path, str]

The path to the directory containing the image files. Files will be searched recursively.

sort_methodOptional[List[str]], optional

A list specifying how to sort each part of the filename. Each element can be either “text” (lexicographical) or “numeric” (numerically). By default, None, which will use “numeric” if numeric parts are detected.

delimiterOptional[str], optional

The delimiter to split filenames into components, by default None.

key_indexUnion[int, List[int]], optional

The index (or list of indices) of the part(s) of the filename to use for sorting. If a list is provided, files will be sorted based on multiple parts in sequence. Thus, first by the part at index 0, then by the part at index 1, and so on. By default 0.

reversebool, optional

Whether to sort in reverse order, by default False.

Raises

NotADirectoryError

If the path is not a directory.

__getitem__(index)

Retrieve the TIFF file at the specified index.

Parameters:

index (int)

Return type:

numpy.ndarray

__str__()

Return type:

str

root_dir

Parameters:

• path (minerva.utils.typing.PathLike)

• sort_method (Optional[List[str]])

• delimiter (Optional[str])

• key_index (Union[int, List[int]])

• reverse (bool)

class minerva.data.readers._Reader

Base class for readers. Readers define an ordered collection of data and provide methods to access it. This class primarily handles:

1. Definition of data structure and storage.

2. Reading data from the source.

The access is handled by the __getitem__ and __len__ methods, which should be implemented by a subclass. Readers usually returns a single item at a time, that can be a single image, a single label, etc.

abstract __getitem__(index)

Retrieve an item from the reader at the specified index.

Parameters

indexint

Index of the item to retrieve.

Returns

Any

An item from the reader.

Parameters:

index (int)

Return type:

Any

abstract __len__()

Get the length of the reader.

Returns

int

The length of the reader.

Return type:

int