Streaming (battdat.streaming)#

Retrieve data in smaller chunks from a large HDF5 file

battdat.streaming.iterate_cycles_from_file(hdf5_path: Path | str | HDFStore, make_dataset: bool = False, key: str = '/raw_data') → Iterator[DataFrame | CellDataset]#

Stream single-cycle datasets from the HDF5 file

Parameters:

  • hdf5_path – Path to the data file

  • make_dataset – Whether to form a BatteryDataset for each cycle, including the metadata from the source file.

  • key – Which table to read

Yields:

All rows belonging to each cycle from the “raw_data” section fo the HDF5 file.

battdat.streaming.iterate_records_from_file(hdf5_path: Path | str | HDFStore, key: str = '/raw_data') → Iterator[Dict[str, str | float | int]]#

Stream individual records from a file

Parameters:

  • hdf5_path – Path to the data file

  • key – Which table to read

Yields:

Individual rows from the “raw_data” section of the HDF5 file

HDF5 Streaming (b.streaming.hdf5)#

Streaming tools related to the HDF5 format

class battdat.streaming.hdf5.HDF5Writer(hdf5_output: ~pathlib.Path | str | ~tables.file.File, metadata: ~battdat.schemas.BatteryMetadata = <factory>, schema: ~battdat.schemas.column.ColumnSchema = <factory>, complevel: int = 0, complib: str = 'zlib', key: str = '', buffer_size: int = 32768, _file: ~tables.file.File | None = None, _dtype: ~numpy.dtype | None = None, _table: ~tables.table.Table | None = None, _write_buffer: ~typing.List[~typing.Dict] | None = None)#

Bases: AbstractContextManager

Tool to write raw time series data to an HDF5 file incrementally

Writes data to the raw_data key of a different dataset.

buffer_size: int = 32768#

Number of rows to collect in memory before writing to disk

complevel: int = 0#

Compression level. Can be between 0 (no compression) and 9 (maximum compression). Ignored if data table already exists

complib: str = 'zlib'#

Compression algorithm. Consult read_hdf() for available options. Ignored if data table already exists

flush() → int#

Write the current row buffer to the file

Returns:

Number of rows written

hdf5_output: Path | str | File#

File or already-open HDF5 file in which to store data

key: str = ''#

Name of the root group in which to store the data. Ignored if hdf5_output is a File.

metadata: BatteryMetadata#

Metadata describing the cell

schema: ColumnSchema#

Schema describing columns of the cell

write_row(row: Dict[str, str | float | int]) → int#

Add a row to the data file

Parameters:

row – Row to be added to the HDF5 file

Returns:

Number of rows written to file. Writes only occur when a write buffer has filled