datalad_next.itertools.align_pattern

Yield data chunks that contain a complete pattern, if it is present

align_pattern makes it easy to find a pattern (str, bytes, or bytearray) in data chunks. It joins data-chunks in such a way, that a simple containment-check (e.g. pattern in chunk) on the chunks that align_pattern yields will suffice to determine whether the pattern is present in the stream yielded by the underlying iterable or not.

To achieve this, align_pattern will join consecutive chunks to ensures that the following two assertions hold:

Each chunk that is yielded by align_pattern has at least the length of the pattern (unless the underlying iterable is exhausted before the length of the pattern is reached).
The pattern is not split between two chunks, i.e. no chunk that is yielded by align_pattern ends with a prefix of the pattern (unless it is the last chunk that the underlying iterable yield).

The pattern might be present multiple times in a yielded data chunk.

Note: the pattern is compared verbatim to the content in the data chunks, i.e. no parsing of the pattern is performed and no regular expressions or wildcards are supported.

>>> from datalad_next.itertools import align_pattern
>>> tuple(align_pattern([b'abcd', b'e', b'fghi'], pattern=b'def'))
(b'abcdefghi',)
>>> # The pattern can be present multiple times in a yielded chunk
>>> tuple(align_pattern([b'abcd', b'e', b'fdefghi'], pattern=b'def'))
(b'abcdefdefghi',)

Use this function if you want to locate a pattern in an input stream. It allows to use a simple in-check to determine whether the pattern is present in the yielded result chunks.

The function always yields everything it has fetched from the underlying iterable. So after a yield it does not cache any data from the underlying iterable. That means, if the functionality of align_pattern is no longer required, the underlying iterator can be used, when align_pattern has yielded a data chunk. This allows more efficient processing of the data that remains in the underlying iterable.

Parameters:

iterable (Iterable) -- An iterable that yields data chunks.
pattern (str | bytes | bytearray) -- The pattern that should be contained in the chunks. Its type must be compatible to the type of the elements in iterable.

Yields:

str | bytes | bytearray -- data chunks that have at least the size of the pattern and do not end with a prefix of the pattern. Note that a data chunk might contain the pattern multiple times.