pyphi.parallel.backends.local_process.LocalMapReduce#

class pyphi.parallel.backends.local_process.LocalMapReduce(map_func, iterables, reduce_func, reduce_kwargs, chunksize, sequential_threshold=1, size_func=None, shortcircuit_func=<function false>, shortcircuit_callback=None, shortcircuit_callback_args=None, ordered=False, map_kwargs=None, progress=True, desc='', total=None, snapshot=None)[source]#

Bases: object

Single-machine parallelization using loky’s reusable executor.

Items are grouped into chunks (evenly, or cost-balanced when a size_func is given), each chunk is submitted to a worker as one future, and the per-chunk result lists are concatenated and reduced. Loky’s cloudpickle support lets functions defined in __main__ (e.g. in a Jupyter notebook) be serialized to workers, and its reusable pool keeps per-task overhead low (roughly 1-5 ms). A short-circuit predicate stops collection early and cancels the remaining futures. Progress is reported through LocalProgressBar, which renders in both terminals and notebooks.

Parameters:

  • map_func (Callable)

  • iterables (tuple[Iterable, ...])

  • reduce_func (Callable)

  • reduce_kwargs (dict)

  • chunksize (int)

  • sequential_threshold (int)

  • size_func (Callable[..., float] | None)

  • shortcircuit_func (Callable)

  • shortcircuit_callback (Callable | None)

  • shortcircuit_callback_args (Any)

  • ordered (bool)

  • map_kwargs (dict | None)

  • progress (bool)

  • desc (str)

  • total (int | None)

  • snapshot (Any | None)

run()[source]#

Execute the parallel computation.

Return type:

Any