pyphi.parallel.backends.dask#
Distributed-cluster scheduler backed by dask.distributed.
Distributes map-reduce workloads across a Dask cluster. The cluster is
user-provided: create and connect a distributed.Client — to a
distributed.LocalCluster, a dask_jobqueue cluster (HTCondor, Slurm,
PBS, …), or a scheduler address — before computing, and select the backend
with config.parallel_backend = "dask". The backend resolves the active
client with distributed.get_client().
Workers receive the caller’s configuration snapshot with each chunk and
install it idempotently, so distributed results reflect the caller’s
configuration regardless of worker process state. Items are grouped into
cost-balanced chunks (one future per chunk) to amortize network overhead;
collection preserves the same determinism guarantees as the local process
backend (submission-order prefixes under ordered or an active
short-circuit predicate).
A map_reduce call that executes inside a Dask worker task runs its
items sequentially in that task rather than submitting to the cluster:
blocking on subtasks from within a task can deadlock a fully occupied
worker pool, and worker slots are typically single-core. Consequently only
the outermost parallel level of a nested computation is distributed.
dask.distributed is imported inside DaskScheduler.map_reduce();
importing this module is free and requires no optional dependencies.
Classes
Scheduler backed by a user-provided |