Estimate the cost before you run#
Analyses grow faster than exponentially with the number of units. Count the work first; it is far cheaper than discovering the answer by waiting.
import pyphi
pyphi.config.progress_bars = False
The practical ceilings#
What |
About |
|---|---|
system integrated information \(\varphi_s\) |
10–12 units |
distinctions and relations (the Φ-structure) |
6–8 units |
These are for fully connected substrates on one core; sparser connectivity raises them, since absent connections shrink every search. (Under the earlier IIT 3.0, the cause–effect structure and its Φ reach about 10–12 units.)
Count the work#
pyphi.cost.estimate_analysis() walks the same enumerations the
analysis would, without computing any φ. It needs no state.
substrate = pyphi.examples.iit4_2023_fig6d_substrate()
pyphi.cost.estimate_analysis(substrate)
| Units | 6 |
|---|---|
| State space | 64 |
| System partitions | = 7,896 |
| Specified-state evaluations | 128 |
| Mechanisms | 63 |
| Purview evaluations | = 7,938 |
| Mechanism partition sweeps | = 21,104,202 |
| Relations | closed form |
| Possible distinctions | 63 |
| Possible relations | ~10^18 |
| Capped | False |
The counting walk has its own budget, limit (one million steps by
default). On ten or more fully connected units it can take tens of seconds
and stop early, reporting capped=True; the counts are then lower bounds.
A small budget shows what that looks like on this six-unit substrate:
estimate = pyphi.cost.estimate_analysis(substrate, limit=1_000)
estimate.capped, estimate.mechanism_partition_sweeps
(True, 16452)
Raise the budget for an exact count when you need one:
estimate = pyphi.cost.estimate_analysis(substrate, limit=10_000_000)
estimate.capped, estimate.mechanism_partition_sweeps
(False, 21104202)
compute="sia" counts only the system-partition search; "distinctions"
only the distinction axis; the default counts everything. The
system-partition count grows fastest with the number of units and is the
axis to watch for \(\varphi_s\).
pyphi.cost.estimate_analysis(substrate, compute="sia").system_partitions
7896
What to reduce#
Connectivity. Pass the real connectivity matrix if you know it.
The candidate system. Analyze a subset (
subset=) rather than the whole substrate;substrate.complexes(state)still has to consider every subset.What you compute.
compute="sia"for \(\varphi_s\) alone.Settings. The cost-reduction table in Computational complexity lists the partition schemes and short-circuit options and what each gives up.
Cores and clusters. Run computations in parallel divides the constants; Run a sweep as an HTCondor campaign shards one analysis across machines.
Where to go next#
Computational complexity derives the scaling.
Search across grains has its own pre-flight estimate.