Colony & Microfluidic Phenotype Quantification in_progress
Investigation report Β· colonies Β· generated 2026-08-17 13:20 UTC Β· for expert review β results below reflect completed runs.
Investigation acceptance: in-progress. 1 of 25 acceptance criteria passing. code-computed from member-study verdicts
π Executive summary in-progress Compute foundation built and HPC-deployable; phenotype quantification underway. The colony grows and divides correctly (natural division 1->2->4),β¦
Does whole-cell E. coli phenotype survive colony embedding, and match microfluidic-device data, across a cost/fidelity ladder of cell models? Runs N whole-cell (or cheaper) agents in one pymunk 2D device geometry and measures a shared phenotype panel (growth rate, size-at-division, added length, inter-division time). Part A (colonies-01..03) is the enabling compute foundation β correct division, characterised cost, bounded memory on Linux; Part B (colonies-04..09) is the phenotype science.
Question. Do the emergent single-cell phenotypes of the whole-cell E. coli model β adder size-homeostasis, the inter-division-time distribution, and the growth rate β survive embedding into a physically-packed, dividing microcolony, and do they match microfluidic-device data, across a cost/fidelity ladder of cell models (simple viva-munk agent -> growth surrogate -> full 55-process WCM) run in the canonical devices (mother machine, daughter machine, free colony)?
Hypothesis. Single-cell phenotype is a property of the cell model, not the container: the WCM's adder size-homeostasis, inter-division-time CV, and growth rate are preserved when cells are embedded in a pymunk 2D colony that grows, packs, and divides, and they are reproduced (to tier-appropriate fidelity) by the cheaper simple-agent and growth-surrogate tiers. The compute foundation needed to run this β daughter hydration through the engine, natural division, coherent physics, and bounded memory β is infrastructure, not the result.
Each acceptance criterion is a behaviour test declared in a study: a measured field from the run (e.g. closure_gap_size) compared against an explicit pass_if band (a numeric threshold/range). The per-criterion result, each studyβs gate verdict, and this roll-up are computed in code from the run outcomes (deterministic) β not human judgement. Expand a row to see the field, the passing band, and the observed value.
| Study | Behavior | Metric (field Β· pass-if β observed) | Result |
|---|---|---|---|
| colonies-01-hpc-readiness | daughters-hydrated | post_division_advance pass if {"op":"all_daughters_advance"} | in-progress |
| colonies-01-hpc-readiness | per-cell-cost-within-2x-reference | per_cell_wall_ratio pass if {"op":"ratio_at_most","ratio":2} | passing |
| colonies-02-parallel-multigen-perf | natural-division-2-generations | in-progress | |
| colonies-02-parallel-multigen-perf | ray-lifts-gil-ceiling | in-progress | |
| colonies-03-wcm-rss-leak | leak-localized | in-progress | |
| colonies-04-device-phenotype-harness | factory-yields-runnable-cell-per-tier | β | in-progress |
| colonies-04-device-phenotype-harness | geometry-builders-run-with-simple-agents | β | in-progress |
| colonies-04-device-phenotype-harness | extractor-recovers-known-division-stats | β | in-progress |
| colonies-04-device-phenotype-harness | harness-produces-phenotype-panel | β | in-progress |
| colonies-05-mother-machine | mother-machine-runs-and-divides | β | in-progress |
| colonies-05-mother-machine | running-animation-rendered | β | in-progress |
| colonies-05-mother-machine | size-at-division-distribution | β | in-progress |
| colonies-05-mother-machine | interdivision-time-distribution | β | in-progress |
| colonies-05-mother-machine | added-size-distribution | β | in-progress |
| colonies-06-daughter-machine | daughter-machine-runs-and-divides | β | in-progress |
| colonies-06-daughter-machine | running-animation-rendered | β | in-progress |
| colonies-06-daughter-machine | size-at-division-distribution | β | in-progress |
| colonies-06-daughter-machine | interdivision-time-distribution | β | in-progress |
| colonies-06-daughter-machine | added-size-distribution | β | in-progress |
| colonies-08-wcm-daughter-machine | whole-cell-runs-and-divides-in-device | β | in-progress |
| colonies-08-wcm-daughter-machine | running-animation-rendered | β | in-progress |
| colonies-08-wcm-daughter-machine | preliminary-phenotypes-extracted | β | in-progress |
| colonies-09-wcm-mother-machine | whole-cell-runs-and-divides-in-device | β | in-progress |
| colonies-09-wcm-mother-machine | running-animation-rendered | β | in-progress |
| colonies-09-wcm-mother-machine | preliminary-phenotypes-extracted | β | in-progress |
𧬠Biology β the mechanism this investigation models The object is a growing microcolony of E. coli, every agent embedding the full whole-cell model (the 55-process baseline that grows, transcribes, translates, replicates itsβ¦
The object is a growing microcolony of E. coli, every agent embedding the full whole-cell model (the 55-process baseline that grows, transcribes, translates, replicates its chromosome, and divides) in a 2D pymunk environment that resolves physical packing. A cell elongates as its dry mass accumulates to the division threshold (~one cell cycle), then splits into two daughters placed along its long axis that reset and regrow (see the colony-animation GIF). No biological model change was made; the compute work is infrastructural β daughter hydration, division counts and placement, realistic physics, streaming a bounded per-cell phenotype panel to zarr, and characterising the compute and memory cost of running many whole cells at once. The scientific payload is the phenotype quantification: does the whole-cell model reproduce the size-homeostasis (adder), inter-division-time, and growth-rate statistics that microfluidic devices measure β and do cheaper cell-model tiers reproduce them well enough to substitute at colony scale?
π¬ Scientific argument 6 for Β· 2 against Whole-cell single-cell phenotype (adder size-homeostasis, inter-division-time distribution, growth rate) is preserved under colony embedding andβ¦
Main claim. Whole-cell single-cell phenotype (adder size-homeostasis, inter-division-time distribution, growth rate) is preserved under colony embedding and division and is quantifiable across a model-tier ladder and device geometries. The compute foundation that enables this is single-machine-correct, cost-flat, and memory-bounded on the Linux HPC target β the memory growth earlier read as an HPC blocker is largely a macOS allocator artifact.
Evidence for
- Per-cell wall flat ~57 ms/tick/cell across N=1,2,4,8 (per-cell-cost-within-2x PASS; 2026-07-26 remote re-run). [colonies-01 F-01]
- pymunk physics ~free (<=0.2 ms/tick at N=8); cost is the inner EcoliWCM. [colonies-01 F-04]
- Per-cell RSS is additive, with sim_data @lru_cache-shared within a process β ~291 MB/cell on current main (numpy footprint flat across N=1..4 confirms sharing), atop an ~888 MB per-actor baseline. Reconciled HPC budget ~700 cells/node (64 Ray actors Γ ~11 cells, RAM- and GIL-realtime-feasible), superseding 384 (450 MB/cell, pre-recorder) and an unsupported ~1000. [colonies-01 F-03; hardening re-measurement 2026-07-27]
- Natural division works after fixing mother-removal (id/key match) and daughter placement (viva-munk daughter_locations, no velocity kick) 1->2->4, daughters replace the mother in place. [colonies-02; colony-animation GIF]
- The Ray protocol runs cells in separate OS processes, ~3.5x faster at N=16, lifting the single-process GIL ceiling. [colonies-02 ray-lifts-gil-ceiling PASS]
- The per-tick RSS growth is localized to the elongation step (82%) + mass-listener (17%); the real colony on Linux/glibc grows 0.225 MB/tick vs ~1.6 on macOS (7x less; end-to-end container run), and the truly-retained residual is that ~0.225 MB/tick scipy-LSODA rwork (arena churn fully reclaimed on Linux). [colonies-03; leak hunt 2026-07-26]
Evidence against
- The residual ~0.2 MB/tick scipy-LSODA retention is real and platform-independent; over a full cell cycle (~3000 ticks) that is ~0.6 GB per cell, so very long single-process WCM runs still need the solver-level fix or periodic trimming.
- Full-colony (55-process) Linux boundedness is now validated end-to-end β the real colony, built for glibc in a Linux container, grew RSS 0.225 MB/tick vs ~1.6 on the macOS mini (7x less), confirming the arena-churn-is-a-macOS-artifact finding. The residual 0.225 MB/tick equals the real scipy-LSODA retention and persists on Linux (~0.6 GB/cell-cycle), so very long multi-generation WCM runs still want the solver-level fix (BDF-first in equilibrium/two-component) or periodic trimming; a run on an actual HPC node remains the last confirmation before a hard budget.
Caveats
- Single-machine, single-seed (0) compute measurements; cross-node scaling is a follow-up.
- Part B phenotype distributions are held for comparison against experimental microfluidic data; the surrogate tier and real mother-machine data are planned.
- The ~700 cells/node budget assumes ~11 cells packed per Ray actor (sim_data shared within the actor). One-cell-per-actor (max GIL parallelism) makes each actor pay a full ~888 MB sim_data baseline and is RAM-bound at ~215 cells/node. The right packing is a deployment decision, not a fixed number.
Open questions & decisions needed
Investigation roadmap
- β colonies-01-hpc-readiness (1 passed Β· 3 pending)
- β½ colonies-02-parallel-multigen-perf (5 pending)
- β½ colonies-03-wcm-rss-leak (3 pending)
- β½ colonies-04-device-phenotype-harness
- β½ colonies-05-mother-machine
- β½ colonies-06-daughter-machine
- β½ colonies-08-wcm-daughter-machine
- β½ colonies-09-wcm-mother-machine
Studies
Each study is collapsed to a one-glance control panel β scan top to bottom, then click any panel to expand its full detail.
1.HPC scaling readiness for the colony compositeβ
Passingβ Not runTests: 1β Β· 3β³β
PassedIs the pure whole-cell colony composite ready to scale on HPC?1/1 tests passingCaveat Single-machine only; cross-node HPC scaling is colonies-02's job.
Biology
Per-cell wall time is flat at 70β74 ms across N β {1, 2, 4, 8} (slight decrease with N as Python/import overhead amortizes). Composite scales linearly on one machine.
The GIL is the dominant bottleneck at colony scale. Process-bigraph composites are single-threaded by design β one process uses ~1 CPU core regardless of how many cells it holds. Scaling means more processes, not more cells/process.
Per-cell RSS is ADDITIVE and sim_data is shared WITHIN a process: each extra cell costs far less than a fresh sim_data load, because v2ecoli/core.py::_load_cache_bundle_cached is @lru_cache'd by cache_dir and ecoli_baseline deep-copies only initial_state per cell. Original N-sweep (commit 2f950d9, emit_cells=True): ~450 MB/cell atop a ~1 GB baseline β 384 cells/node. HARDENING re-measurement (2026-07-27, current main, bounded ColonyPhenotypeRecorder + emit_cells=False): ~291 MB/cell atop an ~888 MB baseline; the gc-visible numpy footprint stays flat (+39 MB) as N goes 1β4, directly confirming sim_data is shared by reference and the per-cell increment is mutable state, not a duplicated bundle. Per-cell RSS DROPPED ~450β~291 MB because the bounded phenotype recorder replaced the in-RAM cells-map accumulation. Reconciled current-main budget: ~700 cells/node (see expected.summary), superseding both the stale 384 and the unsupported ~1000 in the investigation executive.
pymunk 2D physics is essentially free at the N values we tested (β€ 0.2 ms/tick at N=8). All wall-time cost is the inner EcoliWCM.
Three independent bugs blocked daughter hydration through the engine and had to be fixed before the Decide phase could run.
RSS grows continuously during a single long N=1 run at ~5 MB/sim-second (1126 MB at tick 0 β 7016 MB by tick 1050, before division). Originally read as either accumulating state in the EcoliWCM internal composite or a leak; visible only in long runs, not in the 60s N-sweep windows.
Characterised by the 2026-07-26 leak hunt (supersedes the leak framing): the growth is localized to the inner WCM's polypeptide-elongation step (~82%) + mass-listener (~17%), and is NOT an unbounded native leak. Only ~0.2 MB/tick is truly retained (tracemalloc-visible: scipy LSODA integrator work arrays held per solve); the remainder is macOS allocator arena churn β large per-tick numpy working arrays that are freed but not returned to the OS. Linux validation (glibc container) showed RSS growth of only 0.017 MB/tick with no fix and 0.000 with periodic malloc_trim(0), versus ~1.6 MB/tick on the macOS mini. On the Linux HPC target the colony's memory is bounded, so the cells-per-node budget stands; the ~5 MB/s reading here was substantially a macOS measurement artifact.
Overview
This study asks whether is the pure whole-cell colony composite ready to scale on HPC? Can N. We recorded 3 findings confirm the expected biology, 3 novel computational results. Gate decision: Passed. Gate cleared.
Purpose & background (study design)
Detailed findings
Infrastructure / computational findings (5)
Technical details
test:per-cell-cost-within-2x-referencensweep-n8 Β· runs: 6Technical details
run:nsweep-n8nsweep-n8 Β· runs: 6Technical details
run:nsweep-n8Characterised by the 2026-07-26 leak hunt (supersedes the leak framing): the growth is localized to the inner WCM's polypeptide-elongation step (~82%) + mass-listener (~17%), and is NOT an unbounded native leak. Only ~0.2 MB/tick is truly retained (tracemalloc-visible: scipy LSODA integrator work arrays held per solve); the remainder is macOS allocator arena churn β large per-tick numpy working arrays that are freed but not returned to the OS. Linux validation (glibc container) showed RSS growth of only 0.017 MB/tick with no fix and 0.000 with periodic malloc_trim(0), versus ~1.6 MB/tick on the macOS mini. On the Linux HPC target the colony's memory is bounded, so the cells-per-node budget stands; the ~5 MB/s reading here was substantially a macOS measurement artifact.
bio-1cell-natural-division Β· runs: 7Technical details
run:bio-1cell-natural-divisionMethodological findings (1)
Conclusion verdicts
Three-track verdict β each result is computed from canonical fields (gate evaluator, run status, finding tiers). The basis is the author's rationale.
Conditions β what we set up to test it
Baseline
v2ecoli.composites.ecoli_colony.ecoli_colony0out/cache23011What we ran (6 simulations)
One row per concrete run: the model composite, what changes vs the reference baseline, the condition / length, and its status.
| Simulation | Composite | Changes vs baseline | Run | CLI | Status |
|---|---|---|---|---|---|
| build-smoke-n2 feeds: daughters-hydrated two-generations-complete | ecoli_colony | reference baseline | pure-WC, 2 cells, env_size=30 Β· 90 min Β· 1 seed | vwb run study colonies-01-hpc-readiness | ready |
| nsweep-n1 feeds: reference-perf-recorded | ecoli_colony | n_cells=1 | pure-WC, single cell, baseline reference Β· 90 min Β· 1 seed | vwb run study colonies-01-hpc-readiness | gated |
| nsweep-n2 feeds: per-cell-cost-within-2x-reference | ecoli_colony | n_cells=2 | pure-WC, 2 cells Β· 90 min Β· 1 seed | vwb run study colonies-01-hpc-readiness | gated |
| nsweep-n4 feeds: per-cell-cost-within-2x-reference | ecoli_colony | n_cells=4 | pure-WC, 4 cells Β· 90 min Β· 1 seed | vwb run study colonies-01-hpc-readiness | gated |
| nsweep-n8 feeds: per-cell-cost-within-2x-reference | ecoli_colony | n_cells=8 | pure-WC, 8 cells β best-effort on laptop Β· 90 min Β· 1 seed | vwb run study colonies-01-hpc-readiness | gated |
| bio-1cell-natural-division feeds: daughters-hydrated per-cell-cost-within-2x-reference | ecoli_colony | n_cells=1 | pure-WC, 1 cell, run past one natural division Β· 75 min Β· 1 seed | vwb run study colonies-01-hpc-readiness | ready |
Measurements (8 readouts)
Quantities we extract from each simulation run to evaluate the study's tests.
| Readout | Status | Path | Description |
|---|---|---|---|
| colony_cell_count | derived-needed | cells (cells) | Number of live agents in the colony at each tick (live cell IDs in
state['cells'] or state['agents'] β depends on the composite's wiring).
β blocked by req-1-perf-harness |
| per_cell_dry_mass | available | cells.<id>.mass (fg) | Per-cell dry mass series, from each cell's `mass` store driven by EcoliWCM._read_outputs. |
| division_events | derived-needed | derived (events) | Per-cell wall-clock + sim-time stamps of division events, recorded
by the perf harness when it observes the {_remove, _add} structural
update on the cells map.
β blocked by req-1-perf-harness |
| wall_seconds | derived-needed | derived (seconds) | End-to-end wall time of the run. β blocked by req-1-perf-harness |
| peak_rss_mb | derived-needed | derived (megabytes) | Peak resident set size of the run process, via psutil. β blocked by req-1-perf-harness |
| per_tick_latency_ms | derived-needed | derived (milliseconds) | Distribution of wall-time per composite tick (one row per tick in
runs.db). Reduced to {p50, p95, p99, max} in the Decide report.
β blocked by req-1-perf-harness |
| per_cell_update_ms | derived-needed | derived (milliseconds) | Time spent inside each EcoliWCM.update() per tick, summed over cells.
Captured by monkey-patching EcoliWCM.update in the harness, or by a
lightweight timing decorator.
β blocked by req-1-perf-harness |
| pymunk_step_ms | derived-needed | derived (milliseconds) | Time spent inside the pymunk physics step per tick. Same instrumentation
pattern as per_cell_update_ms but on the multibody process.
β blocked by req-1-perf-harness |
Visualisations from the latest run
Model changes
No biological model changes. The only code work is (a) fix daughter hydration in the colony composite, (b) build the perf harness as a sim driver under sims/.
Technical details
base_model: v2ecoli.composites.ecoli_colony.ecoli_colonymodified_processes:
[{"name":"EcoliWCM._handle_division","why":"Current emission of {agents: {_remove, _add}} does not appear to\nhydrate daughter EcoliWCMs through the outer composite engine\n(reports/colony_report.py works around this by manually building\nstandalone EcoliWCMs for daughter IDs). Fix is required for Build\nacceptance.\n","status":"required","requirement_id":"req-2-daughter-hydration-fix"}]Key assumptions
- Pure-WC colony at N=2 is feasible to run past 2 generations within ~90 min sim time on a developer laptop (40-min doubling Γ 2 + slack).
- The dominant per-tick cost is the inner EcoliWCM update, not pymunk integration (validated by per-tick split in the harness).
- Peak RSS measured with psutil after each run captures the steady-state RAM cost per cell accurately enough for HPC extrapolation (Β±20%).
- Single-machine N-sweep extrapolates monotonically to per-node HPC sizing β i.e. no surprise breakdowns from network or filesystem at HPC scale that aren't visible here. (This is the assumption colonies-02 will test directly.)
Build / fix list (2)
Concrete engineering work to fully exercise this study.
req-1-perf-harnessstudies/colonies-01-hpc-readiness/sims/run.py perf harnessharnessMopen- nsweep-* runs
- all derived readouts
Implementation detail
A driver that takes (sim_name, n_cells, duration_min, seed), builds the colony composite, runs it tick-by-tick, and writes per-tick rows to studies/colonies-01-hpc-readiness/runs.db with these columns: tick, sim_time, wall_ms, per_cell_update_ms_sum, pymunk_step_ms, live_cell_count, rss_mb. Also writes one row to a `runs` table per completed run with totals.- Decide DB schema (one ticks table, one runs table).
- Wrap EcoliWCM.update + pymunk step with timing.
- Use psutil for RSS sampling.
- Add a --sim-name flag matching simulation_set entries.
req-2-daughter-hydration-fixHydrate daughter EcoliWCMs through the composite engine on structural _addbridge_fixMopen- daughters-hydrated test
- two-generations-complete test
- all nsweep-n{2,4,8} runs
Implementation detail
Today reports/colony_report.py manually instantiates standalone EcoliWCMs for daughter cell IDs (~lines 520-540) because the structural _add inside EcoliWCM._handle_division doesn't appear to make the engine run daughters as embedded processes. Fix the bridge OR the engine wiring so daughters tick normally after division.- Reproduce the issue with a minimal pure-WC N=2 run and capture exactly what the engine does (or doesn't do) at the division tick.
- Decide whether the fix is in EcoliWCM._handle_division (different update shape) or in how the colony document declares `cells` (must accept process nodes via _add).
- Verify the fix unblocks `daughters-hydrated` test.
Limitations
- Single-machine only; cross-node HPC scaling is colonies-02's job.
- 90 min sim window is one full doubling plus a buffer for the second division, not an N-generation run.
- Per-cell EcoliWCM internal cache is shared (same cache_dir path); RAM extrapolation may underestimate HPC RAM if HPC runs use per-cell caches.
- pymunk is single-threaded; GIL contention may dominate before any HPC-relevant bottleneck shows up.
Conclusion synthesis
Read-only synthesis derived from the study's canonical fields (findings, limitations, follow-up proposals).
- Per-cell wall time is flat at 70β74 ms across N β {1, 2, 4, 8} (slight decrease with N as Python/import overhead amortizes). Composite scales linearly on one machine.
- The GIL is the dominant bottleneck at colony scale. Process-bigraph composites are single-threaded by design β one process uses ~1 CPU core regardless of how many cells it holds. Scaling means more processes, not more cells/process.
- Per-cell RSS is ADDITIVE and sim_data is shared WITHIN a process: each extra cell costs far less than a fresh sim_data load, because v2ecoli/core.py::_load_cache_bundle_cached is @lru_cache'd by cache_dir and ecoli_baseline deep-copies only initial_state per cell. Original N-sweep (commit 2f950d9, emit_cells=True): ~450 MB/cell atop a ~1 GB baseline β 384 cells/node. HARDENING re-measurement (2026-07-27, current main, bounded ColonyPhenotypeRecorder + emit_cells=False): ~291 MB/cell atop an ~888 MB baseline; the gc-visible numpy footprint stays flat (+39 MB) as N goes 1β4, directly confirming sim_data is shared by reference and the per-cell increment is mutable state, not a duplicated bundle. Per-cell RSS DROPPED ~450β~291 MB because the bounded phenotype recorder replaced the in-RAM cells-map accumulation. Reconciled current-main budget: ~700 cells/node (see expected.summary), superseding both the stale 384 and the unsupported ~1000 in the investigation executive.
- pymunk 2D physics is essentially free at the N values we tested (β€ 0.2 ms/tick at N=8). All wall-time cost is the inner EcoliWCM.
- Three independent bugs blocked daughter hydration through the engine and had to be fixed before the Decide phase could run.
- RSS grows continuously during a single long N=1 run at ~5 MB/sim-second (1126 MB at tick 0 β 7016 MB by tick 1050, before division). Originally read as either accumulating state in the EcoliWCM internal composite or a leak; visible only in long runs, not in the 60s N-sweep windows.
Characterised by the 2026-07-26 leak hunt (supersedes the leak framing): the growth is localized to the inner WCM's polypeptide-elongation step (~82%) + mass-listener (~17%), and is NOT an unbounded native leak. Only ~0.2 MB/tick is truly retained (tracemalloc-visible: scipy LSODA integrator work arrays held per solve); the remainder is macOS allocator arena churn β large per-tick numpy working arrays that are freed but not returned to the OS. Linux validation (glibc container) showed RSS growth of only 0.017 MB/tick with no fix and 0.000 with periodic malloc_trim(0), versus ~1.6 MB/tick on the macOS mini. On the Linux HPC target the colony's memory is bounded, so the cells-per-node budget stands; the ~5 MB/s reading here was substantially a macOS measurement artifact.
- {"n1":73.6,"n2":73.2,"n4":72.7,"n8":70.4}
- pymunk_step_ms β 0.1 ms/tick (free); ecoli_ms grows linearly with N
- {"n1_mb":1508,"n2_mb":2013,"n4_mb":2946,"n8_mb":4712}
- pymunk_step_ms β€ 0.2
- After all three fixes, pure-WC N=2 force-divided to 4 hydrated daughters that all advance on the next tick.
- rss_mb climbs monotonically across the pre-division window
- Single-machine only; cross-node HPC scaling is colonies-02's job.
- 90 min sim window is one full doubling plus a buffer for the second division, not an N-generation run.
- Per-cell EcoliWCM internal cache is shared (same cache_dir path); RAM extrapolation may underestimate HPC RAM if HPC runs use per-cell caches.
- pymunk is single-threaded; GIL contention may dominate before any HPC-relevant bottleneck shows up.
Pipeline-gate decision
Passed- per-cell-cost-within-2x-reference
Pipeline gate & conclusion logic (technical)
Prerequisites: none (root study)
Enables: β
Proceed when: Build-phase acceptance test passes (N=2 pure-WC β 4 daughters, no manual hydration) AND the N-sweep completes for at least N β {1,2,4}. N=8 is best-effort on a laptop and may be deferred to the HPC follow-up.
2.Parallel multi-generation colony β natural-division perf & Ray scalingπ§ͺ Preliminaryβ Not runTests: 5β³β
PassedStarting from one whole-cell E.
Biology
One whole-cell agent divides naturally (no forced same-tick division) through >=3 generations: 1 -> 2 -> 4 -> 6 -> 8 -> 10 -> 12 -> 14 cells (7 division events) in the sequential run, with all cells advancing on subsequent ticks. Resolves the colonies-01 PASS-narrow caveat (which validated FORCED division only). Daughter EcoliWCMs hydrate natively via the bridge _remove/_add structural update.
Per-cell EcoliWCM wall does not drift across generations. The cleanest comparison (low-RSS static N-sweep, 60-tick windows): per-cell wall is flat at 54-59 ms across N in {1,2,4,8,16} (65->54 ms, a slight decrease as overhead amortizes). In the growing sequential run the 1-cell gen (55.3 ms) vs 2-cell gen (56.3 ms/cell) differ by only +1.8%. Apparent inflation to ~80 ms/cell at 14 cells is swap pressure (F-03) + O(N) pymunk collision cost, not WCM-internal accumulation. Resolves F-06 for wall time: no per-cell wall drift.
CORRECTED by the 2026-07-26 leak hunt + Linux validation (supersedes the earlier "unbounded native leak" reading recorded below). Per-process RSS attribution on the mini localized the per-tick growth to the inner WCM's polypeptide-elongation step (~82%) plus the mass-listener (~17%) -- it is NOT an unbounded native leak. It decomposes into two parts: (1) a small real retention of ~0.2 MB/tick (tracemalloc-visible), the scipy LSODA integrator work arrays (rwork/iwork) held per solve_ivp(method="LSODA") in the equilibrium / two-component / tRNA-charging ODEs -- ~0.6 GB over a ~3000-tick cell cycle, a small named optimization target (solver reuse / method change / periodic trim); and (2) allocator ARENA retention -- the scary ~1.4 MB/tick was the elongation step's large per-tick numpy working arrays (buildSequences/polymerize, ~1.3 MB) which are freed but macOS does not return to the OS, so mini RSS climbs (invisible to tracemalloc). Linux validation (colima glibc container, faithful reproduction of both sources) grew 0.017 MB/tick WITHOUT any fix and 0.000 with periodic malloc_trim(0), versus ~1.6 MB/tick on the macOS mini. So on the Linux HPC target the colony's memory is BOUNDED; the OOM / unbounded-leak framing below was substantially a macOS measurement artifact. The macOS-mini figures in `observed` (7.7 MB/sim-s, ~19 GB at first division, 27.5 GB peak, OOM at tick 8161) are retained as the original mini measurements, now understood as arena churn, not a Linux-relevant leak. Also corrected: an intermediate "emitter is 94% of the leak" reading was a warm-process artifact and is WRONG -- the growth is intrinsic to comp.run() (elongation), as above.
The process-bigraph Ray protocol (ray:EcoliWCM, parallel_processes=True) lifts the single-process GIL ceiling. Static N-sweep median wall (realtime ratio): local scales linearly 65/118/237/466/865 ms (ratio 0.07->0.87) for N=1/2/4/8/16, crossing realtime near N~18 (the colonies-01 ceiling). Ray scales sub-linearly 54/59/81/115/245 ms (ratio 0.05->0.25): per-cell wall drops 54->15 ms as cells solve concurrently across cores. At N=16 Ray is 3.5x faster and only 0.25 realtime, with headroom to ~50-60 cells before realtime. Confirmed in the growing colony too: 2 cells cost the same total wall (~59 ms) as 1 cell.
Under Ray the binding constraint flips from CPU/GIL to aggregate RAM, and RAM is worse than sequential. The ~7.7 MB/sim-s per-cell leak (F-03) is transport-independent: it moves into the actor process (one actor reached ~19 GB by tick 2338, mirroring sequential). The Ray protocol also pre-spawns a ~13-actor pool, each carrying a ~590 MB WCM baseline (~7.6 GB fixed overhead). At just 2 live cells the two daughter actors were each ~10 GB and climbing (~34 GB distinct actor RSS), so a growing colony OOMs the machine even faster than sequential. The main process stays flat ~0.7 GB (WCM offloaded). RSS sawtooths at division (mother actor returns to pool). Refined by the 2026-07-26 leak hunt (see F-03): the per-actor climb that drove the ~19/34 GB figures is the same macOS allocator-arena artifact and is Linux-bounded (0.017 MB/tick on glibc), so on the HPC target aggregate actor RAM does NOT run away with generation age. The real, transport- specific Ray cost is the FIXED pool baseline recorded here -- ~590 MB/idle pool actor over a ~13-actor pool (~7.6 GB fixed) plus ~154 MB/cell steady state (colonies-01 static sweep) -- not an unbounded per-cell leak. CPU/GIL is the constraint Ray lifts (F-04); RAM is a bounded, budgetable overhead.
Ray and sequential produce the same colony trajectory. First division occurs at tick 2338 under both transports (exact, not just modulo RNG): the EcoliWCM is deterministic given its seed and the transport does not alter seeding, so division timing and per-cell mass are transport- invariant. Per-cell wall under Ray at 2 cells is 29.7 ms (vs 59.8 ms at 1 cell): the two cells solve concurrently.
The colonies-01 excessive-cell-movement artifact has two coupled causes, both confirmed by a 12-tick 2-cell diagnostic sweep over (jitter_per_second, init_mass): (1) jitter_per_second=0.5 is ~5000x viva-munk's 1e-4 default; (2) build_microbe's density(0.02)-seeded body mass is ~0.04 (tiny), so jitter impulse/mass flings the cell. Legacy (0.5, None): mean move 9-10 um/tick. Either fix alone tames it to ~0.13 um/tick; the chosen (1e-4, 200) fixes both motion and fg-unit coherence (body mass ~200 fg). These are the run.py / study.yaml defaults.
Overview
This study asks whether starting from one whole-cell E. coli agent dividing naturally through 3-4. We recorded 7 findings confirm the expected biology. Gate decision: Ready to run. Execute the simulation_set to gather evidence.
Purpose & background (study design)
Detailed findings
Infrastructure / computational findings (7)
Technical details
test:natural-division-2-generationsTechnical details
test:per-cell-wall-drift-within-20pctTechnical details
test:per-cell-rss-drift-boundedTechnical details
test:ray-lifts-gil-ceilingTechnical details
test:ray-biology-consistencyTechnical details
test:ray-biology-consistencyTechnical details
test:(build_task) jitter-mass-diagnosticConclusion verdicts
Three-track verdict β each result is computed from canonical fields (gate evaluator, run status, finding tiers). The basis is the author's rationale.
Conditions β what we set up to test it
Baseline
v2ecoli.composites.ecoli_colony.ecoli_colony0out/cache13011What we ran (3 simulations)
One row per concrete run: the model composite, what changes vs the reference baseline, the condition / length, and its status.
| Simulation | Composite | Changes vs baseline | Run | CLI | Status |
|---|---|---|---|---|---|
| seq-1cell-4div | β | reference baseline | 180 min Β· 1 seed | vwb run study colonies-02-parallel-multigen-perf | β |
| ray-1cell-4div | β | n_cells=1 parallel_processes=true jitter_per_second=0.0001 init_mass=200 | 180 min Β· 1 seed | vwb run study colonies-02-parallel-multigen-perf | β |
| ray-nsweep-static | β | n_cells_sweep=[2,4,8,16] parallel_processes=true | 5 min Β· 1 seed | vwb run study colonies-02-parallel-multigen-perf | β |
Visualisations from the latest run
Conclusion synthesis
Read-only synthesis derived from the study's canonical fields (findings, limitations, follow-up proposals).
- One whole-cell agent divides naturally (no forced same-tick division) through >=3 generations: 1 -> 2 -> 4 -> 6 -> 8 -> 10 -> 12 -> 14 cells (7 division events) in the sequential run, with all cells advancing on subsequent ticks. Resolves the colonies-01 PASS-narrow caveat (which validated FORCED division only). Daughter EcoliWCMs hydrate natively via the bridge _remove/_add structural update.
- Per-cell EcoliWCM wall does not drift across generations. The cleanest comparison (low-RSS static N-sweep, 60-tick windows): per-cell wall is flat at 54-59 ms across N in {1,2,4,8,16} (65->54 ms, a slight decrease as overhead amortizes). In the growing sequential run the 1-cell gen (55.3 ms) vs 2-cell gen (56.3 ms/cell) differ by only +1.8%. Apparent inflation to ~80 ms/cell at 14 cells is swap pressure (F-03) + O(N) pymunk collision cost, not WCM-internal accumulation. Resolves F-06 for wall time: no per-cell wall drift.
- CORRECTED by the 2026-07-26 leak hunt + Linux validation (supersedes the earlier "unbounded native leak" reading recorded below). Per-process RSS attribution on the mini localized the per-tick growth to the inner WCM's polypeptide-elongation step (~82%) plus the mass-listener (~17%) -- it is NOT an unbounded native leak. It decomposes into two parts: (1) a small real retention of ~0.2 MB/tick (tracemalloc-visible), the scipy LSODA integrator work arrays (rwork/iwork) held per solve_ivp(method="LSODA") in the equilibrium / two-component / tRNA-charging ODEs -- ~0.6 GB over a ~3000-tick cell cycle, a small named optimization target (solver reuse / method change / periodic trim); and (2) allocator ARENA retention -- the scary ~1.4 MB/tick was the elongation step's large per-tick numpy working arrays (buildSequences/polymerize, ~1.3 MB) which are freed but macOS does not return to the OS, so mini RSS climbs (invisible to tracemalloc). Linux validation (colima glibc container, faithful reproduction of both sources) grew 0.017 MB/tick WITHOUT any fix and 0.000 with periodic malloc_trim(0), versus ~1.6 MB/tick on the macOS mini. So on the Linux HPC target the colony's memory is BOUNDED; the OOM / unbounded-leak framing below was substantially a macOS measurement artifact. The macOS-mini figures in `observed` (7.7 MB/sim-s, ~19 GB at first division, 27.5 GB peak, OOM at tick 8161) are retained as the original mini measurements, now understood as arena churn, not a Linux-relevant leak. Also corrected: an intermediate "emitter is 94% of the leak" reading was a warm-process artifact and is WRONG -- the growth is intrinsic to comp.run() (elongation), as above.
- The process-bigraph Ray protocol (ray:EcoliWCM, parallel_processes=True) lifts the single-process GIL ceiling. Static N-sweep median wall (realtime ratio): local scales linearly 65/118/237/466/865 ms (ratio 0.07->0.87) for N=1/2/4/8/16, crossing realtime near N~18 (the colonies-01 ceiling). Ray scales sub-linearly 54/59/81/115/245 ms (ratio 0.05->0.25): per-cell wall drops 54->15 ms as cells solve concurrently across cores. At N=16 Ray is 3.5x faster and only 0.25 realtime, with headroom to ~50-60 cells before realtime. Confirmed in the growing colony too: 2 cells cost the same total wall (~59 ms) as 1 cell.
- Under Ray the binding constraint flips from CPU/GIL to aggregate RAM, and RAM is worse than sequential. The ~7.7 MB/sim-s per-cell leak (F-03) is transport-independent: it moves into the actor process (one actor reached ~19 GB by tick 2338, mirroring sequential). The Ray protocol also pre-spawns a ~13-actor pool, each carrying a ~590 MB WCM baseline (~7.6 GB fixed overhead). At just 2 live cells the two daughter actors were each ~10 GB and climbing (~34 GB distinct actor RSS), so a growing colony OOMs the machine even faster than sequential. The main process stays flat ~0.7 GB (WCM offloaded). RSS sawtooths at division (mother actor returns to pool). Refined by the 2026-07-26 leak hunt (see F-03): the per-actor climb that drove the ~19/34 GB figures is the same macOS allocator-arena artifact and is Linux-bounded (0.017 MB/tick on glibc), so on the HPC target aggregate actor RAM does NOT run away with generation age. The real, transport- specific Ray cost is the FIXED pool baseline recorded here -- ~590 MB/idle pool actor over a ~13-actor pool (~7.6 GB fixed) plus ~154 MB/cell steady state (colonies-01 static sweep) -- not an unbounded per-cell leak. CPU/GIL is the constraint Ray lifts (F-04); RAM is a bounded, budgetable overhead.
- Ray and sequential produce the same colony trajectory. First division occurs at tick 2338 under both transports (exact, not just modulo RNG): the EcoliWCM is deterministic given its seed and the transport does not alter seeding, so division timing and per-cell mass are transport- invariant. Per-cell wall under Ray at 2 cells is 29.7 ms (vs 59.8 ms at 1 cell): the two cells solve concurrently.
- The colonies-01 excessive-cell-movement artifact has two coupled causes, both confirmed by a 12-tick 2-cell diagnostic sweep over (jitter_per_second, init_mass): (1) jitter_per_second=0.5 is ~5000x viva-munk's 1e-4 default; (2) build_microbe's density(0.02)-seeded body mass is ~0.04 (tiny), so jitter impulse/mass flings the cell. Legacy (0.5, None): mean move 9-10 um/tick. Either fix alone tames it to ~0.13 um/tick; the chosen (1e-4, 200) fixes both motion and fg-unit coherence (body mass ~200 fg). These are the run.py / study.yaml defaults.
- {"division_ticks":[2338,4706,4707,6975,6976,7075,7101],"cells_final":14}
- {"seq_gen0_ms":55.3,"seq_gen1_per_cell_ms":56.3,"drift_pct":1.8,"nsweep_local_per_cell_ms":{"n1":65,"n2":59,"n4":59,"n8":58,"n16":54}}
- {"macos_mini_single_cell_leak_mb_per_s":7.7,"macos_mini_single_cell_rss_at_first_division_gb":19,"macos_mini_peak_rss_gb":27.5,"macos_mini_oom_tick":8161,"no_emitter_control_mb_per_tick":7.06,"elongation_share_pct":82,"mass_listener_share_pct":17,"real_retention_scipy_lsoda_mb_per_tick":0.2,"linux_glibc_mb_per_tick_no_fix":0.017,"linux_glibc_mb_per_tick_with_malloc_trim":0,"macos_mini_mb_per_tick":1.6}
- {"local_wall_ms":{"n1":65,"n2":118,"n4":237,"n8":466,"n16":865},"ray_wall_ms":{"n1":54,"n2":59,"n4":81,"n8":115,"n16":245},"local_realtime_cross_n":18,"ray_realtime_cross_n_est":55}
- {"single_actor_rss_at_first_division_gb":19,"actor_pool_size":13,"per_actor_baseline_mb":590,"two_cell_distinct_actor_rss_gb":34,"main_proc_rss_gb":0.7}
- {"seq_first_division_tick":2338,"ray_first_division_tick":2338,"ray_per_cell_ms_n1":59.8,"ray_per_cell_ms_n2":29.7}
- {"legacy_0p5_None_move_um":9,"fix_1e-4_200_move_um":0.13,"fix_body_mass_fg":200}
Pipeline-gate decision
Ready to runPipeline gate & conclusion logic (technical)
Prerequisites: none (root study)
Enables: β
Proceed when: colonies-01-hpc-readiness.gate_status == passed (satisfied) AND per_cell_ms / per_cell_rss recorded as projection inputs.
3.Inner EcoliWCM RSS leak β localize and fixπ§ͺ Preliminaryβ Not runTests: 3β³β
PassedWhere does the RSS growth in a single EcoliWCM come from, and is it bounded on the Linux HPC target so per-cell RSS budgets are trustworthy?
Biology
The colonies-02 ~7.7 MB/sim-s per-cell RSS leak is the inner per-cell RAMEmitter, not chromosome_history and not "emitter-independent". Each embedded cell's inner Composite (built by baseline() in the EcoliWCM bridge) gets a default RAMEmitter whose no-override branch (_helpers.py) captures `bulk` (~25k-molecule array) + four unique- molecule node arrays every tick into an unbounded history list that is never read. colonies-02's no-emitter probe nulled only the outer colony emitter, missing the per-cell inner emitters.
Setting the inner RAMEmitter to 'minimal' capture (global_time + listeners only) for embedded cells bounds per-cell RSS. A single EcoliWCM built through the bridge (set_ram_emitter_capture('minimal')) holds RSS essentially flat.
The fix only narrows what the inner RAMEmitter records (a read-only history sink); the emit_schema does not feed back into simulation state, so division timing and masses are unchanged by construction. The full colony re-run (first division still tick 2338) is the empirical re-confirmation, deferred with the no-OOM run above.
The inner-RAMEmitter fix (F-01/F-02) is real and bounds a single bare WCM, but it is not the dominant leak for a multi-cell colony. Re-running colonies-02 on current main with all three contributory fixes live (#239 inner emitter, emit_cells=False outer emitter, viva-munk #11 pymunk shape churn) still leaks ~0.5-1 MB/tick/cell: the count=6 plateau climbed 5.1 -> 22.3 GB at a constant 6 cells and OOMs ~gen 3, unchanged from before the fixes. Refined by the 2026-07-26 leak hunt: the "C-level dominant leak" call stands as a macOS observation but is not an unbounded native leak. Per-process attribution localizes it to elongation (~82%) + mass-listener (~17%); the C-level growth is macOS allocator arena retention of elongation's per-tick numpy arrays plus ~0.2 MB/tick of retained scipy LSODA work arrays. On Linux/glibc it is bounded (0.017 unmitigated, 0.000 with malloc_trim). See F-06.
The dominant per-tick growth is native and localized to the LLVM/Numba JIT layer. A controlled staged run (colonies-02 sims/mem_measure.py) shows numpy-object bytes flat within each cell-count plateau while native memory climbs (decomposition figure), and macOS `leaks` (MallocStackLogging) finds the true unreachable leak with stacks in llvmlite's LLVM pass pipeline (LLVMPY_buildFunctionSimplificationPipeline). So the WCM's JIT compilation is the source, not the emitters (#239), the outer emitter (emit_cells), the pymunk shapes (viva-munk #11), or a retained numpy array. Refined by the 2026-07-26 leak hunt: the LLVM/Numba JIT stacks that macOS `leaks` surfaced were a one-time/small true-leak signal (~13 MB), not the per-tick driver. The per-tick growth is intrinsic to comp.run() elongation (~82%) + mass-listener (~17%): macOS arena retention of per-tick numpy arrays plus ~0.2 MB/tick of scipy LSODA work arrays. It is Linux-bounded. So #253 (JIT recompilation) is not the load-bearing fix; the actionable targets are periodic malloc_trim(0) and scipy-LSODA solver reuse. See F-06.
The colony per-tick RSS growth is localized and Linux-bounded, not an unbounded native leak (this is the corrected diagnosis that reconciles F-01/F-02/F-04/F-05). Per-process RSS attribution on the mini pins the growth to the inner WCM's polypeptide-elongation step (~82%) + mass-listener (~17%). Two components: (1) a small real retention ~0.2 MB/tick (tracemalloc-visible) = scipy LSODA integrator work arrays (rwork/iwork) held per solve from solve_ivp(method="LSODA") in the equilibrium / two-component / tRNA-charging ODEs (~0.6 GB over a ~3000-tick cycle); (2) the scary ~1.4 MB/tick = macOS allocator arena retention of elongation's large per-tick numpy working arrays (buildSequences/polymerize, ~1.3 MB), freed but not returned to the OS, invisible to tracemalloc β a macOS artifact. The earlier "emitter is 94% of the leak" reading was a warm-process artifact and is wrong.
Overview
This study asks whether where does the RSS growth in a single EcoliWCM come from, and is it bounded. We recorded 5 findings confirm the expected biology. Gate decision: Ready to run. Execute the simulation_set to gather evidence.
Purpose & background (study design)
Detailed findings
Infrastructure / computational findings (6)
Conclusion verdicts
Three-track verdict β each result is computed from canonical fields (gate evaluator, run status, finding tiers). The basis is the author's rationale.
Conditions β what we set up to test it
Baseline
v2ecoli.composites.ecoli_baseline.ecoli_baseline0out/cacheWhat we ran (2 simulations)
One row per concrete run: the model composite, what changes vs the reference baseline, the condition / length, and its status.
| Simulation | Composite | Changes vs baseline | Run | CLI | Status |
|---|---|---|---|---|---|
| profile-1cell-predivision | β | reference baseline | 45 min Β· 1 seed | vwb run study colonies-03-wcm-rss-leak | β |
| postfix-seq-1cell-4div | β | n_cells=1 parallel_processes=false jitter_per_second=0.0001 init_mass=200 | 180 min Β· 1 seed | vwb run study colonies-03-wcm-rss-leak | β |
Conclusion synthesis
Read-only synthesis derived from the study's canonical fields (findings, limitations, follow-up proposals).
- The colonies-02 ~7.7 MB/sim-s per-cell RSS leak is the inner per-cell RAMEmitter, not chromosome_history and not "emitter-independent". Each embedded cell's inner Composite (built by baseline() in the EcoliWCM bridge) gets a default RAMEmitter whose no-override branch (_helpers.py) captures `bulk` (~25k-molecule array) + four unique- molecule node arrays every tick into an unbounded history list that is never read. colonies-02's no-emitter probe nulled only the outer colony emitter, missing the per-cell inner emitters.
- Setting the inner RAMEmitter to 'minimal' capture (global_time + listeners only) for embedded cells bounds per-cell RSS. A single EcoliWCM built through the bridge (set_ram_emitter_capture('minimal')) holds RSS essentially flat.
- The fix only narrows what the inner RAMEmitter records (a read-only history sink); the emit_schema does not feed back into simulation state, so division timing and masses are unchanged by construction. The full colony re-run (first division still tick 2338) is the empirical re-confirmation, deferred with the no-OOM run above.
- The inner-RAMEmitter fix (F-01/F-02) is real and bounds a single bare WCM, but it is not the dominant leak for a multi-cell colony. Re-running colonies-02 on current main with all three contributory fixes live (#239 inner emitter, emit_cells=False outer emitter, viva-munk #11 pymunk shape churn) still leaks ~0.5-1 MB/tick/cell: the count=6 plateau climbed 5.1 -> 22.3 GB at a constant 6 cells and OOMs ~gen 3, unchanged from before the fixes. Refined by the 2026-07-26 leak hunt: the "C-level dominant leak" call stands as a macOS observation but is not an unbounded native leak. Per-process attribution localizes it to elongation (~82%) + mass-listener (~17%); the C-level growth is macOS allocator arena retention of elongation's per-tick numpy arrays plus ~0.2 MB/tick of retained scipy LSODA work arrays. On Linux/glibc it is bounded (0.017 unmitigated, 0.000 with malloc_trim). See F-06.
- The dominant per-tick growth is native and localized to the LLVM/Numba JIT layer. A controlled staged run (colonies-02 sims/mem_measure.py) shows numpy-object bytes flat within each cell-count plateau while native memory climbs (decomposition figure), and macOS `leaks` (MallocStackLogging) finds the true unreachable leak with stacks in llvmlite's LLVM pass pipeline (LLVMPY_buildFunctionSimplificationPipeline). So the WCM's JIT compilation is the source, not the emitters (#239), the outer emitter (emit_cells), the pymunk shapes (viva-munk #11), or a retained numpy array. Refined by the 2026-07-26 leak hunt: the LLVM/Numba JIT stacks that macOS `leaks` surfaced were a one-time/small true-leak signal (~13 MB), not the per-tick driver. The per-tick growth is intrinsic to comp.run() elongation (~82%) + mass-listener (~17%): macOS arena retention of per-tick numpy arrays plus ~0.2 MB/tick of scipy LSODA work arrays. It is Linux-bounded. So #253 (JIT recompilation) is not the load-bearing fix; the actionable targets are periodic malloc_trim(0) and scipy-LSODA solver reuse. See F-06.
- The colony per-tick RSS growth is localized and Linux-bounded, not an unbounded native leak (this is the corrected diagnosis that reconciles F-01/F-02/F-04/F-05). Per-process RSS attribution on the mini pins the growth to the inner WCM's polypeptide-elongation step (~82%) + mass-listener (~17%). Two components: (1) a small real retention ~0.2 MB/tick (tracemalloc-visible) = scipy LSODA integrator work arrays (rwork/iwork) held per solve from solve_ivp(method="LSODA") in the equilibrium / two-component / tRNA-charging ODEs (~0.6 GB over a ~3000-tick cycle); (2) the scary ~1.4 MB/tick = macOS allocator arena retention of elongation's large per-tick numpy working arrays (buildSequences/polymerize, ~1.3 MB), freed but not returned to the OS, invisible to tracemalloc β a macOS artifact. The earlier "emitter is 94% of the leak" reading was a warm-process artifact and is wrong.
- Live profiler measured the inner RAMEmitter history at ~5.15 MB/row, growing 1:1 with ticks (9.9 MB @ 2 rows -> 9324 MB @ 1802 rows), tracking the RSS climb. Confirmed by reading process_bigraph Emitter.inputs() == config['emit'] (the emitter only stores its emit_schema ports).
- Bridge-path profile_leak.py (commit bd1a11fe) RSS: 1.3 GB @ 21 min -> 1.7 GB @ 48 min -> 1.9 GB @ 62 min (~0.25 MB/s, legitimate cell growth), vs the unfixed path +24 GB by tick 1800 (~7.7 MB/s). ~30x reduction; no monotonic emitter-driven climb.
- tracemalloc(4) over 150 ticks at 2 forced-division cells: RSS climbed 1396 -> 1586 MB (+1.3 MB/tick), leak reproduced, yet tracemalloc accounted for only ~1-2 MB of retained Python allocations (largest single site +0.08 MB; all transient WCM compute: counts_deriver/translation_deriver np.bincount, scipy-sparse .dot). So the dominant allocator is C-level / native (numpy/scipy array buffers or a C-extension retaining or fragmenting memory), invisible to tracemalloc (it tracks only CPython's allocator, not numpy's). gc object-count scans likewise found only small symptoms (pymunk Segments -> #11), not the megabytes.
- leaks @2cells/400ticks: ~846 MB live, ~13 MB true leak, top stacks all LLVM PassBuilder (GVN/JumpThreading/LoopUnswitch/LICM). numpy-bytes delta ~0 within plateaus; tracemalloc <2 MB of a +151 MB/300-tick growth. Figures: colonies-02 charts/03_memory_decomposition.svg (native vs numpy), charts/01_rss_vs_tick_by_ncells.svg (per-tick slope by N).
- Linux/glibc validation (colima container, faithful reproduction of both sources): RSS grew 0.017 MB/tick WITHOUT any fix and 0.000 MB/tick WITH periodic malloc_trim(0), versus ~1.6 MB/tick on the macOS mini. Per-process attribution: elongation ~82%, mass-listener ~17%. tracemalloc localizes the ~0.2 MB/tick retained component to scipy/integrate/_ode.py work arrays.
Pipeline-gate decision
Ready to runPipeline gate & conclusion logic (technical)
Prerequisites: none (root study)
Enables: β
Proceed when: Met: the 2026-07-26 hunt localizes >=~80% of the growth to named sites (elongation ~82% + mass-listener ~17%; real retention = scipy LSODA work arrays) and shows the growth is Linux-bounded (0.017 MB/tick unmitigated, 0.000 with malloc_trim). Gate proceeds.
4.Device harness + simple-agent phenotype baselineπ§ͺ Preliminaryβ Not runNo tests declaredβ Not runDoes the shared harness β cell-tier factory, geometry builders, and tier-agnostic phenotype extractor β recover a correct, uniform single-cell phenotype panel (growth rate, size-at-division, added length, inter-division time, adder size-homeostasis) across the three device geometries, validated first with cheap simple viva-munk agents before spending WCM compute?
Overview
This study asks whether does the shared harness β cell-tier factory, geometry builders, and. No simulations have run yet β the study is still in its design phase. Gate decision: Ready to run. Execute the simulation_set to gather evidence.
Purpose & background (study design)
Conditions β what we set up to test it
Baseline
v2ecoli.composites.ecoli_colony.ecoli_colony0230Technical context (model changes Β· implementation tasks Β· follow-ups Β· limitations Β· refs)
5.Mother machine (viva-munk) β device run + phenotype distributionsπ§ͺ Preliminaryβ Not runNo tests declaredβ Not runDo the core emergent single-cell phenotypes β size-at-division, inter-division time, added length, the adder size-homeostasis relation, and growth rate β stay measurable when simple viva-munk agents run in the canonical mother-machine device geometry?
Overview
This study asks whether do the core emergent single-cell phenotypes β size-at-division,. No simulations have run yet β the study is still in its design phase. Gate decision: Ready to run. Execute the simulation_set to gather evidence.
Purpose & background (study design)
Conditions β what we set up to test it
Baseline
v2ecoli.composites.ecoli_colony.ecoli_colony08Visualisations from the latest run
Technical context (model changes Β· implementation tasks Β· follow-ups Β· limitations Β· refs)
6.Daughter machine (viva-munk) β device run + phenotype distributionsπ§ͺ Preliminaryβ Not runNo tests declaredβ Not runDo the core emergent single-cell phenotypes β size-at-division, inter-division time, added length, the adder size-homeostasis relation, and growth rate β stay measurable when simple viva-munk agents run in the canonical daughter-machine device geometry?
Overview
This study asks whether do the core emergent single-cell phenotypes β size-at-division,. No simulations have run yet β the study is still in its design phase. Gate decision: Ready to run. Execute the simulation_set to gather evidence.
Purpose & background (study design)
Conditions β what we set up to test it
Baseline
v2ecoli.composites.ecoli_colony.ecoli_colony01Visualisations from the latest run
Technical context (model changes Β· implementation tasks Β· follow-ups Β· limitations Β· refs)
7.Whole-cell baseline (v2ecoli) in a daughter machine β device runπ§ͺ Preliminaryβ Not runNo tests declaredβ Not runRun the REAL v2ecoli whole-cell BASELINE \u2014 the full 55-process EcoliWCM \u2014 as a single cell inside the daughter-machine device geometry (chamber + absorbing wall), letting it grow and divide naturally.
Overview
This study asks whether run the REAL v2ecoli whole-cell BASELINE \u2014 the full 55-process EcoliWCM \u2014 as a. No simulations have run yet β the study is still in its design phase. Gate decision: Ready to run. Execute the simulation_set to gather evidence.
Purpose & background (study design)
A bounded per-cell phenotype panel (size, added length, division/birth times, growth rate) is streamed to zarr by the ColonyPhenotypeRecorder \u2014 a process, not an emitter, so recording stays cheap. Memory is characterised and Linux-bounded, not a blocker: the inner WCM's per-tick RSS growth is localized to the polypeptide-elongation step + mass-listener and is dominated on the dev mini by a macOS allocator artifact (~1.6 MB/tick), but is only ~0.017 MB/tick on glibc and ~0.000 with periodic malloc_trim. WCM-tier runs are therefore length-capped on the dev mini yet effectively unbounded on the Linux HPC target. On the mini this run spans only a few generations, so phenotype distributions are PRELIMINARY (small n); full distributions await the Linux HPC run and experimental data (colonies-09). The value here is the genuine whole-cell baseline running inside the device.
Conditions β what we set up to test it
Baseline
v2ecoli.composites.ecoli_colony.ecoli_colony01Visualisations from the latest run
Technical context (model changes Β· implementation tasks Β· follow-ups Β· limitations Β· refs)
8.Whole-cell baseline (v2ecoli) in a mother machine β device runπ§ͺ Preliminaryβ Not runNo tests declaredβ Not runRun the REAL v2ecoli whole-cell BASELINE \u2014 the full 55-process EcoliWCM \u2014 as one whole cell per channel inside the mother-machine device geometry (narrow dead-end channels + a flow channel that washes out cells crossing the top), letting them grow and divide naturally.
Overview
This study asks whether run the REAL v2ecoli whole-cell BASELINE \u2014 the full 55-process EcoliWCM \u2014 as. No simulations have run yet β the study is still in its design phase. Gate decision: Ready to run. Execute the simulation_set to gather evidence.
Purpose & background (study design)
Heavier than the daughter-machine whole-cell run (colonies-08): N whole cells run simultaneously (one per channel), kept small (2 channels). A bounded per-cell phenotype panel (size, added length, division/birth times, growth rate) is streamed to zarr by the ColonyPhenotypeRecorder \u2014 a process, not an emitter, so recording stays cheap. Memory is characterised and Linux-bounded, not a blocker: the inner WCM's per-tick RSS growth is localized to the polypeptide-elongation step + mass-listener and is dominated on the dev mini by a macOS allocator artifact (~1.6 MB/tick), but is only ~0.017 MB/tick on glibc and ~0.000 with periodic malloc_trim. WCM-tier runs are therefore length-capped on the dev mini yet effectively unbounded on the Linux HPC target. On the mini this run spans only a few generations, so phenotype distributions are PRELIMINARY (small n); full distributions await the Linux HPC run and experimental data (colonies-10). The value is the whole-cell baseline running inside the confined-channel device.
Conditions β what we set up to test it
Baseline
v2ecoli.composites.ecoli_colony.ecoli_colony02Visualisations from the latest run
Technical context (model changes Β· implementation tasks Β· follow-ups Β· limitations Β· refs)
Appendices
Method-grading and verification detail β kept at the back, after the main narrative.
How the verdict is computed β acceptance criteria & gating matrix
Each acceptance criterion is a behaviour test declared in a study: a measured field from the run (e.g. closure_gap_size) compared against an explicit pass_if band (a numeric threshold/range). The per-criterion result, each studyβs gate verdict, and this roll-up are computed in code from the run outcomes (deterministic) β not human judgement. Expand a row to see the field, the passing band, and the observed value.
| Acceptance criterion | Gating study | Result |
|---|---|---|
| daughters-hydrated field post_division_advance · passes if {"op":"all_daughters_advance"}After the first division event in the build-smoke-n2 run, both daughter agent IDs appear in the cells map AND both continue to advance (mass changes, EcoliWCM.update called) on subsequent ticks without the manual standalone-WCM workaround that exists in reports/colony_report.py. | colonies-01-hpc-readiness | β in-progress |
| per-cell-cost-within-2x-reference field per_cell_wall_ratio · passes if {"op":"ratio_at_most","ratio":2}For each N>1 sweep run, the per-cell wall-time (wall_seconds / N_cells_at_end) is at most 2Γ the N=1 reference per-cell wall-time. Catches super-linear blow-up that would block HPC scaling. | colonies-01-hpc-readiness | β passing |
| natural-division-2-generations One cell divides NATURALLY (no --force-divide) to 2, then both daughters divide to 4 β all cells advancing on subsequent ticks. Resolves colonies-01's PASS-narrow (forced-division-only) caveat. | colonies-02-parallel-multigen-perf | β in-progress |
| ray-lifts-gil-ceiling Under the Ray protocol, total wall for the growing colony stays sub-realtime past the sequential ~13-cell ceiling (target: scales with cores), with division timing + masses consistent vs sequential. | colonies-02-parallel-multigen-perf | β in-progress |
| leak-localized A profiling run attributes >=~80% of RSS growth to named sites so a fix is targeted, not speculative. MET (2026-07-26): per-process RSS attribution localizes the growth to elongation (~82%) + mass-listener (~17%); the real-retention component is the scipy LSODA integrator work arrays. | colonies-03-wcm-rss-leak | β in-progress |
| factory-yields-runnable-cell-per-tier | colonies-04-device-phenotype-harness | β in-progress |
| geometry-builders-run-with-simple-agents | colonies-04-device-phenotype-harness | β in-progress |
| extractor-recovers-known-division-stats | colonies-04-device-phenotype-harness | β in-progress |
| harness-produces-phenotype-panel | colonies-04-device-phenotype-harness | β in-progress |
| mother-machine-runs-and-divides | colonies-05-mother-machine | β in-progress |
| running-animation-rendered | colonies-05-mother-machine | β in-progress |
| size-at-division-distribution | colonies-05-mother-machine | β in-progress |
| interdivision-time-distribution | colonies-05-mother-machine | β in-progress |
| added-size-distribution | colonies-05-mother-machine | β in-progress |
| daughter-machine-runs-and-divides | colonies-06-daughter-machine | β in-progress |
| running-animation-rendered | colonies-06-daughter-machine | β in-progress |
| size-at-division-distribution | colonies-06-daughter-machine | β in-progress |
| interdivision-time-distribution | colonies-06-daughter-machine | β in-progress |
| added-size-distribution | colonies-06-daughter-machine | β in-progress |
| whole-cell-runs-and-divides-in-device | colonies-08-wcm-daughter-machine | β in-progress |
| running-animation-rendered | colonies-08-wcm-daughter-machine | β in-progress |
| preliminary-phenotypes-extracted | colonies-08-wcm-daughter-machine | β in-progress |
| whole-cell-runs-and-divides-in-device | colonies-09-wcm-mother-machine | β in-progress |
| running-animation-rendered | colonies-09-wcm-mother-machine | β in-progress |
| preliminary-phenotypes-extracted | colonies-09-wcm-mother-machine | β in-progress |
All 25 acceptance criteria are linked to a gating study.
π¬ Evidence & rigor β how well the method defends its claims 2/6 investigation rigor dimensions addressed Β· 4 gap(s)
Deterministic feedback on how well the method defends its claims against a skeptical reader β a method-level judgement, distinct from the per-study model verdicts above. Computed from declared fields, not judged. Gaps are an invitation to add negative controls, replicate across seeds, weigh alternative explanations, state falsifiability, or add an adversarial study.
Per-study rigor
colonies-01-hpc-readiness β 4/12 rigor dimensions addressed Β· 7 gap(s)
colonies-02-parallel-multigen-perf β 4/12 rigor dimensions addressed Β· 7 gap(s)
colonies-03-wcm-rss-leak β 3/12 rigor dimensions addressed Β· 7 gap(s)
colonies-04-device-phenotype-harness β 3/12 rigor dimensions addressed Β· 9 gap(s)
colonies-05-mother-machine β 3/12 rigor dimensions addressed Β· 9 gap(s)
colonies-06-daughter-machine β 3/12 rigor dimensions addressed Β· 9 gap(s)
colonies-08-wcm-daughter-machine β 3/12 rigor dimensions addressed Β· 9 gap(s)
colonies-09-wcm-mother-machine β 3/12 rigor dimensions addressed Β· 9 gap(s)
π Framework scorecard framework-self metrics (n=14 investigations)
Framework-self metrics aggregated across every study and investigation in the workspace β how consistently the framework itself applies its own rigor practices (discriminating controls, emergent-mechanism labelling, threshold provenance, replication, verdict divergence, falsification exposure). Computed deterministically from declared fields by pbg_superpowers.rigor.framework_metrics.
References (0 cited across this investigation)
Union of bibliography.bib_keys and per-behavior cites: across all studies in this investigation. Click DOI or link to open the source.