diff --git a/benchmarks/resources/results/REPORT.md b/benchmarks/resources/results/REPORT.md index 8342d71..631e3b2 100644 --- a/benchmarks/resources/results/REPORT.md +++ b/benchmarks/resources/results/REPORT.md @@ -1,6 +1,6 @@ # Resource-consumption benchmark report -Generated: 2026-11-08T23:35:45.924829+00:00 · results: /home/darrius/project/zerorl/benchmarks/resources/results +Generated: 2026-11-09T22:47:01.283221+00:00 · results: /home/research/projects/zeroRL/benchmarks/resources/results ## Resource consumption (experiment R) @@ -8,14 +8,14 @@ Fixed workloads, device CPU. CPU% = sum over the live process tree (multiprocess | workload | framework | wall (s) | cpu util avg (%) | cpu total (s) | cpu s / 1k | rss peak (MiB) | n | | --- | --- | --- | --- | --- | --- | --- | --- | -| collect | zeroRL | 20.6 ± 2.1 | 603.9 ± 18.2 | 124.2 ± 11.2 | 1.24 ± 0.11 | 932.7 ± 22.8 | 5 | -| collect | SB3 | 17.2 ± 0.4 | 627.4 ± 8.4 | 108.2 ± 1.7 | 1.08 ± 0.02 | 964.8 ± 6.2 | 5 | -| collect | TorchRL | 36.9 ± 3.7 | 447.1 ± 12.5 | 164.7 ± 11.8 | 1.65 ± 0.12 | 983.2 ± 0.6 | 5 | -| ppo | zeroRL | 201.8 ± 7.0 | 619.7 ± 12.8 | 1,250.1 ± 34.6 | 4.77 ± 0.13 | 1,153.2 ± 217.5 | 3 | -| ppo | SB3 | 225.2 ± 4.7 | 639.6 ± 2.1 | 1,440.1 ± 26.2 | 5.49 ± 0.10 | 1,038.6 ± 3.3 | 3 | -| ppo | TorchRL | 479.5 ± 50.4 | 546.8 ± 17.4 | 2,609.5 ± 205.7 | 9.95 ± 0.78 | 1,057.4 ± 3.1 | 3 | - -- GPU telemetry: available (device-level only). +| collect | zeroRL | 25.6 ± 0.2 | 917.0 ± 5.3 | 235.3 ± 0.9 | 2.35 ± 0.01 | 820.5 ± 15.0 | 12 | +| collect | SB3 | 25.6 ± 0.2 | 916.2 ± 7.0 | 235.6 ± 0.8 | 2.36 ± 0.01 | 850.5 ± 5.4 | 10 | +| collect | TorchRL | 34.9 ± 0.5 | 702.7 ± 11.9 | 245.1 ± 3.1 | 2.45 ± 0.03 | 986.1 ± 4.4 | 10 | +| ppo | zeroRL | 219.4 ± 0.9 | 872.7 ± 3.2 | 1,910.9 ± 7.6 | 7.29 ± 0.03 | 1,033.6 ± 4.8 | 6 | +| ppo | SB3 | 263.3 ± 1.1 | 903.3 ± 2.4 | 2,376.0 ± 5.5 | 9.06 ± 0.02 | 1,049.8 ± 2.9 | 6 | +| ppo | TorchRL | 395.5 ± 1.6 | 814.0 ± 1.8 | 3,201.1 ± 7.3 | 12.21 ± 0.03 | 1,068.2 ± 2.9 | 6 | + +- GPU telemetry: unavailable — NVML unavailable or produced no samples. - Energy: unavailable — GPU power telemetry unsupported on this device and no RAPL/powercap energy interface for CPU/whole-system energy. - Raw per-run rows: `results/resources.jsonl` (validation runs kept separately in `resources_validation.*`). @@ -27,17 +27,17 @@ _Memory note: raw values are bytes (v2; v1 rows/sample files used KiB); averages | framework | rss (MiB) | pss (MiB) | uss (MiB) | peak rss — measure phase (MiB) | n | | --- | --- | --- | --- | --- | --- | -| zeroRL | 932.7 ± 22.8 | 930.1 ± 22.7 | 929.6 ± 22.7 | 928.9 ± 23.5 | 5 | -| SB3 | 964.8 ± 6.2 | 962.2 ± 6.1 | 961.6 ± 6.1 | 962.6 ± 6.5 | 5 | -| TorchRL | 983.2 ± 0.6 | 980.6 ± 0.6 | 980.0 ± 0.7 | 983.2 ± 0.6 | 5 | +| zeroRL | 820.5 ± 15.0 | 812.7 ± 15.0 | 811.4 ± 15.0 | 816.4 ± 15.4 | 12 | +| SB3 | 850.5 ± 5.4 | 842.8 ± 5.3 | 841.5 ± 5.3 | 849.3 ± 7.2 | 10 | +| TorchRL | 986.1 ± 4.4 | 978.7 ± 4.5 | 977.3 ± 4.5 | 986.1 ± 4.4 | 10 | **Average (end-to-end, time-weighted)** | framework | rss (MiB) | pss (MiB) | uss (MiB) | n | | --- | --- | --- | --- | --- | -| zeroRL | 927.3 ± 24.2 | 924.8 ± 24.0 | 924.2 ± 24.0 | 5 | -| SB3 | 961.4 ± 5.5 | 958.8 ± 5.5 | 958.2 ± 5.5 | 5 | -| TorchRL | 983.1 ± 0.6 | 980.5 ± 0.6 | 980.0 ± 0.7 | 5 | +| zeroRL | 815.2 ± 15.1 | 807.4 ± 15.1 | 806.1 ± 15.1 | 12 | +| SB3 | 847.9 ± 6.7 | 840.1 ± 6.7 | 838.8 ± 6.7 | 10 | +| TorchRL | 985.1 ± 5.3 | 977.6 ± 5.3 | 976.3 ± 5.3 | 10 | ### Memory (RSS / PSS / USS) — ppo @@ -45,14 +45,14 @@ _Memory note: raw values are bytes (v2; v1 rows/sample files used KiB); averages | framework | rss (MiB) | pss (MiB) | uss (MiB) | peak rss — measure phase (MiB) | n | | --- | --- | --- | --- | --- | --- | -| zeroRL | 1,153.2 ± 217.5 | 1,087.7 ± 123.6 | 1,028.1 ± 33.0 | 1,153.2 ± 217.6 | 3 | -| SB3 | 1,038.6 ± 3.3 | 1,027.5 ± 3.5 | 1,020.2 ± 3.5 | 1,038.6 ± 3.3 | 3 | -| TorchRL | 1,057.4 ± 3.1 | 1,046.2 ± 3.0 | 1,039.1 ± 3.2 | 1,057.4 ± 3.1 | 3 | +| zeroRL | 1,033.6 ± 4.8 | 1,016.4 ± 4.8 | 1,007.0 ± 4.8 | 1,033.6 ± 4.9 | 6 | +| SB3 | 1,049.8 ± 2.9 | 1,032.6 ± 2.9 | 1,023.2 ± 2.9 | 1,049.8 ± 2.9 | 6 | +| TorchRL | 1,068.2 ± 2.9 | 1,051.1 ± 2.8 | 1,041.7 ± 2.8 | 1,068.2 ± 2.9 | 6 | **Average (end-to-end, time-weighted)** | framework | rss (MiB) | pss (MiB) | uss (MiB) | n | | --- | --- | --- | --- | --- | -| zeroRL | 1,024.6 ± 3.4 | 1,013.4 ± 3.3 | 1,006.0 ± 3.5 | 3 | -| SB3 | 1,037.4 ± 3.7 | 1,026.3 ± 3.7 | 1,019.1 ± 3.7 | 3 | -| TorchRL | 1,056.5 ± 2.9 | 1,045.5 ± 2.7 | 1,038.3 ± 2.8 | 3 | +| zeroRL | 1,031.8 ± 4.5 | 1,014.5 ± 4.5 | 1,005.1 ± 4.5 | 6 | +| SB3 | 1,048.4 ± 3.8 | 1,031.3 ± 3.8 | 1,021.9 ± 3.8 | 6 | +| TorchRL | 1,064.6 ± 2.7 | 1,047.6 ± 2.7 | 1,038.3 ± 2.7 | 6 | diff --git a/benchmarks/resources/results/REPORT_humanoidstandupv4.md b/benchmarks/resources/results/REPORT_humanoidstandupv4.md deleted file mode 100644 index eb8c8af..0000000 --- a/benchmarks/resources/results/REPORT_humanoidstandupv4.md +++ /dev/null @@ -1,58 +0,0 @@ -# Resource-consumption benchmark report - -Generated: 2026-11-08T23:35:46.556160+00:00 · results: /home/darrius/project/zerorl/benchmarks/resources/results - -## Resource consumption (experiment R) [humanoidstandupv4] - -Fixed workloads, device CPU. CPU% = sum over the live process tree (multiprocess sums may exceed 100%); RSS summed across tree processes (shared pages may repeat — PSS/USS recorded in the raw rows); GPU = device-level telemetry only (workload does not use the GPU); energy recorded only when reliable telemetry exists. - -| workload | framework | wall (s) | cpu util avg (%) | cpu total (s) | cpu s / 1k | rss peak (MiB) | n | -| --- | --- | --- | --- | --- | --- | --- | --- | -| collect | zeroRL | 112.0 ± 5.2 | 223.0 ± 1.8 | 249.2 ± 9.9 | 2.49 ± 0.10 | 1,269.0 ± 170.4 | 5 | -| collect | SB3 | 105.6 ± 7.4 | 226.7 ± 2.5 | 239.3 ± 13.9 | 2.39 ± 0.14 | 1,632.6 ± 74.2 | 5 | -| collect | TorchRL | 121.9 ± 3.1 | 207.1 ± 1.2 | 251.6 ± 4.0 | 2.52 ± 0.04 | 1,307.8 ± 57.5 | 5 | -| ppo | zeroRL | 421.0 ± 29.7 | 395.7 ± 18.6 | 1,672.8 ± 39.7 | 6.38 ± 0.15 | 1,441.2 ± 270.0 | 3 | -| ppo | SB3 | 416.4 ± 16.1 | 432.5 ± 5.0 | 1,806.2 ± 52.2 | 6.89 ± 0.20 | 1,295.9 ± 9.8 | 3 | -| ppo | TorchRL | 670.3 ± 24.2 | 385.3 ± 3.5 | 2,580.7 ± 88.4 | 9.84 ± 0.34 | 1,335.1 ± 17.1 | 3 | - -- GPU telemetry: available (device-level only). -- Energy: unavailable — GPU power telemetry unsupported on this device and no RAPL/powercap energy interface for CPU/whole-system energy. -- Raw per-run rows: `results/resources_humanoidstandupv4.jsonl` (validation runs kept separately in `resources_validation.*`). - -_Memory note: raw values are bytes (v2; v1 rows/sample files used KiB); averages are time-weighted (trapezoidal); summed RSS may double-count shared pages — PSS/USS are the attributable views; v1 rows show — (never 0)._ - -### Memory (RSS / PSS / USS) — collect - -**Peak (end-to-end)** - -| framework | rss (MiB) | pss (MiB) | uss (MiB) | peak rss — measure phase (MiB) | n | -| --- | --- | --- | --- | --- | --- | -| zeroRL | 1,269.0 ± 170.4 | 1,266.4 ± 170.5 | 1,265.7 ± 170.5 | 1,197.7 ± 121.0 | 5 | -| SB3 | 1,632.6 ± 74.2 | 1,630.4 ± 74.2 | 1,629.6 ± 74.2 | 1,632.6 ± 74.2 | 5 | -| TorchRL | 1,307.8 ± 57.5 | 1,305.6 ± 57.6 | 1,304.8 ± 57.6 | 1,305.3 ± 62.4 | 5 | - -**Average (end-to-end, time-weighted)** - -| framework | rss (MiB) | pss (MiB) | uss (MiB) | n | -| --- | --- | --- | --- | --- | -| zeroRL | 1,203.0 ± 119.9 | 1,200.7 ± 119.8 | 1,199.9 ± 119.8 | 5 | -| SB3 | 1,475.3 ± 64.7 | 1,473.2 ± 64.5 | 1,472.4 ± 64.5 | 5 | -| TorchRL | 1,285.9 ± 59.3 | 1,283.7 ± 59.2 | 1,282.9 ± 59.2 | 5 | - -### Memory (RSS / PSS / USS) — ppo - -**Peak (end-to-end)** - -| framework | rss (MiB) | pss (MiB) | uss (MiB) | peak rss — measure phase (MiB) | n | -| --- | --- | --- | --- | --- | --- | -| zeroRL | 1,441.2 ± 270.0 | 1,376.2 ± 175.7 | 1,316.9 ± 86.2 | 1,441.2 ± 270.0 | 3 | -| SB3 | 1,295.9 ± 9.8 | 1,285.2 ± 9.8 | 1,277.6 ± 9.8 | 1,295.8 ± 9.8 | 3 | -| TorchRL | 1,335.1 ± 17.1 | 1,324.9 ± 16.7 | 1,317.3 ± 16.7 | 1,321.3 ± 21.1 | 3 | - -**Average (end-to-end, time-weighted)** - -| framework | rss (MiB) | pss (MiB) | uss (MiB) | n | -| --- | --- | --- | --- | --- | -| zeroRL | 1,284.7 ± 1.2 | 1,273.9 ± 1.6 | 1,266.0 ± 2.0 | 3 | -| SB3 | 1,295.8 ± 9.8 | 1,285.2 ± 9.8 | 1,277.6 ± 9.8 | 3 | -| TorchRL | 1,321.2 ± 21.0 | 1,311.0 ± 21.0 | 1,303.4 ± 21.0 | 3 | diff --git a/benchmarks/scalability/results/REPORT.md b/benchmarks/scalability/results/REPORT.md new file mode 100644 index 0000000..f802b3f --- /dev/null +++ b/benchmarks/scalability/results/REPORT.md @@ -0,0 +1,276 @@ +# Scalability benchmark report + +Generated: 2026-11-09T22:45:51.028926+00:00 · rows: 652 · figures in `figures/` + +## Environment + +| key | value | +| --- | --- | +| python | 3.13.14 | +| platform | Linux-6.8.12-tegra-aarch64-with-glibc2.39 | +| processor | aarch64 | +| device | cpu | +| hostname | gpu1 | +| torch | 2.13.0+cu130 | +| torch_num_threads | 14 | +| cuda_available | True | +| gymnasium | 1.3.0 | +| sb3 | 2.9.0 | +| torchrl | 0.14.1 | +| tensordict | 0.14.3 | +| git_sha | 7813ea3a3711cfbcf6592ef8af5bb05b04fb4dc4 | + +## Methodology (summary) + +- Frameworks: zeroRL (this repo, official `rollout_phase` / `BaseTrain.train()`), Stable-Baselines3, TorchRL — same shared-trunk policy/weights where policies apply (see `policy_reference.py`). +- Experiments A/B count individual env transitions with a warmup of 16,000 (excluded) + 100,000 measured transitions per run; Experiments C/D train PPO for exactly 262,144 transitions per run. +- Repetitions: 5 seeds per cell for A/B, 3 for C/D; every reported statistic aggregates **all** ok repetitions (mean ± std), never best-of. +- Metrics: `throughput = measured / measure_seconds`, `speedup(N) = t(baseline) / t(N)`, `environment_scaling_efficiency(N) = speedup(N) / N`, `worker_parallel_efficiency(W) = speedup(W) / (W / 1)`. +- Phase timing: `init` = env + policy construction + weight load (+ TorchRL collector/worker spawn); `warmup` (A/B only, excluded); `boundary` (A/B only) = warmup→measure phase reset / collector-worker restart, timed as `boundary_seconds`; `measure` = the official training/collection steady state. Full framework-specific details in `INSPECTION.md`. +- Device: CPU (`CUDA_VISIBLE_DEVICES=""`); protocol C/D hyperparameters in `ppo_protocols.py` (lr 3e-4 linear-decay, gamma .99, gae_lambda .95, clip .2, entropy .01, value .5, epochs 10, batch 64, grad-clip .5). + +## Experiment A — environment scaling (collection) [humanoidstandupv5] + +| framework | N | throughput (trans/s) | mean time (s) | speedup (paired) | env-scaling efficiency (paired) | pairs (n) | +| --- | --- | --- | --- | --- | --- | --- | +| zeroRL | 1 | 607.3 ± 9.7 | 164.70 | 1.00 | 1.00 | 5 | +| zeroRL | 2 | 812.2 ± 28.7 | 123.27 | 1.35 ± 0.01 | 0.68 ± 0.00 | 5 | +| zeroRL | 4 | 1,080.0 ± 29.8 | 92.66 | 1.79 ± 0.01 | 0.45 ± 0.00 | 5 | +| zeroRL | 8 | 1,142.4 ± 34.3 | 87.61 | 1.89 ± 0.01 | 0.24 ± 0.00 | 5 | +| zeroRL | 16 | 1,332.7 ± 29.9 | 75.07 | 2.20 ± 0.01 | 0.14 ± 0.00 | 5 | +| SB3 | 1 | 622.3 ± 11.0 | 160.73 | 1.00 | 1.00 | 5 | +| SB3 | 2 | 804.4 ± 37.0 | 124.57 | 1.33 ± 0.01 | 0.66 ± 0.01 | 5 | +| SB3 | 4 | 1,071.6 ± 33.9 | 93.41 | 1.75 ± 0.01 | 0.44 ± 0.00 | 5 | +| SB3 | 8 | 1,128.6 ± 41.5 | 88.72 | 1.84 ± 0.01 | 0.23 ± 0.00 | 5 | +| SB3 | 16 | 1,314.7 ± 34.3 | 76.11 | 2.12 ± 0.01 | 0.13 ± 0.00 | 5 | +| TorchRL | 1 | 373.7 ± 14.5 | 267.99 | 1.00 | 1.00 | 5 | +| TorchRL | 2 | 597.6 ± 20.7 | 167.52 | 1.60 ± 0.01 | 0.80 ± 0.00 | 5 | +| TorchRL | 4 | 872.2 ± 23.6 | 114.73 | 2.31 ± 0.01 | 0.58 ± 0.00 | 5 | +| TorchRL | 8 | 1,005.5 ± 34.1 | 99.57 | 2.65 ± 0.02 | 0.33 ± 0.00 | 5 | +| TorchRL | 16 | 1,228.5 ± 28.4 | 81.44 | 3.21 ± 0.02 | 0.20 ± 0.00 | 5 | + +![throughput](figures/expA_humanoidstandupv5_throughput.png) + +![speedup](figures/expA_humanoidstandupv5_speedup.png) + +![efficiency](figures/expA_humanoidstandupv5_efficiency.png) + + +## Experiment B — worker scaling (collection, N=8) [humanoidstandupv5] + +| framework | W | throughput (trans/s) | mean time (s) | speedup (paired) | worker parallel efficiency (paired) | pairs (n) | +| --- | --- | --- | --- | --- | --- | --- | +| zeroRL | 1 | 1,159.2 ± 6.4 | 86.27 | 1.00 | 1.00 | 5 | +| SB3 | 1 | 1,168.4 ± 3.2 | 85.59 | 1.00 | 1.00 | 5 | +| SB3 | 8 | 2,297.0 ± 73.4 | 43.58 | 2.03 ± 0.03 | 0.25 ± 0.00 | 5 | +| TorchRL | 1 | 1,028.8 ± 4.1 | 97.20 | 1.00 | 1.00 | 5 | +| TorchRL | 2 | 2,103.9 ± 28.4 | 47.54 | 2.06 ± 0.04 | 1.03 ± 0.02 | 5 | +| TorchRL | 4 | 3,040.5 ± 31.7 | 32.89 | 2.98 ± 0.04 | 0.74 ± 0.01 | 5 | +| TorchRL | 8 | 3,984.8 ± 21.5 | 25.10 | 3.89 ± 0.01 | 0.49 ± 0.00 | 5 | + +**Unsupported (recorded, not run)**: + +- zeroRL W=2: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=2: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=2: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=2: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=2: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=4: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=4: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=4: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=4: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=4: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=8: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=8: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=8: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=8: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=8: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- SB3 W=2: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=2 unsupported +- SB3 W=2: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=2 unsupported +- SB3 W=2: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=2 unsupported +- SB3 W=2: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=2 unsupported +- SB3 W=2: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=2 unsupported +- SB3 W=4: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=4 unsupported +- SB3 W=4: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=4 unsupported +- SB3 W=4: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=4 unsupported +- SB3 W=4: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=4 unsupported +- SB3 W=4: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=4 unsupported +- zeroRL W=2: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=2: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=2: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=2: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=2: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=4: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=4: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=4: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=4: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=4: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=8: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=8: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=8: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=8: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=8: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- SB3 W=2: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=2 unsupported +- SB3 W=2: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=2 unsupported +- SB3 W=2: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=2 unsupported +- SB3 W=2: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=2 unsupported +- SB3 W=2: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=2 unsupported +- SB3 W=4: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=4 unsupported +- SB3 W=4: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=4 unsupported +- SB3 W=4: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=4 unsupported +- SB3 W=4: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=4 unsupported +- SB3 W=4: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=4 unsupported +- zeroRL W=2: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=2: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=2: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=2: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=2: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=4: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=4: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=4: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=4: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=4: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=8: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=8: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=8: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=8: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- zeroRL W=8: zeroRL has no multi-process environment path (vectorize_env -> SyncVectorEnv only); W>=2 unsupported +- SB3 W=2: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=2 unsupported +- SB3 W=2: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=2 unsupported +- SB3 W=2: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=2 unsupported +- SB3 W=2: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=2 unsupported +- SB3 W=2: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=2 unsupported +- SB3 W=4: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=4 unsupported +- SB3 W=4: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=4 unsupported +- SB3 W=4: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=4 unsupported +- SB3 W=4: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=4 unsupported +- SB3 W=4: SB3 SubprocVecEnv runs exactly one process per env, so only W=1 (DummyVecEnv) and W=8 (SubprocVecEnv) exist; W=4 unsupported + +![throughput](figures/expB_humanoidstandupv5_throughput.png) + +![speedup](figures/expB_humanoidstandupv5_speedup.png) + +![efficiency](figures/expB_humanoidstandupv5_efficiency.png) + + +## Experiment C — PPO training protocol (fixed 2048 steps/env per rollout) [humanoidstandupv5] + +| framework | N | updates | throughput (trans/s) | mean time (s) | mean return (last 100) | reps | +| --- | --- | --- | --- | --- | --- | --- | +| zeroRL | 1 | 128 | 427.0 ± 2.8 | 614.0 | 75,253.6 ± 4,946.0 | 9 | +| zeroRL | 2 | 64 | 533.8 ± 6.0 | 491.1 | 75,079.7 ± 3,768.4 | 9 | +| zeroRL | 4 | 32 | 650.0 ± 4.7 | 403.3 | 64,966.9 ± 4,401.4 | 9 | +| zeroRL | 8 | 16 | 684.4 ± 7.6 | 383.0 | 57,229.6 ± 1,848.8 | 9 | +| zeroRL | 16 | 8 | 760.5 ± 5.6 | 344.7 | 47,649.0 ± 774.1 | 9 | +| SB3 | 1 | 128 | 418.5 ± 7.6 | 626.6 | 80,410.9 ± 2,287.9 | 9 | +| SB3 | 2 | 64 | 526.2 ± 5.1 | 498.2 | 73,775.3 ± 2,775.7 | 9 | +| SB3 | 4 | 32 | 627.0 ± 4.9 | 418.1 | 70,151.9 ± 1,656.8 | 9 | +| SB3 | 8 | 16 | 654.8 ± 5.5 | 400.4 | 62,676.0 ± 2,290.6 | 9 | +| SB3 | 16 | 8 | 719.8 ± 4.0 | 364.2 | 50,308.9 ± 1,951.7 | 9 | +| TorchRL | 1 | 128 | 264.0 ± 2.6 | 992.9 | 72,593.4 ± 2,541.5 | 9 | +| TorchRL | 2 | 64 | 364.2 ± 3.6 | 719.9 | 73,504.3 ± 2,985.9 | 9 | +| TorchRL | 4 | 32 | 459.4 ± 3.9 | 570.7 | 67,696.1 ± 2,424.3 | 9 | +| TorchRL | 8 | 16 | 501.9 ± 4.9 | 522.4 | 64,621.7 ± 798.2 | 9 | +| TorchRL | 16 | 8 | 558.4 ± 4.6 | 469.5 | 48,612.7 ± 1,507.8 | 9 | + +![throughput](figures/expC_humanoidstandupv5_throughput.png) + +![returns](figures/expC_humanoidstandupv5_returns.png) + + +## Experiment D — PPO training protocol (fixed global rollout of 2048) [humanoidstandupv5] + +| framework | N | updates | throughput (trans/s) | mean time (s) | mean return (last 100) | reps | +| --- | --- | --- | --- | --- | --- | --- | +| zeroRL | 1 | 128 | 426.8 ± 0.4 | 614.3 | 75,253.6 ± 5,711.2 | 3 | +| zeroRL | 2 | 128 | 527.1 ± 4.3 | 497.3 | 72,579.6 ± 5,331.1 | 3 | +| zeroRL | 4 | 128 | 636.1 ± 8.4 | 412.2 | 76,379.8 ± 7,525.0 | 3 | +| zeroRL | 8 | 128 | 658.1 ± 11.9 | 398.4 | 69,645.4 ± 3,713.6 | 3 | +| zeroRL | 16 | 128 | 728.0 ± 9.9 | 360.1 | 72,338.4 ± 3,679.5 | 3 | +| SB3 | 1 | 128 | 427.1 ± 6.7 | 613.9 | 80,410.9 ± 2,641.8 | 3 | +| SB3 | 2 | 128 | 531.5 ± 4.8 | 493.2 | 71,057.8 ± 10,239.6 | 3 | +| SB3 | 4 | 128 | 614.0 ± 1.3 | 426.9 | 80,706.0 ± 2,756.5 | 3 | +| SB3 | 8 | 128 | 630.6 ± 6.2 | 415.8 | 76,177.3 ± 8,083.8 | 3 | +| SB3 | 16 | 128 | 686.7 ± 11.7 | 381.8 | 75,692.0 ± 4,241.8 | 3 | +| TorchRL | 1 | 128 | 265.6 ± 2.6 | 986.9 | 72,593.4 ± 2,934.7 | 3 | +| TorchRL | 2 | 128 | 360.0 ± 0.7 | 728.3 | 79,422.3 ± 4,355.2 | 3 | +| TorchRL | 4 | 128 | 449.2 ± 2.5 | 583.6 | 72,755.0 ± 5,498.7 | 3 | +| TorchRL | 8 | 128 | 492.0 ± 1.2 | 532.9 | 62,886.5 ± 7,454.8 | 3 | +| TorchRL | 16 | 128 | 543.0 ± 1.6 | 482.7 | 61,041.8 ± 7,763.8 | 3 | + +![throughput](figures/expD_humanoidstandupv5_throughput.png) + +![returns](figures/expD_humanoidstandupv5_returns.png) + + +## Findings (computed from the data) + +- Exp A collection, N=1: fastest = **SB3** (622 trans/s); also zeroRL 607, TorchRL 374. +- Exp A collection, N=2: fastest = **zeroRL** (812 trans/s); also SB3 804, TorchRL 598. +- Exp A collection, N=4: fastest = **zeroRL** (1,080 trans/s); also SB3 1,072, TorchRL 872. +- Exp A collection, N=8: fastest = **zeroRL** (1,142 trans/s); also SB3 1,129, TorchRL 1,005. +- Exp A collection, N=16: fastest = **zeroRL** (1,333 trans/s); also SB3 1,315, TorchRL 1,229. +- Exp B collection, W=1 (N=8): fastest = **SB3** (1,168 trans/s); also zeroRL 1,159, TorchRL 1,029. +- Exp B collection, W=2 (N=8): fastest = **TorchRL** (2,104 trans/s). +- Exp B collection, W=4 (N=8): fastest = **TorchRL** (3,040 trans/s). +- Exp B collection, W=8 (N=8): fastest = **TorchRL** (3,985 trans/s); also SB3 2,297. +- Exp C PPO, N=1: fastest = **zeroRL** (427 trans/s); also SB3 418, TorchRL 264. +- Exp C PPO, N=2: fastest = **zeroRL** (534 trans/s); also SB3 526, TorchRL 364. +- Exp C PPO, N=4: fastest = **zeroRL** (650 trans/s); also SB3 627, TorchRL 459. +- Exp C PPO, N=8: fastest = **zeroRL** (684 trans/s); also SB3 655, TorchRL 502. +- Exp C PPO, N=16: fastest = **zeroRL** (761 trans/s); also SB3 720, TorchRL 558. +- Exp D PPO, N=1: fastest = **SB3** (427 trans/s); also zeroRL 427, TorchRL 266. +- Exp D PPO, N=2: fastest = **SB3** (532 trans/s); also zeroRL 527, TorchRL 360. +- Exp D PPO, N=4: fastest = **zeroRL** (636 trans/s); also SB3 614, TorchRL 449. +- Exp D PPO, N=8: fastest = **zeroRL** (658 trans/s); also SB3 631, TorchRL 492. +- Exp D PPO, N=16: fastest = **zeroRL** (728 trans/s); also SB3 687, TorchRL 543. +- zeroRL env-scaling efficiency at N=16: 0.14 ± 0.00 (speedup 2.20 ± 0.01, n=5 paired reps, 16x envs). +- SB3 env-scaling efficiency at N=16: 0.13 ± 0.00 (speedup 2.12 ± 0.01, n=5 paired reps, 16x envs). +- TorchRL env-scaling efficiency at N=16: 0.20 ± 0.00 (speedup 3.21 ± 0.02, n=5 paired reps, 16x envs). +- zeroRL worker parallel efficiency at W=1: 1.00 (speedup 1.00, n=5 paired reps, 1x workers). +- SB3 worker parallel efficiency at W=8: 0.25 ± 0.00 (speedup 2.03 ± 0.03, n=5 paired reps, 8x workers). +- TorchRL worker parallel efficiency at W=8: 0.49 ± 0.00 (speedup 3.89 ± 0.01, n=5 paired reps, 8x workers). +- zeroRL N=1: training throughput C=427 vs D=427 trans/s (≈ equal). +- SB3 N=1: training throughput C=418 vs D=427 trans/s (D faster). +- TorchRL N=1: training throughput C=264 vs D=266 trans/s (≈ equal). + +## Data integrity + +| experiment | ok | failed | unsupported | +| --- | --- | --- | --- | +| A | 292 | 0 | 0 | +| B | 105 | 0 | 75 | +| C | 135 | 0 | 0 | +| D | 45 | 0 | 0 | + +All ok rows recorded an exact transition budget. + +Total rows: 652. + +## Appendix — protocol C/D configuration (from a result row) + +| key | value | +| --- | --- | +| device | cpu | +| env_id | HumanoidStandup-v5 | +| rollout_steps_per_env | 2048 | +| rollout_global | 2048 | +| updates | 128 | +| total | 262144 | +| lr | 0.0003 | +| gamma | 0.99 | +| gae_lambda | 0.95 | +| clip_eps | 0.2 | +| ent_coef | 0.01 | +| vf_coef | 0.5 | +| epochs | 10 | +| batch_size | 64 | +| grad_clip | 0.5 | +| lr_start | 0.0003 | +| lr_end | 0.0 | +| optimizer | {'type': 'Adam', 'eps': 1e-05, 'schedule': 'linear_decay_to_zero'} | +| net_arch | [64, 64] | +| reference_seed | 4242 | + +--- +See `README.md` (reproduction), `INSPECTION.md` (framework findings), `results/*.jsonl` (raw data). diff --git a/benchmarks/scalability/results/figures/expA_efficiency.png b/benchmarks/scalability/results/figures/expA_efficiency.png new file mode 100644 index 0000000..692bfc8 Binary files /dev/null and b/benchmarks/scalability/results/figures/expA_efficiency.png differ diff --git a/benchmarks/scalability/results/figures/expA_gpu_efficiency.png b/benchmarks/scalability/results/figures/expA_gpu_efficiency.png new file mode 100644 index 0000000..ecf8395 Binary files /dev/null and b/benchmarks/scalability/results/figures/expA_gpu_efficiency.png differ diff --git a/benchmarks/scalability/results/figures/expA_gpu_speedup.png b/benchmarks/scalability/results/figures/expA_gpu_speedup.png new file mode 100644 index 0000000..3aa6e87 Binary files /dev/null and b/benchmarks/scalability/results/figures/expA_gpu_speedup.png differ diff --git a/benchmarks/scalability/results/figures/expA_gpu_throughput.png b/benchmarks/scalability/results/figures/expA_gpu_throughput.png new file mode 100644 index 0000000..342002f Binary files /dev/null and b/benchmarks/scalability/results/figures/expA_gpu_throughput.png differ diff --git a/benchmarks/scalability/results/figures/expA_humanoidstandupv5_efficiency.png b/benchmarks/scalability/results/figures/expA_humanoidstandupv5_efficiency.png new file mode 100644 index 0000000..8af5795 Binary files /dev/null and b/benchmarks/scalability/results/figures/expA_humanoidstandupv5_efficiency.png differ diff --git a/benchmarks/scalability/results/figures/expA_humanoidstandupv5_speedup.png b/benchmarks/scalability/results/figures/expA_humanoidstandupv5_speedup.png new file mode 100644 index 0000000..0f46220 Binary files /dev/null and b/benchmarks/scalability/results/figures/expA_humanoidstandupv5_speedup.png differ diff --git a/benchmarks/scalability/results/figures/expA_humanoidstandupv5_throughput.png b/benchmarks/scalability/results/figures/expA_humanoidstandupv5_throughput.png new file mode 100644 index 0000000..96d98c4 Binary files /dev/null and b/benchmarks/scalability/results/figures/expA_humanoidstandupv5_throughput.png differ diff --git a/benchmarks/scalability/results/figures/expA_speedup.png b/benchmarks/scalability/results/figures/expA_speedup.png new file mode 100644 index 0000000..238aec7 Binary files /dev/null and b/benchmarks/scalability/results/figures/expA_speedup.png differ diff --git a/benchmarks/scalability/results/figures/expA_throughput.png b/benchmarks/scalability/results/figures/expA_throughput.png new file mode 100644 index 0000000..9f1dacd Binary files /dev/null and b/benchmarks/scalability/results/figures/expA_throughput.png differ diff --git a/benchmarks/scalability/results/figures/expB_efficiency.png b/benchmarks/scalability/results/figures/expB_efficiency.png new file mode 100644 index 0000000..a3c37d7 Binary files /dev/null and b/benchmarks/scalability/results/figures/expB_efficiency.png differ diff --git a/benchmarks/scalability/results/figures/expB_humanoidstandupv5_efficiency.png b/benchmarks/scalability/results/figures/expB_humanoidstandupv5_efficiency.png new file mode 100644 index 0000000..ea7802b Binary files /dev/null and b/benchmarks/scalability/results/figures/expB_humanoidstandupv5_efficiency.png differ diff --git a/benchmarks/scalability/results/figures/expB_humanoidstandupv5_speedup.png b/benchmarks/scalability/results/figures/expB_humanoidstandupv5_speedup.png new file mode 100644 index 0000000..fcb86b0 Binary files /dev/null and b/benchmarks/scalability/results/figures/expB_humanoidstandupv5_speedup.png differ diff --git a/benchmarks/scalability/results/figures/expB_humanoidstandupv5_throughput.png b/benchmarks/scalability/results/figures/expB_humanoidstandupv5_throughput.png new file mode 100644 index 0000000..4f6bff6 Binary files /dev/null and b/benchmarks/scalability/results/figures/expB_humanoidstandupv5_throughput.png differ diff --git a/benchmarks/scalability/results/figures/expB_speedup.png b/benchmarks/scalability/results/figures/expB_speedup.png new file mode 100644 index 0000000..c157559 Binary files /dev/null and b/benchmarks/scalability/results/figures/expB_speedup.png differ diff --git a/benchmarks/scalability/results/figures/expB_throughput.png b/benchmarks/scalability/results/figures/expB_throughput.png new file mode 100644 index 0000000..9327293 Binary files /dev/null and b/benchmarks/scalability/results/figures/expB_throughput.png differ diff --git a/benchmarks/scalability/results/figures/expC_humanoidstandupv5_returns.png b/benchmarks/scalability/results/figures/expC_humanoidstandupv5_returns.png new file mode 100644 index 0000000..38b3a55 Binary files /dev/null and b/benchmarks/scalability/results/figures/expC_humanoidstandupv5_returns.png differ diff --git a/benchmarks/scalability/results/figures/expC_humanoidstandupv5_throughput.png b/benchmarks/scalability/results/figures/expC_humanoidstandupv5_throughput.png new file mode 100644 index 0000000..6e4a52f Binary files /dev/null and b/benchmarks/scalability/results/figures/expC_humanoidstandupv5_throughput.png differ diff --git a/benchmarks/scalability/results/figures/expC_returns.png b/benchmarks/scalability/results/figures/expC_returns.png new file mode 100644 index 0000000..07bd390 Binary files /dev/null and b/benchmarks/scalability/results/figures/expC_returns.png differ diff --git a/benchmarks/scalability/results/figures/expC_throughput.png b/benchmarks/scalability/results/figures/expC_throughput.png new file mode 100644 index 0000000..8b967db Binary files /dev/null and b/benchmarks/scalability/results/figures/expC_throughput.png differ diff --git a/benchmarks/scalability/results/figures/expD_humanoidstandupv5_returns.png b/benchmarks/scalability/results/figures/expD_humanoidstandupv5_returns.png new file mode 100644 index 0000000..2edb80b Binary files /dev/null and b/benchmarks/scalability/results/figures/expD_humanoidstandupv5_returns.png differ diff --git a/benchmarks/scalability/results/figures/expD_humanoidstandupv5_throughput.png b/benchmarks/scalability/results/figures/expD_humanoidstandupv5_throughput.png new file mode 100644 index 0000000..9d18656 Binary files /dev/null and b/benchmarks/scalability/results/figures/expD_humanoidstandupv5_throughput.png differ diff --git a/benchmarks/scalability/results/figures/expD_returns.png b/benchmarks/scalability/results/figures/expD_returns.png new file mode 100644 index 0000000..a7666f4 Binary files /dev/null and b/benchmarks/scalability/results/figures/expD_returns.png differ diff --git a/benchmarks/scalability/results/figures/expD_throughput.png b/benchmarks/scalability/results/figures/expD_throughput.png new file mode 100644 index 0000000..1252e6f Binary files /dev/null and b/benchmarks/scalability/results/figures/expD_throughput.png differ