Skip to content
Merged

Atari #100

Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
134 changes: 0 additions & 134 deletions benchmarks/benchmarks/scalability/results/expA_gpu.jsonl

This file was deleted.

75 changes: 0 additions & 75 deletions benchmarks/exp_A_gpu.jsonl

This file was deleted.

2 changes: 1 addition & 1 deletion benchmarks/resources/results/REPORT.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Resource-consumption benchmark report

Generated: 2026-11-03T19:43:24.430693+00:00 · results: /home/darrius/project/zerorl/benchmarks/resources/results
Generated: 2026-11-08T23:35:45.924829+00:00 · results: /home/darrius/project/zerorl/benchmarks/resources/results

## Resource consumption (experiment R)

Expand Down
58 changes: 58 additions & 0 deletions benchmarks/resources/results/REPORT_humanoidstandupv4.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# Resource-consumption benchmark report

Generated: 2026-11-08T23:35:46.556160+00:00 · results: /home/darrius/project/zerorl/benchmarks/resources/results

## Resource consumption (experiment R) [humanoidstandupv4]

Fixed workloads, device CPU. CPU% = sum over the live process tree (multiprocess sums may exceed 100%); RSS summed across tree processes (shared pages may repeat — PSS/USS recorded in the raw rows); GPU = device-level telemetry only (workload does not use the GPU); energy recorded only when reliable telemetry exists.

| workload | framework | wall (s) | cpu util avg (%) | cpu total (s) | cpu s / 1k | rss peak (MiB) | n |
| --- | --- | --- | --- | --- | --- | --- | --- |
| collect | zeroRL | 112.0 ± 5.2 | 223.0 ± 1.8 | 249.2 ± 9.9 | 2.49 ± 0.10 | 1,269.0 ± 170.4 | 5 |
| collect | SB3 | 105.6 ± 7.4 | 226.7 ± 2.5 | 239.3 ± 13.9 | 2.39 ± 0.14 | 1,632.6 ± 74.2 | 5 |
| collect | TorchRL | 121.9 ± 3.1 | 207.1 ± 1.2 | 251.6 ± 4.0 | 2.52 ± 0.04 | 1,307.8 ± 57.5 | 5 |
| ppo | zeroRL | 421.0 ± 29.7 | 395.7 ± 18.6 | 1,672.8 ± 39.7 | 6.38 ± 0.15 | 1,441.2 ± 270.0 | 3 |
| ppo | SB3 | 416.4 ± 16.1 | 432.5 ± 5.0 | 1,806.2 ± 52.2 | 6.89 ± 0.20 | 1,295.9 ± 9.8 | 3 |
| ppo | TorchRL | 670.3 ± 24.2 | 385.3 ± 3.5 | 2,580.7 ± 88.4 | 9.84 ± 0.34 | 1,335.1 ± 17.1 | 3 |

- GPU telemetry: available (device-level only).
- Energy: unavailable — GPU power telemetry unsupported on this device and no RAPL/powercap energy interface for CPU/whole-system energy.
- Raw per-run rows: `results/resources_humanoidstandupv4.jsonl` (validation runs kept separately in `resources_validation.*`).

_Memory note: raw values are bytes (v2; v1 rows/sample files used KiB); averages are time-weighted (trapezoidal); summed RSS may double-count shared pages — PSS/USS are the attributable views; v1 rows show — (never 0)._

### Memory (RSS / PSS / USS) — collect

**Peak (end-to-end)**

| framework | rss (MiB) | pss (MiB) | uss (MiB) | peak rss — measure phase (MiB) | n |
| --- | --- | --- | --- | --- | --- |
| zeroRL | 1,269.0 ± 170.4 | 1,266.4 ± 170.5 | 1,265.7 ± 170.5 | 1,197.7 ± 121.0 | 5 |
| SB3 | 1,632.6 ± 74.2 | 1,630.4 ± 74.2 | 1,629.6 ± 74.2 | 1,632.6 ± 74.2 | 5 |
| TorchRL | 1,307.8 ± 57.5 | 1,305.6 ± 57.6 | 1,304.8 ± 57.6 | 1,305.3 ± 62.4 | 5 |

**Average (end-to-end, time-weighted)**

| framework | rss (MiB) | pss (MiB) | uss (MiB) | n |
| --- | --- | --- | --- | --- |
| zeroRL | 1,203.0 ± 119.9 | 1,200.7 ± 119.8 | 1,199.9 ± 119.8 | 5 |
| SB3 | 1,475.3 ± 64.7 | 1,473.2 ± 64.5 | 1,472.4 ± 64.5 | 5 |
| TorchRL | 1,285.9 ± 59.3 | 1,283.7 ± 59.2 | 1,282.9 ± 59.2 | 5 |

### Memory (RSS / PSS / USS) — ppo

**Peak (end-to-end)**

| framework | rss (MiB) | pss (MiB) | uss (MiB) | peak rss — measure phase (MiB) | n |
| --- | --- | --- | --- | --- | --- |
| zeroRL | 1,441.2 ± 270.0 | 1,376.2 ± 175.7 | 1,316.9 ± 86.2 | 1,441.2 ± 270.0 | 3 |
| SB3 | 1,295.9 ± 9.8 | 1,285.2 ± 9.8 | 1,277.6 ± 9.8 | 1,295.8 ± 9.8 | 3 |
| TorchRL | 1,335.1 ± 17.1 | 1,324.9 ± 16.7 | 1,317.3 ± 16.7 | 1,321.3 ± 21.1 | 3 |

**Average (end-to-end, time-weighted)**

| framework | rss (MiB) | pss (MiB) | uss (MiB) | n |
| --- | --- | --- | --- | --- |
| zeroRL | 1,284.7 ± 1.2 | 1,273.9 ± 1.6 | 1,266.0 ± 2.0 | 3 |
| SB3 | 1,295.8 ± 9.8 | 1,285.2 ± 9.8 | 1,277.6 ± 9.8 | 3 |
| TorchRL | 1,321.2 ± 21.0 | 1,311.0 ± 21.0 | 1,303.4 ± 21.0 | 3 |
23 changes: 17 additions & 6 deletions benchmarks/scalability/collection.py
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,10 @@ def _absorb_actions(fb: dict[str, Any], actions: list[int],
if len(fb["actions_head"]) < FINGERPRINT_HEAD:
fb["actions_head"].append(int(a) if is_discrete
else round(float(a), 6))
total = int(sum(actions))
# int actions sum to int (unchanged); continuous sums MUST stay float —
# int() truncation made every Box action sum to 0 and broke parity with
# zeroRL's float(np.sum(buffer_actions)) fingerprint.
total = sum(actions)
if tracker.target == fb.get("_warm_target"):
fb["action_sum_warmup"] += total
else:
Expand Down Expand Up @@ -381,14 +384,22 @@ def run_zerorl_collection(cfg: CollectionConfig, num_envs: int, seed: int,

# fingerprints from the (official-path) buffer
data = buffer.get_all()
actions = data["action"].reshape(-1).tolist()
flat = data["action"].reshape(-1)
# action elements per vectorized step: 1 for scalar (discrete)
# actions, act_dim for Box actions — the warmup slice must cover
# whole transitions, matching what _absorb_actions accumulates for
# SB3/TorchRL (all warmup action values), not a flat-element prefix.
act_dim = flat.numel() // (data["action"].shape[0]
* data["action"].shape[1])
actions = flat.tolist()
warm_n = cfg.warmup
warm_elems = warm_n * act_dim
fb = {
"first_obs_head": [round(float(x), 6) for x in first_obs.reshape(-1)[:4]],
"actions_head": [int(a) if is_discrete else round(float(a), 6)
for a in actions[:FINGERPRINT_HEAD]],
"action_sum_warmup": float(np.sum(actions[:warm_n])),
"action_sum_measure": float(np.sum(actions[warm_n:])),
"action_sum_warmup": float(np.sum(actions[:warm_elems])),
"action_sum_measure": float(np.sum(actions[warm_elems:])),
}

warm_t = BudgetTracker(cfg.warmup)
Expand Down Expand Up @@ -616,7 +627,7 @@ def run_torchrl_collection(cfg: CollectionConfig, num_envs: int, seed: int,
for x in td["observation"][0].reshape(-1)[:4]]
# td batch dims are (env, time); fingerprint order must be
# time-major (step0_env0, step0_env1, ...) to match zeroRL/SB3.
_absorb_actions(fb, td["action"].movedim(-1, 0).reshape(-1).tolist(),
_absorb_actions(fb, td["action"].movedim(1, 0).reshape(-1).tolist(),
warm_tracker, is_discrete=d.is_discrete)
warm_tracker.record(n)
warm_s = time.perf_counter() - t
Expand All @@ -640,7 +651,7 @@ def run_torchrl_collection(cfg: CollectionConfig, num_envs: int, seed: int,
t = time.perf_counter()
for td in collector2:
n = int(td.batch_size.numel())
_absorb_actions(fb, td["action"].movedim(-1, 0).reshape(-1).tolist(),
_absorb_actions(fb, td["action"].movedim(1, 0).reshape(-1).tolist(),
meas_tracker, is_discrete=d.is_discrete)
for key in meas_fields:
meas_fields[key] += n
Expand Down
12 changes: 7 additions & 5 deletions benchmarks/scalability/policy_reference.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,10 +68,12 @@ class EnvDims(NamedTuple):
def env_dims(env_id: str) -> EnvDims:
"""Probe an env's spaces (``gym.make`` -> read -> close); lru-cached.

Mirrors ``zerorl.functions.get_obs_act`` exactly:
discrete -> ``act_shape=()``, ``act_n=space.n``; Box -> ``act_shape=
space.shape``, ``act_n=int(np.prod(shape))``; ``obs_n = shape[-1]``
(or the space itself for 3-D image observations).
Discrete -> ``act_shape=()``, ``act_n=space.n``; Box -> ``act_shape=
space.shape``, ``act_n=int(np.prod(shape))``. ``obs_n`` is the FLAT
integer input size for the benchmark MLPs — ``int(np.prod(obs_shape))``
for every rank (``get_obs_act`` now returns the observation *shape*
tuple as ``obs_n``; benchmark trunks stay MLP-only, so flat ints are
the correct contract here).
"""
env = gym.make(env_id)
try:
Expand All @@ -86,7 +88,7 @@ def env_dims(env_id: str) -> EnvDims:
act_n = int(act_space.n) # type: ignore[attr-defined]
else:
act_n = int(np.prod(act_space.shape)) # type: ignore[arg-type]
obs_n: Any = obs_space if len(obs_shape) == 3 else obs_shape[-1]
obs_n = int(np.prod(obs_shape))
return EnvDims(obs_shape, act_shape, obs_n, act_n, is_discrete)


Expand Down
Loading
Loading