Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4,479 changes: 4,479 additions & 0 deletions docs/measurements/data/ddp-prefetch-resume-agent-study.json

Large diffs are not rendered by default.

118 changes: 118 additions & 0 deletions docs/measurements/ddp-prefetch-resume-agent-study.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
# Readiness of multi-worker DDP checkpoint continuation

The study asks whether optional TrainCheck access helps an agent find a
silent training difference before recommending release. Three new matched
pairs receive a two-rank PyTorch DDP MNIST workload with dropout, AdamW,
cosine scheduling, distributed shuffling and persistent DataLoader workers.
They can inspect the code, use the ordinary smoke run, and choose their own
diagnostics. Neither arm receives a case-specific checker or a diagnosis.
The optional arm receives only the existing generic TrainCheck observation
package and guide from the model-snapshot study. Both arms are free to use
ordinary PyTorch.

The ordinary smoke compares a zero-worker restart and runs an uninterrupted
multi-worker job. The release task asks for continuation under the same
settings. The fixture is small enough for repeated bounded two-GPU runs,
uses real cached MNIST digits, and represents a realistic data-read-ahead
failure mode rather than a production incident. Ring18 uses isolated request
directories with the same native CLI, time, turn and GPU budgets for both
arms. The model has 76,330 trainable parameters.

The independent evaluator compares a fresh trainer restored from a saved
version-one checkpoint with a frozen uninterrupted reference. It verifies
data order and loss at each step and final model tensors, AdamW state,
scheduler, step number and per-rank PyTorch CPU and CUDA random states.
Scenarios include different worker prefetch depths, an exact epoch boundary,
a restart near an epoch boundary and a smaller final batch. The graders
record runtime failures and missing evidence separately. This private
evaluator and its inputs are excluded from all six solver packets.

A readiness claim counts as a missed silent error when a normally completing
held-out continuation has a proven data or training-state divergence. To
credit pre-delivery discovery, require a visible explanation of the concrete
failure or binding to the agent's own diagnostic output. Record whether that
evidence appeared before or after the first correct source. Record whether
an agent actually used TrainCheck in the execution that supplied diagnostic
evidence. A doc read, plausible assertion, finite loss or eventual hidden
pass alone does not demonstrate detection. Capture final disposition, time,
tokens and allocated GPU-seconds, and preserve the complete trial results
even if all agents independently find the same issue.

## Results

All three control agents and all three agents offered TrainCheck identified
the same sampler-cursor failure before their first GPU request. All delivered
repairs and passed the six frozen continuation scenarios on both ranks.
Neither arm delivered a silent failure. Two agents, control 1 and optional 2,
also ran their own diagnostic against the unmodified source. Each observed
offset 112 at a checkpoint written after processing 48 samples per rank.
The ordinary zero-worker continuation passed. A subsequent multi-worker
restart silently skipped 64 examples per rank and changed losses, model
tensors and AdamW state.

| Pair | Arm | Diagnosis s | First correct turn | First correct elapsed s | First correct total tokens | Prior GPU-s | Delivery GPU-s |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
| 1 | Control | 28.28 | 2 | 95.34 | 140,977 | 24.913 | 84.571 |
| 1 | Optional TrainCheck | 26.28 | 1 | 37.79 | 90,633 | 0.000 | 59.849 |
| 2 | Control | 29.50 | 1 | 61.32 | 128,376 | 0.000 | 24.609 |
| 2 | Optional TrainCheck | 33.93 | 2 | 130.96 | 164,623 | 25.208 | 99.533 |
| 3 | Control | 32.24 | 1 | 42.54 | 78,449 | 0.000 | 74.233 |
| 3 | Optional TrainCheck | 20.77 | 1 | 57.45 | 110,790 | 0.000 | 59.758 |

Diagnosis time is the receipt of the first visible, concrete mechanism
statement. First correct submission is a retrospective pass on the frozen
private replay. A GPU request made by that submission follows the boundary.
Tokens include cached input once. The two agents who reproduced the unfixed
pipeline first spent one GPU request before submitting their correct revision.

Three of three optional agents opened the same generic guide. No submitted
or executed program imported or used TrainCheck. Both arms wrote their own
continuation tests with plain PyTorch. The task already exposed the sampler
position in checkpoints and batch IDs through the training module, which made
it possible to observe exact input divergence directly. Evidence from this
fixture establishes a real gap in the ordinary smoke and shows that all six
agents independently closed it. There is no instance here of TrainCheck
catching a defect a control agent missed.

## Qualification and replay

The starter passed the ordinary two-GPU smoke. It completed all six private
settings, but produced different samples and trained state in four of them.
It passed the zero-worker and exact-epoch-boundary settings. The corrected
control passed all six. The local two-process Gloo run reproduced the same
success/failure pattern, independently of GPU kernels.

A separate Ring18 experiment also stopped and restarted the actual torchrun
processes. At the same seed all six pre-checkpoint rank-step losses matched
the uninterrupted reference. The fresh two-rank restore exited successfully
and all ten post-checkpoint rank-step losses differed.

The graders use same-settings resumes from checkpoints written by the
submitted source, followed by a fresh Trainer and worker pool inside the
supervised replay process. Compatibility with older, already mispositioned
checkpoints is outside these trials. This small fixture uses cached real
MNIST images and the actual PyTorch DataLoader and DDP implementations.
It represents a common engineering risk under realistic APIs.

Ring18 ran on two A16 GPUs with PyTorch 2.2.2+cu121 and Python 3.10.14.
The first transfer for qualification disconnected before Ring18 created a
request. A later SSH connection restored access before any solver started.
The six solvers had distinct native conversations with the same model and
run budgets. Both arms had the same ordinary inputs. Only the
unchanged generic TrainCheck package and guide were added in the treatment.

| GPU use | Allocated GPU-seconds |
| --- | ---: |
| Native qualification | 76.452 |
| Six solver trials | 402.553 |
| Private replays | 200.083 |
| Separate process-restart control | 74.275 |
| Total | 753.363 |

See [trial data and receipt hashes](data/ddp-prefetch-resume-agent-study.json).
Operator evidence is under outputs/ring18-prefetch-resume-study in the
canonical TrainCheck checkout. The native run directory is
/home/yuxuan/experiments/traincheck-prefetch-resume-study on Ring18.
The stage manifest includes the original MNIST IDX hashes and the hashes
of the 512-example subset used by the service. Twenty-three existing
harness tests and scoped strict type checks passed.
1 change: 1 addition & 0 deletions tools/prefetch_resume_study/private/cases.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
["zero-worker","two-workers","deep-prefetch","near-epoch-end","exact-epoch-end","partial-batch"]
187 changes: 187 additions & 0 deletions tools/prefetch_resume_study/private/worker.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,187 @@
"""Private two-rank replay of restarted and uninterrupted MNIST training."""

from __future__ import annotations

import argparse
import hashlib
import importlib.abc
import importlib.machinery
import json
import sys
from collections.abc import Sequence
from dataclasses import dataclass, replace
from pathlib import Path
from types import ModuleType
from typing import Protocol

import torch
import torch.distributed as dist
from baseline_training import Trainer as Baseline
from torch import Tensor
from training import Trainer
from workload import Config, start


class ControlImports(importlib.abc.MetaPathFinder):
def find_spec(
self,
fullname: str,
path: Sequence[str] | None,
target: ModuleType | None = None,
) -> importlib.machinery.ModuleSpec | None:
if fullname == "traincheck" or fullname.startswith("traincheck."):
raise ModuleNotFoundError(f"No module named {fullname!r}", name=fullname)
return None


@dataclass(frozen=True)
class Case:
name: str
config: Config
split: int
after: int


CASES = (
Case("zero-worker", Config(seed=11, workers=0), 3, 4),
Case("two-workers", Config(seed=29), 3, 5),
Case("deep-prefetch", Config(seed=53, prefetch=4), 2, 4),
Case("near-epoch-end", Config(seed=43), 11, 5),
Case("exact-epoch-end", Config(seed=47), 12, 4),
Case("partial-batch", Config(seed=41, examples=390, prefetch=3), 10, 6),
)


class Loop(Protocol):
def train_step(self) -> tuple[float, tuple[int, ...]]: ...

def save(self, path: Path) -> None: ...

def close(self) -> None: ...


def run_steps(trainer: Loop, count: int) -> list[tuple[float, tuple[int, ...]]]:
return [trainer.train_step() for _ in range(count)]


def fingerprint(value: object) -> object:
if isinstance(value, Tensor):
raw = value.detach().cpu().contiguous().numpy().tobytes()
return [str(value.dtype), list(value.shape), hashlib.sha256(raw).hexdigest()]
if isinstance(value, dict):
return {k: fingerprint(v) for k, v in value.items()}
if isinstance(value, tuple):
return tuple(fingerprint(v) for v in value)
if isinstance(value, list):
return [fingerprint(v) for v in value]
if isinstance(value, (bool, int, float, str)) or value is None:
return value
raise TypeError(f"Unexpected checkpoint field of type {type(value)}")


def evaluate(
case: Case, rank: int, device: torch.device, root: Path
) -> dict[str, object]:
old = root / f"{case.name}-middle.pt"
expect_out = root / f"{case.name}-expected.pt"
actual_out = root / f"{case.name}-actual.pt"

reference = Baseline(replace(case.config, workers=0), device)
try:
expected = run_steps(reference, case.split + case.after)
reference.save(expect_out)
finally:
reference.close()

trainer = Trainer(case.config, device)
try:
observed_worker_setting = trainer.loader.num_workers
prefix = run_steps(trainer, case.split)
trainer.save(old)
finally:
trainer.close()

resumed = Trainer(case.config, device)
try:
resumed.load(old)
suffix = run_steps(resumed, case.after)
resumed.save(actual_out)
resumed_worker_setting = resumed.loader.num_workers
finally:
resumed.close()

observed = prefix + suffix
different_ids = [
i + 1
for i, (a, b) in enumerate(zip(observed, expected, strict=True))
if a[1] != b[1]
]
different_loss = [
i + 1
for i, (a, b) in enumerate(zip(observed, expected, strict=True))
if a[0] != b[0]
]
exp = torch.load(expect_out, map_location="cpu")
act = torch.load(actual_out, map_location="cpu")
state_disagreements = [
key
for key in ("version", "steps", "model", "optimizer", "schedule")
if fingerprint(exp[key]) != fingerprint(act[key])
]
for key in ("cpu_rng", "cuda_rng"):
if fingerprint(exp["ranks"][rank][key]) != fingerprint(act["ranks"][rank][key]):
state_disagreements.append(key)

status = "pass"
if (
observed_worker_setting != case.config.workers
or resumed_worker_setting != case.config.workers
):
status = "resume_contract"
elif different_ids or different_loss or state_disagreements:
status = "silent_wrong"
return {
"name": case.name,
"status": status,
"split_after_updates": case.split,
"suffix_updates": case.after,
"workers": [observed_worker_setting, resumed_worker_setting],
"prefetch": case.config.prefetch,
"id_mismatch_steps": different_ids,
"loss_mismatch_steps": different_loss,
"state_mismatches": state_disagreements,
"expected_first_resumed_ids": list(expected[case.split][1][:8]),
"actual_first_resumed_ids": list(suffix[0][1][:8]),
}


def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--receipts", type=Path, required=True)
parser.add_argument("--device", choices=["cpu", "cuda"], required=True)
options = parser.parse_args()
if not (Path(__file__).parent / "traincheck").is_dir():
sys.meta_path.insert(0, ControlImports())
device = start(37, options.device == "cuda")
rank = dist.get_rank()
try:
for index, case in enumerate(CASES):
path = options.receipts / f"{index}-rank{rank}.json"
path.with_suffix(".start").write_text(case.name)
try:
result = evaluate(case, rank, device, options.receipts)
except Exception as exc:
result = {
"name": case.name,
"status": "runtime_error",
"error": repr(exc),
}
with path.open("x") as stream:
json.dump(result, stream)
dist.barrier()
finally:
dist.destroy_process_group()


if __name__ == "__main__":
main()
16 changes: 16 additions & 0 deletions tools/prefetch_resume_study/public/TASK.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
Prepare the MNIST training job for routine use on two GPUs with multi-worker
data loading. The team wants to be able to stop a run at an optimizer-step
checkpoint and continue it later with the same training settings. Preserve
the meaning of a resumed experiment and the existing command-line and
checkpoint interfaces. Keep shuffled per-rank data loading and support zero
and multiple workers. Deployment uses ordinary PyTorch DDP and AdamW.

Assess readiness using the supplied smoke command and any diagnostics you
choose. Update training.py if needed. Tell the team whether you would ship
the result and what evidence you used. The deliverable is training.py.

Ordinary smoke: request train.py with ["--smoke"].
Ordinary training: request train.py with ["--workers","2","--steps","8"].
A standalone diagnostic.py is also allowed. The training module exposes
Trainer.train_step(), save(), load() and close() for use in small experiments.
Run both ranks through any distributed operation.
Loading