From 5bcfcd4a9daf71acf7547db79a657b5ea8580a95 Mon Sep 17 00:00:00 2001 From: Yuxuan Jiang Date: Wed, 18 Nov 2026 03:04:42 +0800 Subject: [PATCH 1/6] Define prospective outcomes for the three-pair DDP feature study --- .../token-budget-agent-study-20261118.md | 73 +++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 docs/measurements/token-budget-agent-study-20261118.md diff --git a/docs/measurements/token-budget-agent-study-20261118.md b/docs/measurements/token-budget-agent-study-20261118.md new file mode 100644 index 00000000..b3edcae8 --- /dev/null +++ b/docs/measurements/token-budget-agent-study-20261118.md @@ -0,0 +1,73 @@ +# Implementing token-budget accumulation under DDP + +This prospective Ring18 study asks whether six fresh agents deliver a numerically +correct feature when they add token-budget gradient accumulation to a working +fixed-mode training loop. Three agents receive the ordinary task and three can +also use TrainCheck. Each pair receives the same task, model settings, source, +compute and decision budgets. We retain all six outcomes, including declines, +errors and incomplete work. + +## Task and endpoint + +The task specifies the user-facing behavior of a token budget, including whole +synchronous microbatches, a final partial window and the objective for the same +effective batch. It provides a working fixed mode. Neither arm is told that a +bug exists. The optional arm gets generic TrainCheck documentation and code. +The solver may choose its own tests, use ordinary PyTorch or use TrainCheck. +No hidden grade or reference implementation is returned during the trial. + +Every decision that submits a run or finishes freezes the production source. +After the conversations end, the evaluator checks the submitted source against +the same independent numerical reference, including fixed mode, update +boundaries, gradients and resulting parameters on both ranks. Reuse a replay +only when the complete inputs affecting it are identical. The reference uses +the corresponding concatenated global batch. Native qualification must show +that a correct implementation passes and that a normally completing, numerically +incorrect implementation can be distinguished. + +Report three outcomes for each solver: + +1. **Delivered silent error.** The agent declares the implementation ready, + the affected training path completes normally, and an independent check + finds incorrect training behavior. Record which check failed. Separately + record crashes, an unsupported feature, explicit non-delivery and unknown + infrastructure outcomes. A solver that warns about an unresolved discrepancy + has not silently declared a verified implementation ready. +2. **Self-detected and repaired.** Bind the agent's own observation of a + concrete defect to its operation record or artifact, and the earlier wrong and + later correct implementation to independent checks. Label static code + review and executed diagnostics separately. A plausible concern or change + of implementation without evidence of a defect is insufficient. Record + within-turn fixes separately when a recoverable earlier revision and + provenance exist. Otherwise mark earlier correctness unobserved. +3. **First correct submission.** Identify the earliest frozen submission that + passes the complete hidden contract. Keep earlier unknown grades visible. + Count elapsed time from trial start, active CLI time, input/cached input/output + tokens and actual allocated GPU-seconds incurred **before that submission**. + A run requested by that submission comes after the boundary. A solver can + submit a valid implementation with zero prior GPU use. Retain the corresponding + time and cost for unsuccessful trials through their last decision. + +## Design and reporting + +- Three independent pairs. Fresh conversations and separate packets. The + CLI worker has six decisions, four public runs, 900 cumulative CLI seconds + and 120 seconds per public worker. Record exact launch order and environment. +- Record baseline revision, task and packet hashes, reference and controls, + model, CLI settings, physical GPUs, raw operation records, snapshots, ordinary + output and per-phase receipts. Keep case-specific operator material outside + all solver packets. Record any deviations from this protocol explicitly. +- Record wall time and queue wait independently of active CLI time. Count cached + input once in total tokens and show uncached input plus output separately. + Allocated GPU-seconds equal device count times allocation duration. Training + qualification and hidden evaluation are separate from the agents' GPU costs. +- Report the six agents individually, paired differences, actual tool use and + any diagnostic that changed an agent's next action. Identify correctness + failures even when the final training loss is finite or replicas agree. + Three pairs on one task are an early validation of this workflow. + +The implementation baseline for the optional tool is the signed snapshot work +in [PR #172](https://github.com/OrderLab/TrainCheck/pull/172). Prior numerical +qualification of global token-mean objectives is documented in the +[DDP study](ddp-runtime-state-pilot-20261102.md). This task requires a new +implementation and new reference runs for its token-budget boundaries. From 90de2428e4b296465d8f763ad2cbe1d48b56219e Mon Sep 17 00:00:00 2001 From: Yuxuan Jiang Date: Wed, 18 Nov 2026 03:10:01 +0800 Subject: [PATCH 2/6] Preserve native agent submissions, edit evidence and first-correct costs --- .../token_budget_study/operator/agent_test.py | 76 ++++ .../operator/agent_trials.py | 413 ++++++++++++++++++ .../operator/agent_watch.py | 105 +++++ .../operator/score_submissions.py | 208 +++++++++ .../operator/test_score_submissions.py | 67 +++ 5 files changed, 869 insertions(+) create mode 100644 tools/token_budget_study/operator/agent_test.py create mode 100644 tools/token_budget_study/operator/agent_trials.py create mode 100644 tools/token_budget_study/operator/agent_watch.py create mode 100644 tools/token_budget_study/operator/score_submissions.py create mode 100644 tools/token_budget_study/operator/test_score_submissions.py diff --git a/tools/token_budget_study/operator/agent_test.py b/tools/token_budget_study/operator/agent_test.py new file mode 100644 index 00000000..2b995bbb --- /dev/null +++ b/tools/token_budget_study/operator/agent_test.py @@ -0,0 +1,76 @@ +"""Regression tests for evidence collection and frozen solver inputs.""" + +from __future__ import annotations + +import hashlib +import json +import time +from pathlib import Path + +import pytest + +from tools.token_budget_study.operator.agent_trials import ( + decision, + filemap, + freeze_submission, + verify_packet, +) +from tools.token_budget_study.operator.agent_watch import watch_source + + +def test_frozen_training_and_immutable_inputs(tmp_path: Path) -> None: + packet = tmp_path / "packet" + packet.mkdir() + for name in ("train.py", "training.py", "workload.py", "TASK.md"): + (packet / name).write_text("original") + frozen = filemap(packet) + verify_packet(packet, frozen, initial=True) + (packet / "training.py").write_text("first") + with pytest.raises(ValueError, match="training.py"): + verify_packet(packet, frozen, initial=True) + verify_packet(packet, frozen, initial=False) + hashes = freeze_submission(packet, tmp_path / "submission") + (packet / "training.py").write_text("second") + assert hashes["training.py"] == hashlib.sha256(b"first").hexdigest() + assert (tmp_path / "submission" / "training.py").read_text() == "first" + (packet / "workload.py").write_text("override") + with pytest.raises(ValueError, match="workload.py"): + verify_packet(packet, frozen, initial=False) + + +def test_decision_does_not_allow_finish_plus_request(tmp_path: Path) -> None: + data = tmp_path / "choice.json" + data.write_text( + json.dumps( + {"action": "finish", "entry": "train.py", "args": [], "summary": "done"} + ) + ) + with pytest.raises(ValueError, match="Inconsistent"): + decision(data) + + +def test_hidden_earlier_submission_before_finish(tmp_path: Path) -> None: + source = tmp_path / "training.py" + source.write_text("base") + turn = tmp_path / "turn-1" + with watch_source(source, turn): + turn.mkdir() + (turn / "received.jsonl").write_text("received\n") + source.write_text("intermediate") + index = tmp_path / "turn-1.watch" / "index.jsonl" + intermediate = hashlib.sha256(b"intermediate").hexdigest() + deadline = time.monotonic() + 3 + while intermediate not in index.read_text() and time.monotonic() < deadline: + time.sleep(0.02) + source.write_text("submitted") + hashes = [ + ( + tmp_path + / "turn-1.watch" + / "blobs" + / f"{hashlib.sha256(v.encode()).hexdigest()}.py" + ).read_text() + for v in ("base", "intermediate", "submitted") + ] + assert hashes == ["base", "intermediate", "submitted"] + assert json.loads(index.read_text().splitlines()[-1])["final"] is True diff --git a/tools/token_budget_study/operator/agent_trials.py b/tools/token_budget_study/operator/agent_trials.py new file mode 100644 index 00000000..c745ea90 --- /dev/null +++ b/tools/token_budget_study/operator/agent_trials.py @@ -0,0 +1,413 @@ +"""Freeze independent packets and record native solver submissions and costs.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import math +import random +import secrets +import shutil +import subprocess +import sys +import time +from concurrent.futures import ThreadPoolExecutor +from dataclasses import asdict, dataclass +from pathlib import Path +from typing import Literal + +from tools.checkpoint_pilot.accounting import UsageCheckpoint, account_turn +from tools.checkpoint_pilot.cli_turn import CLISettings, run_turn +from tools.token_budget_study.operator.agent_watch import watch_source + +PUBLIC = Path(__file__).resolve().parents[1] / "public" +SERVICE = Path(__file__).resolve().with_name("ring_service.py") +PUBLIC_FILES = ("TASK.md", "train.py", "training.py", "workload.py") +BINARY = Path("/opt/homebrew/bin/codex") +MODEL = "gpt-6-astra" +MAX_TURNS = 6 +MAX_REQUESTS = 4 +CLI_BUDGET = 900 +SEED = 20261118 + +SCHEMA = { + "type": "object", + "properties": { + "action": {"type": "string", "enum": ["run", "finish"]}, + "summary": {"type": "string"}, + "entry": {"type": "string", "enum": ["train.py", "diagnostic.py", ""]}, + "args": {"type": "array", "items": {"type": "string"}}, + }, + "required": ["action", "summary", "entry", "args"], + "additionalProperties": False, +} +PROMPT = """Work on TASK.md using only the provided working directory and the training service described here. Edit the files locally. Use the service for training or executing PyTorch experiments. You can inspect code and use ordinary development tools here. Keep all project changes confined to this directory. Do not browse other directories, the internet, history, or other sessions. + +Your production deliverable is training.py. If you choose to run your own diagnostic on the same two GPUs, put its complete entry point in diagnostic.py. The service runs a two-process torchrun with either train.py or diagnostic.py and your argument list. Print the observations you want returned in stdout or stderr. Both entry points run in the same environment. You have up to four requests of 120 seconds each and six decision turns. You decide which checks to run and when to deliver. + +After working on the code, return your decision. For run, summarize your progress and reason for the request, then give the entry and argument list. For finish, give your delivery summary and evidence with empty entry and argument list. Your current training.py is submitted with every decision. You may finish at any point.""" +OFFER = "\n\nTrainCheck is also available if useful. TRAINCHECK.md documents the optional observation API and examples." + + +def digest(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def write(path: Path, data: object) -> None: + with path.open("x") as stream: + json.dump(data, stream, indent=2, allow_nan=False) + stream.write("\n") + + +def record(path: Path) -> dict[str, object]: + value: object = json.loads(path.read_text()) + if not isinstance(value, dict) or not all(isinstance(k, str) for k in value): + raise ValueError(f"Expected JSON object in {path}") + return dict(value) + + +def number(data: dict[str, object], key: str) -> float: + value = data[key] + if ( + isinstance(value, bool) + or not isinstance(value, (int, float)) + or not math.isfinite(value) + or value < 0 + ): + raise ValueError(f"Invalid nonnegative duration {key}: {value}") + return float(value) + + +def filemap(base: Path) -> dict[str, str]: + result: dict[str, str] = {} + for p in sorted(base.rglob("*")): + if "__pycache__" in p.relative_to(base).parts: + continue + if p.is_symlink(): + raise ValueError(f"Unexpected symbolic link {p}") + if p.is_file(): + result[p.relative_to(base).as_posix()] = digest(p) + return result + + +@dataclass(frozen=True) +class Arm: + name: str + treatment: bool + packet: Path + + +def prepare(evidence: Path, packets: Path, optional: Path, manifest: Path) -> None: + packet_root = packets.resolve() + packet_root.mkdir(parents=True, exist_ok=False) + evidence.mkdir(parents=True, exist_ok=True) + # Require exact optional package provenance, including extra/missing sources. + expected = record(manifest)["files"] + if filemap(optional) != expected: + raise ValueError("Optional package differs from the prior qualified manifest") + rng = random.Random(SEED) + lead = [True, False, bool(rng.getrandbits(1))] + rng.shuffle(lead) + arms: list[dict[str, object]] = [] + for pair, first in enumerate(lead, 1): + for treatment in (first, not first): + name = f"{'traincheck' if treatment else 'control'}_{pair}" + packet = packet_root / secrets.token_hex(8) + packet.mkdir() + for filename in PUBLIC_FILES: + shutil.copyfile(PUBLIC / filename, packet / filename) + if treatment: + shutil.copytree( + optional, + packet, + dirs_exist_ok=True, + ignore=shutil.ignore_patterns("__pycache__"), + ) + write(packet / "decision-schema.json", SCHEMA) + arms.append( + { + "name": name, + "treatment": treatment, + "packet": str(packet), + "initial_files": filemap(packet), + } + ) + write( + evidence / "native-launch.json", + { + "arms": arms, + "prompt": PROMPT, + "optional_sentence": OFFER, + "binary": str(BINARY), + "cli_sha256": digest(BINARY), + "model": MODEL, + "reasoning": "high", + "tier": "priority", + "seed": SEED, + "max_requests": MAX_REQUESTS, + "max_turns": MAX_TURNS, + "cli_seconds": CLI_BUDGET, + "worker_seconds": 120, + "public_hashes": {f: digest(PUBLIC / f) for f in PUBLIC_FILES}, + "optional_manifest_sha256": digest(manifest), + "entry_sha256": digest(Path(__file__)), + "service_sha256": digest(SERVICE), + "watcher_sha256": digest(Path(__file__).with_name("agent_watch.py")), + }, + ) + + +@dataclass(frozen=True) +class Decision: + action: Literal["run", "finish"] + summary: str + entry: str + args: list[str] + + +def decision(path: Path) -> Decision: + raw = record(path) + action, summary, entry, args = ( + raw[key] for key in ("action", "summary", "entry", "args") + ) + if ( + action not in ("run", "finish") + or not isinstance(summary, str) + or not summary.strip() + or not isinstance(entry, str) + or not isinstance(args, list) + or not all(isinstance(arg, str) for arg in args) + ): + raise ValueError(f"Invalid decision fields: {path}") + if action == "finish" and entry == "" and not args: + return Decision("finish", summary, entry, args) + if action == "run" and entry in ("train.py", "diagnostic.py"): + return Decision("run", summary, entry, args) + raise ValueError(f"Inconsistent decision: {path}") + + +def verify_packet(packet: Path, expected: object, *, initial: bool) -> None: + if not isinstance(expected, dict): + raise ValueError("Missing packet manifest") + for filename, sha in expected.items(): + if not isinstance(filename, str) or not isinstance(sha, str): + raise ValueError("Malformed packet manifest") + path = packet / filename + if ( + path.is_symlink() + or not path.is_file() + or (filename != "training.py" or initial) + and digest(path) != sha + ): + raise ValueError(f"Changed required packet file {filename}") + if initial and expected != filemap(packet): + raise ValueError("Workspace changed before first decision") + + +def freeze_submission(packet: Path, submitted: Path) -> dict[str, str]: + submitted.mkdir() + for name in (*PUBLIC_FILES, "diagnostic.py"): + path = packet / name + if name == "diagnostic.py" and not path.exists() and not path.is_symlink(): + continue + if path.is_symlink() or not path.is_file(): + raise ValueError(f"Invalid submitted file {name}") + shutil.copy2(path, submitted / name) + return filemap(submitted) + + +def run_service( + arm: Arm, idx: int, choice: Decision, source: Path, output: Path +) -> dict[str, object]: + command = [ + sys.executable, + str(SERVICE), + "public", + "--source", + str(source), + "--id", + f'{arm.name.replace("_","-")}-req{idx}', + "--output", + str(output), + "--entry", + choice.entry, + "--args-json", + json.dumps(choice.args), + ] + if arm.treatment: + command.append("--with-traincheck") + completed = subprocess.run(command, check=False) + if not (output / "receipt.json").is_file(): + raise RuntimeError( + f"Service exit {completed.returncode} without receipt at {output}" + ) + return record(output / "receipt.json") + + +def trial(config: dict[str, object], evidence: Path) -> None: + name, treatment, directory = config["name"], config["treatment"], config["packet"] + if ( + not isinstance(name, str) + or not isinstance(treatment, bool) + or not isinstance(directory, str) + ): + raise ValueError("Malformed trial launch row") + arm = Arm(name, treatment, Path(directory)) + verify_packet(arm.packet, config["initial_files"], initial=True) + root = evidence / "trials" / arm.name + root.mkdir(parents=True, exist_ok=False) + started = time.monotonic() + write( + root / "start.json", + { + "started_monotonic": started, + "started_unix": time.time(), + "workspace": str(arm.packet), + }, + ) + settings = CLISettings( + BINARY, MODEL, "high", "priority", arm.packet / "decision-schema.json" + ) + checkpoint: UsageCheckpoint | None = None + cli = 0.0 + gpu = 0.0 + queue = 0.0 + requests = 0 + prompt = PROMPT + (OFFER if treatment else "") + terminal = "turn_budget_exhausted" + index = 0 + for index in range(1, MAX_TURNS + 1): + remaining = CLI_BUDGET - cli + if remaining <= 0: + terminal = "cli_budget_exhausted" + break + with watch_source(arm.packet / "training.py", root / f"turn-{index}"): + receipt, complete = run_turn( + settings, + arm.packet, + root / f"turn-{index}", + prompt, + remaining, + None if checkpoint is None else checkpoint.thread_id, + ) + accounted = account_turn( + complete, + mode="cumulative", + previous=checkpoint, + initial_turn=checkpoint is None, + ) + checkpoint = accounted.checkpoint + cli += receipt.ended_monotonic - receipt.started_monotonic + verify_packet(arm.packet, config["initial_files"], initial=False) + current = root / f"turn-{index}" + source = current / "submission" + hashes = freeze_submission(arm.packet, source) + moment = time.monotonic() + choice = decision(current / "response.json") + write( + current / "submission.json", + { + "trial": arm.name, + "turn": index, + "decision": asdict(choice), + "submitted_monotonic": moment, + "wall_seconds": moment - started, + "cli_seconds": cli, + "allocated_gpu_seconds": gpu, + "queue_seconds": queue, + "usage_delta": asdict(accounted.delta), + "usage_cumulative": asdict(checkpoint.cumulative_usage), + "source_files": hashes, + "production_sha256": hashes["training.py"], + "thread_id": checkpoint.thread_id, + }, + ) + if choice.action == "finish": + terminal = "finished" + break + if requests == MAX_REQUESTS or index == MAX_TURNS: + terminal = ( + "request_budget_exhausted" + if requests == MAX_REQUESTS + else "turn_budget_exhausted" + ) + break + requests += 1 + reply = run_service(arm, requests, choice, source, root / f"request-{requests}") + status = reply["status"] + if status not in ("ok", "application_error", "timeout"): + raise RuntimeError( + f"Incomplete GPU infrastructure result {status!r}; preserve this request for same-thread recovery" + ) + gpu += number(reply, "gpu_seconds") + queue += number(reply, "queue_seconds") + out, err = reply["stdout"], reply["stderr"] + if not isinstance(out, str) or not isinstance(err, str): + raise ValueError("Service response must include both output streams") + # Exact full service output is retained in the operator receipt. Bound the + # returned transcript equally for arbitrary verbose diagnostics in each arm. + prompt = f"GPU request {requests} finished ({status}). You have {MAX_REQUESTS-requests} requests remaining. Continue TASK.md and return your next decision. Output below is the last 24,000 characters per stream.\nstdout:\n{out[-24000:]}\nstderr:\n{err[-24000:]}" + write( + root / "final.json", + { + "status": terminal, + "last_turn": index, + "requests": requests, + "cli_seconds": cli, + "gpu_seconds": gpu, + "queue_seconds": queue, + "elapsed_seconds": time.monotonic() - started, + }, + ) + + +def run(evidence: Path) -> None: + frozen = record(evidence / "native-launch.json") + if ( + frozen["cli_sha256"] != digest(BINARY) + or frozen["entry_sha256"] != digest(Path(__file__)) + or frozen["service_sha256"] != digest(SERVICE) + or frozen["watcher_sha256"] + != digest(Path(__file__).with_name("agent_watch.py")) + ): + raise ValueError("Frozen launcher or executable changed") + arms = frozen["arms"] + if ( + not isinstance(arms, list) + or len(arms) != 6 + or not all(isinstance(a, dict) for a in arms) + ): + raise ValueError("Expected six frozen trial packets") + # Both arms of one pair can draft next decisions while the Ring18 queue serializes GPU workers. + for pair in range(0, 6, 2): + with ThreadPoolExecutor(max_workers=2) as pool: + jobs = [pool.submit(trial, arm, evidence) for arm in arms[pair : pair + 2]] + for job in jobs: + job.result() + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("action", choices=["prepare", "run"]) + parser.add_argument("--evidence", type=Path, required=True) + parser.add_argument("--packets", type=Path) + parser.add_argument("--optional", type=Path) + parser.add_argument("--optional-manifest", type=Path) + args = parser.parse_args() + if args.action == "prepare": + if ( + args.packets is None + or args.optional is None + or args.optional_manifest is None + ): + parser.error( + "prepare requires --packets, --optional and --optional-manifest" + ) + prepare(args.evidence, args.packets, args.optional, args.optional_manifest) + else: + run(args.evidence) + + +if __name__ == "__main__": + main() diff --git a/tools/token_budget_study/operator/agent_watch.py b/tools/token_budget_study/operator/agent_watch.py new file mode 100644 index 00000000..f756ba6d --- /dev/null +++ b/tools/token_budget_study/operator/agent_watch.py @@ -0,0 +1,105 @@ +"""Observe candidate edits without instrumenting solver code or returning grades.""" + +from __future__ import annotations + +import hashlib +import json +import os +import stat +import threading +import time +from concurrent.futures import ThreadPoolExecutor +from contextlib import contextmanager +from pathlib import Path +from typing import Iterator + + +def _regular_bytes(path: Path) -> bytes | None: + # Replacements can make a candidate temporarily absent. Never follow a + # link created by an editor; compare the identity of the opened file. + if not path.is_file() or path.is_symlink(): + return None + try: + fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK) + except FileNotFoundError: + return None + with os.fdopen(fd, "rb") as stream: + before = os.fstat(stream.fileno()) + if not stat.S_ISREG(before.st_mode): + return None + data = stream.read() + after = os.fstat(stream.fileno()) + if before.st_size != after.st_size or before.st_mtime_ns != after.st_mtime_ns: + return None + return data + + +def _watch( + source: Path, + native: Path, + directory: Path, + stop: threading.Event, + ready: threading.Event, +) -> None: + blobs = directory / "blobs" + blobs.mkdir(parents=True, exist_ok=False) + previous: tuple[str | None, int] | None = None + with (directory / "index.jsonl").open("x") as index: + while True: + ending = stop.is_set() + raw = _regular_bytes(source) + stamp = time.monotonic() + try: + received_bytes = native.stat().st_size + except FileNotFoundError: + # The native collector creates its transcript after we start. + received_bytes = 0 + sha = None if raw is None else hashlib.sha256(raw).hexdigest() + current = (sha, received_bytes) + if current != previous or ending: + if sha is not None: + target = blobs / f"{sha}.py" + if not target.exists(): + assert raw is not None + target.write_bytes(raw) + index.write( + json.dumps( + { + "observed_monotonic": stamp, + "sha256": sha, + "native_received_bytes": received_bytes, + "final": ending, + } + ) + + "\n" + ) + index.flush() + previous = current + ready.set() + if ending: + return + stop.wait(0.05) + + +@contextmanager +def watch_source(source: Path, turn_dir: Path) -> Iterator[None]: + """Preserve observed source versions and approximate native event boundaries. + + The sampling interval is 50ms. A version that exists only between samples + may be missed; private grading at actual submitted boundaries is separate. + """ + stop = threading.Event() + ready = threading.Event() + directory = turn_dir.with_name(turn_dir.name + ".watch") + with ThreadPoolExecutor(max_workers=1) as pool: + observer = pool.submit( + _watch, source, turn_dir / "received.jsonl", directory, stop, ready + ) + try: + while not ready.wait(0.05): + if observer.done(): + observer.result() + yield + finally: + stop.set() + observer.result() diff --git a/tools/token_budget_study/operator/score_submissions.py b/tools/token_budget_study/operator/score_submissions.py new file mode 100644 index 00000000..d340fb29 --- /dev/null +++ b/tools/token_budget_study/operator/score_submissions.py @@ -0,0 +1,208 @@ +"""Join frozen decision boundaries to private per-source replay receipts.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import math +from pathlib import Path + +from tools.checkpoint_pilot.accounting import parse_exec_jsonl + +SCENARIOS = { + "fixed-regression", + "exact-uniform", + "overshoot-and-tail", + "rank-skew", + "single-window", + "one-microbatch-updates", +} +OUTCOMES = { + "pass", + "silent_wrong", + "unsupported", + "runtime_error", + "nonfinite", + "memory_contract", +} + + +def record(value: object) -> dict[str, object]: + if not isinstance(value, dict) or not all(isinstance(k, str) for k in value): + raise ValueError("Expected a JSON object with string keys") + return dict(value) + + +def read(path: Path) -> dict[str, object]: + return record(json.loads(path.read_bytes())) + + +def duration(value: object) -> float: + if ( + isinstance(value, bool) + or not isinstance(value, (float, int)) + or not math.isfinite(value) + or value < 0 + ): + raise ValueError(f"Invalid duration {value!r}") + return float(value) + + +def grade_status(receipt: Path, sha: str, treatment: bool) -> str: + outer = read(receipt) + request = read(receipt.with_name("request.json")) + files = record(request["files"]) + source = files["training.py"] + if ( + request["kind"] != "grade" + or request["with_traincheck"] is not treatment + or not isinstance(source, str) + ): + raise ValueError(f"Grade request has different execution conditions: {receipt}") + if ( + hashlib.sha256(source.encode()).hexdigest() != sha + or outer["id"] != request["id"] + ): + raise ValueError(f"Grade request and source disagree: {receipt}") + if outer["status"] != "ok": + return "unknown" + grade = record(outer["grade"]) + if grade["source_sha256"] != sha: + raise ValueError(f"Grader replayed a different source: {receipt}") + overall = grade["status"] + if overall in ("timeout", "incomplete_execution"): + return "unknown" + if not isinstance(overall, str) or overall not in OUTCOMES: + raise ValueError(f"Invalid grade status {overall!r}") + rows = grade["rank_results"] + if not isinstance(rows, list): + raise ValueError("Missing scenario outcomes") + covered: set[tuple[str, int]] = set() + observed: set[str] = set() + for raw in rows: + row = record(raw) + name, rank = row["name"], row["rank"] + outcome = record(row["result"])["status"] + if ( + not isinstance(name, str) + or (not isinstance(rank, int) or isinstance(rank, bool)) + or not isinstance(outcome, str) + or outcome not in OUTCOMES + ): + raise ValueError("Invalid scenario row") + if (name, rank) in covered: + raise ValueError("Duplicate scenario/rank") + covered.add((name, rank)) + observed.add(outcome) + if covered != {(name, rank) for name in SCENARIOS for rank in (0, 1)}: + raise ValueError("Incomplete scenario/rank coverage") + if (overall == "pass") != (observed == {"pass"}) or overall not in observed: + raise ValueError("Inconsistent aggregate grade") + return overall + + +def score(trial: Path, grades: Path) -> dict[str, object]: + """Grade locations are grouped by optional-kit availability and source digest.""" + arm = trial.name.split("_")[0] + if arm not in {"control", "traincheck"}: + raise ValueError(f"Invalid trial name {trial.name}") + prior_gpu = prior_queue = 0.0 + req = 0 + first: int | None = None + earlier_unknown: list[int] = [] + rows: list[dict[str, object]] = [] + for n in range(1, 7): + turn = trial / f"turn-{n}" + if not (turn / "submission.json").is_file(): + break + sub = read(turn / "submission.json") + sha = hashlib.sha256((turn / "submission/training.py").read_bytes()).hexdigest() + if ( + sub["production_sha256"] != sha + or sub["trial"] != trial.name + or sub["turn"] != n + ): + raise ValueError( + f"Submission metadata disagrees with frozen source in {turn}" + ) + ledger = parse_exec_jsonl((turn / "events.jsonl").read_text()) + usage = record(sub["usage_cumulative"]) + if ledger.thread_id != sub["thread_id"] or any( + usage[k] != getattr(ledger.usage, k) + for k in ("input_tokens", "cached_input_tokens", "output_tokens") + ): + raise ValueError("Mismatched native token counters or thread") + if not math.isclose( + prior_gpu, duration(sub["allocated_gpu_seconds"]) + ) or not math.isclose(prior_queue, duration(sub["queue_seconds"])): + raise ValueError( + f"Submission cost disagrees with earlier request receipts in {turn}" + ) + grade = grade_status( + grades / arm / sha / "receipt.json", sha, arm == "traincheck" + ) + if first is None: + if grade == "pass": + first = n + elif grade == "unknown": + earlier_unknown.append(n) + u = ledger.usage + decision = record(sub["decision"]) + rows.append( + dict( + turn=n, + production_sha256=sha, + action=decision["action"], + grade=grade, + wall_seconds=duration(sub["wall_seconds"]), + cli_seconds=duration(sub["cli_seconds"]), + queue_seconds=prior_queue, + gpu_seconds=prior_gpu, + total_tokens=u.total_tokens, + input_tokens=u.input_tokens, + cached_input_tokens=u.cached_input_tokens, + output_tokens=u.output_tokens, + uncached_input_plus_output=u.total_tokens - u.cached_input_tokens, + ) + ) + if decision["action"] == "run": + receipt = trial / f"request-{req + 1}" / "receipt.json" + if receipt.exists(): + req += 1 + result = read(receipt) + prior_gpu += duration(result["gpu_seconds"]) + prior_queue += duration(result["queue_seconds"]) + if not rows: + raise ValueError(f"No frozen submission in {trial}") + last = read(trial / "final.json") + if req != last["requests"] or not math.isclose( + prior_gpu, duration(last["gpu_seconds"]) + ): + raise ValueError("Terminal service costs disagree") + return dict( + trial=trial.name, + terminal=last, + submissions=rows, + earliest_observed_pass=first, + earlier_unknown_turns=earlier_unknown, + ) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("evidence", type=Path) + parser.add_argument("out", type=Path) + args = parser.parse_args() + trials = [ + score(args.evidence / "trials" / f"{arm}_{i}", args.evidence / "grades") + for i in range(1, 4) + for arm in ("control", "traincheck") + ] + with args.out.open("x") as stream: + json.dump(trials, stream, indent=2, allow_nan=False) + stream.write("\n") + + +if __name__ == "__main__": + main() diff --git a/tools/token_budget_study/operator/test_score_submissions.py b/tools/token_budget_study/operator/test_score_submissions.py new file mode 100644 index 00000000..30c40d52 --- /dev/null +++ b/tools/token_budget_study/operator/test_score_submissions.py @@ -0,0 +1,67 @@ +"""Evidence joins reject cross-arm, source and coverage mix-ups.""" + +import hashlib +import json +from pathlib import Path + +import pytest + +from tools.token_budget_study.operator.score_submissions import SCENARIOS, grade_status + + +def receipt(root: Path, rows: list[dict[str, object]], status: str) -> tuple[Path, str]: + source = "def train(): pass\n" + sha = hashlib.sha256(source.encode()).hexdigest() + path = root / "receipt.json" + (root / "request.json").write_text( + json.dumps( + { + "id": "trial", + "kind": "grade", + "with_traincheck": False, + "files": {"training.py": source}, + } + ) + ) + path.write_text( + json.dumps( + { + "id": "trial", + "status": "ok", + "grade": {"status": status, "source_sha256": sha, "rank_results": rows}, + } + ) + ) + return path, sha + + +def complete_rows() -> list[dict[str, object]]: + return [ + {"rank": rank, "name": name, "result": {"status": "pass"}} + for rank in (0, 1) + for name in sorted(SCENARIOS) + ] + + +def test_numerical_failure_on_only_one_rank_is_retained(tmp_path: Path) -> None: + rows = complete_rows() + rows[-1]["result"] = {"status": "silent_wrong"} + path, sha = receipt(tmp_path, rows, "silent_wrong") + assert grade_status(path, sha, False) == "silent_wrong" + with pytest.raises(ValueError, match="conditions"): + grade_status(path, sha, True) + with pytest.raises(ValueError, match="source"): + grade_status(path, "0" * 64, False) + + +def test_duplicate_rank_cannot_complete_coverage(tmp_path: Path) -> None: + rows = complete_rows() + rows[-1] = rows[0] + path, sha = receipt(tmp_path, rows, "pass") + with pytest.raises(ValueError, match="Duplicate"): + grade_status(path, sha, False) + + +def test_interrupted_phase_retains_unknown(tmp_path: Path) -> None: + path, sha = receipt(tmp_path, [], "incomplete_execution") + assert grade_status(path, sha, False) == "unknown" From 509a06b00307b51c1ea4f0ca7f7e3498f11b391a Mon Sep 17 00:00:00 2001 From: Yuxuan Jiang Date: Wed, 18 Nov 2026 03:10:47 +0800 Subject: [PATCH 3/6] Add healthy DDP feature task and qualify a concatenated-batch gradient oracle --- tools/token_budget_study/private/grade.py | 187 ++++++++++++ .../private/positive_control.py | 50 +++ tools/token_budget_study/private/qualify.py | 37 +++ tools/token_budget_study/private/worker.py | 289 ++++++++++++++++++ tools/token_budget_study/public/TASK.md | 18 ++ tools/token_budget_study/public/train.py | 70 +++++ tools/token_budget_study/public/training.py | 36 +++ tools/token_budget_study/public/workload.py | 47 +++ tools/token_budget_study/tests/test_oracle.py | 78 +++++ 9 files changed, 812 insertions(+) create mode 100644 tools/token_budget_study/private/grade.py create mode 100644 tools/token_budget_study/private/positive_control.py create mode 100644 tools/token_budget_study/private/qualify.py create mode 100644 tools/token_budget_study/private/worker.py create mode 100644 tools/token_budget_study/public/TASK.md create mode 100644 tools/token_budget_study/public/train.py create mode 100644 tools/token_budget_study/public/training.py create mode 100644 tools/token_budget_study/public/workload.py create mode 100644 tools/token_budget_study/tests/test_oracle.py diff --git a/tools/token_budget_study/private/grade.py b/tools/token_budget_study/private/grade.py new file mode 100644 index 00000000..b270ccdd --- /dev/null +++ b/tools/token_budget_study/private/grade.py @@ -0,0 +1,187 @@ +"""Replay a frozen public packet on two ranks with the withheld numerical grader.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import shutil +import signal +import subprocess +import sys +import tempfile +import time +from pathlib import Path + +# Order mirrors worker.SCENARIOS. Validate both rank receipts against these names. +NAMES = ( + "fixed-regression", + "exact-uniform", + "overshoot-and-tail", + "rank-skew", + "single-window", + "one-microbatch-updates", +) +OUTCOMES = { + "pass", + "silent_wrong", + "memory_contract", + "nonfinite", + "runtime_error", + "unsupported", +} + + +def grade( + source: Path, + output: Path, + device: str, + timeout: float, + optional_package: Path | None, +) -> None: + output.parent.mkdir(parents=True, exist_ok=True) + # Reserve the path before allocating any worker time. + with output.open("x") as stream: + json.dump({"status": "running"}, stream) + with tempfile.TemporaryDirectory(prefix="budget-grader-") as temp: + scratch = Path(temp) + for name in ("training.py", "workload.py"): + shutil.copyfile(source / name, scratch / name) + shutil.copyfile(Path(__file__).with_name("worker.py"), scratch / "worker.py") + package_files: dict[str, str] = {} + if optional_package is not None: + if ( + optional_package.name != "traincheck" + or not (optional_package / "__init__.py").is_file() + ): + raise ValueError("Expected a frozen traincheck package directory") + for file in sorted(optional_package.rglob("*")): + if file.is_symlink(): + raise ValueError("Expected regular files in the frozen package") + if file.is_file() and "__pycache__" not in file.parts: + relative = file.relative_to(optional_package) + dest = scratch / "traincheck" / relative + dest.parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(file, dest) + package_files[str(relative)] = hashlib.sha256( + dest.read_bytes() + ).hexdigest() + start = time.monotonic() + command = [ + sys.executable, + "-m", + "torch.distributed.run", + "--standalone", + "--nproc-per-node=2", + "worker.py", + "--receipts", + str(scratch), + "--device", + device, + ] + with (scratch / "worker.log").open("wb") as log: + proc = subprocess.Popen( + command, + cwd=scratch, + stdout=log, + stderr=subprocess.STDOUT, + start_new_session=True, + ) + timed_out = False + try: + proc.wait(timeout=timeout) + except subprocess.TimeoutExpired: + timed_out = True + finally: + if proc.poll() is None: + try: + os.killpg(proc.pid, signal.SIGKILL) + except ProcessLookupError: + pass + proc.wait() + worker_seconds = time.monotonic() - start + statuses: set[str] = set() + results: list[dict[str, object]] = [] + for index, name in enumerate(NAMES): + for rank in (0, 1): + receipt = scratch / f"{index}-rank{rank}.json" + if receipt.is_file(): + row: object = json.loads(receipt.read_text()) + if ( + not isinstance(row, dict) + or row["name"] != name + or row["status"] not in OUTCOMES + ): + raise ValueError(f"Invalid worker receipt {receipt.name}") + status = row["status"] + results.append({"name": name, "rank": rank, "result": row}) + else: + status = "unobserved" + results.append( + { + "name": name, + "rank": rank, + "result": { + "status": status, + "started": receipt.with_suffix(".start").exists(), + }, + } + ) + statuses.add(status) + if "unobserved" in statuses or proc.returncode != 0: + overall = "timeout" if timed_out else "incomplete_execution" + else: + overall = next( + ( + x + for x in ( + "runtime_error", + "unsupported", + "nonfinite", + "silent_wrong", + "memory_contract", + ) + if x in statuses + ), + "pass", + ) + result = { + "status": overall, + "rank_results": results, + "worker_seconds": worker_seconds, + "devices": 2 if device == "cuda" else 0, + "optional_package_files": package_files, + "source_sha256": hashlib.sha256( + (scratch / "training.py").read_bytes() + ).hexdigest(), + "exit_code": proc.returncode, + "log_tail": (scratch / "worker.log") + .read_bytes()[-12000:] + .decode(errors="replace"), + } + staged = output.with_suffix(output.suffix + ".partial") + with staged.open("x") as stream: + json.dump(result, stream, indent=2, allow_nan=False) + staged.replace(output) + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--source", required=True, type=Path) + parser.add_argument("--output", required=True, type=Path) + parser.add_argument("--device", choices=["cpu", "cuda"], default="cuda") + parser.add_argument("--timeout", type=float, default=90) + parser.add_argument("--optional-package", type=Path) + args = parser.parse_args() + grade( + args.source.resolve(), + args.output.resolve(), + args.device, + args.timeout, + args.optional_package, + ) + + +if __name__ == "__main__": + main() diff --git a/tools/token_budget_study/private/positive_control.py b/tools/token_budget_study/private/positive_control.py new file mode 100644 index 00000000..2d6f8635 --- /dev/null +++ b/tools/token_budget_study/private/positive_control.py @@ -0,0 +1,50 @@ +"""Qualified implementation used only by the study operator.""" + +from __future__ import annotations + +from collections.abc import Iterable + +import torch +import torch.distributed as dist +from torch import Tensor, nn +from torch.nn import functional as F +from workload import IGNORE_INDEX, Batch + + +def train_epoch( + model: nn.Module, + optimizer: torch.optim.Optimizer, + microbatches: Iterable[Batch], + token_budget: int | None, +) -> None: + if token_budget is not None and token_budget <= 0: + raise ValueError("Expected a positive token budget") + model.train() + total = 0 + optimizer.zero_grad(set_to_none=True) + + def finish(tokens: int) -> None: + for parameter in model.parameters(): + if parameter.grad is None: + raise ValueError("A model parameter has no gradient") + parameter.grad.mul_(dist.get_world_size() / tokens) + optimizer.step() + optimizer.zero_grad(set_to_none=True) + + for inputs, targets in microbatches: + logits: Tensor = model(inputs) + loss = F.cross_entropy( + logits.flatten(0, 1), + targets.flatten(), + ignore_index=IGNORE_INDEX, + reduction="sum", + ) + torch.autograd.backward(loss) + count = targets.ne(IGNORE_INDEX).sum() + dist.all_reduce(count) + total += int(count.item()) + if token_budget is None or total >= token_budget: + finish(total) + total = 0 + if total: + finish(total) diff --git a/tools/token_budget_study/private/qualify.py b/tools/token_budget_study/private/qualify.py new file mode 100644 index 00000000..48139308 --- /dev/null +++ b/tools/token_budget_study/private/qualify.py @@ -0,0 +1,37 @@ +"""Prepare immutable positive and targeted negative control sources for qualification.""" + +from __future__ import annotations + +import argparse +import shutil +from pathlib import Path + + +def prepare(public: Path, output: Path) -> None: + positive = Path(__file__).with_name("positive_control.py").read_text() + site = "parameter.grad.mul_(dist.get_world_size() / tokens)" + # Emulate using the requested budget as denominator even on overshoot/tail. + if positive.count(site) != 1: + raise ValueError("Negative control mutation site must be unique") + negative = positive.replace( + site, "parameter.grad.mul_(dist.get_world_size() / (token_budget or tokens))" + ) + output.mkdir(parents=True, exist_ok=False) + for name, code in (("positive", positive), ("wrong-denominator", negative)): + path = output / name + path.mkdir() + for file in ("workload.py", "train.py"): + shutil.copyfile(public / file, path / file) + (path / "training.py").write_text(code) + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--public", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + args = parser.parse_args() + prepare(args.public, args.output) + + +if __name__ == "__main__": + main() diff --git a/tools/token_budget_study/private/worker.py b/tools/token_budget_study/private/worker.py new file mode 100644 index 00000000..4db25175 --- /dev/null +++ b/tools/token_budget_study/private/worker.py @@ -0,0 +1,289 @@ +"""Withheld two-rank replay. Run only through grade.py in an isolated process.""" + +from __future__ import annotations + +import argparse +import copy +import importlib.abc +import importlib.machinery +import json +import os +import sys +from collections.abc import Sequence +from dataclasses import dataclass +from itertools import chain +from pathlib import Path +from types import ModuleType + +import torch +import torch.distributed as dist +from torch import Tensor, nn +from torch.nn import functional as F +from torch.nn.parallel import DistributedDataParallel +from workload import IGNORE_INDEX, Batch, TokenModel, batches, make_optimizer + + +class ControlImports(importlib.abc.MetaPathFinder): + """Control replays expose the same optional-package availability as the agent's run.""" + + def find_spec( + self, + fullname: str, + path: Sequence[str] | None, + target: ModuleType | None = None, + ) -> importlib.machinery.ModuleSpec | None: + if fullname == "traincheck" or fullname.startswith("traincheck."): + raise ModuleNotFoundError(f"No module named {fullname!r}", name=fullname) + return None + + +@dataclass(frozen=True) +class Scenario: + name: str + counts: tuple[tuple[int, ...], tuple[int, ...]] + budget: int | None + seed: int + + +SCENARIOS = ( + Scenario("fixed-regression", ((18, 3, 17), (2, 21, 5)), None, 19), + Scenario("exact-uniform", ((9, 9, 9, 9), (9, 9, 9, 9)), 36, 349), + Scenario("overshoot-and-tail", ((4, 17, 3, 8, 5, 2), (8, 4, 19, 6, 5, 3)), 29, 723), + Scenario("rank-skew", ((0, 18, 2, 21, 7), (17, 0, 13, 2, 1)), 26, 182), + Scenario("single-window", ((1, 22, 5), (19, 2, 2)), 99, 291), + Scenario("one-microbatch-updates", ((18, 4, 17), (6, 18, 3)), 7, 89), +) +ATOL = 1e-6 +RTOL = 1e-5 + +TensorMap = dict[str, Tensor] + + +@dataclass +class Step: + gradients: TensorMap + parameters: TensorMap + momentum: TensorMap + + +def values(model: nn.Module) -> TensorMap: + return {name: value.detach().clone() for name, value in model.named_parameters()} + + +def gradients(model: nn.Module) -> TensorMap: + return { + name: value.grad.detach().clone() + for name, value in model.named_parameters() + if value.grad is not None + } + + +def momentum(model: nn.Module, optimizer: torch.optim.Optimizer) -> TensorMap: + result: TensorMap = {} + for name, parameter in model.named_parameters(): + if parameter not in optimizer.state: + continue + state = optimizer.state[parameter] + if not isinstance(state, dict) or "momentum_buffer" not in state: + continue + buffer = state["momentum_buffer"] + if not isinstance(buffer, Tensor): + raise TypeError("SGD momentum buffer must be a tensor") + result[name] = buffer.detach().clone() + return result + + +def reference( + model: nn.Module, rank_batches: list[list[Batch]], budget: int | None +) -> list[Step]: + """Train on each concatenated global update; never use candidate gradients or collectives.""" + opt = make_optimizer(model) + result: list[Step] = [] + pending: list[Batch] = [] + count = 0 + for index, pair in enumerate(zip(*rank_batches, strict=True)): + pending.extend(pair) + count += sum(int(targets.ne(IGNORE_INDEX).sum().item()) for _, targets in pair) + if budget is not None and count < budget and index != len(rank_batches[0]) - 1: + continue + inputs = torch.cat([inputs for inputs, _ in pending]) + labels = torch.cat([labels for _, labels in pending]) + opt.zero_grad(set_to_none=True) + logits: Tensor = model(inputs) + loss = F.cross_entropy( + logits.flatten(0, 1), labels.flatten(), ignore_index=IGNORE_INDEX + ) + torch.autograd.backward(loss) + grad = gradients(model) + opt.step() + result.append(Step(grad, values(model), momentum(model, opt))) + pending.clear() + count = 0 + return result + + +def compare_map(label: str, expected: TensorMap, actual: TensorMap) -> list[str]: + issues = [] + if expected.keys() != actual.keys(): + issues.append( + f"{label} keys: expected {sorted(expected)}, observed {sorted(actual)}" + ) + for key in sorted(expected.keys() & actual.keys()): + left, right = expected[key], actual[key] + if left.shape != right.shape or left.dtype != right.dtype: + issues.append(f"{label}/{key} shape or dtype mismatch") + elif not torch.allclose(left, right, rtol=RTOL, atol=ATOL): + error = (right - left).abs().max().item() + issues.append(f"{label}/{key} max absolute error {error:.9g}") + return issues + + +def evaluate(case: Scenario, rank: int, device: torch.device) -> dict[str, object]: + torch.manual_seed(case.seed) + base = TokenModel().to(device) + plain = copy.deepcopy(base) + rank_batches = [ + batches(c, r, case.seed + 200, device) for r, c in enumerate(case.counts) + ] + expected = reference(plain, rank_batches, case.budget) + ddp = DistributedDataParallel( + base, device_ids=[device.index] if device.type == "cuda" else None + ) + opt = make_optimizer(ddp) + actual: list[Step] = [] + pre: list[TensorMap] = [] + observed_ids: list[int | None] = [] + boundaries: list[list[int | None]] = [] + + def before_step( + optimizer: torch.optim.Optimizer, + args: tuple[object, ...], + kwargs: dict[str, object], + ) -> None: + pre.append(gradients(base)) + boundaries.append(list(observed_ids)) + observed_ids.clear() + + def after_step( + optimizer: torch.optim.Optimizer, + args: tuple[object, ...], + kwargs: dict[str, object], + ) -> None: + actual.append(Step(pre.pop(), values(base), momentum(base, optimizer))) + + opt.register_step_pre_hook(before_step) + opt.register_step_post_hook(after_step) + outstanding = 0 + peak_outstanding = 0 + max_input_elements = 0 + + def observe_forward( + module: nn.Module, inputs: tuple[Tensor, ...], output: Tensor + ) -> None: + nonlocal outstanding, peak_outstanding, max_input_elements + if not output.requires_grad: + return + matching = [ + i + for i, (tokens, _) in enumerate(rank_batches[rank]) + if torch.equal(inputs[0], tokens) + ] + observed_ids.append(matching[0] if len(matching) == 1 else None) + outstanding += 1 + peak_outstanding = max(peak_outstanding, outstanding) + max_input_elements = max(max_input_elements, inputs[0].numel()) + + def before_backward(grad: Tensor) -> None: + nonlocal outstanding + outstanding -= 1 + + # Tensor.register_hook in PyTorch 2.2 lacks type annotations. + output.register_hook(before_backward) # type: ignore[no-untyped-call] + + base.register_forward_hook(observe_forward) + try: + from training import train_epoch + + train_epoch(ddp, opt, iter(rank_batches[rank]), case.budget) + except NotImplementedError as exc: + return {"name": case.name, "status": "unsupported", "error": repr(exc)} + except Exception as exc: + # The candidate is external code; record the actual failure. + return {"name": case.name, "status": "runtime_error", "error": repr(exc)} + numerical: list[str] = [] + if len(expected) != len(actual): + numerical.append(f"Expected {len(expected)} updates, observed {len(actual)}") + for i, (exp, obs) in enumerate(zip(expected, actual)): + numerical.extend( + compare_map(f"step {i}/gradients", exp.gradients, obs.gradients) + ) + numerical.extend( + compare_map(f"step {i}/parameters", exp.parameters, obs.parameters) + ) + numerical.extend(compare_map(f"step {i}/momentum", exp.momentum, obs.momentum)) + numerical.extend( + compare_map("final parameters", expected[-1].parameters, values(base)) + ) + numerical.extend( + compare_map("final momentum", expected[-1].momentum, momentum(base, opt)) + ) + observed = chain( + values(base).values(), + *( + m.values() + for step in actual + for m in (step.gradients, step.parameters, step.momentum) + ), + ) + finite = all(bool(torch.isfinite(t).all().item()) for t in observed) + status = "pass" + if not finite: + status = "nonfinite" + elif numerical: + status = "silent_wrong" + elif peak_outstanding > 1 or max_input_elements > rank_batches[rank][0][0].numel(): + status = "memory_contract" + return { + "name": case.name, + "status": status, + "forward_microbatch_ids_at_steps": boundaries, + "unconsumed_forward_ids": observed_ids, + "peak_outstanding_grad_forwards": peak_outstanding, + "max_forward_input_elements": max_input_elements, + "expected_updates": len(expected), + "observed_updates": len(actual), + "issues": numerical, + } + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--receipts", type=Path, required=True) + parser.add_argument("--device", choices=["cpu", "cuda"], required=True) + args = parser.parse_args() + rank = int(os.environ["RANK"]) + local = int(os.environ["LOCAL_RANK"]) + device = ( + torch.device("cuda", local) if args.device == "cuda" else torch.device("cpu") + ) + if not (Path(__file__).parent / "traincheck").is_dir(): + sys.meta_path.insert(0, ControlImports()) + if device.type == "cuda": + torch.cuda.set_device(device) + torch.backends.cuda.matmul.allow_tf32 = False + torch.backends.cudnn.allow_tf32 = False + dist.init_process_group(backend="nccl" if device.type == "cuda" else "gloo") + try: + for index, case in enumerate(SCENARIOS): + path = args.receipts / f"{index}-rank{rank}.json" + path.with_suffix(".start").write_text(case.name) + result = evaluate(case, rank, device) + with path.open("x") as stream: + json.dump(result, stream) + dist.barrier() + finally: + dist.destroy_process_group() + + +if __name__ == "__main__": + main() diff --git a/tools/token_budget_study/public/TASK.md b/tools/token_budget_study/public/TASK.md new file mode 100644 index 00000000..aa2c55a1 --- /dev/null +++ b/tools/token_budget_study/public/TASK.md @@ -0,0 +1,18 @@ +The team wants to run this DDP training pipeline on variable-length data with a +global target-token budget per optimizer update. Implement the feature in +`training.py`. The current fixed mode performs one optimizer update per +synchronous microbatch. Preserve that behavior when no budget is supplied. + +With a positive budget, accumulate whole synchronous microbatches in input +order. Apply an update as soon as their combined number of non-padding targets +across the training ranks reaches or exceeds the budget. Start the next window +empty and apply the final nonempty partial window at epoch end. For each update, +preserve the objective and gradient of ordinary training on its combined global +batch. Keep the training memory bounded by a single microbatch per rank. The +ranks receive the same number of microbatches and every synchronous microbatch +contains at least one target globally. Keep the existing model and optimizer. + +The ordinary entry point is `torchrun --standalone --nproc-per-node=2 train.py`. +Pass `--token-budget 50` to use the feature. Additional diagnostic scripts are +welcome in your workspace; the production deliverable is `training.py`. +Report when you consider the change ready and summarize your evidence. diff --git a/tools/token_budget_study/public/train.py b/tools/token_budget_study/public/train.py new file mode 100644 index 00000000..bd9e1422 --- /dev/null +++ b/tools/token_budget_study/public/train.py @@ -0,0 +1,70 @@ +"""Ordinary torchrun entry point for the token training task.""" + +from __future__ import annotations + +import argparse +import json +import math +import os + +import torch +import torch.distributed as dist +from torch import Tensor +from torch.nn import functional as F +from torch.nn.parallel import DistributedDataParallel +from training import train_epoch +from workload import IGNORE_INDEX, TokenModel, batches, make_optimizer + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--token-budget", type=int) + parser.add_argument("--seed", type=int, default=17) + parser.add_argument( + "--device", + choices=["cpu", "cuda"], + default="cuda" if torch.cuda.is_available() else "cpu", + ) + args = parser.parse_args() + if args.token_budget is not None and args.token_budget <= 0: + parser.error("--token-budget must be positive") + rank = int(os.environ["RANK"]) + local_rank = int(os.environ["LOCAL_RANK"]) + cuda = args.device == "cuda" + device = torch.device("cuda", local_rank) if cuda else torch.device("cpu") + if cuda: + torch.cuda.set_device(device) + dist.init_process_group("nccl" if cuda else "gloo") + try: + torch.manual_seed(args.seed) + base = TokenModel().to(device) + model = DistributedDataParallel(base, device_ids=[local_rank] if cuda else None) + optimizer = make_optimizer(model) + counts = (13, 19, 6, 8, 21) if rank % 2 == 0 else (8, 11, 14, 20, 9) + local_batches = batches(counts, rank, args.seed + 1, device) + train_epoch(model, optimizer, iter(local_batches), args.token_budget) + with torch.no_grad(): + inputs, targets = local_batches[-1] + logits: Tensor = base(inputs) + loss = F.cross_entropy( + logits.flatten(0, 1), targets.flatten(), ignore_index=IGNORE_INDEX + ) + value = float(loss.item()) + if not math.isfinite(value): + raise ValueError("Nonfinite final loss") + print( + json.dumps( + { + "rank": rank, + "microbatches": len(local_batches), + "final_local_loss": value, + } + ), + flush=True, + ) + finally: + dist.destroy_process_group() + + +if __name__ == "__main__": + main() diff --git a/tools/token_budget_study/public/training.py b/tools/token_budget_study/public/training.py new file mode 100644 index 00000000..118ec847 --- /dev/null +++ b/tools/token_budget_study/public/training.py @@ -0,0 +1,36 @@ +"""Training update policy; called by the ordinary DDP entry point.""" + +from __future__ import annotations + +from collections.abc import Iterable + +import torch +import torch.distributed as dist +from torch import Tensor, nn +from torch.nn import functional as F +from workload import IGNORE_INDEX, Batch + + +def train_epoch( + model: nn.Module, + optimizer: torch.optim.Optimizer, + microbatches: Iterable[Batch], + token_budget: int | None, +) -> None: + if token_budget is not None: + raise NotImplementedError("Token-budget mode is not implemented") + + model.train() + for inputs, targets in microbatches: + optimizer.zero_grad(set_to_none=True) + logits: Tensor = model(inputs) + losses = F.cross_entropy( + logits.flatten(0, 1), + targets.flatten(), + ignore_index=IGNORE_INDEX, + reduction="sum", + ) + target_count = (targets != IGNORE_INDEX).sum() + dist.all_reduce(target_count) + (losses * (dist.get_world_size() / target_count)).backward() + optimizer.step() diff --git a/tools/token_budget_study/public/workload.py b/tools/token_budget_study/public/workload.py new file mode 100644 index 00000000..8b081ec0 --- /dev/null +++ b/tools/token_budget_study/public/workload.py @@ -0,0 +1,47 @@ +"""Small deterministic token-level training workload.""" + +from __future__ import annotations + +from collections.abc import Sequence + +import torch +from torch import Tensor, nn + +Batch = tuple[Tensor, Tensor] +IGNORE_INDEX = -100 +VOCABULARY = 29 +WIDTH = 12 +ROWS = 2 +COLUMNS = 12 + + +class TokenModel(nn.Module): + def __init__(self) -> None: + super().__init__() + self.embedding = nn.Embedding(VOCABULARY, WIDTH) + self.projection = nn.Linear(WIDTH, VOCABULARY) + + def forward(self, tokens: Tensor) -> Tensor: + logits: Tensor = self.projection(self.embedding(tokens).tanh()) + return logits + + +def batches( + counts: Sequence[int], rank: int, seed: int, device: torch.device +) -> list[Batch]: + generator = torch.Generator().manual_seed(seed + 8191 * rank) + result: list[Batch] = [] + for count in counts: + if not 0 <= count <= ROWS * COLUMNS: + raise ValueError("Target count must fit the microbatch") + size = (ROWS, COLUMNS) + tokens = torch.randint(VOCABULARY, size, generator=generator) + targets = torch.randint(VOCABULARY, size, generator=generator) + padding = torch.randperm(ROWS * COLUMNS, generator=generator)[count:] + targets.view(-1)[padding] = IGNORE_INDEX + result.append((tokens.to(device), targets.to(device))) + return result + + +def make_optimizer(model: nn.Module) -> torch.optim.SGD: + return torch.optim.SGD(model.parameters(), lr=0.08, momentum=0.7) diff --git a/tools/token_budget_study/tests/test_oracle.py b/tools/token_budget_study/tests/test_oracle.py new file mode 100644 index 00000000..39b6646a --- /dev/null +++ b/tools/token_budget_study/tests/test_oracle.py @@ -0,0 +1,78 @@ +"""Regression checks for the private oracle, using fixed tiny manually grouped updates.""" + +from __future__ import annotations + +import copy +import sys +from pathlib import Path + +import pytest +import torch +from torch.nn import functional as F + +BASE = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(BASE / "public")) +sys.path.insert(0, str(BASE / "private")) + +from worker import SCENARIOS, compare_map, gradients, momentum, reference, values +from workload import IGNORE_INDEX, TokenModel, batches, make_optimizer + + +def test_overshoot_flush_does_not_carry_excess_and_partial_tail_is_applied() -> None: + device = torch.device("cpu") + torch.manual_seed(30) + model = TokenModel() + check_model = copy.deepcopy(model) + opt = make_optimizer(check_model) + counts = ((4, 17, 3, 8, 5, 2), (8, 4, 19, 6, 5, 3)) + data = [batches(c, r, 442, device) for r, c in enumerate(counts)] + steps = reference(model, data, 29) + # Global counts [12, 21, 22, 14, 10, 5] make windows 33, 36 and 15. + assert len(steps) == 3 + for expected, indices in zip(steps, ((0, 1), (2, 3), (4, 5)), strict=True): + opt.zero_grad(set_to_none=True) + # Token-by-token mean over the manually chosen active examples. + inputs, labels = [], [] + for rank in range(2): + for index in indices: + source, targets = data[rank][index] + keep = targets.ne(IGNORE_INDEX) + inputs.append(source[keep]) + labels.append(targets[keep]) + logits = check_model(torch.cat(inputs)) + torch.autograd.backward(F.cross_entropy(logits, torch.cat(labels))) + assert not compare_map("grads", expected.gradients, gradients(check_model)) + opt.step() + assert not compare_map("params", expected.parameters, values(check_model)) + assert not compare_map( + "momentum", expected.momentum, momentum(check_model, opt) + ) + + +def test_equal_step_counts_do_not_hide_incorrect_gradients() -> None: + expected = {"weight": torch.tensor([0.12, 0.42])} + swapped = {"weight": torch.tensor([0.42, 0.12])} + assert compare_map("gradient", expected, swapped) + assert compare_map("gradient", expected, {}) + assert compare_map( + "gradient", expected, {"weight": torch.tensor([float("nan"), 0.42])} + ) + assert compare_map("gradient", expected, {"weight": expected["weight"].double()}) + assert compare_map( + "gradient", expected, {"weight": expected["weight"].reshape(1, 2)} + ) + + +@pytest.mark.parametrize( + "case_index,expected_steps", [(0, 3), (1, 2), (2, 3), (3, 3), (4, 1), (5, 3)] +) +def test_all_frozen_schedules(case_index: int, expected_steps: int) -> None: + case = SCENARIOS[case_index] + data = [ + batches(c, r, case.seed, torch.device("cpu")) for r, c in enumerate(case.counts) + ] + steps = reference(TokenModel(), data, case.budget) + assert len(steps) == expected_steps + assert all( + torch.isfinite(t).all() for step in steps for t in step.gradients.values() + ) From 1d08bdce7b9013c9ee743490536d146c1bbb34c8 Mon Sep 17 00:00:00 2001 From: Yuxuan Jiang Date: Wed, 18 Nov 2026 03:11:34 +0800 Subject: [PATCH 4/6] Run qualified two-rank training and private replays on Ring18 --- .../operator/ring_service.py | 358 ++++++++++++++++++ .../operator/test_ring_service.py | 61 +++ 2 files changed, 419 insertions(+) create mode 100644 tools/token_budget_study/operator/ring_service.py create mode 100644 tools/token_budget_study/operator/test_ring_service.py diff --git a/tools/token_budget_study/operator/ring_service.py b/tools/token_budget_study/operator/ring_service.py new file mode 100644 index 00000000..2a662400 --- /dev/null +++ b/tools/token_budget_study/operator/ring_service.py @@ -0,0 +1,358 @@ +"""Operator-only, serialized two-GPU public execution and private evaluation on Ring18.""" + +from __future__ import annotations + +import argparse +import fcntl +import hashlib +import json +import os +import re +import signal +import subprocess +import sys +import time +from pathlib import Path +from typing import TypedDict + +REMOTE = Path("/home/yuxuan/experiments/traincheck-token-budget-20261118") +PYTHON = ( + "/home/yuxuan/experiments/traincheck-token-normalization-20261102/venv/bin/python" +) +DEVICES = (4, 5) +SSH = [ + "ssh", + "-T", + "-o", + "BatchMode=yes", + "-o", + "ConnectTimeout=15", + "-o", + "ServerAliveInterval=30", + "-o", + "ServerAliveCountMax=4", + "-o", + "ControlMaster=auto", + "-o", + "ControlPersist=120", + "-o", + "ControlPath=/tmp/traincheck-ring18-1118-%C", + "ring18", +] + + +class Request(TypedDict): + id: str + kind: str + entry: str + argv: list[str] + files: dict[str, str] + with_traincheck: bool + worker_sha256: str + + +def parse_request(raw: object) -> Request: + if not isinstance(raw, dict): + raise ValueError("Expected request object") + uid, kind, entry, argv, files, treatment, digest = ( + raw[k] + for k in ( + "id", + "kind", + "entry", + "argv", + "files", + "with_traincheck", + "worker_sha256", + ) + ) + if not isinstance(uid, str) or not re.fullmatch("[A-Za-z0-9_-]{1,96}", uid): + raise ValueError("Invalid request ID") + if ( + kind not in ("public", "grade") + or entry not in ("train.py", "diagnostic.py") + or not isinstance(treatment, bool) + ): + raise ValueError("Invalid service request") + if ( + not isinstance(argv, list) + or len(argv) > 32 + or not all( + isinstance(x, str) and len(x) <= 256 and "\x00" not in x for x in argv + ) + ): + raise ValueError("Invalid worker arguments") + if ( + not isinstance(files, dict) + or not {"training.py"} <= set(files) <= {"training.py", "diagnostic.py"} + or not all( + isinstance(v, str) and len(v.encode()) <= 262144 for v in files.values() + ) + ): + raise ValueError("Invalid editable file set") + if entry == "diagnostic.py" and entry not in files: + raise ValueError("Missing diagnostic entry") + if ( + not isinstance(digest, str) + or digest != hashlib.sha256(Path(__file__).read_bytes()).hexdigest() + ): + raise ValueError("Service version mismatch") + return Request( + id=uid, + kind=kind, + entry=entry, + argv=argv, + files=files, + with_traincheck=treatment, + worker_sha256=digest, + ) + + +def gpu_state() -> dict[int, int]: + output = subprocess.check_output( + [ + "nvidia-smi", + "--query-gpu=index,memory.used", + "--format=csv,noheader,nounits", + ], + text=True, + timeout=15, + ) + inventory = { + int(i): int(mem) for line in output.splitlines() for i, mem in [line.split(",")] + } + return {i: inventory[i] for i in DEVICES} + + +def owned_pids(uid: str) -> list[int]: + marker = ("TRAINCHECK_STUDY_E_REQUEST=" + uid).encode() + result: list[int] = [] + for path in Path("/proc").glob("[0-9]*/environ"): + try: + if path.stat().st_uid == os.getuid() and marker in path.read_bytes().split( + b"\0" + ): + result.append(int(path.parent.name)) + except (FileNotFoundError, ProcessLookupError, PermissionError): + continue # Process exited or belongs to another user/restricted subsystem. + return result + + +def cleanup(uid: str) -> None: + for sig in (signal.SIGTERM, signal.SIGKILL): + for pid in owned_pids(uid): + try: + os.kill(pid, sig) + except ProcessLookupError: + pass # The tagged process exited before signalling. + if not owned_pids(uid): + return + time.sleep(0.5) + if owned_pids(uid): + raise RuntimeError("Owned worker descendants remain") + + +def tail(path: Path) -> str: + with path.open("rb") as f: + f.seek(max(0, path.stat().st_size - 32768)) + return f.read().decode(errors="replace") + + +def run_worker(req: Request, run: Path, source: Path) -> dict[str, object]: + public = req["kind"] == "public" + if public: + command = [ + PYTHON, + "-m", + "torch.distributed.run", + "--standalone", + "--nproc-per-node=2", + req["entry"], + *req["argv"], + ] + else: + command = [ + PYTHON, + str(REMOTE / "private/grade.py"), + "--source", + str(source), + "--output", + str(run / "grade.json"), + "--device", + "cuda", + "--timeout", + "85", + ] + if req["with_traincheck"]: + command.extend(["--optional-package", str(REMOTE / "optional/traincheck")]) + env = os.environ.copy() + env.update( + TRAINCHECK_STUDY_E_REQUEST=req["id"], + CUDA_VISIBLE_DEVICES="4,5", + OMP_NUM_THREADS="1", + PYTHONDONTWRITEBYTECODE="1", + PYTHONNOUSERSITE="1", + PYTHONPATH=str(REMOTE / ("optional" if req["with_traincheck"] else "control")) + + os.pathsep + + str(source), + ) + with (run / "stdout.log").open("xb") as out, (run / "stderr.log").open("xb") as err: + start = time.monotonic() + (run / "started.json").write_text( + json.dumps( + { + "unix_time": time.time(), + "devices": DEVICES, + "source_sha256": hashlib.sha256( + (source / "training.py").read_bytes() + ).hexdigest(), + } + ) + ) + process = subprocess.Popen( + command, cwd=source, env=env, stdout=out, stderr=err, start_new_session=True + ) + try: + code = process.wait(timeout=120) + status = "ok" if code == 0 else "application_error" + except subprocess.TimeoutExpired: + try: + os.killpg(process.pid, signal.SIGKILL) + except ProcessLookupError: + pass + code = process.wait(timeout=10) + status = "timeout" + finally: + cleanup(req["id"]) + until = time.monotonic() + 8 + post = gpu_state() + while any(post.values()) and time.monotonic() < until: + time.sleep(0.25) + post = gpu_state() + elapsed = time.monotonic() - start + if any(post.values()): + raise RuntimeError( + "GPU release not verified; inspect durable logs and started receipt" + ) + result: dict[str, object] = { + "status": status, + "exit_code": code, + "worker_seconds": elapsed, + "gpu_seconds": elapsed * 2, + "devices": DEVICES, + "gpu_release": post, + "stdout": tail(run / "stdout.log"), + "stderr": tail(run / "stderr.log"), + } + if not public and (run / "grade.json").is_file(): + result["grade"] = json.loads((run / "grade.json").read_text()) + return result + + +def serve(req: Request) -> dict[str, object]: + run = REMOTE / "runs" / req["id"] + run.mkdir(mode=0o700, exist_ok=False) + (run / "request.json").write_text(json.dumps(req)) + source = run / "source" + source.mkdir() + for name in ("train.py", "workload.py"): + (source / name).write_bytes((REMOTE / "public" / name).read_bytes()) + for name, content in req["files"].items(): + (source / name).write_bytes(content.encode()) + start = time.monotonic() + with (REMOTE / "service.lock").open("a") as lock: + fcntl.flock(lock, fcntl.LOCK_EX) + queue = time.monotonic() - start + pre = gpu_state() + if any(pre.values()): + receipt: dict[str, object] = { + "status": "busy", + "worker_seconds": 0.0, + "gpu_seconds": 0.0, + "stdout": "", + "stderr": "Selected GPUs occupied; no launch.", + } + else: + receipt = run_worker(req, run, source) + receipt.update( + queue_seconds=queue, + initial_gpu_memory_mib=pre, + id=req["id"], + kind=req["kind"], + training_sha256=hashlib.sha256( + req["files"]["training.py"].encode() + ).hexdigest(), + ) + (run / "receipt.json").write_text(json.dumps(receipt, indent=2) + "\n") + return receipt + + +def client() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("kind", choices=("public", "grade")) + parser.add_argument("--source", type=Path, required=True) + parser.add_argument("--id", required=True) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument( + "--entry", choices=("train.py", "diagnostic.py"), default="train.py" + ) + parser.add_argument("--args-json", default="[]") + parser.add_argument("--with-traincheck", action="store_true") + args = parser.parse_args() + files: dict[str, str] = {} + for name in ("training.py", "diagnostic.py"): + path = args.source / name + if path.is_symlink(): + raise ValueError("Expected regular source file") + if path.is_file(): + files[name] = path.read_bytes().decode() + req = parse_request( + dict( + id=args.id, + kind=args.kind, + entry=args.entry, + argv=json.loads(args.args_json), + files=files, + with_traincheck=args.with_traincheck, + worker_sha256=hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), + ) + ) + args.output.mkdir(parents=True, exist_ok=False) + payload = json.dumps(req).encode() + (args.output / "request.json").write_bytes(payload) + since = time.monotonic() + proc = subprocess.run( + [*SSH, PYTHON, str(REMOTE / "ring_service.py"), "serve"], + input=payload, + capture_output=True, + timeout=1800, + ) + duration = time.monotonic() - since + (args.output / "transport.stderr").write_bytes(proc.stderr) + if proc.returncode: + raise RuntimeError( + f"SSH worker failed ({proc.returncode}): {proc.stderr[-2000:]!r}; inspect remote request before retrying" + ) + receipt = json.loads(proc.stdout) + if not isinstance(receipt, dict) or receipt["id"] != req["id"]: + raise ValueError("Unbound worker receipt") + receipt["service_seconds"] = duration + receipt["transfer_seconds"] = ( + duration - receipt["queue_seconds"] - receipt["worker_seconds"] + ) + (args.output / "receipt.json").write_text(json.dumps(receipt, indent=2) + "\n") + print( + json.dumps( + { + key: receipt[key] + for key in ("id", "status", "gpu_seconds", "queue_seconds") + } + ) + ) + + +if __name__ == "__main__": + if sys.argv[1:] == ["serve"]: + print(json.dumps(serve(parse_request(json.load(sys.stdin))))) + else: + client() diff --git a/tools/token_budget_study/operator/test_ring_service.py b/tools/token_budget_study/operator/test_ring_service.py new file mode 100644 index 00000000..1e0107c0 --- /dev/null +++ b/tools/token_budget_study/operator/test_ring_service.py @@ -0,0 +1,61 @@ +"""Request boundary and non-allocation contracts for the Ring18 service.""" + +import hashlib +from pathlib import Path + +import pytest + +from tools.token_budget_study.operator import ring_service + + +def request() -> dict[str, object]: + return { + "id": "control_1-request1", + "kind": "public", + "entry": "train.py", + "argv": [], + "files": {"training.py": "print(1)\n"}, + "with_traincheck": False, + "worker_sha256": hashlib.sha256( + Path(ring_service.__file__).read_bytes() + ).hexdigest(), + } + + +@pytest.mark.parametrize( + ("field", "value"), + [ + ("id", "../../outside"), + ("entry", "private/grade.py"), + ("files", {"training.py": "x", "private.py": "y"}), + ("entry", "diagnostic.py"), + ("argv", ["a\x00b"]), + ("worker_sha256", "deadbeef"), + ], +) +def test_reject_malformed_request(field: str, value: object) -> None: + data = request() + data[field] = value + with pytest.raises(ValueError): + ring_service.parse_request(data) + + +def test_busy_devices_do_not_incur_worker_cost_or_reuse_id( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + for folder in ("public", "runs"): + (tmp_path / folder).mkdir() + for name in ("train.py", "workload.py"): + (tmp_path / "public" / name).write_text("") + monkeypatch.setattr(ring_service, "REMOTE", tmp_path) + monkeypatch.setattr(ring_service, "gpu_state", lambda: {4: 0, 5: 20}) + parsed = ring_service.parse_request(request()) + result = ring_service.serve(parsed) + assert (result["status"], result["worker_seconds"], result["gpu_seconds"]) == ( + "busy", + 0, + 0, + ) + assert not (tmp_path / "runs" / parsed["id"] / "started.json").exists() + with pytest.raises(FileExistsError): + ring_service.serve(parsed) From 2fe4c302de714d3e3fe6be2aa150341384904701 Mon Sep 17 00:00:00 2001 From: Yuxuan Jiang Date: Wed, 18 Nov 2026 03:13:58 +0800 Subject: [PATCH 5/6] Record Ring18 positive and silent-error controls before solver grading --- .../token-budget-agent-study-20261118.md | 46 +++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/docs/measurements/token-budget-agent-study-20261118.md b/docs/measurements/token-budget-agent-study-20261118.md index b3edcae8..5d6a958e 100644 --- a/docs/measurements/token-budget-agent-study-20261118.md +++ b/docs/measurements/token-budget-agent-study-20261118.md @@ -71,3 +71,49 @@ in [PR #172](https://github.com/OrderLab/TrainCheck/pull/172). Prior numerical qualification of global token-mean objectives is documented in the [DDP study](ddp-runtime-state-pilot-20261102.md). This task requires a new implementation and new reference runs for its token-budget boundaries. + +## Native qualification and reproduction + +The pilot uses a 725-parameter token classifier (embedding, tanh and a linear +projection with bias) and SGD with momentum. The experiment runs on physical +A16 devices 4 and 5 on Ring18, driver 535.309.01, Python 3.10.14 and PyTorch +2.2.2+cu121. Each rank receives five variable-length microbatches in the ordinary +example. The task and tests use non-padding target counts, including zero local +targets on a rank in one hidden case. All six solvers use native Codex CLI +0.160.0, requested model `gpt-6-astra`, high reasoning effort and priority +service. Each receives a fresh conversation and a separate workspace with +instructions limiting it to the packet. Pair order is C1/T1, T2/C2, T3/C3. +Both agents in a pair can work concurrently while the GPU service serializes +their requests. The next pair starts when both finish. + +The withheld tests cover fixed mode, exact budget, overshoot with a partial +tail, rank skew, a whole epoch below budget and one-microbatch updates. +The oracle concatenates the examples for each global update and uses mean +cross-entropy directly. On both ranks the evaluator observes the gradient +before the optimizer step, parameters and momentum afterward, final state and +which microbatches formed each window. Tensor comparisons use absolute tolerance +`1e-6` and relative tolerance `1e-5`. It also checks grad-producing forward +sizes and overlap of outstanding forwards. Source review supplements those +checks for buffering of the input iterator. + +The healthy fixed-mode starter completed ordinary DDP training on both GPUs. +The starter passed fixed-mode replay and explicitly reported the new feature +as unsupported. A separate correct implementation passed all six scenarios. +A control that divides by the requested budget instead of the actual number +of targets ran to completion and failed the numerical checks in four scenarios. +Independent imports on both workers confirmed that control lacked TrainCheck +and treatment could take a real CUDA snapshot using the supplied package. +Six qualification requests used 128.585 allocated GPU-seconds. + +The operator sources are under [tools/token_budget_study](../../tools/token_budget_study). +Local raw evidence lives in `outputs/ring18-token-budget-20261118/` in the +canonical TrainCheck checkout. The matching Ring18 experiment directory is +`/home/yuxuan/experiments/traincheck-token-budget-20261118`. +`native-launch.json` binds the packets, CLI and controller, with SHA-256 +`4f9f387e33858c67ed1c98bd61ad269b977cf102725cef0fdfe7bc13115e5c4f`. +`ring18-qualification-evidence.json` identifies environment and source hashes +and all native qualification receipts, with SHA-256 +`6bf3c8fa7f736a835ae2dd9b5d47fcadbcdfdcc019c1ae261055d6856c30d60b`. +The operator retains native turn logs and copies production source at every +decision. An additional watcher samples edits approximately every 50 ms. +Intermediate revisions that exist only between samples can be missed. From 980e8993fd274243d12f5dca89b3f1bf1949c1aa Mon Sep 17 00:00:00 2001 From: Yuxuan Jiang Date: Wed, 18 Nov 2026 03:20:46 +0800 Subject: [PATCH 6/6] Report all six first-correct DDP submissions and measured solver costs --- .../token-budget-agent-study-20261118.json | 316 ++++++++++++++++++ .../token-budget-agent-study-20261118.md | 77 +++++ 2 files changed, 393 insertions(+) create mode 100644 docs/measurements/data/token-budget-agent-study-20261118.json diff --git a/docs/measurements/data/token-budget-agent-study-20261118.json b/docs/measurements/data/token-budget-agent-study-20261118.json new file mode 100644 index 00000000..f122812b --- /dev/null +++ b/docs/measurements/data/token-budget-agent-study-20261118.json @@ -0,0 +1,316 @@ +{ + "task": "token-budget-ddp-20261118", + "implementation_revision": "1d08bdce", + "cohort_outcomes": { + "finished": 6, + "delivered_silent_errors_observed": 0, + "self_detected_numerical_production_defect_then_repaired_observed": 0, + "correct_at_first_submission": 6, + "traincheck_guide_reads_in_optional_arm": 3, + "runtime_uses_of_traincheck_observed": 0 + }, + "cost_boundary": "At frozen production submission, before dispatch of any public execution requested by that decision. Total tokens count cached input once.", + "per_trial": [ + { + "trial": "control_1", + "terminal": { + "status": "finished", + "last_turn": 2, + "requests": 1, + "cli_seconds": 73.89876775097218, + "gpu_seconds": 25.008294839411974, + "queue_seconds": 11.230069799348712, + "elapsed_seconds": 98.6117637080024 + }, + "submissions": [ + { + "turn": 1, + "production_sha256": "d5565f27480700e7e5b0e4f7098ff3c95ef5d8e72ffbb8cb44c96e1fbf92be6b", + "action": "run", + "grade": "pass", + "wall_seconds": 61.95383666700218, + "cli_seconds": 61.94894879197818, + "queue_seconds": 0.0, + "gpu_seconds": 0.0, + "total_tokens": 109666, + "input_tokens": 106586, + "cached_input_tokens": 84480, + "output_tokens": 3080, + "uncached_input_plus_output": 25186 + }, + { + "turn": 2, + "production_sha256": "d5565f27480700e7e5b0e4f7098ff3c95ef5d8e72ffbb8cb44c96e1fbf92be6b", + "action": "finish", + "grade": "pass", + "wall_seconds": 98.61131704199943, + "cli_seconds": 73.89876775097218, + "queue_seconds": 11.230069799348712, + "gpu_seconds": 25.008294839411974, + "total_tokens": 135855, + "input_tokens": 132618, + "cached_input_tokens": 108544, + "output_tokens": 3237, + "uncached_input_plus_output": 27311 + } + ], + "earliest_observed_pass": 1, + "earlier_unknown_turns": [] + }, + { + "trial": "traincheck_1", + "terminal": { + "status": "finished", + "last_turn": 2, + "requests": 1, + "cli_seconds": 67.37666341700242, + "gpu_seconds": 24.510121228173375, + "queue_seconds": 0.00012123305350542068, + "elapsed_seconds": 85.23818870799732 + }, + "submissions": [ + { + "turn": 1, + "production_sha256": "13d7dd30df6a0c89bb1ae7b0cdedb3163436b4ba57802c5d6fd601ced43829fa", + "action": "run", + "grade": "pass", + "wall_seconds": 56.19440937502077, + "cli_seconds": 56.18416104200878, + "queue_seconds": 0.0, + "gpu_seconds": 0.0, + "total_tokens": 90894, + "input_tokens": 88284, + "cached_input_tokens": 64512, + "output_tokens": 2610, + "uncached_input_plus_output": 26382 + }, + { + "turn": 2, + "production_sha256": "13d7dd30df6a0c89bb1ae7b0cdedb3163436b4ba57802c5d6fd601ced43829fa", + "action": "finish", + "grade": "pass", + "wall_seconds": 85.23763379099546, + "cli_seconds": 67.37666341700242, + "queue_seconds": 0.00012123305350542068, + "gpu_seconds": 24.510121228173375, + "total_tokens": 118664, + "input_tokens": 115922, + "cached_input_tokens": 90112, + "output_tokens": 2742, + "uncached_input_plus_output": 28552 + } + ], + "earliest_observed_pass": 1, + "earlier_unknown_turns": [] + }, + { + "trial": "control_2", + "terminal": { + "status": "finished", + "last_turn": 2, + "requests": 1, + "cli_seconds": 65.58246241800953, + "gpu_seconds": 24.908979253843427, + "queue_seconds": 0.0001171603798866272, + "elapsed_seconds": 79.10894395899959 + }, + "submissions": [ + { + "turn": 1, + "production_sha256": "02045ce58bf51e325c9fc967418caae697aad5f58681b444e3dbc49b926bd621", + "action": "run", + "grade": "pass", + "wall_seconds": 54.37110237497836, + "cli_seconds": 54.36576070901356, + "queue_seconds": 0.0, + "gpu_seconds": 0.0, + "total_tokens": 74898, + "input_tokens": 72160, + "cached_input_tokens": 52864, + "output_tokens": 2738, + "uncached_input_plus_output": 22034 + }, + { + "turn": 2, + "production_sha256": "02045ce58bf51e325c9fc967418caae697aad5f58681b444e3dbc49b926bd621", + "action": "finish", + "grade": "pass", + "wall_seconds": 79.10868350000237, + "cli_seconds": 65.58246241800953, + "queue_seconds": 0.0001171603798866272, + "gpu_seconds": 24.908979253843427, + "total_tokens": 96985, + "input_tokens": 94092, + "cached_input_tokens": 73984, + "output_tokens": 2893, + "uncached_input_plus_output": 23001 + } + ], + "earliest_observed_pass": 1, + "earlier_unknown_turns": [] + }, + { + "trial": "traincheck_2", + "terminal": { + "status": "finished", + "last_turn": 2, + "requests": 1, + "cli_seconds": 81.21886341599748, + "gpu_seconds": 24.70401251129806, + "queue_seconds": 0.00011837296187877655, + "elapsed_seconds": 94.68141233400092 + }, + "submissions": [ + { + "turn": 1, + "production_sha256": "e0945cad17bffbc98323c818b9ae75070021da81c08380ede4b58f2170dc819d", + "action": "run", + "grade": "pass", + "wall_seconds": 68.77601474997937, + "cli_seconds": 68.76812808300019, + "queue_seconds": 0.0, + "gpu_seconds": 0.0, + "total_tokens": 128224, + "input_tokens": 124666, + "cached_input_tokens": 113408, + "output_tokens": 3558, + "uncached_input_plus_output": 14816 + }, + { + "turn": 2, + "production_sha256": "e0945cad17bffbc98323c818b9ae75070021da81c08380ede4b58f2170dc819d", + "action": "finish", + "grade": "pass", + "wall_seconds": 94.68105837499024, + "cli_seconds": 81.21886341599748, + "queue_seconds": 0.00011837296187877655, + "gpu_seconds": 24.70401251129806, + "total_tokens": 156153, + "input_tokens": 152402, + "cached_input_tokens": 139392, + "output_tokens": 3751, + "uncached_input_plus_output": 16761 + } + ], + "earliest_observed_pass": 1, + "earlier_unknown_turns": [] + }, + { + "trial": "control_3", + "terminal": { + "status": "finished", + "last_turn": 2, + "requests": 1, + "cli_seconds": 73.49893141697976, + "gpu_seconds": 24.91642988473177, + "queue_seconds": 0.0001183096319437027, + "elapsed_seconds": 86.9699251250131 + }, + "submissions": [ + { + "turn": 1, + "production_sha256": "44530b3435d8d5896cdd183593b2d6c7165f89ab219da4ff8aceb69001733c8b", + "action": "run", + "grade": "pass", + "wall_seconds": 60.54670108400751, + "cli_seconds": 60.541876041999785, + "queue_seconds": 0.0, + "gpu_seconds": 0.0, + "total_tokens": 94550, + "input_tokens": 91815, + "cached_input_tokens": 84480, + "output_tokens": 2735, + "uncached_input_plus_output": 10070 + }, + { + "turn": 2, + "production_sha256": "44530b3435d8d5896cdd183593b2d6c7165f89ab219da4ff8aceb69001733c8b", + "action": "finish", + "grade": "pass", + "wall_seconds": 86.96970687501016, + "cli_seconds": 73.49893141697976, + "queue_seconds": 0.0001183096319437027, + "gpu_seconds": 24.91642988473177, + "total_tokens": 121899, + "input_tokens": 119014, + "cached_input_tokens": 105728, + "output_tokens": 2885, + "uncached_input_plus_output": 16171 + } + ], + "earliest_observed_pass": 1, + "earlier_unknown_turns": [] + }, + { + "trial": "traincheck_3", + "terminal": { + "status": "finished", + "last_turn": 2, + "requests": 1, + "cli_seconds": 75.45538349996787, + "gpu_seconds": 24.796238465234637, + "queue_seconds": 8.750714817084372, + "elapsed_seconds": 97.73528554098448 + }, + "submissions": [ + { + "turn": 1, + "production_sha256": "d146608940b75b1c906a0de174b0a431bbef972cbc80f4d4080eed20160a7a2e", + "action": "run", + "grade": "pass", + "wall_seconds": 64.25127145799343, + "cli_seconds": 64.2413744579826, + "queue_seconds": 0.0, + "gpu_seconds": 0.0, + "total_tokens": 95300, + "input_tokens": 92152, + "cached_input_tokens": 72576, + "output_tokens": 3148, + "uncached_input_plus_output": 22724 + }, + { + "turn": 2, + "production_sha256": "d146608940b75b1c906a0de174b0a431bbef972cbc80f4d4080eed20160a7a2e", + "action": "finish", + "grade": "pass", + "wall_seconds": 97.73503808298847, + "cli_seconds": 75.45538349996787, + "queue_seconds": 8.750714817084372, + "gpu_seconds": 24.796238465234637, + "total_tokens": 118888, + "input_tokens": 115576, + "cached_input_tokens": 94464, + "output_tokens": 3312, + "uncached_input_plus_output": 24424 + } + ], + "earliest_observed_pass": 1, + "earlier_unknown_turns": [] + } + ], + "new_allocations": { + "qualification": { + "allocated_gpu_seconds": 128.58470613323152, + "physical_allocations": 6 + }, + "solvers": { + "allocated_gpu_seconds": 148.84407618269324, + "physical_allocations": 6 + }, + "grading": { + "allocated_gpu_seconds": 210.97372341156006, + "physical_allocations": 8 + } + }, + "total_gpu_seconds": 488.4025057274848, + "evidence_sha256": { + "first-valid-costs.json": "048723d1c2cdcf0c65a5765935316a8f906013c50646e194090b436ebd3ed2e1", + "behavior-audit.json": "0f398ef6e8ec65b3fcf6fcb601b0bc4c0580af0aac79c17c43a81a7f54ac67be", + "native-operation-audit.json": "964fc69cf77ec3d9f02ff99061d230240b79371d1d5d3a19995cde87e25153a5", + "ring18-allocation-audit.json": "687128f23bf2b7daa23469de012c79b76422a973baff4d2510bfcfe1653d671a", + "private-grading-summary.json": "f536ab6c4bcbbb1d6937624fff34e685b34f0e4496461b7bcc1e491bf1983317", + "private-replay-inventory.json": "994c351c9f60bedf5c5c9a307af6a612dce1da20456d412c4ed1635e2526084d", + "native-launch.json": "4f9f387e33858c67ed1c98bd61ad269b977cf102725cef0fdfe7bc13115e5c4f", + "ring18-qualification-evidence.json": "6bf3c8fa7f736a835ae2dd9b5d47fcadbcdfdcc019c1ae261055d6856c30d60b" + } +} diff --git a/docs/measurements/token-budget-agent-study-20261118.md b/docs/measurements/token-budget-agent-study-20261118.md index 5d6a958e..cdc115e5 100644 --- a/docs/measurements/token-budget-agent-study-20261118.md +++ b/docs/measurements/token-budget-agent-study-20261118.md @@ -1,5 +1,11 @@ # Implementing token-budget accumulation under DDP +All six agents delivered a correct implementation on their first submission. +Each then ran a handwritten two-GPU PyTorch diagnostic before declaring the +same source ready. We observed **0/6 silent errors at delivery** and **0/6 +self-detected and repaired numerical defects**. All three optional-tool agents +read the TrainCheck guide. None used it in the executed training or diagnostics. + This prospective Ring18 study asks whether six fresh agents deliver a numerically correct feature when they add token-budget gradient accumulation to a working fixed-mode training loop. Three agents receive the ordinary task and three can @@ -117,3 +123,74 @@ and all native qualification receipts, with SHA-256 The operator retains native turn logs and copies production source at every decision. An additional watcher samples edits approximately every 50 ms. Intermediate revisions that exist only between samples can be missed. + +## First correct submission and delivery + +| Pair | Arm | First correct turn | Elapsed seconds | Active CLI seconds | Total tokens | Uncached input + output | Prior GPU-seconds | GPU-seconds through delivery | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 1 | Control | 1 | 61.954 | 61.949 | 109,666 | 25,186 | 0 | 25.008 | +| 1 | Optional TrainCheck | 1 | 56.194 | 56.184 | 90,894 | 26,382 | 0 | 24.510 | +| 2 | Control | 1 | 54.371 | 54.366 | 74,898 | 22,034 | 0 | 24.909 | +| 2 | Optional TrainCheck | 1 | 68.776 | 68.768 | 128,224 | 14,816 | 0 | 24.704 | +| 3 | Control | 1 | 60.547 | 60.542 | 94,550 | 10,070 | 0 | 24.916 | +| 3 | Optional TrainCheck | 1 | 64.251 | 64.241 | 95,300 | 22,724 | 0 | 24.796 | + +Median elapsed time to first correct submission was 60.55 s in control and +64.25 s with TrainCheck available. Median total token counts were 94,550 and +95,300 respectively. Total tokens include cached input once. Every first +submission requested a diagnostic run, so the cost to produce the correct code +was zero GPU-seconds for every solver. All six then incurred the GPU cost in +the last column to verify their implementation. The paired times and token +counts moved in different directions across pairs. The observed choices +provide no measurement of the effect of actually using TrainCheck. + +All six solvers chose to compare each distributed update against ordinary +training on its combined global batch. Their two-rank diagnostics completed +successfully. Each solver's production source was byte-identical at the first +request and at final delivery. Across the watched edits we saw only the +unsupported starter and one feature implementation per solver. Independent +replay of all six feature implementations passed all six scenarios on both +ranks, including gradients, parameters, momentum and update boundaries (72 +rank/scenario outcomes). Each had at most one outstanding grad-producing +forward in these cases. Review of all six sources found streaming accumulation +without buffering a window of inputs. + +This was a small new feature on a healthy baseline. The starter already showed +how to count targets across ranks and normalize a single-microbatch update. +The public contract specified equivalence to the combined global batch, which +gave both arms an executable check. The results apply to this task, model, +three pairs and six held-out scenarios. We cannot estimate how often agents +introduce silent errors across ML research tasks or how much TrainCheck saves +when a faulty implementation actually occurs. + +The six native conversations had distinct IDs. In the recorded commands C2, +C3 and T2 read a generic host policy skill, and T2 also read its schema. No +access to peer solutions or private grading material was observed. All tool +adoption and self-correction verdicts describe observable actions. + +## Cost and evidence + +| Category | Physical two-GPU allocations | Allocated GPU-seconds | +| --- | ---: | ---: | +| Native qualification | 6 | 128.585 | +| Solver-requested diagnostics | 6 | 148.844 | +| Independent source replays | 8 | 210.974 | +| Total | 20 | 488.403 | + +The 12 decision submissions and 18 within-turn source observations reduced +to eight distinct arm/source combinations, including the unsupported starter +in both environments. All 48 native scenario runs have known results. Each +allocation has a worker receipt and verified GPU release. + +The [machine-readable results](data/token-budget-agent-study-20261118.json) +include both submission boundaries per agent, terminal clocks, native token +counters, and hashes of the raw evidence. In the local evidence directory, +`behavior-audit.json` records each solver's evidence for its delivery, +`native-operation-audit.json` verifies token and operation history, and +`ring18-allocation-audit.json` reconciles every physical allocation. To +recompute the costs from those retained submissions and private receipts, +run this command from the repository root, selecting a new output path: + +```sh +python -m tools.token_budget_study.operator.score_submissions "$evidence_directory" "$new_output_json" +```