Repository navigation
Compare PR benchmarks on the same runners - #529
Merged
AJenbo merged 2 commits intoNov 5, 2026
Merged
Conversation
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
shuvroroy
marked this pull request as ready for review
November 5, 2026 05:35
Build both revisions with identical settings and distribute matching cases using one shard assignment. Measure each revision with its own fixtures, validate every expected result, and retain the 30% regression threshold. Report new and removed cases and keep main history unchanged. Closes PHPantom-dev#489 Validation: 29 benchmark helper tests, Clippy, formatting, YAML parsing, and independent review of the workflow and measurement paths.
`cargo bench --benches` also compiles the library and binary unit tests in release mode, once per revision; `--bench '*'` builds only the bench targets. Take the base revision from the first parent of the test merge commit being measured instead of `pull_request.base.sha`, and fetch only the two commits that needs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AJenbo
force-pushed
the
fix/paired-benchmark-comparison
branch
from
November 5, 2026 18:00
0c45ca2 to
8de9e79
Compare
Contributor
|
Thanks |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR benchmarks currently compare timings from different GitHub runners against the historical main-branch baseline. Runner speed alone can exceed the 30% threshold and repeatedly block changes such as #527. This measures the base and PR on the same worker while retaining the existing threshold and required check.
Both revisions build with the same toolchain and flags before measurement. A shared manifest keeps matching cases on the same shard when benchmarks are added or moved, and each binary uses its own revision’s fixtures. The comparison requires complete results, shows the raw timings and ratios, and identifies new and removed cases. Main’s historical dashboard is unchanged; latency reports use the job summary with read-only permissions. The extra base build and measurement increase benchmark CI work.
Validation: all 29 helper tests pass, including fixture directories, artifact portability, incomplete results, and threshold boundaries. Clippy, formatting, YAML parsing, and an independent workflow review passed. Full GitHub CI passed, including all six test shards, lint, PHP lint, WebAssembly, scaling, memory, and the four paired benchmark shards and comparison.
Closes #489