Skip to content

PHOENIX-8069 Move PR precommit and branch builds from Jenkins to GitHub Actions - #2674

Merged
virajjasani merged 16 commits into
apache:masterfrom
virajjasani:PHOENIX-8069-master
Nov 19, 2026
Merged

virajjasani merged 16 commits into
apache:masterfrom
virajjasani:PHOENIX-8069-master

Conversation

@virajjasani

@virajjasani virajjasani commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

JIRA: PHOENIX-8069

Why

Phoenix PR precommit (Jenkinsfile.github) and branch builds (Jenkinsfile) run on ci-hadoop.apache.org, which has not been reachable for about a week. This PR moves both builds to GitHub Actions. The Jenkins limits and test grouping are kept, and some limits are made stricter.

What

  • .github/workflows/phoenix-precommit.yml replaces Jenkinsfile.github. It runs on pull_request (not pull_request_target, which ASF disables from November 2026). It runs Yetus 0.13.0, the same release Jenkins used, as 12 jobs:
    • one general check (all Yetus plugins except unit)
    • 10 unit checks, one per test group or slice
    • one test coverage check, after the unit checks
  • .github/workflows/phoenix-branch-build.yml replaces Jenkinsfile. It runs on every push to master and 5.*, and can be started by hand. It builds the HBase 2.5 and 2.6 profiles, 10 jobs per profile, and rebuilds HBase first as dev/rebuild_hbase.sh does today.
  • dev/phoenix-test-group-args.sh (new) prints the Maven arguments that limit a build to one test group.
  • dev/jenkins_precommit_github_yetus.sh and dev/phoenix-personality.sh still work on Jenkins and now also run under GitHub Actions. They pass the test group and the fork counts to Maven.
  • dev/yetus_console_to_md.py is copied from HBase, with pylint fixes, and failed tests are taken from the per-test lines of the Maven output. It writes the Yetus result to the job summary.
  • dev/phoenix-list-tests.sh, dev/phoenix-test-progress.sh and dev/phoenix_check_tests.py (new) list the tests run, show the test progress, and check the test coverage.
  • .github/dependabot.yml keeps the actions up to date, as the ASF GitHub Actions policy requires.

Test split

A GitHub-hosted job can run for at most 6 hours. Jenkins ran every test in one job with an 11 hour timeout. The tests are split by the existing groups in the root pom.xml, and the two large groups are split again by the first letter of the test class name:

Group Jobs
unit tests (surefire) 1
ParallelStatsEnabledTest 1
ParallelStatsDisabledTest 4 (A-C, D-I, J-R, S-Z)
NeedsOwnMiniClusterTest 4 (A-C, D-K, L-P, Q-Z)

Together the jobs run the same tests as the single Jenkins job, with no test left out or run twice.

Limits

Limit Jenkins GitHub Actions
Artifact retention PR: 3 days, branch: 5 days 2 days
Concurrent jobs per workflow 'Hadoop' executors at most 6 (ASF recommended limit: 15)
Concurrent builds per PR 1 1
PR builds per 24 hours 2, manual builds not counted 2, manual re-runs not counted
Concurrent builds per branch not limited 1
HBase rebuild timeout 30 minutes 30 minutes
Job timeout 11 hours 6 hours (GitHub-hosted runner maximum)

Differences from Jenkins

  • Third PR build in 24 hours: Jenkins queued it. GitHub Actions cannot queue a run, so the run fails with a message to re-run it from the GitHub UI. It fails rather than passes so that the PR never shows a passing check for a commit that was not built.
  • Test forks: GitHub-hosted runners have 4 CPUs and 16 GB of memory. The jobs use 4 unit test forks and 3 integration test forks (pom.xml defaults: 8 and 7). The Yetus docker memory limit is 14g (Jenkins: 20g).
  • Yetus commit statuses: not written. Fork PRs get a read-only token, and 12 jobs would overwrite each other's statuses. Each job's check shows its result, and the Yetus report is in the job summary and the uploaded artifact.
  • Build emails: Jenkins emailed each branch build result to commits@. That is not replaced in this PR.

Test checks and reporting

  • Test class names: General check fails if an IT class in a test group cannot be picked up by any unit check job, because its name does not end in IT or does not start with a letter A-Z.
  • Test coverage: a job after the unit checks collects the classes each job ran, and fails if a test class that Maven should run ran in no job or in more than one job.
  • Console output: each unit check prints the start and result of every test class while the tests run, and the list of test classes run when they finish.
  • Job summary: each unit check shows the exact Maven command Yetus ran, and lists every failed test by its full name.

Test fixes

  • SaltedTableWithParallelStatsEnabledIT had the wrong test category, so no execution ran it.
  • PreMatureTimelyAbortScanIt was renamed to PreMatureTimelyAbortScanIT. Failsafe only picks up classes ending in IT.

IT classes that are never run

These IT classes have no test group @Category on the class or a parent class, so Maven has never run them, on Jenkins or here. This PR leaves them as they are. The test class name check lists them as a warning.

  • org.apache.phoenix.end2end.DynamicColumnWildcardIT
  • org.apache.phoenix.end2end.GlobalConnectionTenantTableIT
  • org.apache.phoenix.end2end.IndexScrutinyWithMaxLookbackIT
  • org.apache.phoenix.end2end.MixedCaseInnerJoinAliasesIT
  • org.apache.phoenix.end2end.NonParameterizedIndexScrutinyToolIT
  • org.apache.phoenix.end2end.SyncUpdateCacheFreqIT
  • org.apache.phoenix.end2end.SystemTablesUpgradeIT
  • org.apache.phoenix.end2end.UpgradeIndexToolTablesIT
  • org.apache.phoenix.end2end.UpgradeIndexToolTablesNameSpaceMappingDisabledIT
  • org.apache.phoenix.end2end.index.IndexTwoPhaseCreateIT
  • org.apache.phoenix.end2end.index.LocalIndexIT
  • org.apache.phoenix.end2end.transform.TransformIT
  • org.apache.phoenix.end2end.transform.TransformMonitorIT
  • org.apache.phoenix.end2end.transform.TransformToolIT

Test runs on a fork

The workflows were run on a fork before this PR. The unit tests and the ParallelStatsEnabledTest and ParallelStatsDisabledTest slices took 7 to 36 minutes each on GitHub-hosted runners. One ParallelStatsDisabledTest slice failed on a flaky UpsertSelectIT.testParallelUpsertSelect connection leak assertion, which is not related to this PR. The NeedsOwnMiniClusterTest slices were cancelled before they finished.

Not yet measured

  • How long the NeedsOwnMiniClusterTest slices take, and the total build time with at most 6 jobs at the same time. The slices may need adjusting after the first runs.
  • Total runner minutes against the ASF budget.

Out of scope

  • Removing the Jenkinsfiles. They stay until the GitHub Actions results match Jenkins.
  • Jenkinsfile.yetus (patches attached to JIRA).
  • Backporting the workflows to the 5.x branches.

…ub Actions

Add GitHub Actions workflows that replace the ci-hadoop Jenkins jobs:
* phoenix-precommit.yml replaces Jenkinsfile.github. It runs Yetus 0.13.0 (the
  same release) as a general check plus one unit check per test group.
* phoenix-branch-build.yml replaces Jenkinsfile. It builds the HBase 2.5 and
  2.6 profiles on every push to master and 5.* branches.

A GitHub-hosted job can run for at most 6 hours, so the tests are split by the
existing failsafe groups (ParallelStatsEnabledTest, ParallelStatsDisabledTest,
NeedsOwnMiniClusterTest) and by IT class name ranges, see
dev/phoenix-test-group-args.sh. Together the jobs run the same tests as the
single Jenkins job.

Jenkins limits are kept, or made stricter:
* Build artifacts are kept for 2 days.
* At most 15 jobs run at the same time (ASF recommended limit).
* One build at a time per PR, and at most 2 PR builds in 24 hours. A manual
  re-run is not counted.
* 30 minute HBase rebuild timeout.

Also add Dependabot for GitHub Actions, as the ASF GitHub Actions policy
requires.
…bledIT

The class was tagged @category(ParallelStatsEnabledIT.class) instead of
ParallelStatsEnabledTest, so no failsafe execution selected it and it never ran.
…AbortIT

Failsafe only runs test classes whose name ends in IT, so this test
never ran. The new name differs from the old one by more than case, as
Yetus rejects a patch that renames a file by changing only its case.
The test fails, so comment out its test group @category. Maven then
does not run it, as before it was renamed to end in IT.
…name

The Results section of the Maven output names a failure in a helper or
@after method by the helper, for example
UpsertSelectIT.assertNoConnLeak:114->waitForOpenConnectionsToSettle:129,
which the summary did not recognise as a test, so the Failed Tests table
was missing. Take failures and errors from the per-test lines instead.
The tests are split across several unit check jobs by test group and
class name. Add two checks so that no test class is silently left out:

* General check: before Yetus, fail if an IT class in a test group
  cannot be selected by any job, because its name does not end in IT or
  does not start with a letter A-Z. IT classes with no test group
  @category are listed as a warning, as Maven never runs them.
* Test coverage: a job after the unit checks that collects the classes
  each job ran, and fails if a test class that Maven should run ran in
  no job or in more than one job.
Yetus writes the Maven test output to patch-unit-<module>.txt, so the
console shows nothing while the tests run, which can be hours. Follow
that file while Yetus runs and print the start and the result of each
test class, the failed tests and the build result, with the time.
Up to 15 jobs at the same time is the ASF recommended maximum, not a
target. Run at most 6, so that the Phoenix builds leave runners for the
other ASF projects. The test groups that do not fit wait for a free
slot, so a build takes longer but runs the same tests.
Only the newest commit of a PR needs a build, so a new commit now
cancels the running build of the PR instead of waiting for it. A
cancelled build still counts against the limit of 2 builds per PR in
24 hours.
…erage summary

The test coverage check only listed the classes that should have run
but did not. List every concrete test class that ran in no job, grouped
by the reason: a class-level @ignore, no test group @category, a name
that no job selects, or none of these, which fails the check as before.
… in CI

Each NeedsOwnMiniClusterTest class runs in its own forked JVM, which
failsafe kills after 1800 seconds. On the 4 CPU GitHub-hosted runners
some classes take longer, and the build then fails although all tests
pass: BackwardCompatibilityIT took 5977 seconds, and
GlobalIndexCheckerEventualGenerateIT and GlobalIndexCheckerEventualIT
took 2648 and 2295 seconds.

Make the timeout a property, forkedProcessTimeoutIT, that stays 1800
seconds by default, and set it to 7200 seconds in the GitHub Actions
precommit and branch builds.
A new commit cancels the running build of the PR, and the cancelled
build still counts against the limit, so 2 builds in 24 hours (the
Jenkins limit) is too few. Allow 4.
GitHub Actions now runs the PR precommit and the branch builds, so
remove Jenkinsfile.github, Jenkinsfile and the Docker image that only
the branch build used. Jenkinsfile.yetus, which tests patches attached
to JIRA, is kept.
@apache apache deleted a comment from stoty Nov 19, 2026
@virajjasani
virajjasani requested a review from tkhurana November 19, 2026 04:10
@tkhurana

tkhurana commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

The Test coverage check failed
ERROR: these test classes ran in more than one job: org.apache.phoenix.end2end.SortOrderWithRegionMoves2IT (NeedsOwnMiniClusterTest-Q-Z.txt, ParallelStatsDisabledTest-S-Z.txt) org.apache.phoenix.end2end.SortOrderWithRegionMovesIT (NeedsOwnMiniClusterTest-Q-Z.txt, ParallelStatsDisabledTest-S-Z.txt)
because of the duplicates: SortOrderWithRegionMovesIT and SortOrderWithRegionMoves2IT are @category(NeedsOwnMiniClusterTest). They extend SortOrderIT, which is @category(ParallelStatsDisabledTest).
This is not a regression but it is flagged now in the new checks.

@virajjasani

Copy link
Copy Markdown
Contributor Author

The Test coverage check failed ERROR: these test classes ran in more than one job: org.apache.phoenix.end2end.SortOrderWithRegionMoves2IT (NeedsOwnMiniClusterTest-Q-Z.txt, ParallelStatsDisabledTest-S-Z.txt) org.apache.phoenix.end2end.SortOrderWithRegionMovesIT (NeedsOwnMiniClusterTest-Q-Z.txt, ParallelStatsDisabledTest-S-Z.txt) because of the duplicates: SortOrderWithRegionMovesIT and SortOrderWithRegionMoves2IT are @category(NeedsOwnMiniClusterTest). They extend SortOrderIT, which is @category(ParallelStatsDisabledTest). This is not a regression but it is flagged now in the new checks.

+1. To catch similar issues e.g. duplicate runs of tests, missing runs of tests etc, I have introduced additional coverage step.

@virajjasani

Copy link
Copy Markdown
Contributor Author
Test classes run: 801 in 10 jobs, of 804 test classes in the source tree.

ERROR: these test classes ran in more than one job:
  org.apache.phoenix.end2end.SortOrderWithRegionMoves2IT (NeedsOwnMiniClusterTest-Q-Z.txt, ParallelStatsDisabledTest-S-Z.txt)
  org.apache.phoenix.end2end.SortOrderWithRegionMovesIT (NeedsOwnMiniClusterTest-Q-Z.txt, ParallelStatsDisabledTest-S-Z.txt)

Test classes that did not run in any job: 18.

Not run, because Maven never runs it: no @Category with one of NeedsOwnMiniClusterTest, ParallelStatsDisabledTest, ParallelStatsEnabledTest:
  org.apache.phoenix.end2end.DynamicColumnWildcardIT
  org.apache.phoenix.end2end.GlobalConnectionTenantTableIT
  org.apache.phoenix.end2end.IndexScrutinyWithMaxLookbackIT
  org.apache.phoenix.end2end.MixedCaseInnerJoinAliasesIT
  org.apache.phoenix.end2end.NonParameterizedIndexScrutinyToolIT
  org.apache.phoenix.end2end.PrematureScanTimelyAbortIT
  org.apache.phoenix.end2end.SyncUpdateCacheFreqIT
  org.apache.phoenix.end2end.SystemTablesUpgradeIT
  org.apache.phoenix.end2end.UpgradeIndexToolTablesIT
  org.apache.phoenix.end2end.UpgradeIndexToolTablesNameSpaceMappingDisabledIT
  org.apache.phoenix.end2end.index.IndexTwoPhaseCreateIT
  org.apache.phoenix.end2end.index.LocalIndexIT
  org.apache.phoenix.end2end.transform.TransformIT
  org.apache.phoenix.end2end.transform.TransformMonitorIT
  org.apache.phoenix.end2end.transform.TransformToolIT

Not run, although it should have run:
  org.apache.phoenix.pherf.SchemaReaderIT
  org.apache.phoenix.pherf.workload.mt.TenantTableOperationWorkloadIT
  org.apache.phoenix.pherf.workload.mt.TenantViewOperationWorkloadIT

WARNING: the build failed in NeedsOwnMiniClusterTest-Q-Z.txt, ParallelStatsDisabledTest-D-I.txt, so Maven may have skipped the 3 test classes that should have run.

@virajjasani
virajjasani merged commit 33ccebc into apache:master Nov 19, 2026
1 check passed
virajjasani added a commit that referenced this pull request Nov 19, 2026
virajjasani added a commit that referenced this pull request Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants