Skip to content

ci: run PR risk assessment after Droid Auto Review, approving up to low risk - #112

Merged
markattar-factory merged 1 commit into
mainfrom
droid/risk-install
Nov 20, 2026
Merged

markattar-factory merged 1 commit into
mainfrom
droid/risk-install

Conversation

@factory-droid

@factory-droid factory-droid Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds Factory's PR risk assessment to Droid Auto Review in this repository, with approvals capped at Low. Tracking: VAL-528 (rollout), VAL-581 (review.yaml rule), VAL-589 (concurrency).

  • .github/workflows/droid-review.yml: the same job now runs the risk step (Factory-AI/droid-action/risk@dev-internal) after the review, so the assessment sees how the review ended and the threads it left. Factory decides whether each run skips, holds, or judges, and writes the risk comment and any approval as the Droid GitHub App. The workflow holds no credential that can approve.
    • Adds synchronize and a workflow_dispatch with pr_number. Pushes run only the risk step, which reassesses the new head once every Droid review thread is resolved.
    • Moves the draft check from the job to the review step, so drafts run only the risk step, which withdraws approvals of older commits.
    • The review stays on Factory-AI/droid-action@main with the same inputs.
    • Adds persist-credentials: false on checkout and timeout-minutes: 60 (the review can take 30 minutes and the judge 20 more).
  • .factory/review.yaml (new): one rule, risk_assessment.approve_risk_level: low. The Droid App may approve a pull request it rates minimal or low; medium and high wait for a human. Factory reads the file from the default branch, never from the pull request.

Concurrency

There is one run per pull request, and cancel-in-progress is ${{ github.event.action != 'synchronize' }}.

  • A review event cancels the run in progress, because its own review and risk step cover the new head.
  • A push never cancels. It waits for the run in progress, so a running review posts its threads before the push's risk step reads them, and a newer push replaces one that is still waiting.

Without this, a push during a review cancelled the review. The push run doesn't review, so the pull request would get no review, and its risk step could judge with no threads to hold on. Dispatch and draft runs have their own groups.

Forks (this repository is public)

  • The workflow uses pull_request, not pull_request_target, and no workflow in this repository uses pull_request_target. A run for a fork's pull request therefore gets no secrets (FACTORY_API_KEY is empty) and no OIDC token, so it cannot ask Factory for anything. It also waits for a maintainer to approve the run first: chore(cli): document local marketplace naming #87's run shows action_required.
  • The risk step skips fork pull request runs, because without an OIDC token it would only fail the job.
  • A maintainer's gh workflow run droid-review.yml -f pr_number=<fork PR> runs in this repository with its secrets. Factory's begin then skips the pull request as a fork, so it never judges or approves fork code.

Bot pull requests

  • sync-jp-docs.yml opens and updates its PR with the workflow GITHUB_TOKEN. GitHub starts no pull_request workflows for events caused by that token, so JP sync PRs get no review and no risk comment. A dispatch with pr_number still assesses one by hand. Its last three runs failed, and its last successful run was Oct 23.
  • Some PRs here are opened by the Droid App (app/factory-droid, for example docs(reference): add Droid Execute API reference #93 and docs(reference): add AskUser tool reference #83). The App cannot approve a pull request it opened, so those always need a human approval.

Needed before approvals happen

  • risk_assessment_auto_approve (Statsig) must pass for this repository and the pull request's author. Until it does, assessments comment and never approve.
  • Factory applies the Low cap only while its level-based decision stands (the risk_assessment_level_approval gate on, or a judge that reports a level and no verdict). Otherwise the judge's verdict decides.
  • There are no rulesets and no CODEOWNERS file. Recent merges went in without approvals, so an approval here may be advisory rather than required; classic branch protection isn't readable with the App token.

Testing

  • actionlint: clean on droid-review.yml.
  • .factory/review.yaml checked with Factory's rules reader: status ok, controls riskAssessmentApprovePr: true, riskAssessmentApproveRiskLevel: low.
  • This edits the workflow file, so on this PR the review skips ("workflow file differs from the version on the default branch") and Factory refuses the risk report (403 workflow_not_trusted); the job passes. After merge, run gh workflow run droid-review.yml --ref main -f pr_number=<an open PR>. It should post a risk comment, and an approval only at minimal or low.

Follow-up: Mark asked for the cap to be Minimal instead of Low after this merged; that change is in a separate pull request.

…ow risk

Add Factory's risk step to droid-review.yml and .factory/review.yaml with
approve_risk_level: low. Pushes, drafts and a workflow_dispatch with
pr_number run only the risk step, and a push waits behind a running review
instead of cancelling it (VAL-589).

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@markattar-factory
markattar-factory merged commit ddd7d36 into main Nov 20, 2026
2 checks passed
@markattar-factory
markattar-factory deleted the droid/risk-install branch November 20, 2026 07:55
@factory-droid factory-droid Bot changed the title ci: run PR risk assessment after Droid Auto Review, approving up to low risk ci: run PR risk assessment after Droid Auto Review, approving up to minimal risk Nov 20, 2026
@factory-droid factory-droid Bot changed the title ci: run PR risk assessment after Droid Auto Review, approving up to minimal risk ci: run PR risk assessment after Droid Auto Review, approving up to low risk Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant