Repository navigation
ci: run PR risk assessment after Droid Auto Review, approving up to low risk - #112
Merged
Merged
Conversation
…ow risk Add Factory's risk step to droid-review.yml and .factory/review.yaml with approve_risk_level: low. Pushes, drafts and a workflow_dispatch with pr_number run only the risk step, and a push waits behind a running review instead of cancelling it (VAL-589). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
markattar-factory
approved these changes
Nov 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Factory's PR risk assessment to Droid Auto Review in this repository, with approvals capped at Low. Tracking: VAL-528 (rollout), VAL-581 (review.yaml rule), VAL-589 (concurrency).
.github/workflows/droid-review.yml: the same job now runs the risk step (Factory-AI/droid-action/risk@dev-internal) after the review, so the assessment sees how the review ended and the threads it left. Factory decides whether each run skips, holds, or judges, and writes the risk comment and any approval as the Droid GitHub App. The workflow holds no credential that can approve.synchronizeand aworkflow_dispatchwithpr_number. Pushes run only the risk step, which reassesses the new head once every Droid review thread is resolved.Factory-AI/droid-action@mainwith the same inputs.persist-credentials: falseon checkout andtimeout-minutes: 60(the review can take 30 minutes and the judge 20 more)..factory/review.yaml(new): one rule,risk_assessment.approve_risk_level: low. The Droid App may approve a pull request it rates minimal or low; medium and high wait for a human. Factory reads the file from the default branch, never from the pull request.Concurrency
There is one run per pull request, and
cancel-in-progressis${{ github.event.action != 'synchronize' }}.Without this, a push during a review cancelled the review. The push run doesn't review, so the pull request would get no review, and its risk step could judge with no threads to hold on. Dispatch and draft runs have their own groups.
Forks (this repository is public)
pull_request, notpull_request_target, and no workflow in this repository usespull_request_target. A run for a fork's pull request therefore gets no secrets (FACTORY_API_KEYis empty) and no OIDC token, so it cannot ask Factory for anything. It also waits for a maintainer to approve the run first: chore(cli): document local marketplace naming #87's run showsaction_required.gh workflow run droid-review.yml -f pr_number=<fork PR>runs in this repository with its secrets. Factory's begin then skips the pull request as a fork, so it never judges or approves fork code.Bot pull requests
sync-jp-docs.ymlopens and updates its PR with the workflowGITHUB_TOKEN. GitHub starts nopull_requestworkflows for events caused by that token, so JP sync PRs get no review and no risk comment. A dispatch withpr_numberstill assesses one by hand. Its last three runs failed, and its last successful run was Oct 23.app/factory-droid, for example docs(reference): add Droid Execute API reference #93 and docs(reference): add AskUser tool reference #83). The App cannot approve a pull request it opened, so those always need a human approval.Needed before approvals happen
risk_assessment_auto_approve(Statsig) must pass for this repository and the pull request's author. Until it does, assessments comment and never approve.risk_assessment_level_approvalgate on, or a judge that reports a level and no verdict). Otherwise the judge's verdict decides.Testing
actionlint: clean ondroid-review.yml..factory/review.yamlchecked with Factory's rules reader: statusok, controlsriskAssessmentApprovePr: true,riskAssessmentApproveRiskLevel: low.403 workflow_not_trusted); the job passes. After merge, rungh workflow run droid-review.yml --ref main -f pr_number=<an open PR>. It should post a risk comment, and an approval only at minimal or low.Follow-up: Mark asked for the cap to be Minimal instead of Low after this merged; that change is in a separate pull request.