ort-ci

ONNX Runtime CI Management

Workflows for triggering, re-running, and unblocking CI checks on an ONNX Runtime PR. The repository is microsoft/onnxruntime. As of 2026-07, nearly all CI runs as GitHub Actions workflows (~80+ checks per PR). Only one required check still runs on Azure Pipelines — Linux Android Emulator QNN CI Pipeline (host aiinfra.visualstudio.com). There is also the bot-driven license/cla status check.

Failures are not all the same. Before touching anything, diagnose each failure (see Triage: Diagnose Before Re-running): most failures need a code change and re-running them just fails again; only genuinely transient (network/disk) failures should be re-run via §1. §4 (Azure Pipelines) applies only to the single QNN pipeline.

Before doing anything, inspect current state so you do not queue duplicate runs.

If you make any change, commit and push it, then stop. A push updates the PR head SHA and automatically starts CI for the new commit. Do not manually re-run failures from the old SHA after pushing a fix; that only queues redundant runs against stale code. Manual re-runs are only for transient failures when the PR head has not changed.

Classify a check's provider

GitHub Actions checks have a non-empty workflowName and a detailsUrl on github.com; the Azure Pipelines check has an empty workflowName and a detailsUrl on aiinfra.visualstudio.com:

gh pr view <number> --repo microsoft/onnxruntime --json statusCheckRollup \
  --jq '[.statusCheckRollup[] | {name, host:(.detailsUrl|split("/")[2])}]
        | group_by(.host) | map({host:.[0].host, count:length})'

Gather Context

# PR metadata + all checks grouped by state
gh pr view <number> --repo microsoft/onnxruntime \
  --json number,title,url,state,isDraft,headRefName,headRefOid,baseRefName,statusCheckRollup

# Head / merge SHAs for external CI
gh api repos/microsoft/onnxruntime/pulls/<number> --jq '{head:.head.sha, merge:.merge_commit_sha}'

Inspect statusCheckRollup and note, for each requested check, whether it is missing, queued, in progress, failed, canceled, skipped, or already successful. Do not re-trigger a check that is already queued/in_progress/SUCCESS unless the user explicitly asks.

Quickly list just the failed/pending checks:

gh pr view <number> --repo microsoft/onnxruntime --json statusCheckRollup \
  --jq '.statusCheckRollup[]
        | {name:(.name//.context), status:(.status//.state), conclusion:.conclusion}
        | select(.conclusion!="SUCCESS" and .status!="SUCCESS")'

Triage: Diagnose Before Re-running

Never blindly re-run failed CI. Most failures need a code change and will fail again identically on re-run. Only transient failures should be re-run. The process is: download the failed job's log, read the actual error, classify it, then fix or re-run case by case.

Step 1 — Download the failed log

For a GitHub Actions check (the vast majority), get the run and read only the failed steps:

HEAD_SHA=$(gh api repos/microsoft/onnxruntime/pulls/<number> --jq .head.sha)

# Map failed checks to their workflow run IDs
gh run list --repo microsoft/onnxruntime --commit "$HEAD_SHA" --limit 100 \
  --json databaseId,workflowName,status,conclusion,url \
  --jq '.[] | select(.conclusion=="failure" or .conclusion=="cancelled")'  # cspell:ignore cancelled -- literal GitHub API value

# Dump just the failed steps of a run (grep for the real error)
gh run view <run_id> --repo microsoft/onnxruntime --log-failed > /tmp/ci_<run_id>.log
grep -nE "error:|FAILED|warning:|Traceback|fatal error|No space left|Could not resolve|timed out" \
  /tmp/ci_<run_id>.log | head -50

For the Azure Pipelines QNN check, open its detailsUrl (a dev.azure.com / aiinfra.visualstudio.com build page) and download the job log, or use the Azure DevOps .../builds/<buildId>/timeline + log APIs (see the ci-failure-retrieval skill for the exact requests).

Step 2 — Classify the failure and act

Rules of thumb:

• A compile error: or a [ FAILED ]/FAILED line means fix the code — re-running reruns the same failing commit and fails identically.
• Only re-run when the log shows a network/disk/agent problem and no compile or assertion error.
• When unsure, download the log and read it; do not guess from the check name alone.
• After any code, test, lint, or generated-file fix, commit and push it. CI starts automatically for the new head SHA. Do not use §1 after pushing a change; the failed runs belong to the old SHA and re-running them would test stale code.

1. Re-run Failed GitHub Actions (transient failures only)

The repo ships a helper that re-runs only the GitHub Actions workflows whose latest run for the PR's current head commit failed/canceled — and skips any workflow that already has a newer run queued or in progress. Use it only after triage confirms the failures are transient (network/disk/agent) — see Triage. It is the safest way to retry those without piling on duplicates. Do not use this helper if you changed anything and pushed a new commit; the push already starts CI for the new head SHA.

Script: tools/scripts/rerun_failed_ci.sh

# Dry run first — shows what would be re-run, triggers nothing
./tools/scripts/rerun_failed_ci.sh <number> --dry-run

# Actually re-run the failed/canceled workflows for the PR's head commit
./tools/scripts/rerun_failed_ci.sh <number>

# Explicit repo (auto-detected from cwd when omitted)
./tools/scripts/rerun_failed_ci.sh <number> microsoft/onnxruntime

It prefers gh run rerun <id> --failed (retry only failed jobs) and falls back to a full rerun for fully canceled runs that have no discrete failed jobs. Requires an authenticated gh. Always run --dry-run first and confirm the list looks right before the real run.

To re-run one specific workflow manually:

gh run list --repo microsoft/onnxruntime --commit <head_sha> --limit 100 \
  --json databaseId,workflowName,status,conclusion,url
gh run rerun <run_id> --repo microsoft/onnxruntime --failed   # only failed jobs
gh run rerun <run_id> --repo microsoft/onnxruntime            # full rerun

2. Unblock license/cla (CLA bot)

The license/cla check is posted by Microsoft's CLA bot, independent of the CI pipelines. When it is stuck as "Expected — Waiting for status to be reported", re-trigger only the bot — no CI jobs are re-run — by posting this comment on the PR:

gh pr comment <number> --repo microsoft/onnxruntime \
  --body "@microsoft-github-policy-service rerun"

Then verify it flips to success:

gh pr view <number> --repo microsoft/onnxruntime --json statusCheckRollup \
  --jq '.statusCheckRollup[] | select((.name//.context)=="license/cla")
        | {status, conclusion}'

Expect {"status":"COMPLETED","conclusion":"SUCCESS"}.

3. Fix the "Python format" required check

The required Python format check (job lint-python-format in .github/workflows/lint.yml) runs lintrunner --all-files and fails on any formatting/lint violation. Re-running it will not help — you must fix the code, commit, and push. See docs/Coding_Conventions_and_Standards.md and the ort-lint skill.

# One-time setup (in an activated Python venv)
pip install -r requirements-lintrunner.txt
lintrunner init

# Auto-fix. Prefer changed files; use --all-files to match CI exactly.
lintrunner -a                    # changed files only
lintrunner -a --all-files        # everything (what CI checks)

# Verify clean (no changes reported == pass)
lintrunner --all-files

Then commit and push the formatting fixes; the check re-runs automatically on the new commit:

git add -u && git commit -m "Fix lint" && git push

Notes:

• CI runs lintrunner --all-files, so a local lintrunner -a on only changed files can miss a pre-existing violation the CI reports. If the check still fails, run --all-files locally.
• The same job also covers C++ clang-format and other adapters; the fix is the same (lintrunner -a).

4. Trigger the Azure Pipelines check (QNN Android Emulator only)

As of 2026-07, the only ORT check still on Azure Pipelines is Linux Android Emulator QNN CI Pipeline. Everything else is GitHub Actions (use §1). Trigger it through the PR comment integration:

gh pr comment <number> --repo microsoft/onnxruntime \
  --body "/azp run Linux Android Emulator QNN CI Pipeline"

Then wait briefly and check for a reply from azure-pipelines[bot]:

# Note: through the GraphQL `comments` field (what `gh pr view --json comments`
# uses), the bot's author.login is `azure-pipelines` (no `[bot]` suffix), even
# though it surfaces as azure-pipelines[bot] in the UI and the REST API.
gh pr view <number> --repo microsoft/onnxruntime --json comments \
  --jq '.comments[] | select(.author.login=="azure-pipelines")
        | {createdAt, body}' | tail

• If the bot replies "No pipelines are associated with this pull request", the pipeline is not wired to the comment app — use the direct Azure DevOps API fallback (see the Trigger CI Pipelines section of the private gh-pr-management skill for the dev.azure.com project/definition discovery and POST .../runs payload using refName: refs/pull/<number>/merge and the PR merge_commit_sha).
• Keep it to one batch /azp run comment per attempt; do not spam repeated comments.

5. Fix the "Windows GPU Doc Gen CI" check (operator docs out of date)

The ONNX Runtime Windows GPU Doc Gen CI check (workflow .github/workflows/windows_gpu_doc_gen.yml) builds ORT and runs build.py --gen_doc validate. It fails when the generated operator docs no longer match what's committed — typically after you add/modify an operator or its kernel registrations but forget to regenerate docs/ContribOperators.md / docs/OperatorKernels.md. Re-running will not help; you must update the docs.

The easiest fix is to download the regenerated docs the failed job already produced: on failure the workflow uploads a single artifact named updated-docs that contains both OperatorKernels.md and ContribOperators.md at its top level, so you can replace the committed copies without building locally.

HEAD_SHA=$(gh api repos/microsoft/onnxruntime/pulls/<number> --jq .head.sha)

# Find the failed Doc Gen run
run_id=$(gh run list --repo microsoft/onnxruntime --commit "$HEAD_SHA" --limit 100 \
  --json databaseId,workflowName,conclusion \
  --jq '.[] | select(.workflowName|test("Doc Gen")) | select(.conclusion=="failure") | .databaseId' | head -1)

# Confirm the artifact is present (expect: updated-docs)
gh api repos/microsoft/onnxruntime/actions/runs/$run_id/artifacts --jq '.artifacts[].name'

# gh run download does not overwrite existing files. Remove both stale generated docs first,
# then extract their replacements (the artifact always holds both files).
rm docs/OperatorKernels.md docs/ContribOperators.md
gh run download "$run_id" --repo microsoft/onnxruntime -n updated-docs --dir docs/

Then review, commit, and push — the check re-runs on the new commit:

git diff --stat docs/ContribOperators.md docs/OperatorKernels.md
git add docs/ContribOperators.md docs/OperatorKernels.md
git commit -m "Update operator docs" && git push

Notes:

• The updated-docs artifact always contains both OperatorKernels.md and ContribOperators.md at its top level. Remove both committed files before downloading because gh run download refuses to overwrite them; --dir docs/ then recreates both. Only the file(s) whose generated content changed will show up in git diff after extraction.
• Equivalent local alternative: python tools/ci_build/build.py --config Release --build_dir build/Linux --gen_doc after a build, then commit the regenerated files. Downloading the artifact is faster since it avoids a full build.

Special-Case Check Handling

Safety Rules

• Never trigger release, publish, official, nightly, signing (ESRP), deployment, or package-upload pipelines unless the user explicitly asks for that class of pipeline. Treat names containing release, publish, official, nightly, sign, ESRP, production, or deploy as high-risk and confirm first.
• Always --dry-run the rerun script before the real run.
• Do not re-run green checks or dispatch unrelated workflows.
• Prefer PR merge refs (refs/pull/<number>/merge) over branch refs for external CI so the run validates the merge result.
• Do not claim success until the provider returns a queued/in-progress/completed run ID or URL, or the check flips state in statusCheckRollup.

Verify After Triggering

gh pr view <number> --repo microsoft/onnxruntime --json statusCheckRollup \
  --jq '.statusCheckRollup | group_by(.status//.state)
        | map({state:(.[0].status//.[0].state), count:length})'

Confirm the previously stuck/failed check moved to queued/in_progress (or SUCCESS for the CLA bot), and that no duplicate runs were created for the same head SHA.