openvmm-ci-investigation

Investigating CI Failures

When a PR has failing CI checks, always start by running the investigation script. Do not manually query the GitHub API or download artifacts by hand — the script handles all of that automatically.

Step 1: Run the Script

# Investigate the current branch's PR (no argument needed)
python3 repo_support/investigate_ci.py

# Investigate a specific PR by number
python3 repo_support/investigate_ci.py 2946

# Or by run ID directly
python3 repo_support/investigate_ci.py 23017249697

The script automatically:

1. Resolves the PR to every failed CI run for its head commit. A commit can have multiple workflows (e.g. "OpenVMM PR" and "OpenVMM Docs PR"), and each failing one is investigated in turn — so don't assume a single run covers everything. Only the newest run of each workflow is used, so superseded runs are ignored; a NOTE: <workflow> is still in_progress line means results are incomplete and more failures may appear later.
2. Identifies failed jobs
3. Downloads *-unit-tests-junit-xml artifacts and parses JUnit XML for unit test failures
4. Downloads *-vmm-tests-logs artifacts if they exist
5. Finds tests with petri.failed markers and extracts ERROR/WARN lines
6. If no test or JUnit artifacts exist (build/fmt/clippy failure), shows the tail of the failed job's log

Step 2: Analyze the Script Output

Read the script's output to identify:

• Which tests failed (unit tests and/or VMM tests)
• Error messages and root causes
• Whether it's a build/fmt/clippy failure vs. a test failure

When more than one run failed, the output is split by # RUN n of m banners. Check all of them — a docs failure and a build failure are reported in separate runs.

Then use the information to diagnose the issue and suggest fixes.

Step 3: Diagnose from Logs and Code

Always try to diagnose the failure from CI logs, test code, and error messages first. Most failures can be understood without local reproduction. Read the relevant test source, trace the error through the code, and form a hypothesis before considering local repro.

Step 4: Reproduce Locally (Only If Needed)

If you cannot diagnose the failure from logs alone, ask the user whether they want to attempt local reproduction before proceeding. Do not automatically start building or running tests on the user's machine.

If the user agrees, check whether the failing platform matches the host architecture — you can only reproduce tests locally on the same arch. If it doesn't match, explain this to the user and continue diagnosing from CI logs and test code.

To reproduce locally, load the vmm-tests skill for instructions on running with cargo xflowey vmm-tests-run. Do not use cargo nextest run -p vmm_tests directly — it won't have the required artifacts.

Reference: Manual Commands

Only use these if the script fails or you need to dig deeper into a specific artifact. In normal usage, the script above is sufficient.

If the script isn't available or you need more control, follow these steps:

1. Find the failing run

# Get the run ID for a PR
gh pr checks <PR_NUMBER> -R microsoft/openvmm
# Or list runs for a specific commit
gh run list -R microsoft/openvmm --commit <SHA>

2. Identify the failing job

gh run view <RUN_ID> -R microsoft/openvmm --json jobs \
  -q '[.jobs[] | select(.conclusion == "failure") | {name, databaseId}]'

3. Download test artifacts

Unit test results are stored in JUnit XML artifacts named {platform}-unit-tests-junit-xml. Known platforms include:

• x64-linux
• aarch64-linux
• aarch64-linux-musl

# Download unit test JUnit XML for a platform
gh run download <RUN_ID> -R microsoft/openvmm \
  -n aarch64-linux-unit-tests-junit-xml -D /tmp/junit-xml
# Parse failures from the XML
python3 -c "
import xml.etree.ElementTree as ET, sys
for f in __import__('pathlib').Path(sys.argv[1]).rglob('*.xml'):
    for tc in ET.parse(f).iter('testcase'):
        fail = tc.find('failure')
        if fail is None:
            fail = tc.find('error')
        if fail is not None:
            print(f'FAIL: {tc.get(\"classname\",\"\")}::{tc.get(\"name\",\"\")}')
            print(f'  {fail.get(\"message\",\"\")[:200]}')
" /tmp/junit-xml

VMM test results are stored in artifacts named {platform}-vmm-tests-logs. The known platforms are:

• x64-windows-intel
• x64-windows-intel-tdx
• x64-windows-amd
• x64-windows-amd-snp
• x64-linux
• aarch64-windows

# Download a specific platform's test logs
gh run download <RUN_ID> -R microsoft/openvmm \
  -n x64-windows-amd-snp-vmm-tests-logs -D /tmp/test-logs

4. Find failed tests

Each test gets its own directory inside the artifact. Look for petri.failed marker files (passing tests have petri.passed instead):

find /tmp/test-logs -name "petri.failed"

The petri.failed file contains the error petri recorded for the test. The test's real name is in petri.test, which petri writes before the test body runs — so a directory with petri.test but no result marker is a test that was killed or crashed before it could report.

5. Extract errors from petri.jsonl

The petri.jsonl file in each test directory is the primary structured log. Each line is a JSON object with fields: timestamp, source, severity, message. Filter for ERROR and WARN severity for a quick diagnosis:

python3 -c "
import json, sys
for line in open(sys.argv[1]):
    try:
        e = json.loads(line.strip())
        if e.get('severity') in ('ERROR', 'WARN'):
            print(f'[{e[\"severity\"]}] {e.get(\"source\",\"?\")}: {e.get(\"message\",\"\").strip()[:200]}')
    except: pass
" /tmp/test-logs/<test-dir>/petri.jsonl

Artifact Contents

Unit test JUnit XML

Artifacts named {platform}-unit-tests-junit-xml contain JUnit XML files with <testcase> elements. Failed tests have <failure> or <error> children with message attributes describing the failure. These are the primary artifacts for diagnosing unit test / cargo-nextest failures.

VMM test logs (petri)

Each test directory contains:

• petri.jsonl — Structured JSON Lines log (primary file for investigation)
• petri.log — Plain text version of the test log
• petri.test — The test's name, written before the test body runs
• petri.passed or petri.failed — Pass/fail marker; a failure marker holds the error
• openhcl.log — OpenHCL serial console output, if the test exercised OpenHCL
• hyperv.log — Hyper-V event log, if the test exercises the Hyper-V backend
• openvmm.log — OpenVMM serial console output, if the test exercises the OpenVMM backend
• guest.log, uefi.log — Guest OS serial output
• Sometimes: screenshot_*.png — periodic screenshots of the guest
• Sometimes: dumpfile.dmp

Viewing Results in Browser

Test results are uploaded to Azure Blob Storage and viewable at: https://openvmm.dev/test-results/#/runs/<RUN_ID>_<RUN_ATTEMPT>

Logs are uploaded per re-run attempt, so the run key includes the attempt number (1 for the first attempt). investigate_ci.py prints the correct URL in its summary.

Common Failure Patterns

• Unit test failure: A unit tests job failed. The script downloads JUnit XML artifacts and shows the failing test names and messages. Common causes: new test code that relies on OS capabilities not available in CI (e.g. TAP devices, elevated permissions).
• Formatting / house-rules: The quick check [fmt, clippy x64-linux] job failed. No test artifacts will exist. Run cargo xtask fmt --fix locally and check the job log for the specific rule that failed.
• Docs failure: A job in the separate "OpenVMM Docs PR" workflow failed — either build mdbook guide (broken Guide markdown/links; mdbook logs [ERROR] (mdbook::...)) or build and check docs (rustdoc warnings, which are denied in CI). Reproduce with cargo doc --no-deps -p <package>. No test artifacts will exist.
• TripleFault: VM hit a fatal error during boot. Check petri.jsonl for Hyper-V Worker/Chipset errors. Often infrastructure-related, not caused by the PR's code changes.
• Timeout: Test exceeded its time limit. Check if the VM booted at all.
• Guest assertion failure: Guest-side test failed. Check guest.log.
• Build failure: No test artifacts will exist. Check the job log directly with gh run view <RUN_ID> --job <JOB_ID> --log.

Important API Note

The gh run view --json artifacts flag does not exist. To list artifacts for a run, use the GitHub API directly:

gh api repos/microsoft/openvmm/actions/runs/<RUN_ID>/artifacts