powertoys-verification

When to use this skill

Use this skill whenever you need to verify PowerToys behavior with the winapp CLI and emit a structured PASS / FAIL / BLOCKED verdict with evidence (UIA enumeration, log line, settings.json diff, screenshot, etc.). It runs in two scenarios that share the same drive techniques, helpers, taxonomy and report format — and differ only on what the checklist is and what bits you test:

Step 0 for every run — pick the scenario. Read references/scenarios/index.md (the router + the "bits under test" contract + the verdict-vocabulary mapping), then read the one matching scenario doc (references/scenarios/{module-checklist,pr-validation}.md). For B, also resolve the bits sub-decision ("is the PR's code in the build under test?") before driving. This SKILL.md is the shared engine (drive techniques, helpers, taxonomy, pitfalls) common to both.

Each item produces a PASS / FAIL / BLOCKED verdict with evidence. For A the checklist is supplied; for B you derive it from the PR(s). The skill is the how — independent of any specific checklist.

Required reads (in order)

1. references/scenarios/index.md — read FIRST: the scenario router (A/B), the "bits under test" contract (installed-and-immutable, vs build-and-sideload for a PR whose code isn't in the build under test), and the verdict-vocabulary mapping. Then read the one matching scenario doc: references/scenarios/module-checklist.md (A) · pr-validation.md (B).
2. references/winapp-ui-testing.md — the prerequisite UIA mechanics doc (winapp ui verbs, scripted batch testing, file pickers, accessibility audits, screenshots, click-vs-invoke, PostMessage, SendInput cb=40, stunted-UIA recovery, settings-mutation safety contract). Read this first — this skill assumes you know its content and only adds PT-specific extensions.
3. This SKILL.md — the shared engine: the 3-bucket drive-technique selector (Step 2), classification taxonomy, critical pitfalls, helper-script catalog.
4. references/modules/<module>.md IF IT EXISTS — per-module entry-paths, item-by-item recipes, common BLOCKED traps, fixture lists, source citations. Always check references/modules/ first. If no profile exists, fall back to this SKILL.md and create one after you finish (template in references/modules/README.md).
5. references/explorer-context-menu-flow.md IF your module registers an Explorer right-click entry (PowerRename, File Locksmith, Image Resizer, New+, Preview Pane, RegistryPreview) — shared synthetic-right-click + UIA-invoke + multi-file-selection flow + module-caption table. Helper: scripts/pt-explorer-contextmenu.ps1.
6. references/pre-flight.md — pre-flight checks, bootstrap snippet, state-hygiene cleanup, final wrap-up, hard rules.
7. references/reporting-format.md — per-item table template, top-of-report summary, step-table rules, anti-patterns, worked example.
8. references/environment-setup.md — RDP/sleep/screensaver/session-attachment gotchas. Cite in BLK-ENV verdicts.
9. references/release-checklist/<module>.md — SCENARIO A ONLY — the supplied checklist for the module under test (one file per module; see references/release-checklist/index.md). Each item carries [ADMIN: …] + [CLARITY: …] metadata. This file IS the set of items to verify. For B the checklist is derived from the PR(s) instead (see the scenario doc).

Helper scripts shipped with this skill

Dot-source them all at once in your bootstrap (the Get-ChildItem loop loads every helper — see Step 1 — Bootstrap):

$skill = '<this skill folder>'   # the folder containing SKILL.md, e.g. <PT-repo>\.github\skills\powertoys-verification
Get-ChildItem "$skill\scripts" -Filter '*.ps1' | ForEach-Object { . $_.FullName }

Step 1 — Bootstrap

$module = 'AdvancedPaste'  # or 'CmdPal', 'FZ', 'Peek', ...
# Work out of %TEMP% during the run (keeps screenshots/scratch off OneDrive); move to the
# sign-off archive at the very end (see Step 7).
$workspace = "$env:TEMP\verify-$module-$(Get-Date -Format yyyyMMdd-HHmmss)"
New-Item -ItemType Directory -Path $workspace, "$workspace\artifacts" -Force | Out-Null
$report = "$workspace\verify-$module.md"

# Dot-source helpers
$skill = '<this skill folder>'   # set once at top of your script (the folder containing SKILL.md)
Get-ChildItem "$skill\scripts" -Filter '*.ps1' | ForEach-Object { . $_.FullName }

# Verify environment
"=== Environment ===" | Tee-Object $report -Append
"IsAdmin: $(Test-PtAdmin)" | Tee-Object $report -Append
$rn = Test-PtRunnerAdmin
"PT runner: PID=$($rn.Pid) Elevated=$($rn.Elevated)" | Tee-Object $report -Append

# The checklist source depends on the scenario (see references/scenarios/):
#   A - read the supplied references/release-checklist/<module>.md
#   B - derive 1-3 items from each PR's `gh pr view/diff`; then drive the installed bits, OR
#       build+sideload if the code isn't in the build under test (pr-validation.md)
# Then iterate the items (see Step 6 - Verifier loop).

Step 2 — Drive techniques

Every checklist item boils down to ONE of three intents. Pick the bucket from the verb in the item, then use the best technique inside it. Stop at the first technique that works.

§2.A — Change a setting (single technique)

Edit the JSON file the module reads, wait for the file-watcher debounce, assert, then restore from backup. Zero external tools.

$bk = Backup-PtModuleSettings -ModuleDir AdvancedPaste
try {
    $j = Get-PtModuleSettings -ModuleDir AdvancedPaste
    $j.properties.IsAdvancedAIEnabled.value = $false
    $j | ConvertTo-Json -Depth 12 | Set-Content "$env:LOCALAPPDATA\Microsoft\PowerToys\AdvancedPaste\settings.json"
    Start-Sleep -Seconds 4  # debounce — runner re-reads via file-watcher
    # ... assertion ...
} finally {
    Restore-PtModuleSettings -ModuleDir AdvancedPaste -BackupPath $bk
}

For shell-extension modules (PowerRename, File Locksmith, Image Resizer, New+) edit the module-owned file under %LOCALAPPDATA%\Microsoft\PowerToys\<Module>\, then Restart-PtRunner (and on stubborn handlers, restart Explorer). See pitfall #12 below.

If you need to flip the enabled bit for a whole module, debounce isn't enough — call Restart-PtRunner after the write.

§2.B — Interact with a UI element (2 techniques, most-reliable first)

B1. UIA invoke / set-value — always try first

winapp ui invoke 'SubmitButton' -a PowerToys.Settings
winapp ui set-value 'QueryTextBox' '=2+3*4' -a PowerToys.PowerLauncher
Start-Sleep -Milliseconds 600
winapp ui inspect -a PowerToys.PowerLauncher --depth 7 -i 2>$null

Invoke goes through UIA InvokePattern COM IPC — no foreground steal, no UIPI. See references/winapp-ui-testing.md §CRITICAL — invoke vs click.

B2. PostMessage WM_KEYDOWN/CHAR — when UIA can't reach the target

For elevated targets, AppX windows with stunted UIA trees, or keystrokes that UIA set-value can't dispatch (arrow-key ListView nav, Enter to commit). See references/winapp-ui-testing.md §CRITICAL — Keystroke input that bypasses UIPI (PostMessage). Esc is often filtered by WinUI 3 raw-input hook — use BackButton invoke instead.

§2.C — Trigger a module action (2 techniques, most-reliable first)

Pick by what the item asserts: for "does action Y happen" use C1; for "pressing chord X triggers Y" or "the rebind takes effect", C1 is insufficient (it bypasses the chord) — use C2, or C1 plus a runner-log line proving the chord was accepted.

C1. Named Event signal — preferred

Invoke-PtSharedEvent -Name 'CmdPal.Show'           # opens CmdPal without keyboard
Invoke-PtSharedEvent -Name 'AOT.Pin'               # pins foreground window via AOT
Invoke-PtSharedEvent -Name 'PowerLauncher.Invoke'  # opens PT Run
Invoke-PtSharedEvent -Name 'LightSwitch.Toggle'    # toggles theme
Get-PtSharedEventCatalog | Format-Table            # full list

No synthetic input — it's a SetEvent on the kernel event the module waits on, the same downstream path the runner's hotkey handler signals. Verify the side effect via UIA (winapp ui list-windows -a <module>), a log line (Get-PtRunnerLogTail), or settings.json diff (Get-PtModuleSettings). The event only exists while the owning process runs, so Test-PtSharedEvent doubles as an "is the module alive" check.

C2. SendInput chord — last resort / chord-binding verification

Real synthetic keys. Loud (steals foreground) and fragile, but the only way to prove the activation chord is actually bound. The runner's global keyboard hook catches the chord regardless of focus, so the precondition is just an attached input desktop (pitfall #7; on a detached desktop SendInput returns ACCESS_DENIED and the keys vanish → mark BLK-ENV).

# Precondition: input desktop attached? 0 = detached → don't bother sending, mark BLK-ENV (pitfall #7)
if ([PtFg]::GetForegroundWindow() -eq [IntPtr]::Zero) { throw 'No input desktop — BLK-ENV (pitfall #7)' }

Send-PtChord -Mods 0x5B,0x10 -Key 0x43           # Win+Shift+C → Color Picker  (cb=40 fix is inside the helper)
$line = Wait-PtHotkeyAccepted -ModuleHint 'Color' -TimeoutSec 3
if (-not $line) { throw 'Runner did not log hotkey invocation' }

Rare fallback — a module that uses its own RegisterHotKey and exposes no Named Event. Post WM_HOTKEY (0x312) straight to its message window (find the HWND via EnumWindows+GetClassName through Add-Type — same P/Invoke pattern as pt-foreground-guard.ps1). No current PT module needs this: ZoomIt — the obvious candidate — also waits on Named Events (ZoomIt.Zoom, ZoomIt.Draw, …; source: Zoomit.cpp CreateEventW(ZOOMIT_ZOOM_EVENT)), so drive it with C1.

Different case — sending keys into a specific focused window (e.g. a CmdPal alias like = / < / > that winapp ui set-value can't trigger because it bypasses TextChanged; see references/modules/command-palette.md). Here the keystrokes go to whatever currently has focus, so you must bring the target window foreground first:

Assert-PtForegroundOrAbort -AppId Microsoft.CmdPal.UI   # -AppId = the window you're typing INTO
Send-PtChord -Key 0xBB                                  # '=' (no modifiers) to trigger the calculator alias

The -AppId is whatever window you're targeting — it's not CmdPal-specific. CmdPal is just the worst offender: its AppX foreground-lock drops focus after the first SetForegroundWindow, so without the guard the keys silently leak to your terminal.

Verdict decisions (PASS if behavior matches spec; FAIL if the product is wrong or the checklist item is stale/ambiguous; BLOCKED if you couldn't run the check after ≥2 entry-paths) live in Step 3 — Classification taxonomy below. Don't put verdict logic in Step 2.

Step 3 — Classification taxonomy

Verdicts (assign exactly ONE per item)

BLOCKED reasons

Different failure reasons stay distinct because each drives a different remediation.

Rule of thumb: in your report, separate the two FAIL causes — product FAILs are bugs to file; checklist FAILs are items to rewrite or prune. BLOCKED is only for a concrete, named obstacle (cite it), never a substitute for effort. If a large share of a module's items are checklist-FAILs, the checklist needs an overhaul before re-verifying.

Step 4 — Report format

See references/reporting-format.md for the full template (per-item table, summary, step-table rules, anti-patterns, worked example). Don't paraphrase; copy the templates literally. This includes a mandatory §G Retrospective — a self-reflection on the run itself: list every friction encountered (classified by source — SKILL-UNCLEAR / WINAPP-TOOL-BUG / WINAPP-DOC-UNCLEAR / HELPER-FLAW / PT-PRODUCT / ENVIRONMENT — with severity + minutes/attempts cost + a suggested fix), or write Everything was smooth — no friction encountered. if there was none. This is how the skill improves run over run, so don't skip it.

Step 5 — State hygiene (CRITICAL)

See references/pre-flight.md §State hygiene for the backup/restore pattern and cleanup commands. Always wrap mutations in try { ... } finally { Restore-* }.

Module-specific quick reference

Look for references/modules/<module>.md FIRST. Each per-module profile contains paths, entry-paths, item-by-item recipes, common BLOCKED traps, fixture lists, and source citations specific to that module.

Catalog: see references/modules/README.md. Currently authored: peek.md, power-rename.md, file-locksmith.md, image-resizer.md.

If your module has NO profile yet:

1. Fall back to the generic drive-stack in §2 above.
2. For Explorer-context-menu modules (PowerRename / File Locksmith / Image Resizer / New+ / Preview Pane / RegistryPreview): read references/explorer-context-menu-flow.md first — it has the synthetic-right-click + UIA-invoke pattern with stability rules and module-caption table. Per-module profiles cite it and only document module-specific quirks. The canonical helper is scripts/pt-explorer-contextmenu.ps1.
3. After finishing the verification, create the profile using the template in references/modules/README.md so the next agent benefits from what you learned.

Quick one-liners for modules without dedicated profiles (will be moved to per-module files as they're authored):

• Advanced Paste: Invoke-PtSharedEvent -Name 'AdvancedPaste.ShowUI' + Set-PtClipboardRich + Compare-PtClipboardFormatDiff (see scripts/pt-clipboard-diff.ps1).
• Command Palette: Invoke-PtSharedEvent -Name 'CmdPal.Show' + Invoke-CmdPalQuery (auto-handles degraded state via scripts/pt-cmdpal-recycle.ps1). Settings file via Get-CmdPalSettings. Full profile + AppX/alias/foreground quirks: references/modules/command-palette.md.
• PowerToys Run: Invoke-PtSharedEvent -Name 'PowerLauncher.Invoke' + winapp ui set-value QueryTextBox. Window has 2 HWNDs — filter by width ≥ 800.
• FancyZones: Invoke-PtSharedEvent -Name 'FancyZones.ToggleEditor'. Snap-drag tests are usually BLK-DRAG-REQUIRED; settings verify via settings.json round-trip.
• Light Switch: Invoke-PtSharedEvent -Name 'LightSwitch.Toggle' | LightSwitch.Light | LightSwitch.Dark. Verify via HKCU:\Software\Microsoft\Windows\CurrentVersion\Themes\Personalize\AppsUseLightTheme.
• Always on Top: Invoke-PtSharedEvent -Name 'AOT.Pin'. Verify WS_EX_TOPMOST on pinned HWND.
• Hosts File Editor (admin): Invoke-PtSharedEvent -Name 'Hosts.Show' | Hosts.ShowAdmin.
• GPO (admin): write HKLM:\Software\Policies\PowerToys + Restart-PtRunner + Get-PtRunnerLogTail -Pattern 'GPO sets'. Cleanup: Remove-Item HKLM:\Software\Policies\PowerToys -Recurse -Force.
• Mouse Without Borders: most items BLK-HARDWARE (need 2 physical PCs).
• ZoomIt: most modes inside BlockInput + WDA_EXCLUDEFROMCAPTURE overlay → BLK-OVERLAY-INPUT-BLOCK. Mode triggers: ZoomIt.Zoom, ZoomIt.Draw, ZoomIt.Break, etc.
• Peek: see references/modules/peek.md for the full recipe (CLI back-door + Shell.Application + Ctrl+Space).

Step 6 — Verifier loop per checkbox

For each item in module:
   1. Pick a bucket from the verb in the item (§2.A change a setting / §2.B interact with UI / §2.C trigger an action)
   2. Walk that bucket's techniques top-to-bottom; stop at the first one that drives the item
   3. Compare observed behavior to the spec:
        • matches the spec       → PASS (note the method in Category)
        • product behaves wrong  → FAIL, cause=product (repro + expected/actual + screenshot/log + build)
   4. Checklist item itself is broken — feature removed from source, or spec too ambiguous to judge → FAIL, cause=checklist (cite the source proof / quote the wording)
   5. Couldn't drive it after ≥2 entry-paths → BLOCKED with a concrete reason (§3)
   6. Record verdict + evidence + cleanup
   7. Next item

When done, run state hygiene cleanup, write the report including the §G retrospective, archive the workspace (Step 7), and exit.

Step 7 — Archive the workspace to the sign-off folder (do this LAST)

The live run works out of %TEMP%, but the final deliverable must live in the module sign-off archive so reports persist and sync via OneDrive:

# After the report is written AND the artifact-existence check passes:
$signoff = "$env:OneDrive\PowerToys\Module-Signoff"   # e.g. C:\Users\<you>\OneDrive - Microsoft\PowerToys\Module-Signoff
New-Item -ItemType Directory -Path $signoff -Force | Out-Null
$final = Join-Path $signoff (Split-Path $workspace -Leaf)
Move-Item -Path $workspace -Destination $final -Force
# Report uses RELATIVE artifacts/… paths, so all links stay valid after the move.
Write-Host "Final report: $(Join-Path $final (Split-Path $report -Leaf))"

Print the moved report path (under …\PowerToys\Module-Signoff\) as the last line — never the %TEMP% path.

Invocation & placeholders

This skill auto-activates for either of the two scenarios above (verify a module checklist, or validate a PR — open, merged, or a whole release/hotfix set). Step 0: resolve the scenario via references/scenarios/index.md and set the BITS: contract (for B, also the bits sub-decision: "is the PR's code in the build under test?") before driving anything. Scope rule: one module per run for Scenario A, one PR per report folder for B. Resolve these placeholders:

Execution order: references/scenarios/index.md (pick scenario + set BITS) → the matching references/scenarios/<scenario>.md (inputs, discipline, and build+sideload deploy steps for B when the code isn't in the build) → references/pre-flight.md → per item, the §2 drive-stack (this file) → references/reporting-format.md per-item table → Step 6 verifier loop → references/pre-flight.md §Final wrap-up → Step 7 archive → print the final report path.

What NOT to do

• Do NOT skip Step 0 — drive nothing until you've set the scenario and the BITS: contract. A run that mutates the wrong bits (sideloads when the code is already installed, or drives the stale installed binary when validating unreleased code) is invalid regardless of the verdict.
• Do NOT chain multiple modules in one report (Scenario A) — one module per run. For B, one PR per report folder.
• Do NOT mark an item BLOCKED without a concrete, named obstacle (see §3 and references/pre-flight.md §Hard rules).
• Do NOT invent steps for a VAGUE checklist item — if the spec is too ambiguous to judge, that is FAIL (cause=checklist), not a guess.
• All other rules (foreground guard, always restore mutated state, etc.) live in references/pre-flight.md §Hard rules — follow them.

Critical pitfalls (PT-specific)

Reference, not a sequential step — skim before you start and consult while driving. Numbered for cross-reference only.

1. Each module's own <Module>\settings.json IS hot-reloaded via per-module file watcher (~3s debounce). EXCEPTION — shell-extension/context-menu modules do NOT read this file; see pitfall #12.
2. GPO HKLM vs HKCU: HKLM wins when both are set with conflicting values.
3. HKLM Software\Policies\PowerToys writes require admin — verify with Test-PtAdmin.
4. Stop-Process is policy-blocked in this session unless you pass -Id <int> literally. Always inline the PID.
5. WinUI 3 islands are largely invisible to UIA (QuickAccess flyout, RegistryPreview Monaco editor, Peek WebView2). For these, fall back to screenshot + OCR or settings.json diff.
6. OS-reserved chords (Win+L, Win+Tab) are consumed by Windows before any hook and cannot be injected via SendInput at all.
7. RDP minimized = SendInput denied. Even though quser shows the remote session State=Active, minimizing the mstsc client detaches the session's input desktop. GetForegroundWindow() returns 0; SendInput returns ACCESS_DENIED (5); tests that need synthetic input fail. Same applies to: closed mstsc with X (Disconnected), local PC sleep (RDP TCP drops), remote screensaver/workstation lock, remote machine sleep. Run scripts/pt-session-diagnose.ps1 in pre-flight to detect, and see references/environment-setup.md for the full per-scenario table + powercfg setup commands the user should run before starting the agent. The agent should call Test-PtForeground mid-run before each input-injection-dependent item; if it returns False, mark BLK-ENV with mitigation citation (an environment block — not a product FAIL).
8. winapp ui arg-order quirk: winapp ui inspect --depth N -w $hwnd may intermittently fail to parse --depth as Int64 if -w precedes it. Put -w $hwnd AFTER --depth N or as the first arg before any flag. If you see "Cannot bind argument" or numeric parse errors, swap the order and retry.
9. winapp ui list-windows line wrapping: when window titles or process names are long, output may wrap a single window's HWND <id>: "<title>" ... (proc, PID N) across multiple lines, breaking single-line regexes. Either pipe through Out-String and use a multi-line regex, or use --json (when supported) and parse structured output.
10. De-elevation: launching a NON-elevated (Medium IL) child from an elevated agent shell. The drive-stack only covers gaining more privilege; some items need the opposite. From a High-IL shell you cannot Start-Process a Medium-IL child directly. Use scripts/pt-nonelevated.ps1 (Start-PtNonElevated / Invoke-PtNonElevatedCapture) — a one-shot RunLevel Limited + LogonType Interactive scheduled task that lands on the user's desktop at their filtered token. Confirm with Test-ProcessElevated. Needed for elevation-visibility pairs (File Locksmith L649/L650: non-elevated FL must not see the elevated runner; elevated FL must).
11. Win11 packaged context menus are not observable without real Explorer. Modern PT context-menu entries are packaged IExplorerCommands (sparse MSIX, e.g. File Locksmith CLSID {AAF1E27D-…}). They are NOT enumerable via classic Shell.Application … FolderItem.Verbs() and NOT CoCreate-able from a non-Explorer host (REGDB_E_CLASSNOTREGISTERED). So "verify the entry appears / no longer appears" cannot be pixel-verified by API. Verify instead via the gate flag the entry's GetState reads (e.g. general enabled.<Module>) + a source citation that maps it to ECS_HIDDEN; treat the literal render as BLK-VISUAL-RENDER and recommend a 5-second manual right-click. (Disabling does NOT unregister the package — it stays Status Ok; the entry is hidden dynamically.)
12. Shell-extension modules read a module-OWNED settings file, NOT the PT-store <Module>\settings.json. PowerRename, File Locksmith, Image Resizer, and New+ context-menu handlers and exes run outside the runner (hosted by Explorer / launched on demand) and cannot use the PT-Settings IPC. Each reads its own json in %LOCALAPPDATA%\Microsoft\PowerToys\<Module>\ at process/handler launch (registry-migrated CSettings/Settings classes — lib/Settings.cpp Load→ParseJson). The PT-Settings UI writes the PT-store settings.json (the bool_*/int_* file Get-PtModuleSettings reads); the runner's module DLL syncs PT-store→module-store only on a Settings-UI change event — so the PT-store file can be stale for days and editing it has no effect on the running shell handler. To drive a settings item on these modules, edit the module-owned file directly (drive-stack §2.A) and relaunch the module (or restart runner+Explorer for the menu handlers), then restore.

Pitfall #12 — module-owned files + their key style (verified 2026-06-10 against <PT-repo>\src):

Confirm which file actually drives behavior with a quick A/B: edit the module-owned file → relaunch → observe; if behavior follows, that's the source of truth (PowerRename L394/L395/L396/L397/L409 were all driven this way).

13. Foreground the window under test, and close per-test Explorer windows — for clean recordings AND reliable screenshots. When a module UI opens (e.g. PowerToys.PowerRename.exe launched with file args, or the Settings window), it can open behind a previously-opened Explorer/temp folder that still holds the foreground z-order, so a human watching / a screen recording sees the wrong window. Two complementary rules (do both):
    • (a) Foreground the target before you observe/screenshot/drive it. Call Force-PtForeground -AppId <appId> (scripts/pt-foreground-guard.ps1) right after the window appears — e.g. Force-PtForeground -AppId PowerToys.PowerRename after Start-Process …PowerToys.PowerRename.exe. winapp ui screenshot -w <hwnd> captures by HWND so it technically works when the window is buried, but a recording only shows the top window, and SendInput / coordinate-click fallbacks need real foreground anyway. This is the primary fix. (CmdPal's AppX foreground-lock is the exception — see references/modules/command-palette.md; use Assert-PtForegroundOrAbort.)
    • (b) Close each Explorer window when its item's menu tests end. The "open one menu per fresh Explorer window" rule (context-menu flow) piles up windows that fight for z-order and clutter the capture. Track the HWNDs you open and close them per item (not only in global cleanup): (New-Object -ComObject Shell.Application).Windows() | Where-Object { $_.LocationURL -match '<your-fixture-tag>' } | ForEach-Object { $_.Quit() }. Match your disposable-fixture folder name so you never close the user's own Explorer windows.

If you find another gap during verification, update this skill (add a recipe) AND consider proposing the addition to references/winapp-ui-testing.md if it's generic enough.