validate-skills

Validate that commands documented in skill files actually work. Use when creating, updating, or reviewing skills to ensure all documented commands exit with code 0.

Validating Skills

Verify every executable command in a skill runs successfully on the current OS.

When to Use

After creating or updating a skill that contains executable commands
During skill review to catch stale or broken instructions
When switching OS (e.g. Windows → Linux) to confirm cross-platform commands

Procedure

1. Detect Current OS

Determine which platform commands to extract:

# PowerShell (Windows)
$os = "Windows"

# Bash (Linux / macOS)
OS=$(uname -s)   # "Linux" or "Darwin"

2. Extract Commands

Parse the target skill's SKILL.md and list every shell command for the detected OS:

Many skills document commands in tables with Windows and Linux / macOS columns. Pick the column matching your OS.
If a command contains comments like # Windows or # Linux / macOS, only run the one for your OS.
Placeholder substitution: Replace obvious placeholders (e.g. <path-to-csproj>, <skill-name>) with real values from the repo. If no sensible value exists, skip the command.
Adapt cross-platform commands: Commands like ls -la should be adapted to PowerShell equivalents (Get-ChildItem) on Windows when no native Windows command is documented.

3. Track Results

Use the SQL tool to create a tracking table:

CREATE TABLE skill_commands (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  skill TEXT NOT NULL,
  command TEXT NOT NULL,
  expected_exit INTEGER DEFAULT 0,
  actual_exit INTEGER,
  status TEXT DEFAULT 'pending',
  notes TEXT
);

4. Run Each Command

For every extracted command:

Run it from the repo root
Record the exit code
Classify the result:
- Exit 0 → PASS
- Non-zero + environment issue (missing SDK, no internet) → ENV_ISSUE
- Non-zero + command/docs wrong → ERROR

5. Safety Rules

CRITICAL: Never run unfiltered integration/acceptance tests. They take hours.

test.sh --integrationTest or test.cmd -Integration MUST include a --filter or -p flag.

test.sh -p smoke is acceptable (scoped to smoke tests), but expect it to be slow.

6. Report

After all commands finish, print a summary:

=== Skill Validation Report ===
Skill: <skill-name>
OS: <Linux|Darwin|Windows>
Commands tested: N
PASS: X
ENV_ISSUE: Y (list with reasons)
ERROR: Z (list failed commands with exit codes)

7. Fix or Flag

ERROR (documentation bug): Update the skill's SKILL.md to fix the command.
ENV_ISSUE: Add a troubleshooting note to the skill if the environment prerequisite is not already documented.
PASS: No action needed.