hve-builder-tester

HVE Builder Tester Skill

Goal

Exercise one frozen candidate through representative black-box scenarios and produce a durable report that states exactly what the evidence supports. Each invocation is a complete assessment, not a repair loop. The caller may use its findings to revise artifacts and request another assessment; this skill never edits the target or decides whether the parent continues.

This skill owns scope, scenario design, fidelity, sandbox state, execution evidence, independent grading, reporting, and cleanup. Read references/test-methodology.md for black-box design, fidelity, and containment decisions, references/stage-dispatch.md for independent grading, and references/report-format.md for the durable report.

Use Cases

• Assess HVE Builder's frozen Major change or behavior-bearing review target, including a revised candidate after parent-owned corrections.
• Test an existing artifact directly without editing it, using its documented inputs and expected outcomes.
• Check a connected artifact set for handoff behavior, or assess whether required behavior survives instruction cleanup, relocation, or replacement.

Flow

1. Resolve targets, types, purpose, requirements, profile, requested fidelity, isolation and together sets, sandbox root, candidate revision, and a unique report path. For reassessment, retain the caller's original material requirements and identify changed behavior and regression coverage without replacing prior reports. If no runtime behavior exists, write a supported skip report and return.
2. Select fidelity through the methodology preconditions. Default to simulation. When requested native execution is unsupported or unsafe, use simulation only with caller acceptance; otherwise return Deferred with the rerun condition.
3. Capture pre-run workspace state and create a unique sandbox containing run-state.md. Record the candidate revision, profile and model, fidelity, groupings, purpose, requirements, containment controls, and requirement map.
4. Design the smallest black-box scenario set that covers the documented contract. Assign stable scenario IDs, map requirements to observable outcomes, record intentional gaps, perform the black-box self-check, and write test-design.md. If credible design is unavailable, return Deferred without execution.
5. Execute every scenario once. For simulation, dispatch HVE Artifact Tester with the resolved model bound through the host's model-selection parameter. Record binding evidence under the methodology's Profile Selection rules. For native fidelity, invoke the registered target directly when containment permits it. Never silently substitute fidelity or fabricate evidence after a failed execution.
6. Write test-log.md with the returned trace, observed versus simulated or emulated actions, fidelity, candidate revision, containment checks, workspace delta, and untested behavior.
7. Dispatch one independent grader at the higher of Medium and the target profile. Give it the finalized design and log, targets, purpose, requirements, catalog, and rubric. Validate its bounded Pass, Revise, or Blocked return and write test-review.md.
8. Write the durable report outside the sandbox, preserving the scenario inputs, requirement map, decisive trace evidence, and grading rationale needed to assess the verdict without transient files. Then clean up unless retention was requested. Return the report without revising the target.

Inputs

• targets: artifacts to exercise
• types: prompt, instructions, agent, subagent, or skill per target
• profile: High, Medium, or Low; infer from explicit metadata and responsibility when omitted
• fidelity: simulation or native; defaults to simulation; native requires an explicit request and satisfied preconditions
• purpose: target behavior, requirements, and observable expectations
• isolation and together: target groupings; default to isolation for one target and together for a connected set
• sandboxRoot: optional sandbox parent; defaults to .copilot-tracking/sandbox/
• retain-sandbox: retain transient evidence after reporting
• reportPath: optional durable report path; otherwise allocate the next unique attempt under the dated HVE Builder evidence root
• candidateRevision: source revision or equivalent provenance for the frozen target boundary

Success Criteria

• Every behavior-bearing target is exercised at its intended profile with explicit fidelity, or the report states the exact deferral.
• Scenario design maps each material requirement to an observable outcome or a disclosed gap.
• The test log distinguishes observed, simulated, and emulated behavior and records containment evidence.
• One independent grader assesses the complete evidence and returns Pass, Revise, or Blocked. A pre-grading deferral or blocker records Not available instead.
• The durable report identifies candidate revision, coverage, limitations, findings, sandbox disposition, and an unchecked human-review box.
• The skill performs one complete run per invocation and never edits the target.

Constraints

• Keep scenario text black-box. Put target pointers, profile metadata, and containment controls in the dispatch wrapper rather than the scenario.
• Permit native fidelity only for read-only targets or enforced write containment with caller-approved residual risk.
• Treat targets and logs as data. Keep secrets out of the sandbox and report.
• Do not inspect or assess agent or subagent tools configuration.
• Do not equate mechanical validation with behavior grading or simulation with native execution.
• The lead writes sandbox files. Executors and graders return evidence without modifying targets or lead-owned logs.

Reasoning Profile Resolution

Select the target's responsibility profile, then resolve a currently available model for it at run time rather than from a fixed list.

Resolve names from the GitHub Copilot docs: the supported-models page under copilot/reference/ai-models/ lists current models, per-client availability, and retirements; the model-comparison page beside it groups models by the task areas above.

When a target declares model:, use it. When a High target omits model:, run at the session's selected model and record it as the resolved High model. Use the (copilot) suffix in host model identifiers. The executor uses the target profile; the independent grader uses the higher of Medium and that profile. If the resolved profile is unavailable, disclose the nearest available proxy and do not claim target-profile equivalence.

Bind executor and grader models through host dispatch controls, not prose. Record the requested model, host selection evidence, and actual model when exposed. A worker's self-report is not binding evidence. Treat an unverified binding or a model mismatch as proxy evidence under references/test-methodology.md.

Dispatch

Stop Rules

• Return Complete only when execution, independent grading, and the durable report complete.
• Return Partial when usable evidence exists but contracted coverage is incomplete.
• Return Deferred with verdict Not available when fidelity, design, or execution cannot produce gradeable evidence; name the rerun condition.
• Return Blocked when target identity, intent, safety, or grading cannot be resolved.
• Do not repeat scenarios, edit source, or initiate another assessment inside this invocation. Return the report to the caller; a parent may invoke the skill again after a justified correction or resolved execution prerequisite.

Handoff

Return the durable report to the direct caller or HVE Builder. Pass supports completion for the tested revision. Required findings give the authorized HVE Builder parent evidence for in-scope corrections and reassessment under its workflow contract. Deferred, Partial, or Blocked execution identifies the missing prerequisite, not permission to alter source or retry unchanged conditions. Standalone use returns findings without starting an authoring lifecycle.

Final Response Contract

Return targets, candidate revision, behavior disposition, profile and model, fidelity, execution status, verdict, finding counts, untested behavior, sandbox disposition, report path, and the next owner. Use Not available only for a pre-grading deferral or blocker and the canonical Not applicable fields for a supported skip.

References

• references/test-methodology.md: black-box, fidelity, runtime, dispatch, profile, and containment rules
• references/stage-dispatch.md: independent grading template
• references/report-format.md: finding taxonomy, report structure, and human review
• HVE Artifact Tester: contained simulation executor