6 skills
Build the evaluation harness that gates every fine-tuning run — golden sets, per-failure-mode graders, judge calibration, and base-model baselines. Use when starting a fine-tuning effort, when converting traces into an eval set, or when calibrating a judge against human labels.
Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks.
Use this skill whenever the user wants to evaluate, test, or validate an AI agent, decide whether an agent is ready to ship or go live, choose how to grade an agent's answers (exact match, similarity, meaning, keywords, quality, or custom), design a test set of questions and expected answers, or interpret evaluation results into a go/no-go decision. Invoke it before the user hand-builds tests or declares an agent "done."
Authoring and running behavioral evals for a shopping or merchant agent, covering the case shape, authoring rules, code graders and judges, the run pattern, and poisoned fixtures. Load when writing eval cases or rubrics or deciding how a suite runs.
Draft a firm AI usage policy from published model policies, adapted to your practice profile — a research-and-synthesis tool whose output is a draft for attorney review and adoption, not a finished policy. Use when user says "draft an AI policy", "we need an AI policy", "build an AI usage policy", "our firm needs a GenAI policy", or similar requests to generate a first-cut internal AI policy.
Keep the AI policy current with practice — weekly sweep of saved AIAs, triage results, and vendor reviews to find policy drift, or direct query for a proposed new AI practice. Use when user says "policy sweep", "does our AI policy cover this", "we want to start doing X — does the policy need updating", "run the policy monitor", or on a recurring schedule.
