eval-generator-gen-pages

Eval Generator — Power Apps Generative Pages

Purpose

Answer: "Is this Generative Page correctly structured, does the logic work, and are there security risks?"

Three eval layers are always generated:

1. Presence — static checks that assert code artifacts exist and follow Generative Page conventions
2. Unit — Vitest tests for business logic using a mocked dataApi
3. Security — static analysis across 14 security categories specific to gen page patterns

All three are mandatory.

---

Generative Page Stack — Critical Assumptions

Generative Pages are single-file React/TypeScript components embedded in Model-Driven Apps. They are NOT Code Apps, NOT PCF components. The architecture is fundamentally different from both.

⚠️ NEVER look for: power.config.json, src/generated/services/, @microsoft/power-apps, @microsoft/powerapps-component-framework, context.webAPI.*, context.parameters, initialize(), PowerProvider. These are Code App or PCF patterns — they do not exist in Generative Pages.

⚠️ ALWAYS look for: props.dataApi or dataApi usage, @fluentui/react-components imports, export default component with React TSX, Dataverse operation patterns via dataApi.*.

---

Step 0 — Snapshot Previous Results

Run this first, before collecting inputs or writing any files.

1. Check if evals/results/latest.json exists relative to the project directory.
2. If it does exist:
   • Read it.
   • Count files matching evals/results/snapshots/iter-*.json to determine N (next iteration = count + 1).
   • Write a copy to evals/results/snapshots/iter-<N>-<ISO-timestamp>.json.
   • Update evals/results/snapshots/index.json (create if absent, append if present).
3. If it does not exist — skip silently. This is the first invocation.

---

Step 0b — Collect Inputs

Ask the user (via m_ask_user for structured choices) for:

1. Generative Page file or directory — one of:

   • Path to a single .tsx file (the gen page)
   • Path to a directory containing one or more .tsx gen page files
   • Current working directory (default if already in the right place)

2. Requirements document — OPTIONAL. One of:

   • Local file path (.md, .txt, .docx)
   • OneDrive / SharePoint URL
   • Paste / describe in chat
   • None / Skip — features will be derived from code review (Step 1b)

3. Eval output mode (optional):

   • scaffold+write — write all eval files into an evals/ directory (default ✅)
   • describe-only — print what would be generated, no file writes

⚠️ Ask ALL questions in a SINGLE m_ask_user call before doing any file work.

Determine project directory: If user provided a single file, PROJECT_DIR = the file's parent directory. If a directory was given, PROJECT_DIR = that directory. All evals/ output goes into PROJECT_DIR/evals/.

---

Step 1 — Read Requirements Document (if provided)

If no requirements document was provided, skip to Step 1b.

Local .md or .txt

Use the view tool.

Local .docx

Use the docx skill. Call m_get_skill('docx') first.

OneDrive / SharePoint URL

Use m365_download_file (resolve file ID with m365_search_files first).

---

Step 1b — Code-Review Fallback (No BRD)

Run this step when no requirements document was provided, AND on every re-run.

Read all .tsx files in the project directory (non-recursively first, then src/ if present).

For each gen page file, use view and grep to identify:

Synthesize a feature entry per logical unit with id, title, description, acceptance_criteria, priority, and tables (Dataverse table names involved).

Set manifest.generatedFrom = "code-review".

Re-runs: Repeat Step 1b to pick up new components or new dataApi calls. Match features by id — preserve presence_status for any existing id.

---

Step 2 — Parse Requirements → Feature Manifest

Extract a structured feature list from the requirements document (Step 1) or use the code-derived list from Step 1b.

For each feature, capture:

• id — slugified identifier (e.g. account-list, contact-detail, opportunity-create)
• title — short label
• description — what the feature does
• acceptance_criteria — array of verifiable statements
• priority — high / medium / low (default medium)
• tables — Dataverse table logical names involved (e.g. ["account", "contact"])
• operations — dataApi operations expected: ["queryTable", "retrieveRow", "createRow"]

Track in SQL:

CREATE TABLE IF NOT EXISTS features (
  id TEXT PRIMARY KEY,
  title TEXT,
  description TEXT,
  acceptance_criteria TEXT,
  priority TEXT DEFAULT 'medium',
  tables TEXT,
  operations TEXT,
  presence_status TEXT DEFAULT 'pending',
  mapped_file TEXT
);

Write evals/manifest.json:

{
  "generated": "<ISO timestamp>",
  "project": "<page file name or directory basename>",
  "generatedFrom": "brd | code-review",
  "features": [
    {
      "id": "account-list",
      "title": "Account List",
      "description": "Display a paginated list of Account records using dataApi.queryTable",
      "acceptance_criteria": [
        "Calls dataApi.queryTable('account', { select: [...], pageSize: N })",
        "Shows loading state while data fetches",
        "Shows empty state when no records returned",
        "Handles dataApi errors with a user-friendly message"
      ],
      "priority": "high",
      "tables": ["account"],
      "operations": ["queryTable"],
      "layers": { "presence": true, "unit": true }
    }
  ]
}

---

Step 3 — Audit the Generative Page File(s)

3a. Verify Generative Page structure

Use glob and view to confirm:

• At least one .tsx file exists with export default — the gen page component
• The file imports from @fluentui/react-components (Fluent UI V9)
• The component uses props.dataApi or destructures dataApi from props
• The component does NOT import from @microsoft/power-apps or @microsoft/powerapps-component-framework

If @microsoft/power-apps or power.config.json is found → stop and warn:

"This appears to be a Code App, not a Generative Page. Use the eval-generator-code-app skill instead."

If ComponentFramework or context.webAPI is found → stop and warn:

"This appears to be a PCF component. This skill only supports Power Apps Generative Pages."

3b. Map the gen page structure

Use grep on the .tsx file(s) to identify:

Record matched line numbers in the SQL features table (mapped_file, operations columns).

3c. Flag structural issues

A feature is Not Implemented if its expected dataApi.* call is not found. Mark presence_status = 'not_found'.

Flag these as warnings (not blockers) if found:

• dataApi.queryTable called without a select array → performance risk (fetches all columns)
• dataApi.* calls outside a try/catch block → unhandled async errors
• No loading state detected → poor UX
• No empty state detected → poor UX

---

Step 4 — Generate Presence Checks

Create evals/presence/<feature-id>.check.ts for each feature.

Presence check template

// evals/presence/<feature-id>.check.ts
// AUTO-GENERATED by eval-generator-gen-pages
// Feature: <feature title>
// Description: <feature description>

import { readFileSync, existsSync } from 'fs';
import { join, dirname } from 'path';
import { fileURLToPath } from 'url';
import { globSync } from 'glob';

// Path-anchored to this file's location — works correctly from any working directory
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
const PROJECT_DIR = join(__dirname, '..', '..');  // evals/presence → evals → project dir

interface PresenceResult {
  featureId: string;
  checks: Array<{ name: string; passed: boolean; detail?: string }>;
}

export function check(): PresenceResult {
  const checks: PresenceResult['checks'] = [];

  // Locate gen page .tsx files (non-recursive, then src/ if present)
  const tsxFiles = [
    ...globSync(`${PROJECT_DIR}/*.tsx`),
    ...globSync(`${PROJECT_DIR}/src/**/*.tsx`),
  ];

  if (tsxFiles.length === 0) {
    checks.push({ name: 'gen page .tsx file exists', passed: false,
      detail: 'No .tsx file found in project directory. Expected a generative page component file.' });
    return { featureId: '<feature-id>', checks };
  }

  const allSource = tsxFiles.map(f => readFileSync(f, 'utf-8')).join('\n');

  // ── Check: dataApi is accessed ──
  const dataApiUsed = /props\.dataApi|const\s*\{[^}]*dataApi/.test(allSource);
  checks.push({
    name: 'dataApi prop accessed',
    passed: dataApiUsed,
    detail: dataApiUsed ? undefined : 'dataApi not found in props. Generative pages receive dataApi as a prop from the Power Apps host.',
  });

  // ── Check: dataApi.queryTable called for this feature's table ──
  const queryFound = /dataApi\.queryTable\s*\(\s*['"]<table-name>['"]/.test(allSource);
  checks.push({
    name: 'dataApi.queryTable called for <table-name>',
    passed: queryFound,
    detail: queryFound ? undefined : 'dataApi.queryTable("' + '<table-name>' + '", ...) not found. Feature may not be fetching data.',
  });

  // ── Check: select columns specified (performance best practice) ──
  const selectFound = /dataApi\.queryTable\s*\([^)]*select\s*:/.test(allSource);
  checks.push({
    name: 'select columns specified in queryTable (performance)',
    passed: selectFound,
    detail: selectFound ? undefined : 'queryTable called without select property — fetches ALL columns. Always specify select: [\'col1\', \'col2\'] for performance.',
  });

  // ── Check: try/catch around dataApi call ──
  // Strategy: find dataApi.queryTable and check for surrounding try block
  const tryCatchFound = /try\s*\{[^}]*dataApi\.|dataApi\.[^;]*;\s*}\s*catch/.test(allSource.replace(/\n/g, ' '));
  checks.push({
    name: 'dataApi calls wrapped in try/catch',
    passed: tryCatchFound,
    detail: tryCatchFound ? undefined : 'No try/catch detected around dataApi calls. Unhandled promise rejections will crash the page.',
  });

  // ── Check: loading state ──
  const loadingState = /useState.*[Ll]oading|isLoading|isPending/.test(allSource);
  checks.push({
    name: 'loading state handled',
    passed: loadingState,
    detail: loadingState ? undefined : 'No loading state found. Users see blank content while data fetches.',
  });

  // ── Check: error state ──
  const errorState = /useState.*[Ee]rror|setError|errorMessage/.test(allSource);
  checks.push({
    name: 'error state handled',
    passed: errorState,
    detail: errorState ? undefined : 'No error state found. Connector failures will be silently swallowed.',
  });

  // ── Check: empty state ──
  const emptyState = /\.length\s*===\s*0|rows\.length\s*===\s*0|!rows\.length/.test(allSource);
  checks.push({
    name: 'empty state handled',
    passed: emptyState,
    detail: emptyState ? undefined : 'No empty-state check found. An empty result set may render a blank or broken UI.',
  });

  // ── Check: Fluent UI V9 imported ──
  const fluentV9 = /@fluentui\/react-components/.test(allSource);
  checks.push({
    name: 'Fluent UI V9 (@fluentui/react-components) imported',
    passed: fluentV9,
    detail: fluentV9 ? undefined : 'Fluent UI V9 not imported. Generative pages should use @fluentui/react-components for consistent MDA styling.',
  });

  return { featureId: '<feature-id>', checks };
}

Generative Page-specific presence patterns

⚠️ Never generate presence checks that grep for context.webAPI, initialize(), PowerProvider, src/generated/, or result.value — these are Code App / PCF patterns and will never appear in a Generative Page.

---

Step 5 — Generate Vitest Unit Tests

⛔ This step is MANDATORY on every invocation. Write real unit tests grounded in actual source code. Use it.todo() stubs only when a feature has no testable logic. Never skip this step — even on re-runs.

Re-run behaviour

If evals/unit/ already exists, merge — add tests for new features, preserve passing tests for existing ones. Never delete existing passing test files.

File layout

evals/unit/
  helpers/
    setup.ts          → @testing-library/jest-dom matchers, global mocks
    mocks.ts          → mockDataApi factory (vi.fn() for all 6 methods)
    factories.ts      → createMock<TableName>Row() helpers for Dataverse rows
  <Feature>.test.tsx  → one file per logical feature group

evals/unit/helpers/setup.ts

// evals/unit/helpers/setup.ts
import '@testing-library/jest-dom';
import { vi } from 'vitest';

// Suppress FluentUI console warnings in tests
vi.spyOn(console, 'warn').mockImplementation(() => {});

evals/unit/helpers/mocks.ts

// evals/unit/helpers/mocks.ts
import { vi } from 'vitest';

// mockDataApi — the dataApi prop passed by Power Apps host, fully mocked
export function createMockDataApi() {
  return {
    queryTable: vi.fn().mockResolvedValue({ rows: [], hasMoreRows: false }),
    retrieveRow: vi.fn().mockResolvedValue({}),
    createRow: vi.fn().mockResolvedValue('new-row-id'),
    updateRow: vi.fn().mockResolvedValue(undefined),
    deleteRow: vi.fn().mockResolvedValue(undefined),
    getChoices: vi.fn().mockResolvedValue([]),
  };
}

evals/unit/helpers/factories.ts

⚠️ CRITICAL: Before writing this file, read the actual gen page source to find what fields each dataApi.queryTable result is used for. Factory fields must match the actual column names the component reads from rows. Never invent field names.

// evals/unit/helpers/factories.ts
// Grounded in actual source: fields match what the component reads from dataApi results

// Example — replace with actual fields from source
export function createMockAccountRow(overrides: Partial<Record<string, unknown>> = {}) {
  return {
    accountid: 'acc-001',
    name: 'Contoso Ltd',
    emailaddress1: 'info@contoso.com',
    telephone1: '555-0100',
    statecode: 0,
    ...overrides,
  };
}

Unit test template — per feature

// evals/unit/<feature>.test.tsx
// Feature: <feature title>
// Tests grounded in actual source: <gen page file name>

import { describe, it, expect, vi, beforeEach } from 'vitest';
import { render, screen, waitFor, fireEvent } from '@testing-library/react';
import { createMockDataApi } from './helpers/mocks';
import { createMockAccountRow } from './helpers/factories';

// Import the default export from the gen page file
// Adjust path relative to evals/unit/ — typically ../../PageName.tsx
import GeneratedComponent from '../../<PageName>.tsx';

describe('<feature-id>', () => {
  let dataApi: ReturnType<typeof createMockDataApi>;

  beforeEach(() => {
    dataApi = createMockDataApi();
  });

  it('<feature-id>: renders loading state while data fetches', async () => {
    // Delay resolution so we can catch the loading state
    dataApi.queryTable.mockReturnValue(new Promise(() => {}));
    render(<GeneratedComponent dataApi={dataApi} />);
    // queryBy* returns null instead of throwing — safe to use with || fallback
    // Adjust selectors to match the actual loading indicator in the component
    const loadingIndicator =
      screen.queryByRole('progressbar') ??
      screen.queryByText(/loading/i) ??
      screen.queryByLabelText(/loading/i);
    expect(loadingIndicator).toBeInTheDocument();
  });

  it('<feature-id>: renders account rows returned by dataApi.queryTable', async () => {
    const rows = [createMockAccountRow(), createMockAccountRow({ accountid: 'acc-002', name: 'Fabrikam Inc' })];
    dataApi.queryTable.mockResolvedValue({ rows, hasMoreRows: false });
    render(<GeneratedComponent dataApi={dataApi} />);
    await waitFor(() => {
      expect(screen.getByText('Contoso Ltd')).toBeInTheDocument();
      expect(screen.getByText('Fabrikam Inc')).toBeInTheDocument();
    });
  });

  it('<feature-id>: renders empty state when no rows returned', async () => {
    dataApi.queryTable.mockResolvedValue({ rows: [], hasMoreRows: false });
    render(<GeneratedComponent dataApi={dataApi} />);
    await waitFor(() => {
      // Adjust text to match actual empty-state message in the component
      expect(screen.getByText(/no records|no data|no results/i)).toBeInTheDocument();
    });
  });

  it('<feature-id>: renders error state when dataApi.queryTable rejects', async () => {
    dataApi.queryTable.mockRejectedValue(new Error('Dataverse unavailable'));
    render(<GeneratedComponent dataApi={dataApi} />);
    await waitFor(() => {
      // Adjust to match actual error message shown in component (should be generic, not e.message)
      expect(screen.getByText(/error|failed|unable/i)).toBeInTheDocument();
    });
  });

  it('<feature-id>: calls dataApi.queryTable with correct table name and select columns', async () => {
    dataApi.queryTable.mockResolvedValue({ rows: [], hasMoreRows: false });
    render(<GeneratedComponent dataApi={dataApi} />);
    await waitFor(() => {
      expect(dataApi.queryTable).toHaveBeenCalledWith(
        'account',   // table logical name — verify from source
        expect.objectContaining({ select: expect.arrayContaining(['name']) })
      );
    });
  });
});

it() naming convention — CRITICAL for dashboard traceability

Every it() description MUST begin with the exact feature id from manifest.json, followed by a colon:

it('account-list: renders loading state while data fetches', ...)    // ✅ correct
it('account list: renders loading state', ...)                        // ❌ wrong — space not hyphen
it('renders loading state', ...)                                      // ❌ wrong — no feature ID prefix

The runner extracts the feature ID using /([\w-]+):\s*(.+)$/ and uses it to map test results to dashboard rows. A mismatch causes tests to pass but show "⏭ No tests" per feature in the dashboard.

Post-generation verification (MANDATORY)

After writing all test files, grep each test file for each manifest feature ID to confirm traceability:

For each feature.id in manifest.json:
  grep -c "it\('" + feature.id + ":" in evals/unit/**/*.test.tsx
  → If count === 0: traceability broken — fix the it() names before proceeding

---

Step 5b — Eval Package Setup

The gen page project may not have a package.json with test dependencies. Check:

1. Does PROJECT_DIR/package.json exist?
2. If yes, does it have vitest, @testing-library/react, jsdom in dev dependencies?
3. If no to either: create or patch package.json with the minimal eval harness.

Minimal package.json for evals (create only if needed)

{
  "name": "<project-basename>-evals",
  "private": true,
  "type": "module",
  "scripts": {
    "eval": "tsx evals/runner/run-evals.ts",
    "eval:presence": "tsx evals/runner/presence-runner.ts",
    "eval:security": "tsx evals/runner/security-runner.ts",
    "eval:unit": "vitest run --config evals/vitest.config.ts"
  },
  "devDependencies": {
    "@testing-library/jest-dom": "^6.0.0",
    "@testing-library/react": "^16.0.0",
    "@fluentui/react-components": "^9.0.0",
    "jsdom": "^25.0.0",
    "react": "^17.0.0",
    "react-dom": "^17.0.0",
    "@types/react": "^17.0.0",
    "@types/react-dom": "^17.0.0",
    "tsx": "^4.0.0",
    "vitest": "^2.0.0",
    "glob": "^11.0.0"
  }
}

If package.json already exists, append only the missing dev dependencies and scripts — never overwrite existing ones.

After writing package.json, run npm install to install dependencies.

evals/vitest.config.ts

// evals/vitest.config.ts
import { defineConfig } from 'vitest/config';
import { resolve, dirname } from 'path';
import { fileURLToPath } from 'url';

const __dirname = dirname(fileURLToPath(import.meta.url));
const PROJECT_ROOT = resolve(__dirname, '..');   // evals/ → project root

export default defineConfig({
  root: PROJECT_ROOT,   // CRITICAL: anchors all paths to project root, not evals/
  test: {
    environment: 'jsdom',
    globals: true,
    setupFiles: [resolve(__dirname, 'unit/helpers/setup.ts')],
    include: ['evals/unit/**/*.test.{ts,tsx}'],
    reporters: ['json'],
    outputFile: resolve(__dirname, 'results/unit-results.json'),
  },
  resolve: {
    alias: {
      // Allow tests to import from the project root without relative ../../
      '@': PROJECT_ROOT,
    },
  },
});

⚠️ root: PROJECT_ROOT is mandatory. Without it, Vitest defaults root to evals/, causing all setupFiles and include paths to resolve from evals/evals/... — a path that doesn't exist.

---

Step 6 — Generate Security Runner

Write evals/runner/security-runner.ts:

#!/usr/bin/env tsx
// evals/runner/security-runner.ts
// Static security analysis for Power Apps Generative Pages

import { readFileSync } from 'fs';
import { join, dirname } from 'path';
import { fileURLToPath } from 'url';
import { globSync } from 'glob';

const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
const ROOT = join(__dirname, '..', '..');   // evals/runner → evals → project root

type Severity = 'critical' | 'high' | 'medium' | 'low';

interface SecurityFinding {
  id: string; category: string; severity: Severity;
  file: string; line: number; snippet: string; detail: string;
}

interface SecurityResult {
  layer: 'security';
  passed: number; failed: number; skipped: number;
  findings: SecurityFinding[];
}

const findings: SecurityFinding[] = [];
// Scan all .tsx/.ts files in the project (excluding evals/ itself)
const srcFiles = globSync(`${ROOT}/**/*.ts?(x)`, { ignore: `${ROOT}/evals/**` });

function scan(id: string, category: string, severity: Severity, pattern: RegExp, detail: string, exclude?: RegExp) {
  for (const file of srcFiles) {
    let content: string;
    try { content = readFileSync(file, 'utf-8'); } catch { continue; }
    const lines = content.split('\n');
    lines.forEach((line, i) => {
      if (pattern.test(line) && !(exclude && exclude.test(line))) {
        findings.push({ id, category, severity,
          file: file.replace(ROOT + '\\', '').replace(ROOT + '/', ''),
          line: i + 1, snippet: line.trim().slice(0, 120), detail });
      }
    });
  }
}

// GP-SEC-001: OData Injection — filter string built from user input
scan('GP-SEC-001', 'OData Injection', 'critical',
  /filter\s*:\s*[^,}]*['"`]\s*\+|filter\s*:\s*[^,}]*\$\{/,
  'OData filter string built by concatenation or template literal. User input in filter strings is an injection risk. Use fixed filter strings or validate/sanitize inputs.');

// GP-SEC-002: XSS — dangerouslySetInnerHTML or .innerHTML
scan('GP-SEC-002', 'XSS', 'critical',
  /dangerouslySetInnerHTML|\.innerHTML\s*=/,
  'Direct HTML injection. Use React text rendering or sanitize with DOMPurify. Fluent UI components handle escaping automatically — prefer those over raw HTML.');

// GP-SEC-003: Hardcoded secrets
// Exclude: (1) comment lines, (2) values that are purely alphabetical/kebab-case — real credentials
// always contain digits or special chars. This prevents false positives on TypeScript object maps
// where property keys like `token` hold UI label strings like 'skipToken' or 'badge-token'.
scan('GP-SEC-003', 'Hardcoded Secret', 'critical',
  /(?:apikey|api_key|password|secret|token|client_secret)\s*[:=]\s*['"`][^'"`\s]{6,}/i,
  'Possible hardcoded credential in source. Move secrets to environment variables. In Generative Pages, use Dataverse environment variables or Azure Key Vault references.',
  /^\s*\/\/|(?:apikey|api_key|password|secret|token|client_secret)\s*[:=]\s*['"`][a-zA-Z$_][a-zA-Z$_\-]*['"`]/i);

// GP-SEC-004: PII in console logs
// Exclude: single-string-only console.log calls (e.g. "Email field is required") — UI messages, not variable dumps.
scan('GP-SEC-004', 'PII in Logs', 'high',
  /console\.(log|warn|error|debug)\b.*\b(email|mail|userId|phone|address|password|token|DisplayName|firstname|lastname)/i,
  'Sensitive data written to the browser console. Remove PII from log statements — model-driven app users can open DevTools.',
  /console\.(log|warn|error|debug)\s*\(\s*(?:'[^']*'|"[^"]*"|`[^`]*`)\s*\)/i);

// GP-SEC-005: Raw error surfaced to UI state
scan('GP-SEC-005', 'Error Disclosure', 'high',
  /set\w+\(\s*(e\.message|String\(e\)|e\.toString\(\)|error\.message)/,
  'Raw error message surfaced directly to UI state. Show a generic user-friendly message instead (e.g., "Something went wrong. Please try again."). Technical details may reveal system internals.');

// GP-SEC-006: fetch() to non-Microsoft external URLs
scan('GP-SEC-006', 'Unsafe fetch() Call', 'high',
  /fetch\s*\(\s*['"`]https?:\/\//,
  'fetch() to an external URL. Generative pages should use dataApi methods for all Dataverse operations. External fetch calls bypass Power Platform governance and connector policies.',
  /\.dynamics\.com|\.microsoft\.com|\.microsoftonline\.com|\.sharepoint\.com/);

// GP-SEC-007: Client-side auth bypass
scan('GP-SEC-007', 'Client-Side Auth Bypass', 'high',
  /\b(isAdmin|userRole|hasPermission|canEdit|canDelete|isOwner)\b.*&&.*dataApi\.(create|update|delete)/i,
  'Role check in UI code but no server-side guard before dataApi write. Power Apps security roles should be the authoritative gate — UI checks alone can be bypassed.');

// GP-SEC-008: PII in browser storage
// Removed bare 'user' — too generic (matches user-preferences, user-theme etc.)
scan('GP-SEC-008', 'PII in Browser Storage', 'medium',
  /(localStorage|sessionStorage)\.setItem\s*\([^,]*(?:email|userId|userEmail|userName|token|auth|authToken|mail|profile|password|DisplayName)/i,
  'PII or auth data stored in localStorage/sessionStorage. This data persists across sessions and is accessible to any script on the page. Use Dataverse or in-memory state instead.');

// GP-SEC-009: Unvalidated user input in dataApi filter
// Removed 'value' and 'text' — too generic.
scan('GP-SEC-009', 'Unvalidated Input in Filter', 'medium',
  /dataApi\.(queryTable|retrieveRow)\s*\([^)]*\b(input|userInput|searchTerm|searchQuery|filterValue|rawInput)\b/i,
  'User input variable passed directly to dataApi query. Validate and sanitize before using in queries — while Dataverse has server-side guards, defence-in-depth is best practice.');

// GP-SEC-010: Sensitive data in React state
scan('GP-SEC-010', 'Sensitive Data in State', 'low',
  /useState[^)]*\b(password|token|secret|apiKey|api_key)\b/i,
  'Sensitive data stored in React component state. State is accessible via React DevTools. Consider alternatives like Dataverse secure columns or environment variables.');

// GP-SEC-011: Dynamic code execution
scan('GP-SEC-011', 'Dynamic Code Execution', 'critical',
  /\beval\s*\(/,
  'eval() executes a string as code. If user-controlled content can reach this call, it is a code injection vulnerability. Replace with a safer pattern.');

scan('GP-SEC-011', 'Dynamic Code Execution', 'critical',
  /new\s+Function\s*\(/,
  'new Function() constructs executable code from a string — equivalent to eval(). Avoid or ensure the source string is never user-controlled.');

scan('GP-SEC-011', 'Dynamic Code Execution', 'high',
  /(setTimeout|setInterval)\s*\(\s*['"`]/,
  'setTimeout/setInterval with a string argument evaluates it like eval(). Pass a function reference instead: setTimeout(() => doSomething(), delay).');

// GP-SEC-012: Broader external HTTP clients
scan('GP-SEC-012', 'Unsafe External HTTP Client', 'high',
  /axios\s*\.\s*(get|post|put|patch|delete|request)\s*\(\s*['"`]https?:\/\//i,
  'axios call to an external URL. Use dataApi methods for Dataverse access. External HTTP calls bypass Power Platform governance.',
  /\.dynamics\.com|\.microsoft\.com|\.microsoftonline\.com/);

scan('GP-SEC-012', 'Unsafe External HTTP Client', 'high',
  /new\s+XMLHttpRequest\s*\(\s*\)/,
  'XMLHttpRequest instantiation in a generative page. Dataverse access should always go through dataApi methods, not raw HTTP calls.');

// GP-SEC-013: Sensitive data in URL parameters
scan('GP-SEC-013', 'Sensitive URL Parameters', 'high',
  /useSearchParams[^;]*\b(token|auth|apikey|secret|password|credential)\b/i,
  'Sensitive parameter read from URL query string. Tokens and credentials must never appear in URLs — they are logged by browsers, proxies, and servers.');

scan('GP-SEC-013', 'Sensitive URL Parameters', 'high',
  /[?&](token|auth|apikey|api_key|secret|password|userId|user_id|credential)=/i,
  'Sensitive parameter embedded in a URL string. Use Dataverse record context (props.recordId, props.entityName) for passing identifiers — never URL query strings.');

// GP-SEC-014: queryTable without select (excessive data retrieval)
scan('GP-SEC-014', 'Excessive Data Retrieval', 'medium',
  /dataApi\.queryTable\s*\(\s*['"`]\w+['"`]\s*,\s*\{(?![^}]*select\s*:)/,
  'dataApi.queryTable called without a select property — retrieves ALL columns from the table. This is a performance risk and may expose columns the UI does not need. Always specify select: [\'col1\', \'col2\'].');

const ALL_CATEGORIES = [
  'GP-SEC-001','GP-SEC-002','GP-SEC-003','GP-SEC-004','GP-SEC-005','GP-SEC-006',
  'GP-SEC-007','GP-SEC-008','GP-SEC-009','GP-SEC-010','GP-SEC-011','GP-SEC-012',
  'GP-SEC-013','GP-SEC-014',
];

const hitIds = new Set(findings.map(f => f.id));
const result: SecurityResult = {
  layer: 'security',
  passed: ALL_CATEGORIES.filter(c => !hitIds.has(c)).length,
  failed: hitIds.size,
  skipped: 0,
  findings,
};
process.stdout.write(JSON.stringify(result));

---

Step 7 — Generate the Eval Runners

evals/runner/presence-runner.ts

#!/usr/bin/env tsx
// evals/runner/presence-runner.ts

import { join, dirname } from 'path';
import { fileURLToPath } from 'url';
import { pathToFileURL } from 'url';
import { globSync } from 'glob';

const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
const PRESENCE_DIR = join(__dirname, '..', 'presence');

interface LayerResult {
  layer: 'presence';
  passed: number; failed: number; skipped: number;
  details: Array<{ featureId: string; name: string; status: 'pass' | 'fail' | 'skip'; detail?: string }>;
}

const checkFiles = globSync(`${PRESENCE_DIR}/*.check.ts`).sort();
const details: LayerResult['details'] = [];
let passed = 0, failed = 0;

for (const file of checkFiles) {
  const mod = await import(pathToFileURL(file).href);
  const result = mod.check();
  for (const c of result.checks) {
    const status = c.passed ? 'pass' : 'fail';
    if (c.passed) passed++; else failed++;
    details.push({ featureId: result.featureId, name: c.name, status, detail: c.detail });
  }
}

const output: LayerResult = { layer: 'presence', passed, failed, skipped: 0, details };
process.stdout.write(JSON.stringify(output));

evals/runner/run-evals.ts

⚠️ CRITICAL: Copy the code below EXACTLY. Do not simplify or restructure.

#!/usr/bin/env tsx
// evals/runner/run-evals.ts
// Orchestrates all three eval layers and bakes results into the dashboard.

import { execSync, execFileSync } from 'child_process';
import { readFileSync, writeFileSync, mkdirSync, existsSync, copyFileSync } from 'fs';
import { join, dirname, resolve } from 'path';
import { fileURLToPath, pathToFileURL } from 'url';

const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
const ROOT = join(__dirname, '..', '..');  // evals/runner → evals → project root
const RESULTS_DIR = join(ROOT, 'evals', 'results');
mkdirSync(RESULTS_DIR, { recursive: true });

function run(cmd: string, label: string): string {
  try {
    return execSync(cmd, { cwd: ROOT, encoding: 'utf-8', stdio: 'pipe' });
  } catch (e: any) {
    // Some runners exit 1 on findings (security) or failures (vitest) — capture output anyway
    return e.stdout || e.output?.[1] || '';
  }
}

console.log('▶ Running presence checks...');
const presOut = run('npx tsx evals/runner/presence-runner.ts', 'presence');

console.log('▶ Running unit tests...');
const unitRawOut = run('npx vitest run --config evals/vitest.config.ts --reporter=json', 'unit');

console.log('▶ Running security scan...');
const secOut = run('npx tsx evals/runner/security-runner.ts', 'security');

// Parse presence
let presLayer: any = { layer: 'presence', passed: 0, failed: 0, skipped: 0, details: [] };
try { presLayer = JSON.parse(presOut); } catch { console.warn('Could not parse presence output'); }

// Parse unit (Vitest JSON format)
let unitLayer: any = { layer: 'unit', passed: 0, failed: 0, skipped: 0, details: [] };
try {
  const vitestJson = JSON.parse(unitRawOut);
  const details: any[] = [];
  let p = 0, f = 0, s = 0;
  // Vitest JSON: testResults[].assertionResults[]
  for (const suite of vitestJson.testResults || []) {
    for (const t of suite.assertionResults || []) {
      const m = t.fullName?.match(/([\w-]+):\s*(.+)$/);
      const featureId = m ? m[1] : 'unknown';
      const name = t.fullName || t.title;
      const status = t.status === 'passed' ? 'pass' : t.status === 'pending' ? 'skip' : 'fail';
      const detail = t.failureMessages?.join('\n').slice(0, 500) || undefined;
      details.push({ featureId, name, status, detail });
      if (status === 'pass') p++; else if (status === 'skip') s++; else f++;
    }
  }
  unitLayer = { layer: 'unit', passed: p, failed: f, skipped: s, details };
} catch { console.warn('Could not parse vitest output'); }

// Parse security
let secLayer: any = { layer: 'security', passed: 0, failed: 0, skipped: 0, findings: [] };
try { secLayer = JSON.parse(secOut); } catch { console.warn('Could not parse security output'); }

// Load manifest for feature count
let manifest: any = { features: [] };
const manifestPath = join(ROOT, 'evals', 'manifest.json');
try { manifest = JSON.parse(readFileSync(manifestPath, 'utf-8')); } catch {}

// Load snapshots for iteration history
const SNAP_DIR = join(RESULTS_DIR, 'snapshots');
let snapshots: any[] = [];
try {
  const idx = JSON.parse(readFileSync(join(SNAP_DIR, 'index.json'), 'utf-8'));
  snapshots = idx.map((s: any) => {
    try { return JSON.parse(readFileSync(join(SNAP_DIR, s.file), 'utf-8')); } catch { return null; }
  }).filter(Boolean);
} catch {}

const timestamp = new Date().toISOString();
const evalResults = {
  timestamp,
  project: manifest.project || 'Generative Page',
  iteration: snapshots.length + 1,
  layers: [presLayer, unitLayer, secLayer],
  snapshots,
  summary: {
    totalFeatures: manifest.features.length,
    openBRDGaps: manifest.openBRDGaps || [],
    blockers: manifest.blockers || [],
    securityFindings: secLayer.findings || [],
  },
};

// Write latest.json
writeFileSync(join(RESULTS_DIR, 'latest.json'), JSON.stringify(evalResults, null, 2));

// Bake results into dashboard
const dashPath = join(ROOT, 'evals', 'dashboard', 'index.html');
if (existsSync(dashPath)) {
  let html = readFileSync(dashPath, 'utf-8');
  const dataBlock = `/* BAKED DATA — replaced by npm run eval — do not edit manually */\nvar DATA = ${JSON.stringify(evalResults)};\n/* END BAKED DATA */`;
  html = html.replace(/\/\* BAKED DATA[\s\S]*?END BAKED DATA \*\//m, dataBlock);
  writeFileSync(dashPath, html);
  console.log(`\n✅ Dashboard updated: evals/dashboard/index.html`);
}

// Summary
const totalFailed = presLayer.failed + unitLayer.failed;
console.log(`\nPresence: ${presLayer.passed} passed, ${presLayer.failed} failed`);
console.log(`Unit:     ${unitLayer.passed} passed, ${unitLayer.failed} failed, ${unitLayer.skipped} todo`);
console.log(`Security: ${secLayer.findings?.length || 0} findings`);
if (totalFailed > 0) { console.error(`\n❌ ${totalFailed} eval(s) failed.`); process.exit(1); }
console.log('\n✅ All evals passed.');

---

Step 8 — Generate the Dashboard

Step 8a — Read template

Read the dashboard template from:

path: ~/.copilot/m-skills/eval-generator-gen-pages/dashboard-template.html

Expand ~ to the user's actual home directory. Use PowerShell if needed: (Resolve-Path ~).Path.

Step 8b — Adapt security categories for gen pages

The dashboard template's CATEGORIES array in renderSecurity() must be replaced with the gen-page-specific categories:

var CATEGORIES=[
  {id:'GP-SEC-001',label:'OData Injection',          severity:'critical', what:'filter strings built from user input — injection risk for Dataverse queries'},
  {id:'GP-SEC-002',label:'XSS',                      severity:'critical', what:'dangerouslySetInnerHTML or .innerHTML= — prefer Fluent UI components which escape automatically'},
  {id:'GP-SEC-003',label:'Hardcoded Secret',          severity:'critical', what:'API keys, passwords, or tokens assigned as string literals in source'},
  {id:'GP-SEC-004',label:'PII in Logs',               severity:'high',     what:'Sensitive fields (email, userId, DisplayName) written to the browser console'},
  {id:'GP-SEC-005',label:'Error Disclosure',          severity:'high',     what:'Raw e.message or error.message surfaced directly to UI state'},
  {id:'GP-SEC-006',label:'Unsafe fetch() Call',       severity:'high',     what:'fetch() to non-Microsoft URLs — use dataApi for all Dataverse operations'},
  {id:'GP-SEC-007',label:'Client-Side Auth Bypass',   severity:'high',     what:'Role check in UI before dataApi write without a server-side guard'},
  {id:'GP-SEC-008',label:'PII in Browser Storage',    severity:'medium',   what:'PII or auth data in localStorage/sessionStorage — XSS-exfiltrable'},
  {id:'GP-SEC-009',label:'Unvalidated Filter Input',  severity:'medium',   what:'User input variables in dataApi query options without sanitisation'},
  {id:'GP-SEC-010',label:'Sensitive Data in State',   severity:'low',      what:'password, token, or secret held in React component state'},
  {id:'GP-SEC-011',label:'Dynamic Code Execution',    severity:'critical', what:'eval(), new Function(), or setTimeout/setInterval with string argument'},
  {id:'GP-SEC-012',label:'Unsafe External HTTP Client',severity:'high',    what:'axios, XMLHttpRequest calls to non-Microsoft URLs'},
  {id:'GP-SEC-013',label:'Sensitive URL Parameters',  severity:'high',     what:'token/auth/secret/userId exposed as URL query parameters'},
  {id:'GP-SEC-014',label:'Excessive Data Retrieval',  severity:'medium',   what:'dataApi.queryTable without select — fetches ALL columns, performance + data exposure risk'},
];

Step 8c — Make exactly three substitutions in the template

1. Replace every occurrence of PROJECT_NAME with the gen page file/directory name

2. Replace ITERATION_NUMBER with the current iteration number

3. Replace the FEATURES placeholder:

     /* GENERATED — replace with actual features: { id, title, priority, note? } */
   

   with the actual feature array from the manifest:

   { id: 'account-list', title: 'Account List', priority: 'high' },
   { id: 'contact-detail', title: 'Contact Detail', priority: 'high' },
   

4. Replace the CATEGORIES array in renderSecurity() with the gen-page categories above.

Step 8d — Write evals/dashboard/index.html

Step 8e — Self-verify (THREE checks, all mandatory)

After writing, verify all three:

1. Grep for fetch( — if found: delete and repeat from 8a
2. Grep for FEATURES = [ → confirm next non-whitespace line is NOT /* GENERATED — if placeholder present: delete and repeat
3. Grep for PROJECT_NAME — if still present: delete and repeat

---

Step 9 — Package.json Script

Ensure package.json has these scripts (add if missing, never overwrite existing):

{
  "scripts": {
    "eval": "tsx evals/runner/run-evals.ts",
    "eval:presence": "tsx evals/runner/presence-runner.ts",
    "eval:security": "tsx evals/runner/security-runner.ts",
    "eval:unit": "vitest run --config evals/vitest.config.ts"
  }
}

---

Step 10 — Run Evals

Run:

npm run eval

from PROJECT_DIR. Capture output. If failures are reported, summarise them for the user with specific file and line references.

After a successful run, tell the user to open evals/dashboard/index.html in their browser.

---

Re-run Guard

When evals/ already exists:

⚠️ Unit tests are ALWAYS merged, never skipped on re-runs. Any interpretation of "merge" as "skip unit test generation" is incorrect.

---

Mandatory Checklist (verify before delivering)

• evals/manifest.json written with correct feature IDs
• One .check.ts per feature in evals/presence/
• evals/unit/helpers/setup.ts, mocks.ts, factories.ts written
• Factory fields grounded in actual source column names (never invented)
• One .test.tsx per feature group in evals/unit/
• Every it() name starts with exact manifest.json feature ID + colon
• Post-generation grep confirms traceability for all feature IDs
• evals/runner/security-runner.ts written with GP-SEC-001 through GP-SEC-014
• evals/runner/presence-runner.ts and run-evals.ts written
• evals/vitest.config.ts has root: PROJECT_ROOT set explicitly
• evals/dashboard/index.html written, FEATURES substituted, CATEGORIES substituted
• Dashboard self-verify: no fetch(, no placeholder, no PROJECT_NAME
• npm run eval runs successfully (or failures reported to user)
• Dashboard baked with real results

---

Quality Rules

1. Never invent dataApi call patterns — grep the actual source to confirm what tables and operations are used
2. Never reference Code App patterns — no power.config.json, src/generated/, @microsoft/power-apps, result.value, result.success, initialize()
3. Never reference PCF patterns — no context.webAPI, context.parameters, ComponentFramework
4. Factory fields must come from source — read the .tsx file to see what fields the component accesses from rows before writing factories
5. Feature IDs are immutable — once assigned in manifest.json, every downstream artifact must use them verbatim
6. Error handling in every dataApi call — always generate presence checks that verify try/catch exists
7. select: is always best practice — flag absence of select in queryTable as GP-SEC-014

---

Error Handling