agent-lightning

Agent Lightning

Agent optimization is a search over interacting choices. The useful question is not which architecture is most sophisticated, but which change moves the requested accuracy, cost, latency, and reliability frontier for this agent.

The optimizer's development budget and the resulting agent's per-run cost are different quantities. More development budget creates room to learn; it does not imply that the deployed agent should spend more on every task.

Your remaining development budget is reported at /artifacts/cost_budget.json ({spent, budget, remaining}), refreshed as you work. Read that file to see how much is left, and keep iterating — measure, edit, re-score — while meaningful budget remains; do not stop at the first plausible result. A run is finished not because one change worked, but because further measured changes no longer improve the frontier within the budget you still have. If remaining is large, there is more search to do: ground more cases, probe a lever you have not tested, or add reps to resolve a noisy comparison. Check remaining again after each expensive step so the decision to stop is evidence-based, not a default.

Action space

Reading the evidence

Different failures expose different amounts of information:

• Exceptions, missing artifacts, invalid schemas, and timeouts are objective signals. They can support deterministic checks or focused recovery.
• A valid-looking but wrong answer may expose no label-free repair signal. More calls with the same information can repeat the same mistake.
• Repeated failures across different attempts suggest shared blindness, a contract mismatch, or missing capability. Diverse failures make routing, critique, or selection more plausible.
• A development gain that disappears under validation may come from randomness, memorized examples, training-only fields, or a different deployment path.
• An unavailable measurement is unknown evidence, not proof that a candidate improved or regressed.

Levers interact. More reasoning cannot recover information the model never sees. Repair cannot fix a systematic contract error when the retry receives no new evidence. Failure isolation preserves the batch but does not repair an item. A global model or effort increase and conditional escalation occupy different points on the frontier.

Evaluation context

Agent evaluations are often stochastic. A score can improve while many individual cases regress, and a single strong result can be a lucky draw. Fixed cases, validation splits, repeated runs, frozen primary outputs, and small probes are different ways to reduce uncertainty; their value depends on the decision and the available budget.

Comparisons are easiest to interpret when the checkpoint, cases, model, effort, concurrency, scorer, and execution path are held constant except for the variable being studied. Accuracy is only one result: completion rate, downside, cost, latency, and variance can change the decision.

Development may expose labels, metadata, or tools that do not exist at deployment. An improvement that depends on them is not a deployed improvement. Likewise, measurement plumbing can fail independently of the agent being tested.

Boundaries

• Preserve the target's external interface and deployment environment.
• Do not expose held-out labels or training-only answer fields to the deployed path.
• Treat the scorer and evaluation contract as immutable measurement surfaces.
• Leave a coherent measured checkpoint, not an unfinished or partially tested edit.