Vercel published a guide on November 12, 2026, detailing how developers can build end-to-end evaluations to test whether AI agents complete software tasks correctly.
What it does
Vercel says evals give you a way to measure how well work gets done when AI agents act as users of software like APIs, CLIs, MCP servers, or SDKs.
Before you start
Vercel states you should start with the problem you want to catch and pick the lowest tier in the agentic testing pyramid that can catch it.
Steps
Step 1: Set a goal. Vercel recommends starting with one outcome your users care about, narrow enough to evaluate with a handful of tasks, found in observability, customer support tickets, or your product roadmap.
Step 2: Define metrics. Vercel states you should measure success rate, Pass@k, Pass^k, and efficiency metrics including steps, time, and cost.
Step 3: Design the task set. Vercel advises writing tasks with an instruction, an environment, a verifier, and an optional known-good solution.
Step 4: Run the eval. Vercel explains you can install Harbor, connect a Vercel project, pass tokens, and run the job concurrently in Vercel Sandboxes.
Step 5: Analyze traces. Vercel recommends trace-diving to read agent trajectories and understand why failures occurred.
What the company says
Vercel states that evals are the infrastructure you use to make your product better, providing tools like AI Gateway, Sandbox, Drives, and Blob to build them in one place.
Key facts and where they come from
- Vercel published the blog post on November 12, 2026.
Published: November 12, 2026
- Harbor is an open-source framework from the creators of Terminal-Bench.
Harbor, an open-source framework from the creators of Terminal-Bench
