Google Cloud Developer Relations Lead Casey West outlined four evaluation engineering lessons learned from building agent plugins, highlighting insights gained during the development of the Google Cloud Developer plugin.
Grade Execution Trajectories
Google Cloud found that grading an agent's final text reply against a checklist is flawed because it only shows the final output rather than the actions taken. West noted that automated graders can penalize agents for successful actions if the summary does not explicitly mention them.
Engineers should score raw execution trajectories instead of closing pitches to verify what commands were actually executed, validated, and scoped.
Account for Asynchronous Operations
Telemetry extraction matters when reading tool-call trajectories because flattened timelines can obscure concurrent tool dispatch. West explained that batching multiple tool invocations into a single assistant turn can cause instructions to arrive after execution.
Preserving turn structure in telemetry pipelines helps engineers identify parallel dispatch issues before injected guidance enters the context window.
Compare Live and Plan-Only Runs
Google Cloud recommends running live and plan-only evaluations side-by-side to avoid blind spots. Live runs test actual execution and safety restraint, but permission errors can abort them prematurely.
Plan-only runs strip credentials to expose pure model reasoning and routing intent without getting tripped up by permission errors.
Use Golden Sets as Baselines
Evaluation suites should be treated as continuous integration testing for non-deterministic environments. West stated that teams should capture the whole trajectory and score runs against golden sets curated by human experts.
These baseline runs act as a safety net against model churn and changing agent harnesses.
What to do
- Score raw execution trajectories instead of grading agents on their closing pitches.
- Preserve turn structure in telemetry extractors to identify parallel dispatch issues.
- Run live and plan-only evaluations side by side to test execution restraint and raw reasoning.
- Build golden sets based on what the entire system should actually do in the wild.
Key facts and where they come from
- Google Cloud shared four evaluation engineering lessons from building the Google Cloud Developer plugin.
Here are four eval lessons we learned from building the new plugin that you can apply to your own builds.
- Automated graders that inspect only final text replies can penalize agents for actions not mentioned in summaries.
The grader penalized the agent for doing the right thing simply because it didn’t brag about it in the summary.
- Live runs and plan-only runs should be tested side-by-side to expose both execution restraint and raw reasoning.
Run live and plan-only evaluations side by side. Live runs test actual execution and safety restraint; plan-only runs expose the model's raw reasoning
- Evaluations should use golden sets curated by human experts as durable baselines.
Build golden sets based on what the entire system should actually do in the wild, including your harness, model, and plugins.