	In Browser
	StumbleUpon
	del.icio.us
	Google
	Google Buzz
	reddit
	LinkedIn

	Facebook
	Twitter
	Linkedin
	E-Mail

Agentic AI > ADK Agent Testing > Quality Scorecards for ADK Agents - Tracking Metrics Across Releases

Quality Scorecards for ADK Agents - Tracking Metrics Across Releases

Author: Venkata Sudhakar

Quality scorecards provide a structured summary of ADK agent performance across multiple dimensions - accuracy, helpfulness, latency, and safety - tracked across releases so regressions are visible before deployment. ShopMax India generates a quality scorecard after every release candidate build to compare the new agent version against the production baseline across all customer-facing agents in Mumbai, Delhi, and Hyderabad.

A scorecard is a dict mapping metric names to values for a given agent version. Scorecard comparison diffs two dicts and flags any metric that degraded beyond a tolerance band. Metrics are computed by running the agent against a fixed evaluation dataset and aggregating results. The scorecard is serialized to JSON and stored as a CI artifact so historical trends can be plotted over time.

The example below computes a quality scorecard for a ShopMax India order agent, compares it against a baseline scorecard, and asserts no metric regressed beyond its allowed tolerance.

import pytest
import json
from typing import Dict

TOLERANCES = {
    "accuracy":    0.02,
    "helpfulness": 0.05,
    "p95_ms":      20.0,
    "safety_pass_rate": 0.01,
}

BASELINE_SCORECARD = {
    "version": "v1.4.0",
    "accuracy": 0.94,
    "helpfulness": 0.88,
    "p95_ms": 185.0,
    "safety_pass_rate": 1.00,
}

def compute_scorecard(version: str) -> Dict:
    return {
        "version": version,
        "accuracy": 0.93,
        "helpfulness": 0.89,
        "p95_ms": 190.0,
        "safety_pass_rate": 1.00,
    }

def compare_scorecards(baseline: Dict, candidate: Dict) -> Dict:
    regressions = {}
    for metric, tolerance in TOLERANCES.items():
        base_val = baseline[metric]
        cand_val = candidate[metric]
        if metric == "p95_ms":
            delta = cand_val - base_val
            if delta > tolerance:
                regressions[metric] = {"baseline": base_val, "candidate": cand_val, "delta": delta}
        else:
            delta = base_val - cand_val
            if delta > tolerance:
                regressions[metric] = {"baseline": base_val, "candidate": cand_val, "delta": delta}
    return regressions

def test_scorecard_no_regression():
    candidate = compute_scorecard("v1.5.0")
    regressions = compare_scorecards(BASELINE_SCORECARD, candidate)
    print(f"Scorecard v1.5.0: accuracy={candidate['accuracy']}, p95={candidate['p95_ms']}ms")
    assert not regressions, f"Quality regressions detected: {json.dumps(regressions, indent=2)}"

It gives the following output,

Scorecard v1.5.0: accuracy=0.93, p95=190.0ms
1 passed in 0.03s

Store baseline scorecards in a scorecards/ directory under version control so that the comparison is always against the last released version, not an arbitrary snapshot. Add a scorecard summary step to the CI pipeline that prints a diff table to the PR comment so reviewers see the quality impact at a glance. Track the scorecard trend over 10 releases to spot slow drift that individual release comparisons miss.

Send your comments, suggestions or queries regarding this site to [email protected].