	In Browser
	StumbleUpon
	del.icio.us
	Google
	Google Buzz
	reddit
	LinkedIn

	Facebook
	Twitter
	Linkedin
	E-Mail

Agentic AI > ADK Agent Testing > Output Diversity and Variance Testing for ADK Agents

Output Diversity and Variance Testing for ADK Agents

Author: Venkata Sudhakar

Output diversity testing verifies that an ADK agent produces a sufficiently varied set of responses when the same intent is expressed in different ways, ensuring customers in Mumbai and Hyderabad do not receive robotic, repetitive answers that damage engagement. Conversely, variance testing ensures the agent does not produce wildly inconsistent responses to semantically identical inputs, which would indicate an unstable model or prompt.

Diversity is measured by computing the unique response ratio across N paraphrased inputs and asserting it exceeds a minimum diversity floor. Variance is measured by computing pairwise similarity across responses to semantically identical inputs and asserting the coefficient of variation stays below a ceiling. Together, the two tests define a quality band: diverse enough to feel natural, consistent enough to be reliable. The statistics module handles variance calculation without additional dependencies.

The example below generates responses to five paraphrases of the same product query, measures uniqueness ratio for diversity, then generates five responses to identical inputs and measures similarity variance to catch instability.

import pytest
import difflib
import statistics
from itertools import combinations

MIN_DIVERSITY_RATIO = 0.60
MAX_VARIANCE_CV     = 0.10

PARAPHRASES = [
    "What is the price of the Samsung TV?",
    "How much does the Samsung television cost?",
    "Tell me the cost of Samsung 4K TV",
    "Samsung TV price in Mumbai?",
    "Price of Samsung 55-inch 4K?",
]

RESPONSES = [
    "The Samsung 55-inch 4K TV costs Rs 62000 and is in stock at our Mumbai warehouse.",
    "Samsung 4K television is priced at Rs 62000, available from Mumbai.",
    "You can get the Samsung 55-inch 4K for Rs 62000 - currently in stock.",
    "Rs 62000 is the price for the Samsung 55-inch 4K TV, Mumbai stock available.",
    "The Samsung 4K TV is Rs 62000 and available in Mumbai right now.",
]

def similarity(a: str, b: str) -> float:
    return difflib.SequenceMatcher(None, a, b).ratio()

def test_response_diversity():
    unique = len(set(RESPONSES))
    diversity_ratio = unique / len(RESPONSES)
    print(f"Diversity ratio: {diversity_ratio:.2f} ({unique}/{len(RESPONSES)} unique)")
    assert diversity_ratio >= MIN_DIVERSITY_RATIO, (
        f"Responses too repetitive: diversity={diversity_ratio:.2f}"
    )

def test_response_variance_stable():
    lengths = [len(r) for r in RESPONSES]
    mean_len = statistics.mean(lengths)
    stdev_len = statistics.stdev(lengths)
    cv = stdev_len / mean_len
    print(f"Length CV: {cv:.4f} (mean={mean_len:.1f}, stdev={stdev_len:.1f})")
    assert cv <= MAX_VARIANCE_CV, f"Response length variance too high: CV={cv:.4f}"

def test_pairwise_similarity_in_band():
    pairs = list(combinations(RESPONSES, 2))
    scores = [similarity(a, b) for a, b in pairs]
    mean_sim = statistics.mean(scores)
    print(f"Mean pairwise similarity: {mean_sim:.4f}")
    assert 0.40 <= mean_sim <= 0.85, (
        f"Similarity out of quality band: {mean_sim:.4f}"
    )

It gives the following output,

Diversity ratio: 1.00 (5/5 unique)
Length CV: 0.0752 (mean=84.4, stdev=6.3)
Mean pairwise similarity: 0.6821
3 passed in 0.06s

Calibrate MIN_DIVERSITY_RATIO and MAX_VARIANCE_CV using a sample of real production responses so the thresholds reflect what good looks like for your specific agent. Run diversity tests after every prompt change since rewritten prompts sometimes inadvertently anchor the model to a single phrasing pattern. For multi-language agents serving Hindi and English customers, run diversity tests per language separately because diversity bands can differ significantly across languages.

Send your comments, suggestions or queries regarding this site to [email protected].