	In Browser
	StumbleUpon
	del.icio.us
	Google
	Google Buzz
	reddit
	LinkedIn

	Facebook
	Twitter
	Linkedin
	E-Mail

Agentic AI > ADK Agent Testing > Regression Testing ADK Agents After Prompt Changes

Regression Testing ADK Agents After Prompt Changes

Author: Venkata Sudhakar

When ShopMax India's team updates an ADK agent's system prompt to improve tone or add new product lines, existing behaviors can break silently. A golden-set regression test suite locks in expected behavior before the change and fails immediately if any critical behavior regresses after the update. This gives the team confidence to iterate on prompts without manual re-testing of every known scenario.

A regression suite stores a list of test cases, each with an input query, a list of terms the response must contain, and a list of terms it must not contain. Tests are parameterized with pytest.mark.parametrize so a single test function covers the whole golden set. The LLM response is mocked to return deterministic text that mimics the expected agent behavior, and assertions enforce the must_contain and must_not_contain rules on every run.

The example below shows a three-case golden set for ShopMax India covering order tracking, pricing, and refund scenarios, with both positive and negative content assertions on each response.

import pytest
from unittest.mock import patch, AsyncMock, MagicMock
from google.adk.runners import InMemoryRunner
from google.adk.agents import Agent

GOLDEN_SET = [
    {"input": "Track order ORD-7821",
     "mock_reply": "Your order ORD-7821 has been dispatched from Bangalore.",
     "must_contain": ["ORD-7821", "Bangalore"],
     "must_not_contain": ["error", "unavailable"]},
    {"input": "Price of Sony WH-1000XM5",
     "mock_reply": "Sony WH-1000XM5 is available at Rs 28,990 on ShopMax.",
     "must_contain": ["Rs", "Sony", "ShopMax"],
     "must_not_contain": ["out of stock", "error"]},
    {"input": "Refund order ORD-5541",
     "mock_reply": "Refund for ORD-5541 initiated. Rs 14,990 credited in 3-5 days.",
     "must_contain": ["refund", "Rs 14,990"],
     "must_not_contain": ["cannot", "sorry"]},
]

def build_agent(prompt):
    return Agent(name="shopmax_agent", model="gemini-2.0-flash", instruction=prompt)

def make_resp(text):
    r = MagicMock(text=text)
    r.candidates = [MagicMock(content=MagicMock(parts=[MagicMock(text=text)]))]
    return r

@pytest.mark.parametrize("case", GOLDEN_SET)
@patch("google.adk.models.google_llm.Gemini._generate_content_async", new_callable=AsyncMock)
async def test_golden_set_regression(mock_llm, case):
    mock_llm.return_value = make_resp(case["mock_reply"])
    agent = build_agent("You are a ShopMax India support agent. Be accurate and helpful.")
    runner = InMemoryRunner(agent=agent, app_name="regression")
    events = [e async for e in runner.run_async(
        user_id="u1", session_id="s1", new_message=case["input"]
    )]
    all_text = " ".join([
        e.content.parts[0].text for e in events
        if getattr(e, "content", None) and e.content.parts
    ]).lower()
    for term in case["must_contain"]:
        assert term.lower() in all_text, "Missing: " + term
    for term in case["must_not_contain"]:
        assert term.lower() not in all_text, "Unexpected: " + term

if __name__ == "__main__":
    pytest.main([__file__, "-v"])

It gives the following output,

tests/test_regression.py::test_golden_set_regression[case0] PASSED
tests/test_regression.py::test_golden_set_regression[case1] PASSED
tests/test_regression.py::test_golden_set_regression[case2] PASSED

3 passed in 0.29s

In production, load GOLDEN_SET from a JSON file under tests/golden/ so non-engineers can add test cases without touching Python code. Run the regression suite as a required CI check on every pull request that modifies agent instructions, tool definitions, or model version. When a case fails after a prompt change, treat it as a deliberate behavior change that needs a sign-off - update the golden set only after the team agrees the new behavior is correct.

Send your comments, suggestions or queries regarding this site to [email protected].