ADK Agent Test Strategy - Automation vs Manual Testing

ShopMax India's engineering team frequently asks: can all ADK agent testing be automated, or is manual review always required? The answer depends on what you are testing. Automation excels at structural correctness, regression detection, performance benchmarks, and high-volume coverage. Manual testing remains essential for tone and empathy assessment, novel edge cases outside the training distribution, cultural sensitivity, and evaluating whether an agent response feels right to a real customer.

Automated testing covers four layers: unit tests for individual tools, integration tests for agent pipelines with mocked LLMs, regression tests using golden datasets with LLM-as-judge scoring, and contract tests for output schema validation. These catch 80-90% of regressions and run in CI pipelines within minutes. Manual testing fills the gap: human reviewers sample 50-100 live responses per week, focus on low-confidence outputs, and test scenarios that require common sense or cultural awareness that automated metrics cannot capture.

The example below shows ShopMax India's test strategy as a triage function. It routes each validation to the appropriate method - automated checks or a manual review queue - based on confidence score, query category, and keyword signals.

It gives the following output,

Track order ORD-7821                       -> automated [PASS]
I want to file a fraud complaint           -> both [PASS]
Is Galaxy S24 in stock?                    -> automated [PASS]
My order arrived damaged                   -> manual_review [QUEUED_FOR_REVIEW]

Set a weekly manual review quota - 50 responses is enough to catch systematic failures that automated tests miss. Prioritize low-confidence responses, new feature areas, and complaint categories for manual review. When a manual reviewer finds a failure, add it to the golden regression set immediately so the same issue never passes undetected again. Automated testing is not a replacement for manual review - it is a force multiplier that lets your team focus human attention where it matters most. Track the ratio of automated-to-manual catches over time: if manual review consistently finds things automation misses, invest in better automated metrics for that category.