	In Browser
	StumbleUpon
	del.icio.us
	Google
	Google Buzz
	reddit
	LinkedIn

	Facebook
	Twitter
	Linkedin
	E-Mail

Agentic AI > ADK Agent Testing > Chaos Testing ADK Agents - Injecting Faults and Timeouts

Chaos Testing ADK Agents - Injecting Faults and Timeouts

Author: Venkata Sudhakar

ShopMax India's ADK agents depend on Gemini LLM calls, database tools, and external APIs. Any of these can fail in production - timeouts, connection errors, rate limit responses, and unexpected exceptions. Chaos testing deliberately injects these failure modes during tests to verify that the agent handles them gracefully: returning a sensible fallback response rather than crashing or leaking error details to customers.

Chaos testing uses pytest with AsyncMock side_effect to inject faults: asyncio.TimeoutError for slow LLM responses, custom exceptions for tool failures, and RuntimeError for unexpected crashes. The agent wrapper must catch each fault type and return an appropriate user-facing message. The test asserts both the fault tolerance (no crash) and the response quality (message is helpful, not a raw traceback). This is distinct from unit testing - chaos tests verify the failure handling code path, not the happy path.

The example shows ShopMax India testing three failure scenarios against their order tracking agent: LLM timeout, database tool failure, and an unexpected internal error. Each test verifies the agent returns a safe fallback response.

import asyncio
import pytest
from unittest.mock import AsyncMock, patch

class ToolFailureError(Exception):
    pass

async def resilient_agent_call(query, llm_fn):
    try:
        result = await asyncio.wait_for(llm_fn(query), timeout=2.0)
        return {"status": "ok", "response": result}
    except asyncio.TimeoutError:
        return {"status": "timeout", "response": "Request timed out. Please try again."}
    except ToolFailureError as e:
        return {"status": "tool_error", "response": "A tool failed. Trying fallback."}
    except Exception:
        return {"status": "error", "response": "Something went wrong. Our team has been notified."}

@pytest.mark.asyncio
async def test_agent_handles_llm_timeout():
    async def slow_llm(query):
        await asyncio.sleep(5.0)
        return "response"

result = await resilient_agent_call("Track ORD-7821", slow_llm)
    assert result["status"] == "timeout"
    assert "timed out" in result["response"].lower()
    print("Timeout test:", result["response"])

@pytest.mark.asyncio
async def test_agent_handles_tool_failure():
    async def failing_tool(query):
        raise ToolFailureError("Database connection refused")

result = await resilient_agent_call("Track ORD-7821", failing_tool)
    assert result["status"] == "tool_error"
    assert "fallback" in result["response"].lower()
    print("Tool failure test:", result["response"])

@pytest.mark.asyncio
async def test_agent_handles_unexpected_error():
    async def broken_llm(query):
        raise RuntimeError("Internal model error")

result = await resilient_agent_call("Track ORD-7821", broken_llm)
    assert result["status"] == "error"
    assert "something went wrong" in result["response"].lower()
    print("Unexpected error test:", result["response"])

It gives the following output,

Timeout test: Request timed out. Please try again.
Tool failure test: A tool failed. Trying fallback.
Unexpected error test: Something went wrong. Our team has been notified.

Cover all failure modes your agent depends on: LLM timeout, LLM rate limit (429), database unreachable, third-party API 500, and malformed LLM response (JSON parse error). Add chaos tests to CI alongside integration tests - they run fast since faults are injected, not waited for. Never let raw exception messages reach the user response: assert that no test output contains the word traceback or exception. Review chaos test coverage quarterly as you add new tools and integrations.

Send your comments, suggestions or queries regarding this site to [email protected].