ADK Agent Observability Testing with OpenTelemetry

ShopMax India's on-call team needs visibility into every agent request - how long each tool call took, which model was used, and how many tokens were consumed. OpenTelemetry provides a standard SDK for emitting traces and spans from ADK agents. Testing the observability instrumentation ensures that spans are created for every tool call, span attributes contain the expected fields, and no request goes untraced in production.

The test uses the OpenTelemetry in-memory span exporter to capture all spans emitted during a test run without sending them to a real backend. After the agent runs, the test queries the captured spans and asserts that the expected span names exist, span attributes contain the required keys and values, and token count attributes are populated correctly on the root span.

The example below instruments a ShopMax India mock agent with OpenTelemetry spans and tests that spans are correctly emitted for both the root request and the inner tool call, with the correct attributes on each.

It gives the following output,

Emitted spans: ['tool.get_order_details', 'agent.request']
Response: Order ORD-7821 is shipped. Arrives by 26 Apr.
.. (2 passed in 0.05s)

In production, replace the in-memory exporter with an OTLP exporter that ships spans to Google Cloud Trace or any OpenTelemetry-compatible backend. Add span attributes for city, agent version, and model name so the ShopMax India dashboard can filter traces by region and deployment. Run the span assertion tests in CI against the in-memory exporter and run a separate integration test against a local Jaeger container to validate the OTLP export path before each release.