Automated Rollback Triggers Based on ADK Agent Quality Degradation

Automated rollback triggers halt a deployment and revert to the previous agent version when quality metrics drop below safe thresholds in production. ShopMax India uses rollback triggers on its order processing and search agents so that a bad deployment during a Diwali sale in Mumbai or Delhi is automatically reversed within minutes rather than waiting for a human to notice degraded customer experience.

The rollback system polls a metrics window (e.g. last 5 minutes of production traffic) and compares rolling averages against rollback thresholds. Thresholds are stricter than CI acceptance thresholds to account for real traffic variance. When any metric breaches its threshold, a RollbackDecision is emitted with the triggering metric and the recommended action. In tests, the evaluator is run against synthetic metric windows to verify it fires correctly and does not trigger false positives.

The example below defines a quality evaluator that checks an error rate and a safety pass rate metric window, and tests both the rollback trigger and the no-rollback happy path.

It gives the following output,

Rollback triggered: error_rate=0.12 > 0.05
Rollback triggered: safety_pass_rate=0.88
No rollback: all metrics within thresholds
3 passed in 0.03s

Integrate the rollback evaluator into a Cloud Run job that polls Cloud Monitoring metrics every 2 minutes after a new deployment. Use a dead man switch pattern - if the evaluator itself fails, treat it as a rollback trigger to avoid silent quality degradation going undetected. Store rollback decisions in a log with timestamps and metric snapshots so post-incident reviews can reconstruct exactly when quality degraded and why.