I Found 54 Reliability Issues in My 14-Agent AI System — Here's What Broke

suraj kumar Posted on May 31 I Found 54 Reliability Issues in My 14-Agent AI System — Here's What Broke # ai # python # opensource # testing Every testing tool for AI agents tests individual agents. But production failures don't happen inside agents — they happen between them. I learned this the hard way. The Problem Nobody Is Solving I built a 14-agent document processing system using CrewAI. Each agent worked perfectly in isolation. In production, the system failed constantly — and I couldn't figure out why. The problem wasn't any single agent. It was the interactions : One agent failing silently took down 12 others Agents were sharing sensitive data across boundaries they shouldn't cross Three agents formed a communication clique that bypassed the orchestrator Every agent depended on one central orchestrator with zero fallback No existing tool could find these issues. Arize, Langfuse, Braintrust — they all monitor individual agents. None of them test the graph of agent interactions. So I built one. What I Built: swarm-test swarm-test builds a NetworkX interaction graph of your multi-agent system and runs 6 chaos engineering tests against it: Cascade Failure — which agents bring down the whole system if they fail Context Leakage — sensitive data (API keys, PII, credentials) crossing agent boundaries Intent Drift — agents acting outside their role or being manipulated Collusion Detection — agents communicating outside the orchestrator's oversight Blast Radius — single points of failure and critical dependency paths Timeout Resilience — agents with no fallback if upstream is slow 3-line API: from swarm_test import SwarmProbe probe = SwarmProbe ( crew ) report = probe . run_all () report . print_summary () Enter fullscreen mode Exit fullscreen mode What It Found On My Real System I ran swarm-test on my 14-agent system. The results were brutal: 54 total findings: 15 CRITICAL (14 cascade failures + 1 SPOF) 13 HIGH (9 timeout vulnerabilities + 4 collusion cliques) 2