I Added a 71-Line Black Box to My Python Agent, Then Queried the $200 Crash With DuckDB

S M Tahosin Posted on May 31           I Added a 71-Line Black Box to My Python Agent, Then Queried the $200 Crash With DuckDB # python # ai # debugging # tutorial The incident started with a boring support automation task. Take a user request, search a private document index, summarize the answer, and hand the result to a reviewer. Nothing heroic. The kind of Python agent you build when the demo is over and the real workflow begins. Then one run got stuck in a retry loop. It did not burn $200 before I caught it. The actual test run was cheaper. The problem was the projection: same bad loop, same document search, same model calls, left inside the overnight batch. The estimate landed close to $200 for one avoidable failure. The answer it produced looked polished enough to pass a sleepy review. The trace behind it was not polished at all. The agent had called the right tool with the wrong input, retried against stale context, summarized old results, and kept paying for each turn. That is when I stopped treating the agent like a chat feature. I started treating it like a system that needs a black box. Not a dashboard. Not a full observability stack. Not another hosted service. Just one local file that can answer: What did the agent try? Which tool did it call? What input did the tool receive? Did the tool fail? How long did it take? Did the run cross a cost or turn limit? Can I query the run after everything is over? We will build that black box in plain Python, then use DuckDB to inspect it like a tiny crash database. Before And After Before the fix, debugging looked like this: The final answer is wrong. The model probably hallucinated. Maybe the search tool returned bad data. Maybe the retry loop reused an old message. Maybe the cost spike came from the model call. Enter fullscreen mode Exit fullscreen mode That is not debugging. That is guessing with syntax highlighting. After the fix, debugging looked like this: Turn 1 called search_docs wi