Agentic QA – Open-source middleware to fuzz-test agents for loops
31 points by Saurabh_Kumar_
by Saurabh_Kumar_
0 subcomment
HN, OP here.
I built this because I recently watched my LangChain agent burn through ~$50 of OpenAI credits overnight. It got stuck in a semantic infinite loop (repeating "I am checking..." over and over) which my basic max_iterations check didn't catch because the phrasing was slightly different each time.
Realizing that "Pre-Flight" testing for agents is surprisingly hard, I built a small middleware API (FastAPI + LangChain) to automate this.
What it does:
It acts as an adversarial simulator. You send it your agent's system prompt, and it spins up a 'Red Team' LLM to attack it.
Currently checks for:
Infinite Loops: Semantic repetition detection.
PII Leaks: Attempts social engineering ('URGENT AUDIT') to force the agent to leak fake PII, then checks if it gets blocked.
Prompt Injection: Basic resistance checks.
Tech Stack: Python, FastAPI, Supabase (for logs).
It's open-source and I hosted a live instance on Render if you want to try curl it without installing:
https://agentic-qa-api.onrender.com/docs
Would love feedback on what other failure modes you've seen your agents fall into!
by mikigraf
0 subcomment
Almost thought you found my startup AgenticQA.eu
by giancarlostoro
0 subcomment
I had Claude Code losing its mind because of something outside of its control, one of the formatters used by Zed for Python kept messing with HTML templates, which are insanely sensitive to line breaks in some template specific code statements. Zed kept adding line breaks without reason other than some tool just did it. Claude kept trying to fix it, going to the extreme of using ed to force it, I watched it lose its mind till I asked "I think Zed is formatting the file every time you save?" turns out, yes, yes it was. It wasn't an issue when it used ed, but when Claude or I would change the file again, it would become an issue again.
I don't know what could have saved me, maybe .current_editor should be a file that your agents instructions.md file imports, and your editor updates it, to give Claude context about your tooling.
Hacker News