one thing i'm curious about: how do you handle testing the tool selection logic itself? like the agent choosing WHICH tool to call is often where things break, not the tool execution

we had a support agent that would sometimes call the "refund order" tool when the user just wanted to check order status. the tool worked perfectly, the LLM just kept picking the wrong one. your mock platform lets you verify the tool returns the right data, but does it catch when the agent calls the wrong tool entirely?

also the full-session evaluation vs turn-by-turn is spot on. had a similar issue with a verification flow where each individual turn looked fine in langsmith but the overall flow was completely broken. you'd see "assistant asked for name" (good), "assistant asked for phone" (good), "assistant processed request" (good), but it never actually verified the phone number matched the account

tbh this feels like one of those problems that's obvious in hindsight but nobody builds the tooling for until they get burned in production

One thing that's helped us is tagging each test with an explicit risk class (safety/compliance, business logic, UX) and tracking those buckets over time instead of relying on one pass/fail number. Release decisions get much less hand-wavy when one category starts creeping.

Session-level eval plus risk-bucket trends feels like the right combo for teams shipping agents weekly.

one failure mode that's specific to browser agents and doesn't get much attention: the test environment is too clean. when you run simulations against a controlled setup, the agent never encounters the friction that real sessions do - bot detection challenges, CAPTCHAs, dynamic content that loads differently, fingerprinting checks mid-session. so you end up with agents that pass your test suite but fail in the wild, and the gap is in the environmental assumptions not the agent logic.

the mock tool platform approach is interesting precisely because it sidesteps this - you're testing the agent's decision-making in isolation from the messy runtime. that's valid for catching logic regressions. but i'd be curious how you handle cases where the tool call itself triggers secondary effects in the environment (e.g. the API call changes session state in ways that affect what the agent sees next).

also, does your session-level judge handle cases where the correct behavior is adaptive - where the agent should change strategy mid-session based on what it encountered? that feels like a harder eval problem than a fixed expected outcome.

I have found when using agents to verify agents, that the agent might observe something that a human would immediately find off-putting and obviously wrong but does not raise any flags for the smart-but-dumb agent.