And it doesn't care about robots.txt.

This might be one reason why Claude Code uses XML for tool calling: repeating the tag name in the closing bracket helps it keep track of where it is during inference, so it is less error prone.

Then langchain and structured schemas for the output along w/ a specific system prompt for the LLM. Do you know which open source models work best or do you just use gemini in production?

Also, looking at the docs, Gemini 2.5 flash is getting deprecated by June 17th https://ai.google.dev/gemini-api/docs/deprecations#gemini-2.... (I keep getting emails from Google about it), so might want to update that to Gemini 3 Flash in the examples.

I need to extract article content, determine it's sentiment towards a keyword and output a simple json with article name, url, sentiment and some text around the found keyword.

Currently I'm having problems with the json output, it's not reliable enough and produces a lot of false json.

Even Cloudflares bot filter only blocks some of them.

I'm using honeypot URLs right now to block all crawlers that ignore rel="nofollow", but they appear to have many millions of devices. I wouldn't be surprised if there are a gazillion residential routers, webcams and phones that are hacked to function as a simple doorways.

Things are really getting out of hand.

It may or may not be, but if you want people to actually use this product I’d suggest improving your documentation and replies here to not look like raw Claude output.

I also doubt the premise that about malformed JSON. I have never encountered anything like what you are describing with structured outputs.