One Wikipedia page costs your AI agent 68,000 tokens
14 points by arhamislam5766
by bugalati
0 subcomment
gonna try this out this week, one ask: when a wall does beat it, return "blocked, here's why" instead of empty, avoiding silent fetch failures.
by chonghaoju
1 subcomments
stripping to markdown with Jina Reader or Trafilatura before passing to the agent cuts that 68k down to ~3-5k for most Wikipedia pages, and handles the JS-rendered case too.
by arthurcolle
0 subcomment
one of the early jina.ai products was/is reader api, and they trained ReaderLM for this purpose. definitely a good idea to check out existing implementations
by m463
1 subcomments
> on js-rendered and some anti-bot pages it returns nothing
yeah, and my (human piloted) browser gets blocked by so many websites now. I routinely get "Sorry." when trying to log into hn.
Hacker News