However, the overall capability of the chatbot to fully meet user needs received a lower average score (3.1/5.0), highlighting the need for further improvements.

   Live traffic evaluations are essential for monitoring system behavior, identifying potential issues like hallucinations in production, and understanding performance on diverse live queries.

Would be interesting to know how much money was flushed down the toilet with these experts.

The alignment process goes very quickly once you have all the fish in exactly one barrel. I think pulling data dynamically from the source systems is where this turns into a game of whack-a-mole.

The problem with dynamic fetch is that you don't get any kind of persistent or compounding gains. There are queries that you simply cannot run because you'd chew through your GitHub, et. al., API quotas. It takes over 48h to fully hydrate the database for GitHub items on my current project. But, once that process is complete I can query across things like issue comments and do crosscutting joins with the state of other vendor systems in milliseconds.

I am finding the MSSQL dialect to be quite agreeable to the OAI models. With absolutely no prompting they will bootstrap off information schema and extended description properties every single time. If you design the schema for your audience, the amount of "Jesus prompting" you will require is much better controlled.

Mid-2026, we have very large context windows, and much smarter models than we did in 2024 when this was built. If I were to tackle this today I'd ask a current frontier model to work through the source data and design a hierarchy that would give it the ability to sift through the content itself by drilling down as it sees fit, and I expect it would nail that.

Hmm...

The first sentence makes it seem like they just used to improve sentence structure etc but the second line makes it seem like they used it for 90% of the work. Which one is true?

Not sure how you manage to measure Faithfulness and Answer Relevancy on the live system, without the ground truth.

Good that you have evals in place, but the user satisfaction score might suggest running ablations on the system would be beneficial. I would start by reducing the iterations and unnecessary steps from the agent.

This is an ongoing working project since 2024, I would like to see some KPI metrics to back off any productivity /job satisfaction improvement in the research department, or what have you, at Bayer.

Monthly average token usage would be another interesting information to read about. Paired with any latency numbers (time to first token, for example).

I think right now I'm mostly disappointed with agents writing code as they always degrade the quality of the codebase after a while, and the same goes for writing in general which just requires a ton of editing and mostly just sounds good but doesn't have a lot of substance in the end. I think you can really tell that these systems are trained to just produce plausible streams of text, especially in longer artefacts you notice that locally the inner consistency of what they produce is great but globally it really falls apart, it's like seeing the limits of their "intelligence".

For search however I really like AI, it has improved information retrieval so much for me where before I had to think about which keywords to use and combine and which filters to apply, describing what I'm looking for in plain text and then having the AI find it for me feels magical. Recently I wanted to find an artist that I heard in some old episode of the KEXP runcast (a running podcast), and I didn't remember anything except that it was rap with a kind of monotone voice a fast beat and a strong accent. Googles' agent asked a few clarifying questions and after a few rounds it found the artist for me, Genesis Uwusu. That's why I think Google will win in the AI assisted search market, they just have the best integration between fast and reasonably "smart" agents and high quality search data. Claude or ChatGPT are too slow and don't have fast enough data retrieval it seems, using them for search feels quite sluggish in comparison.

> teaches an O’Reilly course on building production-ready RAG applications

isn't this basically saying that you are a scammer? or am I paranoid?

The data you need to get into context for a small model, vs a big boy frontier model, vs a fine tuned open weight big boy- are all very different. I can understand what they're doing here, and most of the 'why', but- not all of the why.

The model gets it wrong on occasion and I check the input file with Claude/Opus and it just laughs at how simple it is to get the document right

And in the back of my mind I’m thinking why am I not just sending the file through Opus