Given a prompt (or task) how do I evaluate if it is a "simple" task that should be executed by a small model or if it is a complex one that may need a SOTA model?

Are you guys using heuristics? Which one?

It's absolutely the case that routing between frontier models can improve results, mainly because of the alloying effect. Ping ponging between different providers gives the task exposure to different data distributions, and can break models out of non-optimal feedback loops.

That's not to say that's always the right approach, just not clearly wrong. And a small pool does not necessarily 'improve' model routing. The real advice is just 'know why you're routing to each model'.

Especially with guidance to map to improve based on performance - with a large enough volume of tasks/requests, you'd want to maximize the initial pool size to expand the search space in order to determine which is best at each task.

I read this as "here are some thoughts on model routing" -- not first principles I'd advise everyone to live by.