Sabastian Raschka is one of the great LLM researchers/authors. I highly recommend his substack
by souravsspace
0 subcomment
i like your detailed breakdown. Thanks. <3
by alealvarezarg
0 subcomment
Great breakdown. After using Kimi extensively, it's fascinating to see how architectural choices like KDA and NoPE translate into such strong real-world performance. Really impressive engineering.
by gokohl
4 subcomments
Interesting that they went NoPE everywhere — everyone else hedges with RoPE in the local layers. Feels like the linear-attention stuff (Kimi Delta) is quietly doing the positional work so they can get away with it. Curious to see if it holds up at frontier scale.
Hacker News