FRESH

Hacker News

Challenges and Research Directions for Large Language Model Inference Hardware

120 points by transpute

by random3

0 subcomment

David Patterson is such a legend! From RAID to RISC and one of the best books in computer architecture, he's on my personal hall of fame.
Several years ago I was at one of the Berkley AMP Lab retreats at Asilomar, and as I was hanging out, I couldn't figure how I know this person in front of me, until an hour later when I saw his name during a panel :)).
It was always the network. And David Patterson, after RISC, started working on iRAM, that was tackling a related problem.
NVIDIA bought Mellanox/Infiniband, but Google has historically excelled at networking, and the TPU seems to be designed to scale out in the best possible way.

by jauntywundrkind

1 subcomments

> To address these challenges, we highlight four architecture research opportunities: High Bandwidth Flash for 10X memory capacity with HBM-like bandwidth; Processing-Near-Memory and 3D memory-logic stacking for high memory bandwidth; and low-latency interconnect to speedup communication.
High Bandwidth Flash (HBF) got submitted 6 hours ago! It's a great article, fantastic coverage of a wide section of the rapidly moving industry. https://news.ycombinator.com/item?id=46700384 https://blocksandfiles.com/2026/01/19/a-window-into-hbf-prog...
HBF is about having many dozens or hundreds of channels of flash memory. The idea of having Processing Near HBF, spread out, perhaps in mixed 3d design, would be not at all surprising to me. One of the main challenges for HBF is building improved vias, improved stacking, and if that tech advanced the idea of more mixed NAND and compute layers rather than just NAND stacks perhaps opens up too.
This is all really exciting possible next steps.

by zozbot234

0 subcomment

Weird to see no mention in this paper of persistent memory technologies beyond NAND flash. Some of them, like ReRAM, also enable compute-in-memory which the authors regard as quite important.

by HPsquared

5 subcomments

Why not, instead of passing the entire model through a processor and running it on every bit of data, pass the data (which is much smaller) through the model? As in, have compute and memory together in the silicon. Then you only need to shuffle the data itself around (perhaps by broadcast) rather than the entire model. That seems like it would use a LOT less energy.
Or is it not possible to make the algorithms parallel to this degree?
Edit: apparently this is called "compute-in-memory"

by bluehat974

1 subcomments

by suggeststrongid

2 subcomments

by amelius

0 subcomment