FRESH

Hacker News

Home

Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

107 points by NicoConstant

by stymaar

0 subcomment

This is very cool.
I have been lamenting for a while that the memory-bandwidth <-> tps relationship was pretty much working for small models on consumer cards, but not at all on datacenter hardware.
It's great to see that with proper care on the inference engine implementation the relationship can be restored.

by mungoman2

4 subcomments

This looks very interesting. Possible to get those rates without exotic hardware.
But I have to say that the comparison is not really fair. Comparison is done with a 2 B model vs frontier models that are likely 100s of times larger. Also taalas with their 15000 tok/s inference are suspiciously missing from the comparison.
We need to see the comparison with this framework and useful models, which at present seems to mean ~30 B.

by gaeld

1 subcomments

Follow-up reading the most technical and research people here:
Monokernel deep dive (GPU Engineering): http://blog.kog.ai/building-a-single-kernel-latency-optimize...
Delayed Tensor Parallelism (research): http://blog.kog.ai/delayed-tensor-parallelism-for-faster-tra...
To try the speed on the playground: http://playground.kog.ai

by rashkov

0 subcomment

Don't miss trying their demo: https://playground.kog.ai/
Feels like a preview of the future

by bcjdjsndon

0 subcomment

H200 isn't a standard GPU at all

by 867-5309

3 subcomments

> Standard GPUs
> 8× NVIDIA H200

by 0-bad-sectors

2 subcomments

When I read "Standard GPUs" in the title I got excited for a second then I read the article itself..

by cataflam

0 subcomment

Congrats gaeld and team
The demo is very impressive!
disclaimer: I've known the founder for a while, as legitimate as it gets in deep tech, real years of research and engineering behind this, not vaporware

by CastFX

1 subcomments

Looks super promising! A couple of questions:
For new open weights models, will you need to adapt model code and optimization for your inference engine by hand?
It's true that BS=1 is king when it comes to agentic workflows, however these kinds of system serve multiple requests concurrently with dynamic batching. Do you think it will scale as well ?
Any plans to release it open source?
Congratz again for the release

by ilaksh

1 subcomments

Could be amazing, but it's hard to judge if it will really work with say a 27 B model or larger. We can already get pretty good speed with a 2B model.

by robmccoll

1 subcomments

Making these claims on a 2B parameter model seems a bit like seeing linear scalability from 1 to 4 cores and then assuming 256 cores will give you a 256x speedup. Or demonstrating massive improvement on datasets that fit in cache and then assuming the same improvements will be present on problem sizes that span the memory of multiple machines. Something tells me that scaling to larger models will be more difficult than assumed.

by paul-rohan

0 subcomment

I had to test it myself to believe this unreal inference speed.
each time getting 3300+ tps.

by frankensteins

0 subcomment

I have a naive question here - first, the token speed is very impressive. but why this is the highlight? I would prefer the actual performance.

by irishcoffee

1 subcomments

NVIDIA H200 Is not a standard GPU. 8 of them in a box with a cpu and ram costs close to the same as a house.
I am 100% all about using local models instead of sending someone else all my data and paying for the privilege of doing so, this article is misleading.
I can get a 27b model to kick out 40 tok/s on 16 gb vram. This is the area ripe for development.
If you can’t connect a monitor, it isn’t a standard GPU, at least not in the way people have spoken about GPUs until a few years ago.

by bartkappenburg

0 subcomment

Is this the new gateway to a "Model On a Chip"? Is it possible to etch the weights on silicon and get a very efficient way to use a LLM?

by kirtivr

1 subcomments

I can think of real time video, shader generation, real time worldbuilding type problems could require such a high token throughput.
For instant code generatio, 400-500 tok/s should be sufficient, though most frontier models give us closer to 70 tok/s.

by ekianjo

0 subcomment

Title is pure bait. Where is Datacenter GPU gone?

by LoganDark

1 subcomments

I feel the comparison to Groq is unfair. They're running much larger models (orders of magnitude) and still reaching competitive speeds.

by mikdan

0 subcomment

[dead]

by nryoo

0 subcomment

[dead]

0 subcomment

by Jimmymenk2

0 subcomment

[flagged]

by Hfuffzehn

0 subcomment

That's really nice of them.
That means Jensen can add another 30 times faster when comparing Rubin to Blackwell without having to actually do anything.
Hopefully that means he won't have any problem to make another 150 billion in profit in the next year.
Sorry for the sarcasm. Looks like interesting work.