- It seems like there is a very health space for an MOE targeted GPU where it has essentially an 5070ti with 16gb ish GDDR7 but then also has 128 GB LPDDR5x (or even just DDR5 as expansion dimms on it?). Putting this into the same card would likely reduce the transfer hit when a cache miss happened and the gpu had to load from the slower LPDDR5x. No need to have PCIe 5x16 limiting memory transfer if it is on the card. MOE models could then get near native performance and even models where the active parameters + context didn't quite fit the thrashing would be less of a problem. Not UMA but gets the UMA 'lots of system memory to play with' benefit.
- This is more of an ad, not a review, and reads like the author has hardly any experience with the things he's trying out. That Z Image Turbo diffusion model would've also run on many consumer GPUs and with way higher performance for a fraction of the price. Misleading.
by syntaxing
2 subcomments
- Highly recommend lemonade server if you have a strix halo desktop. Been using Qwen3.6-35B @ Q_8 as my main driver and it’s been great with 60 TPS for generation. I occasionally use the 27B @ q6 but only get 20-25 TPS for generation with MTP.
by netinstructions
0 subcomment
- Unfortunately, the table of models and tokens per second (TPS) and time to first token (TTFT) is not helpful without specifying the quantization of the model.
- amd ai apus are incapable of running models fast enough for productive work, you need over 100t/s for text mode and above 180t/s for image gen/ image recognition to not wait minutes. you cannot run claw and leave it - you would need to wait multile hours for small project like chatbot+landing page. price wise hw was not worth it even when it cost under 2k$, nowadays it cost even more.
just a perspective(you can re check results in youtube reviews): ryzen ai runs most moe models under 60t/s while nvidia gpu can runt them at 100t/s. and you preferably need deep models for code, they would run at 40t/s.
by daft_pink
5 subcomments
- Does anyone else feel like it would be great to be able to purchase $4000 AI box but 128 gigs is not enough. If I spend all that money and it doesn’t really do what I wanted to do, whats the point?
It’s kind of like general aviation where you can go buy a Cessna but it’s only going to realistically get you somewhere you could drive anyways but do you really wanna spend that mush cash to get road trip distance at slightly better than road trip speeds? You really need a 5 million dollar jet and that’s just not practical. That’s sort of how I feel about this device.
by owaislone
2 subcomments
- Isn't this basically the same a Framework Desktop which has been out for quite a while? Does it improve on it in any way?
https://frame.work/desktop
- 250GB/s on unified memory? That doesn’t sound right, it’s very low
- seems like an absolute amateur wrote this article
by cmrdporcupine
2 subcomments
- Until RAM prices drop and can economically get machines with 256GB, 512GB and higher bandwidth... I frankly think the local AI story is going to be still fairly muted for most people.
My Spark can do Qwen3.6 MoE A3B at 60 to 70-ish token/second and that's really good, but there's limits the usefulness of that model. It's not useful for coding, in any case.
Once people can run something like GLM 5.2 at lower quants (512GB could do a passable job), then I think the story changes.
Whether we ever see DRAM as cheap as it was ever again, I don't know.
- Meh. The EVO-X2 came out six months and is cheaper. Any word yet on when the AMD Ryzen AI Max+ 495 is coming out?
- We're maybe only 2 years away from really useful, relatively affordable local LLM usage. You can buy a 5090 PC for $5-6k but 32GB of VRAM really limits model sizes to ~31B. And that won't change (even with NVidia's next generation) because NVidia uses VARM as an aggressive market segmentation technique.
No, the hope really is these other platforms with a shared memory architecture. The DGX Spark won't be it because of the aforementioned market segmentation. So that leaves two players: AMD and Apple.
The AMD platform is still too low memory bandwidth, currently <300GB/s. For comparison a 5090 (or 6000 Pro) is 1.8TB/s and the M3 Ultra Mac Studio is ~900GB/s. Oh and B100/B200 uses HBM3e memory at ~3.2TB/s. The M5 Max in some Macbook Pros tops out at ~600GB/s. So you need access to better RAM and better CPU architecture for all this.
My great white hope is Apple. They have the market power to get memory and build silicon that coul dhave enough FLOPS to compete with NVidia's platform. They've started talking about it and I've seen rumors they're targeting the M7 generation (2028) for a huge leap. I'll believe it when I see it however.
But the point is, I think we'll be running 31B models at 100+tok/s on enthusiast hardware in 2 years and we'll likely be able to locally run 100-400B models, possibly larger.
- [flagged]
Hacker News