by d4rkp4ttern
2 subcomments
- You can use llama.cpp server directly to serve local LLMs and use them in Claude Code or other CLI agents. I’ve collected full setup instructions for Gemma4 and other recent open-weight LLMs here, tested on my M1 Max 64 GB MacBook:
https://pchalasani.github.io/claude-code-tools/integrations/...
The 26BA4B is the most interesting to run on such hardware, and I get nearly double the token-gen speed (40 tok/s) compared to Qwen3.5 35BA3B. However the tau2 bench results[1] for this Gemma4 variant lag far behind the Qwen variant (68% vs 81%), so I don’t expect the former to do well on heavy agentic tool-heavy tasks:
[1] https://news.ycombinator.com/item?id=47616761
by seifbenayed1992
1 subcomments
- Local models are finally starting to feel pleasant instead of just "possible." The headless LM Studio flow is especially nice because it makes local inference usable from real tools instead of as a demo.
Related note from someone building in this space: I've been working on cloclo (https://www.npmjs.com/package/cloclo), an open-source coding agent CLI, and this is exactly the direction I'm excited about. It natively supports LM Studio, Ollama, vLLM, Jan, and llama.cpp as providers alongside cloud models, so you can swap between local and hosted backends without changing how you work.
Feels like we're getting closer to a good default setup where local models are private/cheap enough to use daily, and cloud models are still there when you need the extra capability.
ollama launch claude --model gemma4:26b
by martinald
3 subcomments
- Just FYI, MoE doesn't really save (V)RAM. You still need all weights loaded in memory, it just means you consult less per forward pass. So it improves tok/s but not vram usage.
by vbtechguy
1 subcomments
- Here is how I set up Gemma 4 26B for local inference on macOS that can be used with Claude Code.
by pseudosavant
1 subcomments
- I want local models to succeed, but today the gap vs cloud models still seems continually too large. Even with a $2k GPU or a $4k MBP, the quality and speed tradeoff usually isn’t sensible.
Credit to Google for releasing Gemma 4, though. I’d love to see local models reach the point where a 32 GB machine can handle high quality agentic coding at a practical speed.
by jonplackett
1 subcomments
- So wait what is the interaction between Gemma and Claude?
- Seems like this might be a great way to do web software testing. We’ve had Selenium and Puppeteer for a long time but they are a bit brittle with respect to the web design. Change something about the design and there’s a high likelihood that a test will break. Seems like this might be able to be smarter about adapting to changes. That’s also a great use for a smaller model like this.
by asymmetric
1 subcomments
- Is a framework desktop with >48GB of RAM a good machine to try this out?
by Someone1234
6 subcomments
- Using Claude Code seems like a popular frontend currently, I wonder how long until Anthropic releases an update to make it a little to a lot less turn-key? They've been very clear that they aren't exactly champions of this stuff being used outside of very specific ways.
- I could see a future in which the major AI labs run a local LLM to offload much of the computational effort currently undertaken in the cloud, leaving the heavy lifting to cloud-hosted models and the easier stuff for local inference.
- Qwen3-coder has been better for coding in my experience and has similar sizes. Either way, after a bunch of frustration with the quality and price of CC lately I’m happy there are local options.
- omlx gives better performance than ollama on apple silicon
- How well do the Gemma 4 models perform on agentic coding? What are your impressions?
by aetherspawn
1 subcomments
- Can you use the smaller Gemma 4B model as speculative decoding for the larger 31B model?
Why/why not?
by alfiedotwtf
0 subcomment
- PSA: For those getting stuck in a repetitive loop or just stopping without completing a task, try the interactive template. I just tried it now and it's blowing my already impressive results out of the water (llama.cpp):
--jinja --chat-template-file models/templates/google-gemma-4-31B-it-interleaved.jinja
- Running Gemma 4 with llama.cpp and Swival:
$ llama-server
--reasoning auto
--fit on
-hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL
--temp 1.0 --top-p 0.95 --top-k 64
$ uvx swival --provider llamacpp
Done.
- I hate that my M5 with 24 gb has so much trouble with these models. Not getting any good speeds, even with simple models.
by hackerman70000
3 subcomments
- [flagged]
- [flagged]
- [flagged]
by techpulselab
0 subcomment
- [dead]
- [flagged]
by aplomb1026
0 subcomment
- [dead]
- [dead]
- awesome, the lighter the hardware running big softwares the more novelty.
- Did you try the MLX model instead? In general MLX tends provide much better performance than GGUF/Llama.cpp on macOS.
by NamlchakKhandro
3 subcomments
- I don't know why people bother with Claude code.
It's so jank, there are far superior cli coding harness out there
Hacker News