FRESH

Hacker News

Home

Claude Code: connect to a local model when your quota runs out

91 points by fugu2

by paxys

1 subcomments

> Reduce your expectations about speed and performance!
Wildly understating this part.
Even the best local models (ones you run on beefy 128GB+ RAM machines) get nowhere close to the sheer intelligence of Claude/Gemini/Codex. At worst these local models will move you backwards and just increase the amount of work Claude has to do when your limits reset.

by alexhans

2 subcomments

Useful tip.
From a strategic standpoint of privacy, cost and control, I immediately went for local models, because that allowed to baseline tradeoffs and it also made it easier to understand where vendor lock-in could happen, or not get too narrow in perspective (e.g. llama.cpp/open router depending on local/cloud [1] ).
With the explosion of popularity of CLI tools (claude/continue/codex/kiro/etc) it still makes sense to be able to do the same, even if you can use several strategies to subsidize your cloud costs (being aware of the lack of privacy tradeoffs).
I would absolutely pitch that and evals as one small practice that will have compounding value for any "automation" you want to design in the future, because at some point you'll care about cost, risks, accuracy and regressions.
[1] - https://alexhans.github.io/posts/aider-with-open-router.html
[2] - https://www.reddit.com/r/LocalLLaMA

by wkirby

1 subcomments

My experience thus far is that the local models are a) pretty slow and b) prone to making broken tool calls. Because of (a) the iteration loop slows down enough to where I wander off to do other tasks, meaning that (b) is way more problematic because I don't see it for who knows how long.
This is, however, a major improvement from ~6 months ago when even a single token `hi` from an agentic CLI could take >3 minutes to generate a response. I suspect the parallel processing of LMStudio 0.4.x and some better tuning of the initial context payload is responsible.
6 months from now, who knows?

by d4rkp4ttern

0 subcomment

Since Llama.cpp/llama-server recently added support for the Anthropic messages API, running Claude Code with several recent open-weight local models is now very easy. The messy part is what llama-server flags to use, including chat template etc. I've collected all of that setup info in my claude-code-tools [1] repo, for Qwen3-Coder-next, Qwen3-30B-A3B, Nemotron-3-Nano, GLM-4.7-Flash etc.
Among these, I had lots of trouble getting GLM-4.7-Flash to work (failed tool calls etc), and even when it works, it's at very low tok/s. On the other hand Qwen3 variants perform very well, speed wise. For local sensitive document work, these are excellent; for serious coding not so much.
One caviat missed in most instructions is that you have to set CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC = 1 in your ~/.claude/settings.json, otherwise CC's telemetry pings cause total network failure because local ports are exhausted.
[1] claude-code-tools local LLM setup: https://github.com/pchalasani/claude-code-tools/blob/main/do...

by hkpatel3

0 subcomment

Openrouter can also be used with claude code. https://openrouter.ai/docs/guides/claude-code-integration

by TaupeRanger

0 subcomment

God no. "Connect to a 2nd grader when your college intern is too sick to work."

by baalimago

0 subcomment

Or better yet: Connect to some trendy AI (or web3) company's chatbot. It almost always outputs good coding tips

by btbuildem

0 subcomment

I'm confused, wasn't this already available via env vars? ANTHROPIC_BASE_URL and so on, and yes you may have to write a thin proxy to wrap the calls to fit whatever backend you're using.
I've been running CC with Qwen3-Coder-30B (FP8) and I find it just as fast, but not nearly as clever.

by israrkhan

0 subcomment

Using claude code with custom models
Will it work? Yes. Will it produce same quality as Sonnet or Opus? No.

by Animats

0 subcomment

When your AI is overworked, it gets dumber. It's backwards compatible with humans.

by zingar

0 subcomment

I guess I should be able to use this config to point Claude at the GitHub copilot licensed models (including anthropic models). That’s pretty great. About 2/3 of the way through every day I’m forced to switch from Claude (pro license) to amp free and the different ergonomics are quite jarring. Open source folks get copilot tokens for free so that’s another pro license I don’t have to worry about.

by eek2121

0 subcomment

I gotta say, the local models are catching up quick. Claude is definitely still ahead, but things are moving right along.

by mcbuilder

0 subcomment

Opencode has been a thing for a while now

by raw_anon_1111

0 subcomment

Or just don’t use Claude Code and use Codex CLI. I have yet to hit a quota with Codex working all day. I hit the Claude limits within an hour or less.
This is with my regular $20/month ChatGpT subscription and my $200 a year (company reimbursed) Claude subscription.

by esafak

1 subcomments

Or they could just let people use their own harnesses again...

by swyx

0 subcomment

i mean the other obvious answer is to plug in to the other claude code proxies that other model companies have made for you:
https://docs.z.ai/devpack/tool/claude
https://www.cerebras.ai/blog/introducing-cerebras-code
or i guess one of the hosted gpu providers
if you're basically a homelabber and wanted an excuse to run quantized models on your own device go for it but dont lie and mutter under your own tin foil hat that its a realistic replacement