Does anyone know of a good tool for "load balancing" usage across local GPUs?

Why: I have two RTX3090s (24GB). I've been using nvidia-smi to check usage of my RTX3090. Mostly I'm running llama.cpp with unsloth/Qwen3.6-27B-GGUF:Q4_K_M and getting some pretty decent results for a self hosted LLMs (orchestrated via opencode). I'm surprised at how well it is working for a local model. nvidia-smi is great for determining total VRAM usage and nvtop gives a little more insight.

But, I also am doing some experiments with some other non-LLM models (video generation, etc), and want to find a way to timeslice across these GPUs, for example, when my coding is paused.

This "Utilyze" tool appears it would get me better insight into usage of one. Can it be scripted to better utilize my GPUs across a diverse load?

Any suggestions on whether there are existing projects out there? I thought about vibe coding, but wonder if there is existing art.

But if you really care about this, you should actually profile your application. nsight systems makes this pretty simple to do. Dunno how many actually care about having a TUI.

Nvidia’s toolsets and APIs are under-documented, and the commercial-grade hardware itself is super unreliable.

Developers and operators just bear with the whole situation because there is no alternative. To the point that they are ready to jump to things like TPUs or other custom silicon.

Say what you will about Intel, but their documentation and the commercial-grade hardware were top-notch. I wish they find their footing and this time stay humble.

One thing I'd love to see in tools like this is a "memory pressure"view that shows not just current VRAM usage but how close you are to the OOM cliff for the workload you're running. Running quantized LLMs on consumer GPUs (e.g. Q4_K_M Gemma 4 E4B on an 8GB card), you can be at 95% utilization and totally fine, or at 80% and one context spike away from a crash. nvtop and nvidia-smi give you the number but not the headroom.

Whether that's feasible without instrumenting the workload specifically is another question. But it's the metric I actually care about when I'm picking quantization levels.

I don't fully get the 100% utilisation vs. 1-10% real compute. Given you rely on telemetry from users to add new models, are you trying to predict how fast a model should be on vLLM, compared to how it runs in practice? What if users tweak some hyperparameters?

Just testing for now.

Any removal instructions or function for utilyze beyond the manual removal of utilyze & utlz binaries from ~/.local/bin & /usr/local/bin & PATH cleanup for ~/.profile, in particular CAP_SYS_ADMIN capability and reversal for any other changes made?

At the moment (v0.1.3) it is more helpful for compute visualization but keeping track of memory usage/processes/temperature/fan speed/etc. prevent this from becoming a full-on drop-in replacement for `nvidia-smi` for me.

Will further test it.