I'm running a 120B local LLM on 24GB of VRAM, and now it powers my smart home

local-llmmixture-of-expertsself-hostinghome-assistantgpuquantization

Abstraction: Running gpt-oss-120b locally on 24GB VRAM via MoE

Key points:

Runs gpt-oss-120b (OpenAI's largest open-weight model, 117B params / 5.1B active MoE) on an RTX 4090 (24GB VRAM) + 64GB system RAM, hitting ~20 tokens/sec — usable for a voice assistant.
Setup: Proxmox LXCs for llama.cpp and Whisper; Nvidia driver installed on host, LXC given hardware access via cgroup2/mount entries (no full GPU passthrough needed).
Uses Unsloth Q4_K_S GGUF (~60GB); expert (FFN) layers pinned to CPU/system RAM via -ot ".ffn_.*_exps.=CPU", GPU handles attention + dense layers + KV cache. Default context 131,072; can drop to 16K/32K with q4_0 KV cache quantization.
Wired into Home Assistant via HACS "Local LLMs" integration (OpenAI-compatible endpoint) + Whisper STT for a voice pipeline.
Speed trick: only the small activation tensor crosses PCIe per token (not full expert weight matrices); low active param count + quantized experts keep bandwidth bounded. Mixtral 8x7B and gpt-oss are standout MoE models for this CPU/GPU split.

Connections: Mixture Of Experts · GPT Oss · Llama Cpp · Home Assistant · Quantization