I can run models on my 3yo midrange smartphone. Gemma-4-E2B totally runs on there. Bonsai-8B too. But neither is really good for most tasks.
Heck, “a basic model” even runs locally on the 8MB RAM of an ESP32-S3, but it’s utterly worthless at anything.
If you get a bit more into self-hosting AI it quickly becomes obvious that for any actually useful real world tasks you need at least 24GB you can dedicate to the LLM alone, and if this is fully GPU vRAM, the performance is way, way higher than on CPU, even with an NPU.
any actually useful real world tasks you need at least 24GB you can dedicate to the LLM alone, and if this is fully GPU vRAM
Agreed. For anyone who doesn’t have a 24GB card, the AMD 7900 XTX is basically the only sanely priced 24GB GPU left in 2026 and can be had for under a grand new, whereas even a used 3090 is $1200 and $2000 new.
Qwen 3.8 27B runs nicely on my XTX. I have to use Claude at work, and Qwen 3.8 in llama.cpp with the OpenCode desktop app leaves nothing to be desired in comparison. 3.8
27B is about on par with Sonnet 5 according to benchmarks, though I even prefer it over Opus because Qwen writes in clear, understandable language. We are at the point now where anyone with a 24GB GPU has little reason to use data center LLMs.
I can run models on my 3yo midrange smartphone. Gemma-4-E2B totally runs on there. Bonsai-8B too. But neither is really good for most tasks.
Heck, “a basic model” even runs locally on the 8MB RAM of an ESP32-S3, but it’s utterly worthless at anything.
If you get a bit more into self-hosting AI it quickly becomes obvious that for any actually useful real world tasks you need at least 24GB you can dedicate to the LLM alone, and if this is fully GPU vRAM, the performance is way, way higher than on CPU, even with an NPU.
Agreed. For anyone who doesn’t have a 24GB card, the AMD 7900 XTX is basically the only sanely priced 24GB GPU left in 2026 and can be had for under a grand new, whereas even a used 3090 is $1200 and $2000 new.
Qwen 3.8 27B runs nicely on my XTX. I have to use Claude at work, and Qwen 3.8 in llama.cpp with the OpenCode desktop app leaves nothing to be desired in comparison. 3.8 27B is about on par with Sonnet 5 according to benchmarks, though I even prefer it over Opus because Qwen writes in clear, understandable language. We are at the point now where anyone with a 24GB GPU has little reason to use data center LLMs.