Hacker reverse engineers his Samsung TV and finds on-device AI listening for your dog and baby, watching what you watch, and borrowing your soundbar’s mic if the TV doesn’t have one. Then OpenAI banned him for reverse-engineering it.
Source:
Hacker reverse engineers his Samsung TV and finds on-device AI listening for your dog and baby, watching what you watch, and borrowing your soundbar’s mic if the TV doesn’t have one. Then OpenAI banned him for reverse-engineering it.
Source:
I can run Deepseek V4 (or V4.1 with some hacking/dev branches) on a single 3090 at 20 tokens/sec text gen. That’s straight-up better than OpenAI free tier, or the cheap Claude API, not to speak of bonuses from local running (like constrained sampling/grammar, custom sampling, context swapping, raw completion syntax, SGLang-style structured inference, task vectors, no censoring…)
I know 1x 3090 + RAM is kinda cost prohibitive now. But before the RAMpocalypse, that was not an unreasonable home server.
No, it’s not SOTA, but it’s good enough.
Now, the truly mad homelabbers?
They can run 1T models in a similar hybrid inference configuration, with an AMD EPYC server and some janky GPUs. MiMo Pro, GLM, those kind of models get you very close to Claude in most workloads. The gap is maybe months, not years.
And while Claude is good, honestly… OpenAI models are kinda shit. They are deep fried. I’ve never subscribed to them, and I never liked any of the bits I sampled since GPT 4, as they tend to be so sycophantic and slopped.
Are big servers cost effective? Hell no. It makes zero financial sense. There’s no reason to try and host that class of model when they are so dirt cheap over API.
Do I endorse mad homelabs anyway? Absolutely.
Deepseek on a single GPU? How? Q2 or something like colibri?
Exl3 quantization at 3.4bpw or so, KLD very close to the higher Q3/close to Q4 GGUFs.
TabbyAPI as a backend
Split each layer, experts are profiled and hot ones are run on GPU during hybrid inference.
I have a Ryzen 7800 with 128GB DDR5, which is how it fits. The “cold” MoE experts are run on the CPU, but this is very sparse, which is how it stays fast. Batch size of 8192 keeps prompt processing fast even though it is loading experts from RAM for that.
There’s also a lot of system optimization to get it there, like allocating large pages at boot, running the backend at a high priority, keeping background RAM/VRAM usage minimal so it will fit, sizing the quantization just right and such.
And this was not possible… a few weeks ago? DeepseekV4 just came out, and exllama got support for hybrid inference very recently. Before that, I could only get 8-12 toks/sec with a big MoE like this.
Thank you, this is a lot of new names for me and a great starting point for research. This field changes so fast
Exllama is old! Ironically. It was around during the Llama2 days, hence the name.
The paper behind its quantization method is pretty old too, and it had KV cache quantization with hadamard rotation like a year before Google’s TurboQuant paper made it popular. llama.cpp (the engine behind GGUFs) isn’t as cutting edge as people think, unfortunately.
I’m not disagreeing, I just find it amusing how “oldschool” some of this stuff is now.
deleted by creator