Hacker reverse engineers his Samsung TV and finds on-device AI listening for your dog and baby, watching what you watch, and borrowing your soundbar’s mic if the TV doesn’t have one. Then OpenAI banned him for reverse-engineering it.
Source:
Hacker reverse engineers his Samsung TV and finds on-device AI listening for your dog and baby, watching what you watch, and borrowing your soundbar’s mic if the TV doesn’t have one. Then OpenAI banned him for reverse-engineering it.
Source:
What is the honest capability delta between LLMs that can be run locally and the OpenAI model Log4J threw at the TV?
I have a Ultra 9, 5090, and 64GB RAM. I can comfortably run a 32B model which gives me the speed and performance of the best GPT/Claude model from about two years ago. It is functional? Yes. Does it do good work? Meh. Does it compare with the latest code models out now? Not even close.
I’d recommend to try Qwen Flash Next with such hardware. You should have just enough RAM. Even Q3 should be quite good.
I’ve tried it. I find the responses of the Qwen models to be far inferior to the Gemma models.
Interesting, I had the opposite experience. Especially for tool calls. But I heard it depends on the use case: Qwen better at coding and tool calls, Gemma is better for prose.
Like all things llm, the quantisation makes the difference. Which, of course you need the vram for to have good results. Flash next works very well if you can run the higher quants.
You try that Qwen3.8:27B ternary model yet? It’s supposed to be super fast:
https://huggingface.co/penkia/TernaryQuench-Qwen3.8-27B-GGUF
With all that VRAM you’ve got, it should be able to hold a ton of context too.
That fully depends on hardware. The best open weight model is pretty close but requires over a TB of ram to run.
I run a 27-billion parameter model at home. It can do useful work but it more commonly misunderstands code than the big cloud models, so you have to be more rigorous yourself in watching its reasoning and stopping it when it’s going astray. It’s usually possible then to nudge it back on to the right track. Otherwise it’s surprisingly good, enough that I can do a lot without turning to the cloud models. But it is significantly slower than them, sometimes absurdly slow on even very simple tasks, and it can reason round in circles and get nowhere. Still, it usually works well, and if I lost access to the cloud models I wouldn’t feel like I had lost too much.
Same, I run a variety of local models with LM Studio on my 5090. They are already remarkable good. Year or two and they will likely be on par with today’s leading cloud models.
The question is whether anyone will be able to afford the hardware to run them on.
Yes. Not sure if in a year or two but in a few years Chinese accelerators are gonna enter mass production and they’re probably arriving to our shores. At the very least, they’re gonna significantly dampen demand for AMD/NVIDIA’s chips so their prices are gonna fall.
Idk, I guess I was lucky when I built a new computer right before this all got completely crazy. My 5090 is already selling for over 3X what I paid for it. It’s all very sad.
As someone who didn’t catch on in time, grrrr. And harumph.
Dont feel bad, i felt kind of stupid until recently. This card is arguably one of the worst value propositions nvidia has ever produced, just didnt realize how much worse it would get and how quickly.
I’m always looking for better local LLM software. LM Studio looks interesting, your liking it?
LM Studio is good to start with, I do wish it was open source. Internally it just uses llama.cpp except with a GUI for configuration so it’s a bit easier to use. Unsloth Studio is similar to LM Studio except open source, but I haven’t tried it yet.
I actually have LM Studio on a Linux live boot USB along with Ling 3.0 Tiny, and it’s preconfigured to run with the Vulkan backend (no CUDA in a live boot, and might plug it into an Intel or AMD graphics). I haven’t needed to use it yet but I figure it could be nice in a pinch away from home.
It’s the only one I’ve used, but it does what I need and no complaints.
If you’re counting models that can be run locally but are too big to run on consumer hardware, then it’s pretty small. Like, they’re only lagging by a few months. But you you’d need like $40k of hardware to run them effectively.
If you mean just the models that you can run on consumer hardware, like, say qwen3.8-27b, then it’s more like 1-2 years behind. You can compensate for the lack of raw capability with a good harness, agentic loops, and human effort. As much as AI has a reputation for being low-effort, it actually requires a lot of skill to use very effectively.
If I’ve got 128GB of RAM but it’s really slow RAM (DDR3), can I run a good model really slow or will it just not work at all?
It’ll be slow but it’ll run.
Though even 128GB isn’t enough for the biggest models. 512 would open up more options and the very biggest need like 2-4TB, totally out of scope for personal use.
You could try a medium-big MoE model (Mixture of Experts) with less than 100b total parameters. Maybe start with something modest like Qwen3.6-35B-A3B at 8bit quants. That’ll leave you with plenty of memory left over for context size.
Hmm… I guess my board supports up to 256GB; would maxing out the RAM make a big difference? Also, I’ve got 2x16 core Opteron and an AMD Vega 56, if you have any advice related to those.
I don’t think you should go bigger because bigger = slower. It might already be too slow to use. Give a smaller model a try first and see how it runs.
try the Strata engine instead of llama.cpp
https://github.com/Niko1221/Strata/blob/main/docs/MODELS.md#pick-by-memory
You should be able to run Qwen 3.8 Flash Next decently
I guess the question is, what gets priority, RAM or GPU. I can get a dual 192GB 6000 nvidia with 256GB of RAM for $50k, but that still doesn’t get me the capability of top models. Or I could get a 1TB of RAM tower server with maybe one 96GB nvidia 6000 for about the same. Does the RAM capacity mean I can fit the larger model but it’s just going to be slow?
With Strata, things have gotten more flexible. Similar optimizations are coming to llama.cpp eventually.
https://github.com/Niko1221/Strata/blob/main/docs/MODELS.md#pick-by-memory
It says Qwen 3.8 Flash Next can run with just 8GB VRAM and 96GB RAM for a good quant of it. This is a really good model, and the new version 4 should be coming in the next month or so, probably the same size but smarter.
DeepSeek v4.1 Flash is probably even smarter, says it needs 12+GB VRAM and 64GB of RAM. For decent speed I think you’ll want at least 32GB VRAM and 256GB RAM for this one. 96GB VRAM should be very good.
You’re gonna have big diminishing returns chasing for even bigger models, partly because Strata doesn’t support anything bigger and llama.cpp is currently not nearly as optimized. There might be an inference engine I don’t know about for the biggest models.
That’s a good question and it’s hard to say for sure. There’ve been some advancements this year in offloading, using both VRAM and system RAM together. With MoE models and predictive loading you can get pretty good results that way…sometimes. I’ve never tried it myself, though I really should because my 24GB GPU is a serious bottleneck.
You could also think about getting a 512GB Mac Studio (coming soon). Not as fast as a GPU, but maybe faster than a GPU/DDR5 mix? And definitely cheaper per GB.
I realize now that the truly biggest models need way more than $40k of hardware. The biggest Qwen3.8 is 2.4 TRILLION parameters. Yikes.
I think these days we got local models at the level of early Opus 4.6 (early 2026) that can run on a couple of GPUs with 128GB system RAM. Am running the Q4 version (a bit dumber than the standard version) of Qwen 3.8 Flash Next on 2x R9700 with 150-200tps. More GPUs with more system RAM can run the lossless version which beats Opus 4.6 on a few benchmarks if not most.
I can run Deepseek V4 (or V4.1 with some hacking/dev branches) on a single 3090 at 20 tokens/sec text gen. That’s straight-up better than OpenAI free tier, or the cheap Claude API, not to speak of bonuses from local running (like constrained sampling/grammar, custom sampling, context swapping, raw completion syntax, SGLang-style structured inference, task vectors, no censoring…)
I know 1x 3090 + RAM is kinda cost prohibitive now. But before the RAMpocalypse, that was not an unreasonable home server.
No, it’s not SOTA, but it’s good enough.
Now, the truly mad homelabbers?
They can run 1T models in a similar hybrid inference configuration, with an AMD EPYC server and some janky GPUs. MiMo Pro, GLM, those kind of models get you very close to Claude in most workloads. The gap is maybe months, not years.
And while Claude is good, honestly… OpenAI models are kinda shit. They are deep fried. I’ve never subscribed to them, and I never liked any of the bits I sampled since GPT 4, as they tend to be so sycophantic and slopped.
Are big servers cost effective? Hell no. It makes zero financial sense. There’s no reason to try and host that class of model when they are so dirt cheap over API.
Do I endorse mad homelabs anyway? Absolutely.
Deepseek on a single GPU? How? Q2 or something like colibri?
Exl3 quantization at 3.4bpw or so, KLD very close to the higher Q3/close to Q4 GGUFs.
TabbyAPI as a backend
Split each layer, experts are profiled and hot ones are run on GPU during hybrid inference.
I have a Ryzen 7800 with 128GB DDR5, which is how it fits. The “cold” MoE experts are run on the CPU, but this is very sparse, which is how it stays fast. Batch size of 8192 keeps prompt processing fast even though it is loading experts from RAM for that.
There’s also a lot of system optimization to get it there, like allocating large pages at boot, running the backend at a high priority, keeping background RAM/VRAM usage minimal so it will fit, sizing the quantization just right and such.
And this was not possible… a few weeks ago? DeepseekV4 just came out, and exllama got support for hybrid inference very recently. Before that, I could only get 8-12 toks/sec with a big MoE like this.
Thank you, this is a lot of new names for me and a great starting point for research. This field changes so fast
Exllama is old! Ironically. It was around during the Llama2 days, hence the name.
The paper behind its quantization method is pretty old too, and it had KV cache quantization with hadamard rotation like a year before Google’s TurboQuant paper made it popular. llama.cpp (the engine behind GGUFs) isn’t as cutting edge as people think, unfortunately.
I’m not disagreeing, I just find it amusing how “oldschool” some of this stuff is now.
deleted by creator
i am not sure which model he used but the frontier models are better but local ones are catching up. especially since nvidia is pushing local. it comes down to your hardware though. be ause if you fins a higly capable local model, you need to run it. what kind if hardware do you have? do you have an rtx?
It’s for work. I can throw quite a bit of money at it, but only if there’s a point.
If there’s time and budget to play around with, perhaps get a “cheap” 2x R9700 + 128GB RAM system that can run Qwen 3.8 Flash Next and experiment. If it’s useful enough for your use cases, then get big enough hardware that can run this, which is better than QFN. It would also run the unquantized QFN and the next Qwen Flash version.
You could also try them via their APIs or on cloud hardware you rent.
My workplace, a well know NA corporation, has DeepSeek 4.1 Flash deployed on-prem and it’s in active use.
Qwen/Gemma 30B that you can run locally on a regular higher end pc can write some simple software making many mistakes and correcting itself all the time. It can intelligently search in code bases quite well.
GLM that you can self host if you sell your house and buy GPUs for that can one-shot most regular software tasks, like adding some functionality, and can handle larger tasks like “write me an app that looks like x and does y” while making mistakes and correcting itself which takes more time. It can do security and privacy reviews of code bases too.
Claude Opus can one-shot a mobile app on an uncommon platform and the code will usually be well designed and work straight away.
That’s my experience, anyway.
It can never compete with the amount of computer power they have behind.
I recomend anyone with a decent enough computer to try local llms, it will give you a perspective on just the amount of horse power they have to run their models.
I have a 32GB machine with 9070xt with 16GB, of that what matters most (or at all) is the vram, which they’re printing us out or out right denying selling to us. With that I can try running some more ambitious models of 30b parameters MoE.
The result in the best case scenario is a slow token output and not very credible responses.You also get frequently into reasoning loops, and tool using loops, and then run out of context.
I’m sure there’s someways to optimize it that someone more knowledgeable can do, but to me served as a reality check of the discrepancy.
btw, my go to check prompt for models is “how many stars are there in Mario 64”, and most models find it very hard to answer this, even when searching the web.
You should get very fast inference with Qwen 30B on that rig.
A huge problem in this space is that popular inference engines are… generally awful, tech bro slop, and the best ones are in the obscure DIY/hacking realm. The experience you’d get doing a default “Ollama run” vs a tuned rocm exllamav3/vllm setup or something is like night-and day; think 4 tokens/sec of jibberish, vs 40 tokens/s that performs close to unquantized.
Summary: not even close.