Local models are meaningfully useful* for agentic work now. This is a realization I came to over the last day or so.
I wrote a lot of custom mods for a game that recently switched engines. The migration is going rather smooth for most people because of wise architectural choices from the dev early on, and because they prioritized backwards compatibility in the update. But there are still some rough edges, especially if you had your hooks deep into the engine.
I recently vibeforked Unbiased Bob’s fork of Strata to make it compatible, easy to use and optimized for my OS and hardware (NixOS 5800x3d 64gb, 7900xtx 24gb). Sol 6.1 got it working with Qwen 3.8 Flash Next IQ3 XSS (~75GB) at roughly 46tps 164k ctx with enough headroom to launch a heavy browser along with a triple-A game set to low in 720p for testing when I need to (plus space for zram swap)
I set it up in deepseek harness, gave it a 14MB error log in the game’s working directory, then went to sleep. I woke up and it had run for 3 hours. The game launched with only two errors, which it quickly fixed from the new, vastly smaller logfile.
It hammered down every issue, making changes across 30 separate mod repos. Practically a one-shot, and all the features worked as originally intended.
I left a local coding agent running completely on its own for three hours across multiple repos across a dozen auto-compactions with a log file and no ability for it to verify its work besides a compiler. And it produced meaningful results. I can’t speak to the code quality, but it worked. This was completely unthinkable even just two years ago. Even one year ago, the cloud frontier wouldn’t be able to do this on its own, and now a local model can, running on my consumer gaming hardware from one or two gens back.
“Huh… So… We’re here now…" I said to myself, with a bit of stunned disbelief, wondering what the future one or two years from now might hold.
You must log in or # to comment.
I had a similar experience this week. I tried ollama like last year and thought it was kinda cool but ultimately not that helpful.
Over the weekend I got Qwen3.6 35b running Hermes on my gaming PC and it made me a whole back testing application for trading. Everything from planning out the work, breaking it into tasks, spinning up agents for blocks of work, writing tests, the whole shebang.
It took like two work days of me prodding it once in awhile but it’s still impressive
Yeah, we’re here. I’m a professional programmer but I haven’t written an actual line of code in quite a while now, and I’ve been having so much fun with my local model (Qwen3.8-27B Q4) doing all sorts of cleanup tasks on my lifetime’s giant pile of accumulated files. For example, all my generated music - I ended up having Qwen build a GUI music manager and whipped up a website generator for my favourites so I can more easily share them through free hosts without having to hand them over to spotify or whatever.
I’m curious, is the game you mentioned modding Rimworld? I’ve been pondering modding it for some time now, it’s one of my favourite games, if you’ve got experience using AI for that I wouldn’t mind some pointers. It was always a “I know how to do this stuff but it takes way more time and attention than I’ve got available right now” sort of task but AI can help so much there. :) Your description of the “engine changed but compatibility is remarkably good” matches what I’ve seen of Rimworld’s recent 1.7 update.
Yes, it’s Rimworld, you’re right! But I don’t have many pointers for modding with AI because I’m a rimworld modder and have been since alpha. This was just a huge collection of random forks and one-off personal mods that I wanted to update for 1.7 for a personal run.
I have also been using frontier models to write game cheats for various games, replacing my miles of handwritten Lua code in cheat engine tables with native injection mods (easiest in unity and ue4, but I also modded an old dosbox game for a friend, and got a bunch of features for assassins creed black flag resynced on the anvil engine). I haven’t tried doing that with local models yet, but after seeing these results, I might. I suspect that’s code will be substantially worse, though…
As far as modding Rimworld, because most of its JIT C#/mono, and the game files have a lot of data in them already; it seems like models don’t need a lot of hand-holding. I bet downloading the wiki for some additional context would probably be wise, but as far as implementing simple changes, it seems to be perfectly capable with little to no prep. Also I’m using nix so it can get any tooling it needs and set up a dev environment on its own.
Okay, thanks. I was hoping it’d be relatively straightforward to get AI up to speed on it, I just wasn’t sure where to start or whether it’d be a waste of time if I tried. I just want to give my pawns some interesting diseases and it seemed like a daunting bit of setup to get to that point. I’ll take another run at it soon. :)
Rimworld’s an interesting battleground right now in terms of AI, the various fan forums came out strongly against AI and then the new BRUTAL engine came along that incorporates AI-generated code into the base game. So I didn’t think I’d get much help from the forums I’m aware of.
Most agents can read websites, you may not need to download it. Just tell it that that wiki has useful information. Maybe include it in the AGENTS.md file so it remembers.
I imagine my first task for the agent will be to tell it to compile a skill file from all the documentation I’m able to direct it to. I was just hoping that maybe someone had already done that to give it a leg up.
AI creates sounds, not music.
I made a DOS port of Tetris Attack for the first IBM PC. The audio hardware is a single buzzer. I created the engine for music and sound effects, by hand, from scratch… at 4.77 MHz. I had no idea what I was doing. I still don’t. The parts which aren’t that-sounds-right repetition are just turning column heights into notes.
Is that music? That slapdash bespoke effort, in the dire absence of any formal training, on toy hardware? Because if my bullshit counts then I’m not clear how any danceable formulaic realization of proper western music theory would not count… and if my bullshit doesn’t count then I’m not sure what the point is in puffing up human effort over perfectly fine artificial results.
No. lol! It’s not music. It’s computerized sounds. Did you write the music?? Score the different instruments? Wat it your original composition?
Creating music is a SKILL. A talent that someone sharpened over years of repetition, learning, practice, and love. So, no…. You didn’t make music. You programmed a computer to make sounds that OTHER people created.
I can pick up a synthesizer which does nothing but produce prerecorded sounds and hit random things till it sounds good. If someone enjoys it they would call it music.
Slop generated music ain’t any different. Music and art as a whole is in the eye of the behold. Only a raging asshole tries to tell people what is or isn’t art to them.
It doesn’t have to be art to you. But no one else gives a fuck what you or I or anyone else thinks. Because only their own opinion matters when it comes to what they believe is art.
LMAO… the keyboard/synthesiser bullshit? Really?
No dude. Richard Wright of Pink Floyd was a classically trained pianist. He did not “hit random things till it sounded good”.
Ffs! What a fucking parhrtic argument!
No one hits random shit until it sounds good. That’s called noise. You clearly know absolutely nothing about music. Seriously…. Just go sit down.
Alright, I used it to organize my AI-generated sounds.
Feel better? Swapping those words had exactly zero impact on what was actually done. The labels you use don’t make any practical observable difference.
It make the world of difference to people that actually create music. Imagine you spent your lifetime learning and perfecting a craft, then along comes a kid with a 3d printer and calls himself a craftsman.
But that’s not what he said. He didn’t call himself a musician. He said he created music. He did. It’s undeniably music.
A person 3d printing does not call themselves a craftsman. They say they created objects. And they undeniably create objects.
I’m sorry you’re struggling with the AI stuff. As a developer, I’ve gone through stages of grief too. I can’t deny reality tho, this is my livelihood. It’s a tool. I use it.
It’s not music. And it’s not stages of grief. You like it because it does your job for you. I despise it because it allows talentless hacks to pretend they are my equal when I spent my life on this craft. They’re ruining culture. It’s slop.
End of story.
I despise it because it allows talentless hacks to pretend they are my equal when I spent my life on this craft
It really has been all downhill since the year 2000 when Hampton the Hampster’s “The HampsterDance Song” was released, this was PEAK music
Now just about anyone can release some garbage and it’ll go viral eh?
Imagining. In such a situation I’d resist the urge to join a 3d printing forum and interject myself into their conversations with nothing worthwhile to add aside from my “um, actually” semantic quibbling. Because I’m not an ass and if the kid’s having a good time more power to him, it’s no skin off my back.
But you do you I guess.
Yeah, however, imagine that those kids are ruining your industry. But let me guess. You’d be totally fine with that, right?
You see, It’s hacks like… it’s hacks- that pretend to know what they’re doing by punching adjectives into a prompt and producing artificial music, or artificial art that are taking jobs away from talented people that actually work hard to produce their product.
And for the record…. I’m not here to get internet points kid, as a musician of stage and studio for over 30 years….
I’m here to say what needs to be said.
Yeah, however, imagine that those kids are ruining your industry. But let me guess. You’d be totally fine with that, right?
In fact, I am.
I’m a professional programmer. 20 years experience, it’s where I earn my money, got a university degree and everything. And we’re here in a thread about how not only are AI models good at programming, even small local models are good at programming. I use them myself, I give advice on how to use them, I ask advice on how to use them. I think they’re really neat and look forward to using them more, and encourage everyone else to also use them.
Sure, some people have no idea what they’re doing and they end up writing not very good software. My reaction to that is to try to explain to them what they did wrong and how they can get better. Or, if their not-very-good software is still perfectly good for whatever they’re using it for, I give them a little thumbs up and say “good on you, just make sure you know what you’ve got there.”
I’m here to say what needs to be said.
No, you’re here to interject yourself into a conversation that was not at all about AI music to push your personal dislike of AI music and to insult people who don’t share your view of AI music.
It doesn’t need to be said. Certainly not here and now.
Okaybuddy. Not reading all that. AI is for hacks. End of discussion.
Does Qwen still do all the lengthy reasoning steps? Last time I tried to vibe-code some moderately easy webapp, it went through something like 1 - 1.5 million tokens. Which should be about 9h with your 46t/s?
No, I didn’t have that experience. I know that Qwen is known for long reasoning traces, and it certainly does think and even investigate a little bit more than it needs to, but it wasn’t anything crazy (like I’ve seen with the 27b)
Off of memory, the first run was about (not counting cache aggregates) 1.3M in and about 250K out. I’ve since done a bunch of enhancements and changes using that session, so now it’s up to 17 turns, 1544 steps: 901 minutes of LLM time, 130 minutes of tool time. 7.8M in and 1.2M out (115M aggregate if you count cached io). Dropped to about 41tps. But it got work done.
This is absolutely orders of magnitude more than something like sol 6.1 would use, but it isn’t completely outlandish and it can actually perform work; which is what was surprising to me. It still hallucinates a ton too, but it seems to be able to correct itself and ground itself in new information from toolcalls etc much better than models I’ve tried previously, which I suspect is what unlocked this “long running” capability. It also seemed to respond well to steering (most models I’ve tried previously just shit the bed)
Sounds acceptable. I mean as long as you don’t have to intervene all that much. And you said electricity is affordable where you live… So all good.
Yeah, I think hallucinating isn’t all that bad in reasoning steps or in-between. As long as it manages to then home in on the correct thing.
Bonus post: a subagent went off the rails in a subsequent (much harder: reverse engineering) task.
(First and last time I’ve seen that behavior, but I’ve only been running it for a few days)
Hmm.
Hmm.
Ornith is a nice fine-tune of Qwen that works reasonably well on reasonable hardware, Bonsai also seems to be often recommended but I haven’t tried it personally. That said, from everything I’ve seen local agentic code is pretty computationally wasteful in many cases, it iterates heavily and confuses itself a lot and self-corrects often repeatedly. It hurts my soul to imagine the energy waste and environmentally damage if large numbers of people started using this routinely and carelessly.
It is meaningfully useful, I’ll grant you that, and when paired with highly specific and well-bounded tasks it performs quite well. I hate the idea of relying on it too heavily though. If you just throw it at things blind and without you also providing assistance to the model, it will get the job done, eventually, just in a really painful and stupid way. It’s not Claude, but for a locally-running substitute it’s remarkably good, it handles coding assistance very effectively, but it’s a very poor coder when operating completely independently. I find the best and most efficient use is to ask it for suggestions, keeping the pieces as small as reasonably possible, and then implement them myself based on what it suggests (either adjusting the model’s edits or copy-pasting the model’s suggestions). It’s very good at locating the relevant code and it usually suggests a reasonable design, which I typically tweak based on my own experience. Maybe that makes me old-school, maybe I’m a luddite, but that’s what my experience tells me to do. If you don’t have any experience, I guess you’re on your own, good luck with the models. They’re decent, but they’re not brilliant programmers (granted neither am I most days). That’s my opinion and I’m sticking with it until I see evidence to suggest otherwise.
It doesn’t run my GPU any harder than watching anime with my realtime upscalers and my CPU not much more playing lategame modded Minecraft. And I live in a region with very cheap energy with lots of renewables and nuclear. So if it can do work, I would otherwise have to do, for fractions of pennies while I sleep: it’s totally worth it.
It takes electricity. I take moments of a finite life. This is the same reason I didn’t review the code; for this case, I just wanted the result, I just wanted it to work, and I was curious how far a local AI would get. I didn’t actually expect it to finish. I expected to come back to it looping or a complete mess and revert all my repos and then tell Sol to do it in 30 minutes.
They are far less token efficient, and dumber, and far slower than Frontier models running on dedicated hardware. It hallucinates and get things wrong all the time, but it seems to be able to correct itself and get back on track from new information much better than earlier generations of model. That really seems to be the key to its long-running success from what I’ve seen. When I tried this about half a year ago, once it started hallucinating or its context got over ~50k, it was basically game over.
My background is in low-level resource-constrained systems programming, mostly C and modern C++. Yeah, I can code laps around this thing even in languages I don’t know well (not true for frontier) and I’d never trust it to write code that I would own and put my name to… But it can get results. It could do something low-stakes for me, while I slept, that was literally impossible a year ago. And it does it locally. That was mindblowing. I thought this approach (LLMs) would plateau a lot sooner than this. The fact that it can run outside of a datacenter is just a massive cherry on top.
Given that (personally affordable, ingnoring the bubble premium) hardware progress is saturating, how much headspace for improvement do we still have do you think? Would you currently put it at linear-ish, saturating, or too early to tell/not enough data?
the bubble premium
You know, I can absolutely see a future where Nvidia and the other hardware makers turn their eyes back towards consumers as they see the build-out slowing and the upgrade budgets shrinking.
What if in one year, Nvidia puts out a GPU or all in one PC AI server thing that will let you run free models that are as good or better than current frontier ones, at a reasonable speed, and they charge like $3000 or $5000 for the thing?
Well, I guess if they want to be proper capitalists/enshittifiers, they would first sell them for 10x as much to corporate IT departments who can replace their ridiculous tech giant subscriptions with local compute augmentation. On-prem is back, baby!
Then they can rip off the gamers and the home lab enthusiasts for a while.
And finally, at long last, with the economy raked over the coals and a cool trillion in the bank, Jensen will briefly feel whole inside. Surely he will.
It’s really hard to tell because I didn’t even see it getting to this point. I’m running a 75 gigabyte model on a 24 gigabyte GPU and my friend with a 16 gigabyte GPU is actually getting better performance than me. Mostly just because it’s a mixture of experts and Strata’s storage tier caching works well.
I don’t think it’s saturated because we would have to, not quite saturate, but definitely wring out a lot of the gains from kernel and model architecture first before raw compute becomes a limitation.
I expected this approach to AI to plateau before now. There has also been evidence that scale is not the sole key, and that quality and approach and harness all play significant roles. I thought it would have died by now, but since it hasn’t, I suspect it still has a long way to go. Given that I couldn’t have predicted this a year ago, I don’t think I can predict a year from now.
Thanks, food for thought.
The gap between big-boy models and local fare has been closing. That could mean scale is not necessary for further improvement, or it could mean we’re asymptotically approaching some limit to this approach, but either way - existing consumer hardware is sufficient for some witchcraft.
Waiting for the bubble to pop to join the fray. Adding some surplus PV capacity meanwhile.
How is this set up to run? What model? IDE? etc
Details all in the post. Modified Strata backend, qwen 3.8 flash next iq3, deepseek harness. But this isn’t how-to guide
How is DeepSeek vs open code I always see people say DeepSeek is either the best or worst harness by a mile.
They’re also oretty great for stuff that’s simple to write like self-hosting infrastructure/config code like Ansible, SaltStack. I wouldn’t use what comes out without understanding it though.
I use NixOS and my agents seem to love it as much as I do. The ability to read the derivations, evaluate the configs, navigate the nix store, and use nixpkgs from the cmdl is quite awesome for host & environment tinkering/debugging.
Models seem to prefer nix and arch a lot. Don’t really know why. Arch I get but nix always confuses me.
