Efficiency is relative, if there is a solution that uses less resources then the other solution is more efficient, but if there is no other solution then the solution is the most efficient.
Is using fable 5 to do 2+2 efficient? No, because a calculator can do that with less resources
Is using fable 5 to rewrite a code base from zig to rust efficient? Maybe since the only other solution is a human it depends on how you compare human resources to compute resources. Time wise it’ll probably take the human longer since they require breaks.
Time wise it’ll probably take the human longer since they require breaks.
This is only true if your only metric for ‘success’ is lines of code written.
LLMs will output code much faster than a human can type, there’s no doubt about that. But when you consider how terrible even the frontier models still are at the architecture and maintenance side of the process, humans are still way, way more efficient.
There’s a lot of variation in quality with humans though.
I don’t doubt that there are some engineers better then the frontier models at coding considering architecture, maintenance performance etc. Those engineers tend to be more expensive though. I
don’t think an average engineer is better then the frontier models though, and say an entry level engineer fresh out of a boot camp would be significantly worse then even a tier 2 model.
Having worked closely with junior engineers and with high-end models, I strongly disagree. In almost all cases the output of the juniors is on par or better (albeit slower) than the LLMs ,and unlike an LLM, a junior actually learns and becomes much more consistent than the AI over time.
Are you talking about a one shot from the model or using a harness? I agree a junior dev can do better then a one shot, but with a proper harness with adversarial review cycles I don’t think a junior dev could
junior actually learns and becomes much more consistent than the AI over time.
A proper harness will have memory and will get more consistent the more you use it. You can “teach” it by adding skills and having it write it’s learnings either locally or to repo context files.
A proper harness will have memory and will get more consistent the more you use it.
Yeah, by creating a bunch of .md files of questionable quality that get fed into an already-limited context window, on top of a pile of all sorts of other context cruft from the harness, the model provider, and whatever else is propping up the system to give the illusion of intelligence…
I’ve experimented with harnesses quite a bit and I don’t understand the hype. The difference in quality has been middling in my experience, but the token burn has been significantly more — which makes sense when you understand that the more of the context window you use, the worse most models perform.
Efficiency is relative, if there is a solution that uses less resources then the other solution is more efficient, but if there is no other solution then the solution is the most efficient.
Is using fable 5 to do 2+2 efficient? No, because a calculator can do that with less resources
Is using fable 5 to rewrite a code base from zig to rust efficient? Maybe since the only other solution is a human it depends on how you compare human resources to compute resources. Time wise it’ll probably take the human longer since they require breaks.
Currently it is way more cost efficient to use a human
This is only true if your only metric for ‘success’ is lines of code written.
LLMs will output code much faster than a human can type, there’s no doubt about that. But when you consider how terrible even the frontier models still are at the architecture and maintenance side of the process, humans are still way, way more efficient.
There’s a lot of variation in quality with humans though.
I don’t doubt that there are some engineers better then the frontier models at coding considering architecture, maintenance performance etc. Those engineers tend to be more expensive though. I
don’t think an average engineer is better then the frontier models though, and say an entry level engineer fresh out of a boot camp would be significantly worse then even a tier 2 model.
Having worked closely with junior engineers and with high-end models, I strongly disagree. In almost all cases the output of the juniors is on par or better (albeit slower) than the LLMs ,and unlike an LLM, a junior actually learns and becomes much more consistent than the AI over time.
Are you talking about a one shot from the model or using a harness? I agree a junior dev can do better then a one shot, but with a proper harness with adversarial review cycles I don’t think a junior dev could
A proper harness will have memory and will get more consistent the more you use it. You can “teach” it by adding skills and having it write it’s learnings either locally or to repo context files.
Yeah, by creating a bunch of .md files of questionable quality that get fed into an already-limited context window, on top of a pile of all sorts of other context cruft from the harness, the model provider, and whatever else is propping up the system to give the illusion of intelligence…
I’ve experimented with harnesses quite a bit and I don’t understand the hype. The difference in quality has been middling in my experience, but the token burn has been significantly more — which makes sense when you understand that the more of the context window you use, the worse most models perform.