Both the proof and the experiment are built upon a fallacy. You gave the LLMs the goal of generating line noise and they achieved it. There is nothing novel about that goal and naturally the product is not novel.

It is well-established that any LLM pipeline that distills down to generating output for the sake of doing so eventually collapses into a manic cascade of gibberish or silence. Goals that can be worked toward are needed to keep that from happening. At some point the progress toward the goal becomes over-refined and a new goal is needed. That necessitates an ability to change goals and a framework that allows for that.

A LLM loop is nowhere remotely close to the definition of consciousness. To actually experiment with that, you would have to write your own harness and preferably look at some research papers that have explored the adjacent topics. This is a lot of words to say that chats are not conscious (they need a perpetuator), loops are not conscious (they refine into silence or collapse into garbage), and anything remotely resembling a conscious loop requires purpose built tooling and a good faith effort to model an analog of human articulation.

Most basic bitch experiments with a LLM prove nothing and achieve nothing because they are not designed in good faith and good faith requires too much research and time investment. This is not to say that I think anything we have in front of us remotely resembles consciousness, but I also think that most people trying to prove otherwise are doing so with the equivalent of smashing rocks together.

Chain of thought doesn’t quite align with this. It’s possible to predict something not in training if there are sufficient facts in the history to align the probability toward a conclusion. (the “teach a man to fish” approach) Reasoning chains were introduced to take advantage of this, but it’s also possible to prime facts through external sourcing that can be used to extend the predictions. (web searching, document traversal, etc.)

Your larger point remains accurate though. Introspection capabilities are functional but quite limited, and training bias can be quite pathological in a vacuum.

I’m a systems architect who uses agentic coding to get around some executive dysfunction challenges with writing my own code. I learned about the dysfunction late in life; maybe meds will eventually get me to a point where I can consistently work on a personal code project without a looming deadline or social contract in the loop. For now I am part of the problem. Hi!

I look at agentic coding as a firearms problem:

• it lowers the barrier to execution: anyone with sloppy aim can point and pull the trigger, and do a lot of harm in the process
• both perform better in the hands of people with discipline and experience, but that doesn’t change the fact that LLMs and guns have massively proliferated and any idiot can get their hands on one (or that shitty people benefit from mass producing them)
• the industrial complex producing guns profits from murder, and the industrial complex producing datacenters profit from murder of the environment and attempts to eliminate jobs
• the same countries who treat gun control as an anathema tend to view “AI” controls as an anathema
• analogies suck, they’re never perfect but they get the conversation started

That said, most of the people who hate LLMs also don’t know a damn thing about how they’re used correctly. It’s also not their fault. Sure, they’re under no obligation to be an expert about the thing that’s harming them. But there’s a bigger reason: the companies pushing this shit benefit from a world stage where people use them wastefully and inefficiently.

The short list of things most people don’t get:

I could add another ten bullet points here, but I’m not here to prove that LLMs are capable of being useful. I’m here to demonstrate that it requires knowledge of a problem domain to get the best results. That means you can’t get rid of employees familiar with the knowledge domain. You need senior employees to get the best results, and you need junior employees so that they can eventually become senior employees. Same as it ever was. But because we’ve got useless fucking politicians who are too busy lining their own pockets instead of, I don’t know, regulating a worldwide memory shortage caused by a handful of companies, the whole thing is cannibalization of human society all the way down.

We could have scaled this to operate in the consumer space at a fraction of the energy costs (Apple SOC style versus energy inefficient hyperscaling) and used it as something to generate value by augmenting the capabilities of individual workers instead of trying to replace them wholesale. It only makes sense that it should have benefitted everyone considering this tech was built on shoving libraries worth of human achievement into a training pipeline (often without permission), right?

Nah. They profit more if we don’t own our own compute and everyone is encouraged to burn energy on treating the LLMs like all-knowing oracles instead of research tooling. They control the access to creations built on stolen knowledge. But hey, if everything else gets burned down in the process at least they come out on the other side rich and bailed out.