On several occasions, we’ve been exposed to folks that have been sort of lukewarm on our main offerings, but they really, really wanted to use AI to perform a natural language query on their data. And we thought “Okay, if you really want to see it, maybe we can caveat this appropriately and show you what it might look like.”
This was a terrible mistake. It backfired in the most predictable way imaginable – every lukewarm client that saw the chatbot in action, even with us telling them that it was not going to accomplish what they wanted, wanted to buy it immediately. Every other consideration, including millions of dollars that we could plausibly help them achieve by non-AI means, was swept aside. It was like a dark and terrible force seized control of their limbs, plunged their hands into their own chests, and presented their still-beating credit cards to us in grim supplication.
The writing here (and everywhere) is phenomenal, and this section echoes something I’ve seen Cal Newport say before, something along the lines that somebody with no technical knowledge sees one neat trick and suddenly assumes AI will turn into a superintelligence deity.
Warnings about it being “early days” and “hallucinations” both anthropomorphise the bot and provide ready-made excuses for when one of its constant failures are noticed.
Three years later and the “early days” routine is still going strong. Who would have thought?
What AI “decision making” does in our processes is: write the reports exactly like the procedure says they should be written, and fills them with whatever you want to hear - looking up convenient references to back up your opinions. You probably should double check the references yourself to be sure they actually exist (though I think AI is getting better at checking its own work for things like this lately) - but… yeah. Ask it once and it will give you “an opinion” that doesn’t take into account all of the factors that matter to you. So, as you explain all the factors, it also becomes obvious what you want to hear, so after several rounds the obliging AI eventually not only “sees things your way” but can pile on the BS as deep as you ask it to to back up those opinions. What would have taken days to research and write can be done in an hour or so.
It is the ultimate corporate yes-man that they could never quite get in a human. They have built it for C-suites who have control of the money but not their own faculties.
The writing here (and everywhere) is phenomenal, and this section echoes something I’ve seen Cal Newport say before, something along the lines that somebody with no technical knowledge sees one neat trick and suddenly assumes AI will turn into a superintelligence deity.
This matches all too well with what Baldur Bjarnason writes in “The Intelligence Illusion”.
Three years later and the “early days” routine is still going strong. Who would have thought?
What AI “decision making” does in our processes is: write the reports exactly like the procedure says they should be written, and fills them with whatever you want to hear - looking up convenient references to back up your opinions. You probably should double check the references yourself to be sure they actually exist (though I think AI is getting better at checking its own work for things like this lately) - but… yeah. Ask it once and it will give you “an opinion” that doesn’t take into account all of the factors that matter to you. So, as you explain all the factors, it also becomes obvious what you want to hear, so after several rounds the obliging AI eventually not only “sees things your way” but can pile on the BS as deep as you ask it to to back up those opinions. What would have taken days to research and write can be done in an hour or so.
It is the ultimate corporate yes-man that they could never quite get in a human. They have built it for C-suites who have control of the money but not their own faculties.