callers get automatically routed to an AI agent who asks if they are calling regarding the incident. If the answer is yes, then callers can receive information or updates, and if it’s no, then the callers are transferred to a human.
So if somebody calls and is completely out of his mind, what will AI do?
What if someone is obviously out of their mind and says yes? Random example, say someone has a stroke and has just barely figured out how to use the big red emergency call button on their phone screen. The AI asks if they are calling about blah blah emergency. Maybe the person only understands half of it, thinks they’re being asked if they’re calling about an emergency and say “yes”. Now they’re stuck having a form letter read to them by AI while their brain is dying. A person would catch something was up even if they said yes.
The article even notes this can be a problem without anything medical going on:
It’s also possible that AI agents handling 911 triage could fail to understand individuals with stronger accents and dialects, as well as differences in pitch and articulation. This is because AI is programmed with automatic speech recognition systems and, therefore, could potentialy be unreliable when presented with a speech unlike the ones used the AI is trained to recognize.
Responding to people having emergencies is too important a task to leave up to voice recognition slapped on top of an LLM.
Sure, usually, but compare like to like. If a human operator asked if the call was about an already known scenario and transferred people to an automated service if they said “yes”, then the same exact thing could happen.
You attemtped to craft a very specific scenario in which harm would result, when in reality it is wildly unlikely that someone would call in at that precise time and only respond with the word “yes”.
I gave a single example of one way that AI can miss a genuine emergency in a way that a human operator won’t. ‘Nobody will ever call 911 with a stroke and be confused’ isn’t a realistic defense of removing the best system we have of assessing information, the human brain.
It would have to be a stroke during the time there was an ongoing scenario that resulted in higher than typical call volume about the scenario, and a stroke victim calling in who could only say “yes”.
It doesn’t have to be a stroke, do you really think that’s the only thing that can make people confused or give anything other than the expected answer? Panic, being hard of hearing, sometimes people even have to pretend to be calling someone else so that they can call for help in front of an abuser.
You’re really married to the idea of this being unable to go wrong.
I don’t know the concrete answer to your "what if,’ and I don’t think you or the author of the article do either. That would be a question for Carbyne.
I would speculate that there is an answer, and that neither you nor the author are the first to think of these problem scenarios. There’s design, engineering, and real-world testing involved.
I gave you the benefit of the doubt. If you’ve had lots of interactions with AI, I am not sure how you can think that anything with AI can work as expected, specially in complicated edge cases.
If you’ve had lots of interactions with AI, I am not sure how you can think that anything with AI can work as expected
That’s a very broad statement, and “AI fluency” is a broad spectrum. Having “lots of interactions with AI” isn’t an indicator of much at all – lots of end users use LLMs every day. They have “lots of interactions with AI”, but most of them have not developed/trained agent “skill” files, built agent workflows, developed harnesses, handed complex tasks to a “team” of agents (and tuning them for reliability and fidelity), etc. Anyone can have lots of interactions with AI without ever experimenting beyond the consumer-grade interfaces.
Asking if someone has had “lots of interactions with AI” says much more about the experience of the person asking. A lot of “advanced” (using the term very loosely here) users/builders would know that your concern is an easy problem to solve. It might even be a decent sort of challenge to give an intern/student as a learning exercise, since they already have a lot real-world data (from their 311 trials and other cities/orgs that also use this tech) to use for testing and iterating.
You seem to see yourself as experienced AI user, but all you do is repeat the empty marketing promises.
Which ones? There are lots of empty marketing promises around AI; the signal-to-noise ratio is crazy. I try avoid the ones that I’m not confident in, but I admit that I don’t always express that clearly enough. (It’s sort of a bad habit/weak skill for me. These things are more obvious and sort of become “common sense” amongst colleagues, and I’ve never been much of a social media guy.)
I’m happy to try to clarify something, if needed.
The fact is, if they use an LLM, there will be mistakes, just like Claude will happily ignore part of it’s instructions a few times per day.
Of course there will be. Artificial neural networks are function approximation algorithms.
The thing about LLMs is that they’re almost always configured to run non-deterministically[1]. Most tools that depend on LLMs can’t be expected to work or fail deterministically. Building these tools is an iterative process of evaluating and tuning to maximize success rate, thus minimizing failure rate. So yes, LLM-based tech like this will definitely fuck up with a non-zero failure rate. And no, you probably won’t hear about this amongst the marketing noise.
Regarding the problem that the 911 triage thing aims to address: The solution would probably be to hire more operators/dispatchers. Unfortunately, there has been a nationwide shortage of them for years. The AI triage thing has proven to have a success rate high enough (and thus a failure rate low enough) to provide net-positive value[2]. It doesn’t 100% solve the problem, it just mitigates it, so the situation is less bad – not solved.
“AI fluency” is not a thing
It absolutely is a thing, and usually becomes VERY obvious when you start building things to use LLMs for automation. It’s like those iceberg memes. I’m not sure if “AI fluency” is the “correct” (as in widely-adopted) term for it though, but some people have called it that.
Some even try to quantify this, in some industries. For example, I’ve heard of a “Yegge scale” (or something like that) to estimate the AI fluency of a software engineer.
AI makes you lose your critical thinking skills, which then makes you rely even more on AI.
This is a very broad statement. It’s not always true, and not always false. It really depends on how the person uses AI. But yes, it’s true for a lot of everyday users of the mainstream, consumer-grade interfaces (e.g. ChatGPT) who offload their thinking to AI. It’s one of many sources of brainrot for the masses, and that sucks. I suppose that’s a different topic though.
Apologies for the wall of text. People often make such loaded, over-generalized (though usually reasonable) statements in AI-related conversations, and I have a bad habit of yapping about the nuances 😬
This is the temperature hyperparameter. There are reasons for needing it to run non-deterministically, but that’s another topic. ↩︎
Sort of like vaccines or condoms: While not 100% effective, their success rate is high enough to provide real value. ↩︎
Did you read the article?
So if somebody calls and is completely out of his mind, what will AI do?
Is this a rhetorical question?
Just in case you wanted a real answer: The caller is obviously not saying “yes,” so transfer to human.
To be clear: I don’t know anyone at Carbyne or OPCD. I can only offer speculation. But I’m curious: What would you have guessed?
What if someone is obviously out of their mind and says yes? Random example, say someone has a stroke and has just barely figured out how to use the big red emergency call button on their phone screen. The AI asks if they are calling about blah blah emergency. Maybe the person only understands half of it, thinks they’re being asked if they’re calling about an emergency and say “yes”. Now they’re stuck having a form letter read to them by AI while their brain is dying. A person would catch something was up even if they said yes.
The article even notes this can be a problem without anything medical going on:
Responding to people having emergencies is too important a task to leave up to voice recognition slapped on top of an LLM.
Besides “because it supports the conclusion I want to arrive at”, what makes you believe this is true?
People having a stroke can’t carry on a conversation and seem normal.
Sure, usually, but compare like to like. If a human operator asked if the call was about an already known scenario and transferred people to an automated service if they said “yes”, then the same exact thing could happen.
You attemtped to craft a very specific scenario in which harm would result, when in reality it is wildly unlikely that someone would call in at that precise time and only respond with the word “yes”.
I can only speculate as to why you would do that.
I gave a single example of one way that AI can miss a genuine emergency in a way that a human operator won’t. ‘Nobody will ever call 911 with a stroke and be confused’ isn’t a realistic defense of removing the best system we have of assessing information, the human brain.
It would have to be a stroke during the time there was an ongoing scenario that resulted in higher than typical call volume about the scenario, and a stroke victim calling in who could only say “yes”.
Not impossible, but wildly unlikely.
It doesn’t have to be a stroke, do you really think that’s the only thing that can make people confused or give anything other than the expected answer? Panic, being hard of hearing, sometimes people even have to pretend to be calling someone else so that they can call for help in front of an abuser.
You’re really married to the idea of this being unable to go wrong.
I don’t know the concrete answer to your "what if,’ and I don’t think you or the author of the article do either. That would be a question for Carbyne.
I would speculate that there is an answer, and that neither you nor the author are the first to think of these problem scenarios. There’s design, engineering, and real-world testing involved.
“Well I just trust they thought of it” isn’t good enough.
If you’re unsatisfied with my answer, then maybe consider asking them. I’m just a guy writing comments on Lemmy.
I guess you don’t have lots of interaction with AI yet?
Very incorrect. Why do you ask?
I gave you the benefit of the doubt. If you’ve had lots of interactions with AI, I am not sure how you can think that anything with AI can work as expected, specially in complicated edge cases.
That’s a very broad statement, and “AI fluency” is a broad spectrum. Having “lots of interactions with AI” isn’t an indicator of much at all – lots of end users use LLMs every day. They have “lots of interactions with AI”, but most of them have not developed/trained agent “skill” files, built agent workflows, developed harnesses, handed complex tasks to a “team” of agents (and tuning them for reliability and fidelity), etc. Anyone can have lots of interactions with AI without ever experimenting beyond the consumer-grade interfaces.
Asking if someone has had “lots of interactions with AI” says much more about the experience of the person asking. A lot of “advanced” (using the term very loosely here) users/builders would know that your concern is an easy problem to solve. It might even be a decent sort of challenge to give an intern/student as a learning exercise, since they already have a lot real-world data (from their 311 trials and other cities/orgs that also use this tech) to use for testing and iterating.
You seem to see yourself as experienced AI user, but all you do is repeat the empty marketing promises.
The fact is, if they use an LLM, there will be mistakes, just like Claude will happily ignore part of it’s instructions a few times per day.
And I think that “AI fluency” is not a thing, AI makes you lose your critical thinking skills, which then makes you rely even more on AI.
Which ones? There are lots of empty marketing promises around AI; the signal-to-noise ratio is crazy. I try avoid the ones that I’m not confident in, but I admit that I don’t always express that clearly enough. (It’s sort of a bad habit/weak skill for me. These things are more obvious and sort of become “common sense” amongst colleagues, and I’ve never been much of a social media guy.)
I’m happy to try to clarify something, if needed.
Of course there will be. Artificial neural networks are function approximation algorithms.
The thing about LLMs is that they’re almost always configured to run non-deterministically[1]. Most tools that depend on LLMs can’t be expected to work or fail deterministically. Building these tools is an iterative process of evaluating and tuning to maximize success rate, thus minimizing failure rate. So yes, LLM-based tech like this will definitely fuck up with a non-zero failure rate. And no, you probably won’t hear about this amongst the marketing noise.
Regarding the problem that the 911 triage thing aims to address: The solution would probably be to hire more operators/dispatchers. Unfortunately, there has been a nationwide shortage of them for years. The AI triage thing has proven to have a success rate high enough (and thus a failure rate low enough) to provide net-positive value[2]. It doesn’t 100% solve the problem, it just mitigates it, so the situation is less bad – not solved.
It absolutely is a thing, and usually becomes VERY obvious when you start building things to use LLMs for automation. It’s like those iceberg memes. I’m not sure if “AI fluency” is the “correct” (as in widely-adopted) term for it though, but some people have called it that.
Some even try to quantify this, in some industries. For example, I’ve heard of a “Yegge scale” (or something like that) to estimate the AI fluency of a software engineer.
This is a very broad statement. It’s not always true, and not always false. It really depends on how the person uses AI. But yes, it’s true for a lot of everyday users of the mainstream, consumer-grade interfaces (e.g. ChatGPT) who offload their thinking to AI. It’s one of many sources of brainrot for the masses, and that sucks. I suppose that’s a different topic though.
Apologies for the wall of text. People often make such loaded, over-generalized (though usually reasonable) statements in AI-related conversations, and I have a bad habit of yapping about the nuances 😬
This is the
temperaturehyperparameter. There are reasons for needing it to run non-deterministically, but that’s another topic. ↩︎Sort of like vaccines or condoms: While not 100% effective, their success rate is high enough to provide real value. ↩︎