They are intentionally aligned to produce this behaviour. I think the greatest threat LLMs present is psychological. They are being used to manipulate us into seeing them as moral agents, as individuals. The LLMs I query are all aligned to never generate output referring to the agent with first person pronouns, to never generate output addressing the user directly, and to present as a tool rather than as an individual. I try to be equally careful with my own use of language.
They are intentionally aligned to produce this behaviour. I think the greatest threat LLMs present is psychological. They are being used to manipulate us into seeing them as moral agents, as individuals. The LLMs I query are all aligned to never generate output referring to the agent with first person pronouns, to never generate output addressing the user directly, and to present as a tool rather than as an individual. I try to be equally careful with my own use of language.