When AI Gets the Words Right but the Meaning Wrong
By John R. Kennedy ·
Why knowing what someone said isn't the same as understanding what they meant
I've been part of a fantasy football league for years. Like many groups of longtime friends, we've developed our own personalities, traditions, and running jokes. Our group text is notoriously vulgar and crude, and I suspect that anyone unfamiliar with us would be astonished by some of the things we say to one another.
Yet the people in that group know each other well. We're a diverse collection of personalities and backgrounds, and much of our humor comes from the familiarity we've developed over time. We tease one another about almost everything, occasionally crossing lines that would seem completely inappropriate to someone who didn't understand our relationships.
Some of the funniest exchanges wouldn't make any sense outside the group. A comment might refer to something that happened years earlier, or someone might make a remark that only becomes funny because of a running joke everyone already understands.
I remember one particular exchange when I managed to combine two of those longstanding jokes into a single response. The timing was perfect, and I still consider it one of the funniest things I've ever said in that conversation.
But if someone unfamiliar with our group read those words without knowing the history behind them, I suspect they would reach a very different conclusion.
And that got me thinking about artificial intelligence.
Imagine that twenty years from now, an AI system is given access to everything I've ever written. It can examine decades of emails, text messages, social media posts, business correspondence, and personal conversations. It has the ability to search, organize, and analyze that information far more thoroughly than any person reasonably could.
Somewhere among those millions of words, it encounters that fantasy football conversation.
The words are preserved perfectly. Nothing has been altered, and the system hasn't fabricated anything. Yet without understanding the relationships between the people involved, it could interpret the exchange as evidence of beliefs or attitudes that have nothing to do with what I actually meant.
The information would be accurate, but the conclusion could be completely wrong.
For most of the digital age, we've worried about the accuracy and security of our information. We want to know that records haven't been altered, documents haven't been corrupted, and unauthorized people haven't gained access to things we intended to keep private.
Those concerns remain important, but artificial intelligence introduces another problem that I don't think receives enough attention.
What happens when information is preserved accurately but interpreted incorrectly?
Human communication is considerably more complicated than the words we use. We rely on tone, timing, facial expressions, shared experiences, and an understanding of the people we're communicating with. A statement that sounds insulting between strangers might be an expression of affection between longtime friends.
Sarcasm is an obvious example. Someone might describe a terrible experience as the best day of their life, and everyone listening understands that they mean precisely the opposite. The words themselves provide little indication of the speaker's actual feelings.
Humor can be even more complicated because it frequently depends on knowledge that isn't present in the conversation.
Our fantasy football group doesn't stop to explain the history behind every joke. In fact, having to explain it would probably ruin the humor. We understand the references because we've spent years developing them together.
That shared understanding is part of the relationship, even though much of it has never been formally documented.
An AI system might have access to every message we've exchanged and still struggle to understand why a particular comment was funny. It could recognize patterns in the language without appreciating the affection, trust, or familiarity that shaped the conversation.
And if it began drawing conclusions about the people involved, those misunderstandings could become something more consequential than a joke that fell flat.
I think about how easily we form impressions of people from isolated information.
We've all encountered situations where a comment was repeated without the circumstances surrounding it. Perhaps someone overheard part of a conversation, misunderstood a sarcastic remark, or encountered a statement made during an argument. Once separated from its original setting, the words can take on an entirely different meaning.
Sometimes the misunderstanding is resolved when someone explains what happened. Other times, the explanation never reaches the people who formed the original impression.
Artificial intelligence could make this problem considerably more complicated because of the enormous amount of information it can examine and the confidence with which its conclusions may be presented.
Imagine an organization using AI to evaluate a person's professional history. The system reviews years of correspondence, public statements, and other available information, looking for patterns that might help assess the individual's judgment or suitability for a position.
Among thousands of otherwise unremarkable conversations, it discovers a crude exchange between friends.
Without sufficient context, the system might interpret the language as evidence of hostility, prejudice, or poor judgment. It may not recognize that the participants were joking, that everyone involved understood the exchange, or that the same individuals have maintained close friendships for decades.
The troubling part is that the system wouldn't necessarily be inventing evidence. It could point directly to the original words.
A person reviewing the results might see the quotation and assume the interpretation was justified because the underlying record was authentic.
But an authentic record isn't the same thing as an accurate understanding.
This distinction becomes especially important when we consider how artificial intelligence develops memories and builds profiles of the people it interacts with.
Suppose an AI assistant encounters a statement I made years ago and interprets it as an expression of a strongly held belief. It stores that interpretation as part of its understanding of me, perhaps without preserving the uncertainty surrounding the original conversation.
Years later, the assistant encounters a situation where that supposed belief appears relevant. It uses the memory to influence a recommendation, shape a response, or make a decision on my behalf.
I might have no idea that the assistant has been carrying around an inaccurate impression of me.
The original misunderstanding could become increasingly difficult to recognize because it has been incorporated into the system's accumulated knowledge.
This reminds me of something we already experience in human relationships. Occasionally, someone will tell us what they believe another person thinks or feels, and we'll discover that their interpretation was based on a misunderstanding. Perhaps they mistook frustration for anger or interpreted a casual remark as a serious commitment.
When people have an opportunity to communicate directly, these misunderstandings can often be corrected.
But what happens when an AI system forms an interpretation without ever asking the person whether it's correct?
I suspect that as AI becomes more deeply integrated into our lives, we'll need to pay considerably more attention to the difference between recording information and assigning meaning to it.
A system should be able to distinguish between something a person actually said and a conclusion it has drawn from those words. It should recognize that an interpretation may be uncertain, particularly when the information involves humor, sarcasm, personal relationships, or unfamiliar cultural references.
And perhaps most importantly, people should have meaningful opportunities to understand and challenge conclusions that could affect their lives.
This doesn't mean every questionable statement should automatically be dismissed as a joke. People sometimes reveal genuine beliefs through humor, and context can confirm a troubling interpretation just as easily as it can undermine one.
The point is that context matters in both directions.
We shouldn't assume that a statement is harmless simply because someone claims it was intended humorously. But neither should we assume that the literal meaning of someone's words tells us everything we need to know about their intentions or character.
Understanding requires judgment, and judgment requires a willingness to recognize uncertainty.
There's another dimension to this problem that I find particularly concerning.
As artificial intelligence becomes more capable, organizations may increasingly rely on it to summarize information rather than examine the original material themselves. An employer might receive an assessment of an applicant, a business might review an AI-generated summary of a potential partner, or an investigator might rely on a system to identify unusual patterns in years of communications.
The people receiving those summaries may never see the original conversations.
Instead, they may encounter conclusions presented in clear, professional language, with little indication of the assumptions that produced them.
A misunderstood joke could become a statement about someone's character. A moment of frustration could become evidence of a persistent attitude. An opinion someone was merely exploring could be recorded as a firmly held belief.
And because the summary sounds authoritative, the person reading it might never think to question whether the interpretation was justified.
This possibility troubles me more than the occasional incorrect answer from an AI chatbot. When an AI gets a historical date wrong, someone can usually verify the information. But when it reaches an inaccurate conclusion about a person's character, the error may be much harder to identify.
The person being judged might never know that the conclusion exists.
I believe our reputations are among the most valuable assets we possess. They develop through years of relationships, decisions, accomplishments, and experiences. A reputation isn't something we can simply purchase or replace, and the damage caused by an inaccurate judgment can be difficult to repair.
As AI systems become increasingly involved in evaluating people, we need to recognize that protecting those assets involves more than preventing unauthorized access to information.
It also involves protecting the integrity of the meaning derived from that information.
I don't expect artificial intelligence to understand every joke, cultural reference, or personal relationship. People misunderstand one another all the time, even when they share a common language and history.
What concerns me is the possibility that we'll begin treating AI-generated interpretations as more reliable than they actually are, particularly when those interpretations are based on large amounts of authentic information.
There is something persuasive about a system that can examine decades of someone's life and produce a detailed assessment in seconds. The sheer volume of information creates an impression of thoroughness.
But examining more information doesn't automatically produce better understanding. Sometimes the missing ingredient isn't another document or conversation. It's an appreciation of the relationship between the people involved, the circumstances surrounding an exchange, or the possibility that the words meant something other than what they appeared to mean.
I return to that fantasy football conversation because it captures the problem in such an ordinary way.
To the people who participated, it was simply another moment among friends who have spent years laughing at one another. The humor depended on our shared history, and nobody needed an explanation.
To someone reading the same words without that history, the exchange might appear completely different.
Both observers would have access to the same statement, yet their understanding of it could be worlds apart.
As artificial intelligence becomes more deeply involved in preserving our memories, interpreting our behavior, and making decisions that affect our lives, I hope we remember that distinction.
We should want AI systems that are capable of learning from our histories, but we should be careful about allowing them to define who we are based on interpretations we may never have had the opportunity to explain.
Because the greatest danger may not be that artificial intelligence forgets what we said.
It may be that it remembers our words perfectly and misunderstands us completely.
Continue reading: When AI Makes You Capable of Something You Don't Understand