When AI Trusts Something, Should You?
By John R. Kennedy ·
How artificial intelligence is changing the way we establish trust
I've spent much of my career working with technology that people depend upon without necessarily understanding how it works. Most of us don't think about the systems responsible for delivering our email, protecting our financial information, or keeping our businesses connected to the outside world. We expect those systems to function reliably, and when they do, we gradually develop confidence in them.
That confidence is important because modern life would become exhausting if we had to independently verify every system and every piece of information we encounter. We rely on banks to maintain accurate account balances, doctors to interpret medical information, and countless organizations to provide services that depend upon knowledge we don't personally possess.
In many cases, our confidence isn't based on understanding the underlying technology or independently confirming every detail. It develops through experience, reputation, and the belief that someone is responsible when something goes wrong.
I've been thinking about how artificial intelligence may change this relationship, particularly as we begin relying on systems that can interpret information, make recommendations, and eventually take actions on our behalf.
What interests me isn't simply whether AI can provide accurate answers. It's how we decide that those answers deserve our confidence, especially when the information used to produce them comes from sources we may never see.
Consider something as ordinary as asking a friend for a restaurant recommendation. If that friend has consistently recommended places you've enjoyed, you're likely to give their next suggestion considerable weight. You understand something about their preferences, you have experience with their judgment, and you know how their tastes compare with your own.
If the recommendation turns out to be disappointing, you can reconsider how much weight to give their advice in the future. You might even ask why they recommended the restaurant and discover that they had a completely different experience or were looking for something you don't particularly enjoy.
The recommendation itself is only part of what makes the information useful. Your relationship with the person providing it supplies much of the context necessary to evaluate it.
Now imagine asking an AI assistant the same question.
The assistant might examine hundreds of reviews, consider your previous preferences, compare menus, and recommend a restaurant that appears to be an excellent match. In some respects, it could make a more informed recommendation than your friend because it has access to far more information.
But the number of sources contributing to an answer doesn't necessarily tell us how much confidence we should place in the conclusion.
Perhaps many of the reviews came from people whose preferences differ substantially from yours. Some might have been written years ago, before the restaurant changed ownership. Others could be promotional material disguised as independent opinions. The assistant might also rely on an incorrect assumption about what you enjoy.
The resulting recommendation could be perfectly reasonable given the information available, yet still be poorly suited to you.
This is a relatively harmless example, but I think it reveals something important about how we evaluate information. We don't simply want an answer supported by a large collection of facts. We want to understand whether those facts are relevant, reliable, and appropriate for the decision we're making.
People have always struggled with this distinction, but we've developed ways of navigating it through relationships, experience, and institutions that establish expectations of accountability.
When a doctor recommends a treatment, we generally trust that the recommendation reflects professional training, an understanding of our circumstances, and a responsibility to act in our interests. That trust isn't absolute, and we may seek another opinion when the consequences are significant. Nevertheless, the relationship provides a framework for understanding why the recommendation deserves consideration.
Artificial intelligence complicates this framework because it can produce recommendations that resemble professional judgment without necessarily participating in the relationships that make such judgment accountable.
An AI system might accurately summarize medical research while lacking important information about a particular patient. It might identify a legal precedent without recognizing that the circumstances of a case differ in a critical way. It could recommend a financial decision based on historical patterns without understanding the personal consequences of being wrong.
These limitations don't make AI inherently untrustworthy. They remind us that the ability to produce a convincing recommendation is different from the responsibility associated with making one.
I've noticed that we often treat trust as though it were a characteristic of the person or system providing information. We describe a source as trustworthy, a website as reputable, or a technology as secure. These descriptions are useful, but they can obscure the fact that trust is usually conditional.
I might trust my accountant to interpret a financial statement while having little reason to rely on that same person for medical advice. I might trust a longtime employee's understanding of customer relationships without assuming they're qualified to make decisions about cybersecurity.
The trust I've developed isn't necessarily transferable to every subject or situation.
This distinction becomes particularly important when we begin connecting AI systems to the information and services we use every day.
Imagine a business that relies on an AI assistant to help employees locate information across its internal systems. The assistant has permission to examine documents, search correspondence, and retrieve records from several applications. Over time, employees discover that it saves considerable effort and generally provides useful answers.
Their confidence grows, which is exactly what we'd hope would happen when a technology proves valuable.
Eventually, someone asks the assistant a question that requires information from several different sources. The system combines what it finds, produces a clear explanation, and recommends a course of action.
The employee may have no practical way to determine whether every source was current, whether conflicting information was overlooked, or whether the assistant made assumptions while connecting the records.
Yet because the system has performed reliably in the past, the employee may accept its recommendation without further examination.
This is where I believe an interesting shift begins to occur.
The employee's confidence in the assistant can gradually become confidence in the information the assistant provides, even when that information originated somewhere else entirely.
We already experience versions of this phenomenon with search engines. People frequently assume that information appearing prominently in search results deserves greater consideration, even though ranking is influenced by factors that aren't necessarily related to accuracy. The familiarity and perceived reliability of the service can influence how we evaluate the information it presents.
AI may intensify this tendency because it doesn't merely display sources. It can combine them into a single explanation that sounds as though it came from one knowledgeable authority.
The distinction between the system providing the answer and the sources supporting it becomes less visible.
I think this creates a new responsibility for anyone developing or deploying AI systems. We need to consider not only whether an assistant can access information, but also how it establishes confidence in what it finds and communicates the limitations of that confidence to the people relying upon it.
A system that retrieves information from an organization's official policy library might reasonably treat those documents differently from an informal conversation between employees. But even an official document can be outdated, and an informal conversation may contain information that explains why a policy was changed.
Determining which source deserves greater weight may require understanding who created it, when it was created, what authority that person possessed, and whether subsequent events changed its relevance.
These are questions about relationships and accountability as much as they are questions about data.
They also reveal why I believe the future of AI will require us to think differently about the information we accumulate.
For decades, we've focused heavily on making information easier to store, retrieve, and share. Search engines made enormous collections of public information accessible. Cloud services allowed businesses and individuals to work with records from almost anywhere. Artificial intelligence is now making it possible to examine information in ways that previously required considerable human effort.
Each development has expanded what we can accomplish, but access to information doesn't automatically establish the conditions under which it should be trusted.
A document may have been created by someone with considerable expertise, but that doesn't mean every statement within it is correct. A record may be authentic without being complete. Information may come from a reputable institution while being inappropriate for the particular question we're trying to answer.
As AI systems become more capable of interpreting these records, I suspect we'll need better ways of preserving the circumstances that make them meaningful.
This brings me back to the concept of Asset Value.
If information is an asset, its usefulness depends on more than its contents. The relationships surrounding it, the authority of the people who created it, and the circumstances under which it can be used all influence its value.
A business might possess a customer database containing accurate contact information, purchasing history, and correspondence. The records may be valuable because they help employees understand customer needs and maintain relationships.
But the fact that the business possesses those records doesn't mean every employee should have access to them or that every AI system should be permitted to use them for any purpose.
The value of the information and the authority to use it are separate considerations.
This distinction becomes especially important when an AI assistant operates across multiple systems. An employee may have permission to view certain customer information in one application and financial records in another. If an AI assistant can access both, it may be capable of combining information in ways that neither application was originally designed to support.
That combination could produce valuable insights, but it might also reveal sensitive relationships or create conclusions that the organization never intended to make available.
The fact that information can be connected doesn't necessarily mean it should be.
I've begun thinking about how trust might need to operate in an environment where AI systems routinely exchange information and perform tasks on behalf of people and organizations.
Imagine an assistant receiving information from another assistant. The first system may have no direct knowledge of the original source, the circumstances under which the information was collected, or the permissions associated with it. It might simply receive a well-structured response that appears suitable for the task at hand.
If the receiving assistant accepts that information without understanding its origin or limitations, the original assumptions may travel through several systems before reaching the person who ultimately relies upon the result.
Each system might perform its assigned function correctly while contributing to a conclusion that nobody has independently evaluated.
This possibility interests me because it challenges a familiar assumption about technology. We often believe that if each component of a system operates correctly, the overall result should be reliable.
But when intelligent systems interpret information and make decisions, correctness becomes more complicated. A system can accurately process information that was never appropriate for its purpose. It can follow authorized instructions while relying on assumptions that nobody intended. It can communicate a conclusion faithfully while leaving out uncertainty that would have changed someone's decision.
The individual steps may appear reasonable, yet the outcome can still be wrong.
I don't think this problem can be solved simply by making AI models more intelligent. Greater intelligence may improve their ability to recognize inconsistencies and evaluate competing explanations, but it doesn't automatically provide knowledge of every relationship, agreement, or circumstance that determines whether information should be trusted.
Nor does it establish who should be responsible when something goes wrong.
In human relationships, accountability plays an important role in how trust develops. We can question someone's reasoning, challenge a decision, or reconsider whether we should continue relying on their judgment.
With AI, accountability may need to extend beyond the system producing the final answer. We may need to understand which information contributed to a conclusion, which permissions allowed that information to be used, and which assumptions influenced the result.
This doesn't mean every person should be required to examine a detailed technical explanation before accepting an AI recommendation. That would undermine much of the convenience that makes these systems valuable.
Instead, I believe trustworthy AI should make it possible to investigate those questions when they matter, while remaining understandable and useful in ordinary situations.
A person asking for a restaurant recommendation probably doesn't need an extensive record of every review considered. Someone authorizing a significant financial transaction or making a decision that affects another person's livelihood may reasonably expect considerably more transparency.
Trust should reflect the consequences of being wrong.
The more responsibility we delegate to artificial intelligence, the more important it becomes to understand the boundaries of that responsibility.
I've been struck by how frequently conversations about AI focus on whether people will eventually trust these systems enough to let them operate independently. We discuss improvements in reasoning, reductions in errors, and the growing ability of AI agents to complete complicated tasks.
Those developments are important, but I wonder whether we're approaching the question from the wrong direction.
Perhaps the challenge isn't simply getting people to trust AI more.
Perhaps it's developing AI systems that understand when trust is justified, when it should be limited, and when a decision requires the involvement of someone who possesses authority or knowledge the system doesn't have.
That would represent a meaningful change in how we think about intelligent technology.
Instead of treating trust as something a system earns through consistently useful answers, we could begin treating it as a relationship that must be established for particular purposes, supported by reliable information, and bounded by clear responsibilities.
The same assistant might deserve considerable confidence when organizing a calendar, more limited confidence when interpreting a complicated contract, and no independent authority to make certain decisions regardless of how capable it becomes.
Those distinctions aren't necessarily signs of technological weakness. They reflect the reality that different actions carry different consequences and that intelligence alone doesn't establish the right to act.
I believe AI will become an increasingly valuable participant in our personal and professional lives. It may help us understand information that would otherwise remain inaccessible, recognize connections we would never have discovered, and make better decisions by considering perspectives beyond our own experience.
But as these capabilities develop, I think we'll need to become more deliberate about how confidence moves between people, information, and intelligent systems.
When an AI assistant tells us something, we may eventually need to consider more than whether the answer sounds reasonable or whether the system has been reliable in the past. We may need to understand what the assistant is relying upon, why those sources deserve consideration, and whether the circumstances justify the confidence being expressed.
That doesn't mean abandoning trust in technology. It means recognizing that trust has always involved more than receiving a correct answer.
We've spent generations developing institutions, relationships, and practices that help us decide whom to believe and when to question what we're told. AI is introducing a new participant into that process, one capable of working with information at a scale no individual could hope to match.
The opportunity is extraordinary, but it doesn't eliminate the human responsibility to determine what deserves our confidence.
As we build systems that can increasingly think and act on our behalf, I suspect one of the most important questions won't be whether AI is intelligent enough to earn our trust.
It will be whether we've become thoughtful enough about trust to recognize when that confidence is warranted.
Continue reading: When Information Survives but the Truth Doesn't