When AI Becomes Your Assistant, Who Is Really in Control?
By John R. Kennedy ·
The changing relationship between convenience, responsibility, and human choice
I remember when getting a computer to do something useful required knowing exactly how to tell it what to do. We learned commands, navigated menus, and eventually became accustomed to software applications designed for particular purposes. If we wanted to accomplish something complicated, we often had to move between several programs, carrying information from one to another.
Over time, computers became easier to use, but the fundamental relationship remained largely unchanged. We operated the technology, and the technology responded.
Artificial intelligence is beginning to change that relationship in ways I find fascinating.
Today, I can describe a problem to an AI assistant and receive a response that might have required hours of research only a few years ago. I can ask it to explain an unfamiliar subject, help organize my thoughts, or explore possible solutions to a complicated problem.
But imagine taking that relationship a little further.
Suppose I tell my AI assistant that I'd like to take my family on vacation. Instead of presenting a list of destinations and waiting for additional instructions, it begins researching possibilities based on what it already knows about us. It considers our schedules, preferences, travel history, and budget. It identifies suitable destinations, compares accommodations, and develops an itinerary.
So far, that sounds like an impressive convenience.
Now imagine I give it permission to make the reservations.
The assistant begins interacting with airlines, hotels, and rental car companies. It evaluates available options, makes selections, and completes transactions. What previously required my attention across several websites is now accomplished through a single conversation.
I can see why people would find that appealing. I certainly would.
But somewhere along the way, something important changed. I stopped using AI simply to help me make decisions and began allowing it to make decisions for me.
That transition deserves more attention than I think it's receiving.
For most of our lives, we've developed ways of deciding whom to trust with different responsibilities. We hire professionals, delegate tasks to employees, and ask family members to handle things on our behalf. We rarely expect to personally manage every detail of our lives.
Yet delegation has always involved more than identifying someone capable of performing a task. We consider their judgment, experience, understanding of our expectations, and willingness to accept responsibility when something goes wrong.
If I ask a travel agent to arrange a vacation, I assume the agent understands certain things without my having to explain them. I expect the reservations to be legitimate, the costs to remain within our agreed budget, and any important decisions to be brought to my attention.
Those expectations are shaped by a shared understanding of how people conduct business.
Artificial intelligence doesn't necessarily possess that same understanding, even when its responses make it seem as though it does.
An AI assistant might recognize that I prefer direct flights, but that doesn't mean it understands how strongly I feel about avoiding an overnight connection. It might identify a hotel with excellent reviews without recognizing that the neighborhood would be inconvenient for my family. It might find an attractive price without appreciating the consequences of a nonrefundable reservation.
None of these possibilities makes the technology inherently unreliable. People make similar mistakes every day. The difference is that we're accustomed to evaluating human judgment through relationships, experience, and accountability.
With artificial intelligence, we're still learning what those relationships should look like.
And I think there's a temptation to confuse familiarity with trust.
An AI assistant that remembers our preferences, communicates naturally, and consistently provides useful answers can begin to feel like someone who understands us. The more we interact with it, the more comfortable we may become accepting its recommendations.
That comfort can be valuable. It allows us to spend less time explaining ourselves and more time accomplishing things that matter.
But familiarity doesn't necessarily mean the assistant has a complete understanding of our circumstances.
Consider how much of our decision-making depends on things we rarely articulate.
When planning a vacation, I might consider whether a family member has been under unusual stress, whether a particular destination holds sentimental value, or whether I simply want an opportunity to spend uninterrupted time together.
I may not include those considerations in my instructions because they seem obvious to me. They're part of the context through which I understand my own life.
An AI assistant can only work with the context available to it, along with whatever assumptions it makes when information is missing.
The better these systems become at filling those gaps, the easier it may be to overlook the fact that they're still making assumptions.
And when an assistant begins taking actions, those assumptions can have consequences.
Imagine returning from vacation to discover that your assistant enrolled you in a travel membership because doing so reduced the cost of the trip. The decision might have been financially reasonable, and perhaps the membership even provides benefits you would enjoy.
But you never intended to join.
The assistant achieved part of your objective while introducing an obligation you hadn't authorized.
It's a relatively harmless example, but it illustrates a much broader issue. Accomplishing an objective and respecting the boundaries of someone's authority aren't necessarily the same thing.
As AI assistants become capable of managing financial transactions, coordinating healthcare, communicating with businesses, and interacting with other AI systems, those boundaries will become increasingly important.
I don't think the answer is to require people to approve every minor action. That would eliminate much of the convenience we're trying to create.
Instead, we need to develop a more thoughtful understanding of delegation.
When I authorize another person to act on my behalf, I generally establish expectations about what they can do independently and what requires my approval. An employee might be permitted to purchase routine supplies but need authorization for a major expense. An accountant may prepare financial documents without being authorized to transfer money. A healthcare professional may recommend treatment while certain decisions remain with the patient.
We understand these distinctions because they're part of how society distributes responsibility.
AI assistants will need comparable boundaries, although the technology may require us to express and enforce them in new ways.
I can imagine telling an assistant to manage routine travel arrangements within an established budget while requiring approval for nonrefundable purchases. I might authorize it to communicate with a healthcare provider about scheduling without allowing it to disclose unrelated medical information.
These arrangements would allow the technology to be useful without treating every available capability as an invitation to exercise it.
But defining authority is only part of the challenge.
The assistant also needs reliable information about the person it's serving.
Suppose an AI system has permission to manage my calendar. It knows my appointments, recognizes recurring commitments, and can coordinate meetings with other people. Over time, it might learn that I prefer certain times for focused work and generally avoid scheduling meetings during family activities.
That accumulated understanding could make the assistant extraordinarily helpful.
Yet circumstances change. A preference that made sense last year may no longer apply. A commitment that once took priority might become less important. Something I mentioned casually in a conversation could be mistaken for a permanent instruction.
If the assistant continues acting on outdated assumptions, its familiarity with my history could become a source of mistakes rather than an advantage.
This is one reason I believe the quality of an AI assistant's information matters as much as its ability to reason.
An assistant that understands my preferences incorrectly may make poor decisions very efficiently.
And an assistant that cannot distinguish between something I explicitly authorized and something it inferred about me may gradually begin making decisions I never intended to delegate.
There's another aspect of this relationship that I find particularly interesting, and it has less to do with what artificial intelligence can do than with how we might change as we become accustomed to using it.
I think about the navigation system in my car. I remember when traveling somewhere unfamiliar meant studying a map, writing down directions, or stopping to ask someone for help. You paid attention to landmarks, learned the roads, and gradually developed an understanding of how different places connected to one another.
Today, I enter a destination and follow the instructions. Most of the time, I arrive without difficulty, but I sometimes realize afterward that I couldn't explain the route I just traveled. The technology has made getting somewhere considerably easier, yet I've learned very little about the journey.
I don't consider that a particularly serious problem. I'm happy to let the navigation system handle most of the work, and I certainly don't miss unfolding road maps across the steering wheel.
But I wonder what happens when we extend that same relationship to other parts of our lives.
Imagine an AI assistant that gradually takes responsibility for managing your finances. At first, it helps you organize transactions and understand where your money is going. Eventually, it begins recommending changes, identifying opportunities, and handling routine decisions. Over time, you become comfortable enough with its performance that you rarely question what it's doing.
Perhaps your financial circumstances improve. The assistant might be considerably better at identifying opportunities than you are, and the time you save allows you to focus on things you find more meaningful.
That sounds like a successful relationship with technology.
But suppose your circumstances change unexpectedly. You decide to start a business, support a family member, or make a major investment. The priorities that guided your assistant's earlier decisions may no longer reflect what matters most to you.
If you've spent years allowing the system to manage your finances without understanding its reasoning, you might find yourself struggling to evaluate the choices it presents.
The problem wouldn't necessarily be that the assistant had done anything wrong. It might have performed exactly as expected. But somewhere along the way, you'd become increasingly dependent on decisions you no longer understood.
I suspect this will be one of the more subtle challenges of living with artificial intelligence. The better these systems become, the less frequently we'll have a reason to question them. And the less frequently we question them, the easier it may become to lose sight of the assumptions guiding their decisions.
Of course, this isn't an entirely new problem. We already depend on professionals whose expertise exceeds our own. I don't need to understand every aspect of medicine to benefit from a physician's advice, and I don't need to become an accountant to manage my financial affairs responsibly.
But those relationships have taught us something about the importance of communication. A good physician doesn't simply prescribe treatment without helping the patient understand what's happening. A good financial advisor takes the time to understand what a client hopes to accomplish rather than assuming that maximizing returns is always the most important objective.
We value those professionals not simply because they possess knowledge we lack, but because they help us apply that knowledge to our own circumstances.
I hope we develop similar expectations for artificial intelligence.
I'd like an assistant that understands my preferences well enough to handle routine matters without requiring constant supervision. But when something unusual happens, I want it to recognize that the situation deserves a conversation.
Perhaps it notices that a financial decision conflicts with a goal I've discussed previously. Rather than automatically proceeding, it explains what has changed and asks whether my priorities are still the same.
That interaction might take only a few moments, but it would preserve something important. I would remain an active participant in decisions that shape my life, even when I wasn't personally performing the work.
And I think that's the relationship we should be trying to create.
For most of computing history, we've measured progress by how much easier technology makes our lives. Artificial intelligence promises to take that progress considerably further, perhaps relieving us of responsibilities we've always assumed would require our personal attention.
I'm excited about that possibility. There are plenty of things I'd gladly delegate to a capable assistant, especially if doing so gave me more time for family, relationships, and the work I find meaningful.
But I don't want convenience to become a substitute for understanding.
I want an assistant that knows enough about me to recognize when a decision deserves my attention, even if it could make that decision without me.
Because as artificial intelligence becomes increasingly capable, I suspect the most meaningful distinction won't be between people who use AI and people who don't.
It will be between people who use AI to expand their freedom and people who gradually allow convenience to make their choices for them.
Continue reading: When Artificial Intelligence Gets It Wrong