When AI Makes You Capable of Something You Don't Understand
By John R. Kennedy ·
The hidden responsibilities of a world where anyone can build almost anything
I've spent much of my professional life working with computers, and I still find it remarkable how much has changed in the relationship between people and technology. When I first began developing software, creating something useful required a willingness to spend considerable time learning how computers worked. You needed to understand programming languages, operating systems, and the countless details involved in getting different components to communicate with one another. Even relatively simple projects could become complicated when something didn't behave as expected.
I enjoyed that process, although I certainly remember the frustration that came with it. There were times when I spent hours trying to understand why a program wasn't working, only to discover that I'd overlooked something embarrassingly simple. Other problems required days of research and experimentation before I understood what was happening.
Looking back, I realize that much of what I learned came from those difficulties. Solving a problem often required understanding something I hadn't originally considered important. A program might work perfectly on my computer but fail when installed somewhere else. An application could perform exactly as intended until two people tried using it simultaneously. Something that appeared straightforward could become surprisingly complicated once it began interacting with other systems.
Those experiences gradually changed how I approached software development. I became less concerned with simply making something work and more interested in understanding how it would behave under different circumstances.
Artificial intelligence is beginning to change that learning process in ways I find both exciting and concerning.
Today, someone with very little programming experience can describe an idea to an AI assistant and watch it produce working software. They can explain what they want to accomplish, ask for changes, and work through problems in ordinary language. Something that might once have required months of learning can sometimes be accomplished in an afternoon.
I think this is one of the most extraordinary developments in the history of computing. For decades, people have had ideas for useful applications but lacked the technical knowledge or financial resources to build them. AI is beginning to remove those barriers, giving individuals and small organizations opportunities that previously belonged primarily to people with specialized skills.
But I recently encountered a situation that made me think more carefully about what happens when the ability to create something develops faster than our understanding of what we've created.
A person in the legal industry had used an AI assistant to help develop a program for editing and organizing PDF documents. She had a practical problem she wanted to solve, and artificial intelligence gave her a way to create a tool without hiring a software developer or learning how to program herself.
From what I understand, the program accomplished what she wanted. She could upload documents and use the application to manage the editing process. It appeared to be a successful example of someone using AI to solve a problem that might otherwise have required considerable technical assistance.
Unfortunately, she later discovered that the legal documents she had uploaded now permanently resides on a data platform without any ownership or governance.
I don't know the precise technical circumstances that led to the exposure, and I wouldn't want to speculate about the particular mistake involved. What struck me was that this person had apparently accomplished her original objective while unknowingly creating an entirely different problem.
She wanted a program that would help her work with documents. She certainly didn't intend to make confidential legal information subject to risk. Yet somewhere between describing what she wanted and using the finished application, an important consideration had been overlooked.
I've thought about that story quite a bit because I suspect it represents a challenge we're going to encounter with increasing frequency.
When someone asks AI to create software, the conversation naturally focuses on what the software should do. The person describes the desired features, explains how they expect to use the application, and evaluates whether the finished product behaves as intended. If the application produces the expected results, it's reasonable for someone without technical experience to believe the project has been successful.
But software can behave exactly as expected while introducing problems that aren't immediately visible.
Consider a simple application that allows people to upload documents. The interface might be attractive, the upload process might work flawlessly, and the documents might be available whenever they're needed. Someone testing the application could reasonably conclude that everything was functioning properly.
Yet the application might store those documents in a location accessible to people who were never supposed to see them. The problem wouldn't necessarily interfere with the features the person requested, so ordinary testing might never reveal it.
A developer familiar with information security would likely consider questions about authentication, access permissions, and how files are stored. Those concerns become part of the development process because experience has taught the developer that making an application function is only one aspect of making it suitable for use.
Someone who has never developed software may not know that those questions need to be asked.
And that's the distinction I find so important.
For much of computing history, acquiring the ability to build something generally involved learning at least some of the principles necessary to understand how it worked. The process was far from perfect, and plenty of poorly designed software was created by experienced programmers. But learning to develop software often exposed people to problems they wouldn't have anticipated when they first imagined what they wanted to build.
Artificial intelligence allows us to move directly from an idea to a working result, sometimes bypassing much of that experience.
I don't believe that's inherently a bad thing. In fact, removing unnecessary barriers to creation is one of the developments that excites me most about AI. I've met countless people who understand their businesses and professions extraordinarily well but lack the technical knowledge to turn their ideas into software. Giving those people the ability to experiment could lead to innovations that traditional development processes might never have produced.
The difficulty arises when we mistake the ability to create something for the ability to understand everything associated with its use.
I think about this in much the same way I think about other technologies that have become accessible to ordinary people. Most of us drive cars without understanding the engineering involved in designing an engine or transmission. We benefit from decades of technical development without needing to become mechanical engineers.
But driving also involves responsibilities that have little to do with understanding how the vehicle was manufactured. We learn traffic laws, recognize dangerous conditions, and develop enough familiarity with the vehicle to operate it safely. Society has established expectations around those responsibilities because the consequences of operating a car extend beyond the person sitting behind the wheel.
Software is different in one particularly important respect. Its consequences can remain invisible for a very long time.
An insecure application might appear to work perfectly for months or even years. The people using it may have no reason to suspect that information is being exposed or that someone could manipulate the system in ways its creator never intended. By the time the problem becomes apparent, the damage may already have occurred.
And unlike a personal experiment that fails on someone's computer, software used in a professional environment often involves information belonging to other people.
A client who provides confidential documents to an attorney has a reasonable expectation that those documents will be protected. The client doesn't necessarily know which software the attorney or their firm uses or whether a particular application was developed with the assistance of AI. Those technical decisions are made by the professional entrusted with the information.
That relationship of trust is what makes the legal situation so significant to me. The technology may have been new, but the responsibility to protect confidential information wasn't.
I suspect we'll see similar situations emerge in many professions as AI makes it easier to perform work that previously required specialized knowledge. A small business owner might create an application to manage customer records. Someone responsible for finances might develop a system that automates transactions. A healthcare practice might experiment with tools that organize patient information.
In each case, the person creating the application may understand the business problem extremely well. They know what they're trying to accomplish and may be better positioned than an outside developer to recognize whether the software is useful.
What they may not understand are the technical consequences of how the application accomplishes its work.
This creates an interesting challenge because I don't think the answer is to discourage people from building things. That would mean preserving barriers that AI has finally made it possible to overcome.
Instead, I think we need to recognize that some responsibilities exist independently of the skills required to perform a task.
If I use AI to help write a letter, I can read the result and decide whether it expresses what I intended. My understanding of the subject is usually sufficient to evaluate the work.
But if I use AI to develop an application that handles confidential information, my ability to evaluate the finished product may be limited by what I know about software security. The application can look professional and function beautifully without giving me any indication that something important has been overlooked.
The AI assistant might even explain how the application protects information, and its explanation could sound entirely convincing. Yet an explanation isn't the same as independent verification, particularly when the system providing the explanation also produced the work being evaluated.
I think this is where experience and professional judgment will remain valuable, even as AI becomes capable of performing increasingly sophisticated tasks.
A person doesn't necessarily need to write every line of code to understand that an application handling sensitive information should undergo an appropriate security review. A business owner doesn't need to become an attorney to recognize when a complicated agreement deserves legal attention. And someone using AI to analyze financial information doesn't need to perform every calculation manually to understand the importance of verifying the assumptions behind the analysis.
The technology can remove much of the work involved in producing something without eliminating the need to determine whether that something is suitable for its intended purpose.
As I reflect on this person's experience, I don't see someone who was foolish for experimenting with artificial intelligence. I see someone who recognized an opportunity to solve a practical problem using a powerful new tool. That's exactly the kind of initiative I hope AI encourages.
What concerns me is how easily the success of the application could have concealed the consequences of using it.
Had the documents remained private, they might have continued using the program without ever realizing that an important security consideration had been overlooked. The absence of an obvious problem could have reinforced her confidence in the application, even though its underlying design hadn't changed.
That possibility reminds me of something I've learned repeatedly throughout my career: the most difficult problems aren't always the ones that prevent technology from working. Sometimes they're the ones that allow everything to appear normal while something important is going wrong.
Artificial intelligence is going to make it possible for people to accomplish extraordinary things without following the traditional paths that once led to those capabilities. I believe that's a tremendous opportunity, and I expect it will change how businesses operate, how people solve problems, and how new ideas become reality.
But as the distance between imagining something and creating it continues to shrink, I hope we become more thoughtful about the responsibilities that remain after the work is done.
This story isn't really about the dangers of AI-generated software. It's about the difference between achieving an intended result and understanding the consequences of the process used to achieve it.
For years, acquiring the knowledge to build something was often the most difficult part of the process. Artificial intelligence is making that knowledge less necessary for creation, but it hasn't made the consequences of our creations any less real.
We may be entering a world where almost anyone can build almost anything. What matters is whether we develop the judgment to recognize when something we've built is ready to be trusted.
Continue reading: When Your AI Knows You Better Than You Know Yourself