AI’s Biggest Risk Isn’t Rogue Systems; It’s Humans Who Can No Longer Think Without Them

September 28, 2026:

AI’s Biggest Risk Isn’t Rogue Systems; It’s Humans Who Can No Longer Think Without Them

Most conversations about the risks of artificial intelligence eventually arrive at the same question: “Will AI take our jobs?”

It is an understandable concern. And on Wednesday, Sam Altman brought a different version of it to the United Nations Security Council, warning world leaders that humanity could “lose control of the future to AI” if systems grow too fast for governments to follow.

But there is a risk already unfolding that is quieter, more immediate, and happening inside organizations right now. It is not that AI will become too capable. It is that humans will become too dependent on it.

Cornelia Walther, a visiting scholar at the Wharton School, published research this month giving that risk a name: agency decay. She defines it as the gradual erosion of a person’s ability and willingness to observe carefully, think independently, choose deliberately, and act responsibly — not because AI is harmful, but because convenience has become the default setting for cognition. “A tool that first helps us think,” she writes, “can begin to think around us.”

The research backing that diagnosis has been building for months. An MIT Media Lab study found that participants who wrote essays using ChatGPT showed measurably weaker neural connectivity than those who wrote unaided — and 83% could not quote a single sentence from text they had just produced. A randomized controlled trial involving more than 1,200 participants, conducted by researchers at Carnegie Mellon, Oxford, MIT, and UCLA, found that even brief AI assistance reduced independent problem-solving ability once the tool was removed — after as little as ten minutes of use. A BCG global survey of 70 C-suite leaders found that half are already observing skill erosion in their organizations, and more than 60% expect it to pose a material threat within three to five years.

What Does AI Overreliance Actually Cost Workers?

Over the past several years, organizations have adopted AI at dramatically different speeds. Some companies are barely experimenting with ChatGPT. Others are building AI deeply into their operations.

The biggest difference is not access to technology. Almost everyone has access to essentially the same models. The difference is whether people understand how to direct those models.

That distinction matters.

There is a massive difference between using AI to replace thinking and using AI to expand your capacity for it. Imagine two executives: the first asks AI what the company should do, and gradually outsources more analysis, more judgment, more writing, and more decision-making to the machine until AI becomes the source of the thinking. The second develops a clear point of view first, gives the AI context, challenges its assumptions, asks it to explore alternatives, tests its reasoning, and then uses the output to make a better decision.

Both executives are using AI, but only one of them is becoming more capable.

The research literature has a name for what is happening to the first executive: cognitive offloading. It is not new — humans have extended their thinking through external tools since the invention of writing. What makes generative AI categorically different is the nature of the work it substitutes for. Unlike calculators, which handle computation, or search engines, which assist retrieval, large language models perform integrative reasoning — the fundamental cognitive process through which expertise is built. When that process is routinely outsourced, the neural pathways supporting independent judgment begin to atrophy.

Why AI Proficiency Has Never Mattered More

AI proficiency is becoming one of the most important professional skills of the next decade. Not knowing how to write a clever prompt — that is table stakes. The real skill is knowing how to combine human judgment with machine intelligence: providing intent, context, and evaluation while AI handles execution.

When that relationship works well, something interesting happens: people do not become less important. They become dramatically more powerful. A person without formal engineering training can now build software tools that would once have required assembling a technical team. Things that once would have required assembling a technical team can increasingly be explored by one person with a clear idea and the ability to direct AI effectively.

That does not mean engineering expertise has become worthless. It just means that the bottleneck is changing. For decades, execution capability was the barrier because of its associated cost. Anyone who wanted to build software, produce sophisticated analysis, create marketing assets, or automate business processes needed specialized people with specialized skills. AI is rapidly reducing many of those execution barriers.

As that happens, judgment becomes more valuable. Knowing what to build, what question to ask, which information matters, and whether an answer is actually good becomes increasingly important. And that only comes from real, human experience.

PwC’s 2026 Global AI Jobs Barometer, which analyzed more than one billion job postings across six continents, confirmed the labor market shift: the average wage premium for workers with AI skills has reached 62%. More telling is what kind of AI work commands those premiums. Roles where AI automates routine tasks while workers focus on judgment and decision-making are growing at 39%, compared to 17% for roles where AI simply makes tasks easier for non-experts. AI-exposed entry-level positions are now seven times more likely to require traditionally senior-level skills like leadership and judgment than they were a few years ago.

That is where organizations should focus their attention.

AI Deskilling: The Organizational Risk Hiding in Plain Sight

Companies are racing to deploy AI tools, agents, copilots, and automation. But giving employees access to increasingly powerful systems without increasing their ability to direct those systems creates a dangerous imbalance.

The technology gets smarter. The humans do not necessarily become better at using it.

BCG researchers coined the term “distributed de-skilling” to describe what happens when cognitive erosion occurs not in one employee but simultaneously across hundreds or thousands: the organization’s collective ability to function without AI scaffolding quietly disappears. The human fallback that seemed credible at deployment gradually ceases to exist.

The 2026 International AI Safety Report, compiled by researchers across 30 countries, cited one striking example: clinicians’ ability to detect tumors without AI assistance dropped by 6% within just three months of AI support being introduced. The standard deployment model — AI handles routine cases, humans handle exceptions — creates an ironic trap. As Mohammad Jarrahi, an information scientist at the University of North Carolina, put it: organizations end up giving the most cognitively demanding work to employees whose judgment has had fewer opportunities to develop, because AI has been handling the decisions that used to build it.

The organizations that benefit most from AI will not simply be the ones that automate the largest percentage of their workforce. They will be the ones that increase the capability of the people who remain.

The goal should not be to remove humans from every process. The goal should be to remove unnecessary friction between human intention and execution. That requires a different mindset.

How to Use AI Without Losing the Capacity to Think Without It

Instead of asking “What can AI do for us?” the better question is: “What are we capable of doing now that AI exists?” Those questions sound similar, but they lead to very different futures. The first encourages delegation. The second encourages capability.

A Microsoft and Carnegie Mellon University study of 319 knowledge workers found that higher confidence in AI capabilities correlated directly with reduced perceived effort for critical thinking. The workers who trusted the machine most were investing the least mental effort in evaluating its outputs. The researchers identified what they called the key irony of automation: by mechanizing routine tasks and leaving exceptions to the human, organizations deprive workers of the routine decisions that build judgment — leaving them unprepared precisely when a high-stakes exception arrives.

BCG’s research also found that only 5% of companies are generating substantial value from AI, and 70% of what separates those firms from the rest comes from investing in people, not technology. Future-oriented companies upskill more than 50% of their workforce on AI, compared with 20% at lagging firms.

AI is going to become extraordinarily powerful. The models will improve. Agents will become more autonomous. Software will increasingly be able to execute complex work with very little intervention. That makes human agency more important, not less.

Altman told the Security Council that AI should be built for people, “not simply to turn the crank of the machines faster.” Walther’s research suggests the more immediate threat is that people, given the chance to let machines do the thinking, will take it — gradually, willingly, and at measurable cost to their own capacity.

The greatest promise of artificial intelligence is not a world in which humans have less to do. It is a world in which humans are capable of doing far more.


Frequently Asked Questions

What is agency decay from AI use?

Wharton visiting scholar Cornelia Walther defines agency decay as the gradual erosion of a person’s ability and willingness to observe carefully, think independently, choose deliberately, and act responsibly — caused not by AI being harmful, but by convenience becoming the default setting for cognition. MIT’s related concept, “cognitive debt,” describes the measurable neurological cost: participants who wrote essays using ChatGPT showed weaker brain connectivity and were far less able to recall their own work than those who wrote without AI assistance.

How does AI overreliance affect critical thinking at work?

Multiple studies point to the same pattern. A study of 666 participants published in the journal Societies found a strong negative correlation (r = –0.68) between AI tool usage and critical thinking scores. A Microsoft and Carnegie Mellon study of 319 knowledge workers found that the more workers trusted AI outputs, the less mental effort they invested in evaluating them — making them the most likely to miss errors when it mattered most.

What can organizations do to prevent AI deskilling?

BCG researchers recommend treating AI deployment and skill preservation as two separate obligations. Practical steps include creating workflows that require workers to reason through a problem before consulting AI, documenting independent judgments before AI is used, and measuring cognitive performance without AI assistance as a distinct competency. Gartner predicts that by 2026, 50% of global organizations will require AI-free skills assessments to address this gap.

Does using AI regularly make you less capable over time?

The evidence suggests it can, particularly when AI handles integrative reasoning rather than just computation or retrieval. The large-scale Carnegie Mellon, Oxford, MIT, and UCLA randomized trial found that AI assistance impaired independent performance and reduced persistence once the tool was removed — after as little as ten minutes of use. The critical distinction researchers draw: using AI as a sparring partner that stress-tests your own thinking is cognitively protective. Using it as an answer machine is not.

Source link