The Danger of Treating AI as if It Thinks Like Us

Artificial intelligence has become remarkably good at producing the appearance of intelligence. A chatbot can explain a legal principle, summarise a medical paper and offer what sounds like a reasoned answer within seconds. Because the language is fluent and the response feels considered, it is easy to assume that the system has reached its conclusion in something resembling the way a person would.

That assumption carries real risk.

Researchers Teppo Felin and Matthias Holweg argue that much of the discussion around artificial intelligence rests on an overly simple comparison between human minds and computers. Modern AI systems learn statistical relationships from enormous quantities of existing data and use those relationships to estimate likely outcomes or generate plausible responses. Human reasoning can take a different route. People form theories about why things happen, imagine situations that have never existed and search for evidence that might support or challenge an idea.

When prediction looks like reasoning

Large language models are particularly convincing because they operate through language, which is also how people explain ideas and demonstrate expertise. A polished answer can therefore feel more intelligent than a numerical prediction produced by conventional software.

Fluency creates its own authority. When an answer reads well, people may be less inclined to question how it was produced. A confident explanation can appear reliable even when parts of it are inaccurate or based on weak assumptions. Someone with expertise may spot a problem quickly, while a person using the system because they lack that expertise has far less protection.

The consequences vary according to context. A wrong answer to a low-stakes question may mean inconvenience; an inaccurate refund decision can create a compliance problem, while a misleading interpretation in healthcare, finance or law can carry much greater consequences.

The underlying issue is that a system can identify statistical relationships without understanding the causal mechanism behind them. If two factors repeatedly appear together in historical data, an AI system may become highly effective at recognising that relationship. It may still have limited ability to determine why the relationship exists or whether changing circumstances have made the pattern unreliable. Because the response can still sound polished, that limitation is easy to miss.

Automation amplifies the consequences. An error embedded in an automated workflow can be repeated at extraordinary speed, reaching thousands of customers before anyone realises that the underlying logic has failed.

The problem with looking backwards

Felin and Holweg place particular emphasis on AI's dependence on existing data. Modern systems can identify relationships across volumes of information no individual could reasonably process, making them useful where historical patterns remain informative. Retailers forecast demand from previous sales, service teams identify recurring reasons customers make contact, and financial institutions analyse past transactions for signs of fraud.

Some of the hardest decisions organisations face occur when there is little useful precedent.

A new technology can create behaviour that has never previously been measured. An unfamiliar competitor can enter a market with a different business model. Customers may respond to a product in a way that contradicts earlier forecasts. Managers often have to make decisions in these situations, where the available evidence is incomplete and the future cannot be cleanly extrapolated from the past.

Human reasoning has an important role here because people can develop an explanation before all the evidence exists and then look for information that might test it. A scientist may propose a mechanism that has never been directly observed. An entrepreneur may believe customers will value a product category that has never existed. A manager can notice an unexpected change and decide that the established explanation no longer makes sense.

These ideas can be wrong. Their value comes from the ability to imagine another possibility and then create new evidence around it.

Generative AI can contribute to that process. It can locate information, suggest possible explanations and help people examine an idea from several directions. Problems arise when its output gains so much authority that organisations stop searching beyond the patterns already represented in the system. A business could become highly efficient at answering familiar questions while becoming less capable of recognising when the question itself needs to change.

Automation can weaken the people using it

Another danger develops gradually. As AI becomes more accurate, employees may begin handing over parts of their own judgment to systems that are usually right.

A system that performs well day after day changes behaviour. Checking begins to feel unnecessary, particularly in workplaces where speed and productivity are closely measured. Eventually, the generated answer can become the default.

Customer service provides a useful example. An agent using AI-generated response suggestions can move through straightforward enquiries quickly and spend less time searching a knowledge base or rewriting familiar explanations.

The longer-term effect depends on how much thinking remains with the employee. If agents repeatedly receive ready-made answers, they may have fewer opportunities to interpret policies, investigate unusual cases or develop the knowledge needed to recognise when a recommendation is wrong. New employees may become especially dependent on the system because they never experience a version of the job where that knowledge had to be built through repeated problem-solving.

Eventually, the organisation may find that the people expected to supervise the technology have less practical expertise than the employees who performed the same work before automation.

The same issue extends beyond customer service. Lawyers using automated research, analysts relying on generated summaries, clinicians working with decision-support systems and managers reviewing AI recommendations can remain responsible for the final decision while doing progressively less of the reasoning that supports it. Human oversight only works when the person reviewing the output retains enough knowledge and confidence to challenge what the system produces.

A different kind of AI risk

Public discussion about AI risk often focuses on dramatic possibilities such as widespread job displacement, autonomous systems behaving unpredictably or machines eventually exceeding human intelligence across many tasks. Those questions deserve attention, although they can make the more ordinary risks developing inside organisations appear less significant.

Some of those risks are already present.

AI systems increasingly sit between employees and the information they use to make decisions. They summarise documents before anyone reads them, recommend responses before an employee has formed an opinion and influence which information receives attention. Collectively, those interventions can shape how an organisation understands its customers and operating environment.

That influence becomes more consequential when people stop asking how an answer was produced. A statistically plausible response can gradually acquire the same authority as a conclusion supported by evidence, professional experience and an understanding of cause and effect.

Organisations therefore need more than a final approval step labelled human review. Employees need enough expertise to recognise an unusual result, enough freedom to investigate beyond the system and enough support to challenge an automated recommendation.

AI can still transform knowledge work. It can reduce repetitive effort, identify useful patterns and make large quantities of information easier to navigate. Those benefits are more sustainable when organisations remain clear about the system's strengths and when its conclusions deserve closer scrutiny.

As the technology improves, maintaining that discipline may become harder. Better systems will produce fewer obvious errors and more persuasive explanations. Strong performance naturally encourages trust, and trust encourages delegation.

The danger grows when that delegation moves faster than our understanding of how the system reaches its answers.

Source

Felin, T. & Holweg, M. (2024). Theory Is All You Need: AI, Human Cognition, and Causal Reasoning. Strategy Science, 9(4).