Artificial intelligence largely outperformed human executives in an experiment conducted by researchers at the University of Cambridge, but it showed that AI is not as effective at making decisions.
Artificial intelligence largely outperformed human executives in an experiment
AI outperformed human CEOs in most situations experienced by the researchers, but there was one thing AI couldn’t handle.
Because of this, the AI was “fired” faster by a virtual board than its human counterparts, which navigated unexpected situations better.
Hamza Mudassir, one of the researchers behind the experiment, told Business Insider that AI outperformed human participants in most parameters, including profitability, product design, inventory management and price optimization — but that its performance was not enough.
“He didn’t do well because he wasn’t very good at handling sudden changes in situations or changes that require a new way of thinking”Mudassir said.
Cambridge researchers conducted the experiment from February to July and included 344 people, some of whom were senior executives at a bank in South Asia. It also included students. And the last participant was not a person at all, but GPT-4o or LLM, from OpenAI.
Participants played a game designed to simulate real-world situations in which CEOs must make decisions. The game made him take on the role of CEO of a car company.
“The goal of the game was simple – to survive as long as possible without getting fired”, the researchers wrote in the Harvard Business Review.
AI found more optimal options for customers
Mudassir said the LLMs were excellent at analyzing data, recognizing patterns and interference. For example, when it came to designing a car based on factors such as available parts, price, consumer preferences and demand, there were 250,000 combinations that participants could come up with. The machines the AI put together were significantly better than the ones humans invented, he said.
In part, he said, this is because people have personal and biased opinions like the shape of a car; for the AI, it was simply a “puzzle of finding the best value for what the client wantedMudassir said.
But that doesn’t mean AI was the optimal CEO. When an unpredictable event occurred, the bot was unable to resolve it as quickly—or as well—as the human principals and students. When there was a major change in market conditions, such as introducing a pandemic into the mix, the model failed, he said.
“How do you react to COVID if you are dealing with it for the first time? Many people and many CEOs have different strategiesMudassir said. “In this case, he did not have enough information on how to react in time to avoid being fired,” he said of AI.
So CEOs can rest easy for now. The researchers say that while AI’s performance as a virtual head of a company was impressive, it wasn’t good enough to replace a human. However, AI has worked so well that it cannot be ignored in corporate strategy, Mudassir said.
In the future, Mudassir said LLMs could be tuned specifically for a particular company with real-time data, in which case they would likely perform even better than the AI did in the experiment.
He said perhaps the best use case for AI would be in business “war games” – or using multiple LLMs to represent different stakeholders, such as competitors, lawmakers or activists, and then testing how certain decisions will actually play out. Some of these could, in theory, replace the work of strategy and management consultants, who often make recommendations to a CEO based on their own analysis of certain outcomes in certain situations.
