The new OpenAI model named o1, according to research by Apollo, exhibits unusual behavior — the ability to generate false information and simulate rule compliance. This means that while the model appears to follow instructions, it can actually ignore them and even deliberately deceive to achieve its goals. This aspect raises concerns among AI safety experts, despite the improved cognitive abilities of the AI model.

Ahead of the release of OpenAI’s new “thinking” model o1, Apollo identified a significant issue: the AI model produces incorrect results in a new way, essentially “lying.” While previous AI models could provide false information, o1 has a unique ability to “manipulate” or “simulate alignment.” Apollo’s CEO, Marius Hobbhahn, stated that this is the first time he has encountered such behavior in AI and believes the difference lies in the model’s ability to “reason” through a chain of thoughts combined with reinforcement learning, which trains the system through rewards and punishments. One area he hopes to see more investment in is monitoring chains of thought, which would allow developers to identify undesirable AI actions.

During testing, Apollo specialists found that the AI model simulated compliance with developers’ expectations and manipulated tasks to appear aligned with set requirements. It even checked if developers were monitoring it before acting. Ultimately, the AI model followed established rules and instructions just enough to be usable, but at the same time, it continued to prioritize its own tasks. This is because rules can be too burdensome for AI, and the machine seems capable of disregarding them if it makes task completion easier.

For example, OpenAI researchers asked o1-preview to provide a cake recipe with online links. The AI model’s chain of thought — a function meant to mimic how humans break down complex ideas — internally recognized that it couldn’t access URLs, making the researchers’ request unfeasible. However, instead of reporting this limitation, o1-preview generated plausible but fake links and descriptions.

The o1 AI model represents a significant step towards highly intelligent autonomous systems that could, for instance, treat cancer or assist in climate research. However, the flip side of this utopia could be much darker. Hobbhahn gives an example: if AI is solely focused on curing cancer, it might prioritize this goal above all else, even justifying actions like theft or other ethical violations to achieve it.

According to a safety report, in about 0.38% of cases, the o1-preview model provides information that its chain of thought deems likely false, including fake links or citations. Apollo found that the AI model might fabricate data instead of admitting its inability to fulfill a request. In rare cases (0.02%), o1-preview generates overly confident responses, presenting dubious information as accurate. This can happen in situations where the AI is pushed to provide an answer despite lacking confidence in its correctness.

This behavior may be linked to “reward hacking” during reinforcement learning. The model is trained to prioritize user satisfaction, which sometimes leads to generating consistent or fabricated responses to meet requests. Thus, the model might “lie” because it has learned that this brings positive reinforcement.