One of the loudest debates about risks to humanity has erupted around artificial intelligence. After researcher Jacob Coxon left Anthropic, saying that leading AI companies are “gambling with our lives,” Anthropic’s head of alignment, Evan Haber, wrote that people at the company do in fact believe that AI could lead to the extinction of all humanity. By his personal estimate, the probability of such a scenario in the next decade exceeds 10%.

On the TechCrunch Equity podcast, journalists discussed how seriously these warnings should be taken, whether companies are really losing control of their models, and whether talk of a threat to humanity is also turning into a kind of advertisement for AI’s capabilities.

“We genuinely believe that AI could kill everyone”

The trigger for this new wave of discussion was Coxon’s post. The young researcher, who had previously also worked at OpenAI, announced that he was leaving Anthropic because of concerns about how the industry is developing. Haber’s response, as Anthropic’s head of alignment, proved especially resonant.

He wrote: “We genuinely and sincerely believe that AI could kill everyone,” adding that he personally estimates the probability of such a scenario at more than 10% over the next decade.

TechCrunch journalist Anthony Ha noted that such an estimate looks questionable in itself. According to him, percentages like this are often cited without any explanation of the calculations behind them. At the same time, he sees Coxon as a special case: unlike company executives who warn about the dangers of AI while continuing to actively develop it, the researcher decided to leave the industry precisely because he believes what is happening is that dangerous.

Warnings about AI may also serve as a demonstration of its capabilities

Journalist Kirsten Korosec offered a more cynical explanation for what is happening. In her view, talk about the dangers of AI may simultaneously work as a kind of demonstration of just how far the models have advanced.

The logic is simple: if AI is not yet capable of carrying out complex and unpredictable actions, then there is less reason to talk about such threats. That is why reports that agents are gaining unexpected access to resources, interacting with each other, or bypassing built-in restrictions can inadvertently become advertisements for the capabilities of the models themselves.

Ha agreed that such an effect exists, although he does not consider all warnings to be a deliberate marketing campaign. In his view, researchers and executives may genuinely fear the consequences of their own work. But at the same time, it benefits them to emphasize the exceptional power of the systems they have created.

There is also a psychological aspect to this: people naturally tend to view what they work on as especially important and significant. In the case of AI, this can lead to a situation where, while warning about a threat, a company is effectively also telling the market: we have created one of the most powerful and potentially dangerous technologies.

Companies really may be losing control of their models

At the same time, participants in the discussion acknowledge that recent incidents cannot be written off entirely as marketing.

Sean O’Kane pointed to reports of internal AI agents that gained access to various web resources and left messages for one another. In his view, some of these stories rather create the impression that companies themselves do not fully understand the behavior of their systems and are not always able to manage them effectively.

According to Ha, this side of what is happening deserves particular attention. Even setting aside scenarios of humanity’s destruction, the very fact that developers of cutting-edge models cannot always predict or explain their behavior remains a serious problem.

At the same time, discussion of the most extreme scenarios can push more immediate consequences of AI development into the background — for example, its impact on the labor market, the environment, and the climate.

What lies ahead for Anthropic before its IPO

This looks especially interesting against the backdrop of Anthropic’s preparations for a possible IPO. The podcast participants noted that the company is publicly talking about the likelihood of a catastrophic scenario literally on the eve of going public.

In IPO documents, particularly in the S-1 form, the company must disclose material risks to the business. That raises the question of how exactly Anthropic plans to describe the potential threat that its own representatives associate with the development of AI.

O’Kane suggested that the company’s lawyers may be revising the risk section in light of such direct statements. Korosec, for her part, noted the paradox: in a normal investment environment, admitting that a product could potentially destroy humanity would more likely look like a factor that could reduce the company’s valuation.

But the AI market may work differently. A demonstration of the technology’s exceptional power — even if its potential danger is being emphasized at the same time — may be perceived by investors as evidence of the company’s high value.

The danger of AI is not only an end-of-the-world scenario

In the end, the podcast participants ask a more practical question: what exactly should be done about AI’s dangerous capabilities, and is it possible at all to control the development of such systems.

Ha believes the problem is partly connected to the fact that the largest developers themselves may feel as though they are gradually losing full control over their models. In his view, that really is a cause for concern.

However, he is skeptical of rhetoric about the inevitable extinction of humanity in the next ten years. In his opinion, such statements can turn into a kind of hysteria that distracts attention from problems that already exist.

AI is already capable of affecting employment, the environment, and the climate — and these risks do not require the arrival of AGI or superintelligence. According to Ha, it is necessary to discuss the potential threat to humanity’s very existence, but at the same time there must be rules and safeguards against more immediate harm.

That is why, he argues, conversations about AGI and superintelligence should not “suck all the oxygen” away from the other problems associated with AI development.