Artificial intelligence is already capable of continuously monitoring infrastructure, detecting anomalies, and predicting deviations. But the more decisions are delegated to such systems, the more important another question becomes: what should be done if a model is confident in its prediction but wrong? It is precisely the problem of trust in AI that Francesco Lamonaca, a professor at the University of Calabria, described as one of the main challenges for critical infrastructure monitoring systems. 

Speaking at a conference in Yerevan, he noted that the main problem of modern artificial intelligence-based monitoring systems is no longer whether they are capable of detecting an anomaly. What is far more important is understanding how much their conclusions can be trusted when a real decision depends on them.

One number is not enough

According to him, three important changes have taken place recently. First, monitoring is gradually shifting from periodic observation of individual objects to continuous data collection through distributed sensor networks. Second, digital systems now preserve the history of conditions and previous interventions, rather than merely recording a single moment in time. This makes it possible to use accumulated data to predict deviations.

As a result, systems are able to identify unusual behavior even before it becomes clear what exactly has happened. However, this is precisely where a new problem arises, Lamonaca believes. 

“The limitation is no longer what we can measure. The limitation is what we can trust enough to act upon,” the professor noted.

Lamonaca drew attention to a fundamental point: in critical infrastructure management systems, the operator sees many indicators on the screen, but a number by itself is not yet a full-fledged measurement. According to him, each indicator should be accompanied by information about its uncertainty — how much the obtained value may vary and how reliable the result is.

This is especially important in situations when an automated system issues an alarm and the operator’s decision depends on that signal. 

“For it to be a measurement, the number must be accompanied by uncertainty — another number that shows how much that value can vary,” he explained.

In Lamonaca’s view, if a system demonstrates impressive results but the operator does not understand the limits of their reliability, the technology itself does not benefit those who must make decisions in the real world.

AI can be confidently wrong

Another problem is related to predictive models. A system trained on historical data may assess a situation with considerable confidence if it resembles cases it has already encountered. But if it faces a situation it has never seen before, the model itself may not understand the limits of its own experience. That is why, the professor believes, a model’s confidence cannot automatically be equated with its reliability.

Lamonaca emphasized that  artificial intelligence that is confidently wrong poses a serious problem for trust in the system. At the same time, he added that none of this is an argument against the use of technology. On the contrary, systems must be designed with uncertainty and the human factor in mind.

According to Lamonaca, the resilience of critical infrastructure depends not only on the capabilities of sensors, models, and automatic alert systems. Ultimately, it is a human who makes the decision — and that person must understand what exactly the received signal means.

“We need trust beforehand, not afterward,” the professor emphasized.

According to him, model reliability should be part of the methodology itself, not an administrative add-on to an already created system. The operator must be able to compare the data, and for that, information about their uncertainty must be provided together with the data themselves.

Thus, the question that must be asked when developing AI systems for critical infrastructure is not simply, “What can the system detect?” It is far more important to understand: what exactly must it provide to the operator before the operator decides to act on the basis of its signal? It is the problem of trust in such systems, Lamonaca believes, that is becoming one of the key issues of the next stage in the development of AI in critical infrastructure.