Developers of physical artificial intelligence systems are looking for new ways to train robots, and one of the most unusual directions is the use of human brain activity. According to TechCrunch, the American company Encord, together with the German startup Zander Labs, is testing technology that records the brain waves of operators while they perform various physical tasks.

According to the developers, the main bottleneck in the development of humanoid robots today is no longer the architecture of neural networks, but the lack of high-quality training data. Encord believes that such data simply does not exist in sufficient volume, so it has to be created almost from scratch.

How the experiment works

The trials are taking place at Encord’s warehouse in San Leandro, California. During a demonstration, an operator wearing a headset equipped with a camera and sensors that capture the brain's electrical activity carefully disassembles a tower of wooden Jenga blocks. In addition to the first-person video feed, the system receives information about how the person reacts to what is happening—for example, when encountering an error, an unexpected situation, or making a decision.

The developers believe that such data will help robots better understand which actions require maximum concentration and more complex computations. For now, the project is in the pilot testing phase. Encord intends to create the first dataset tagged with brain activity labels, test it on clients' models, and only then decide whether to scale the technology.

Robots need real-world actions, not the internet

Unlike large language models, which were trained on vast amounts of text from the internet, robots require real-world examples of interaction with the physical world. That is why companies are increasingly organizing their own data collection efforts.

Today, the main sources of such information are so-called egocentric video recordings—first-person videos shot by people while performing various tasks—as well as recordings of remote robot control.

From pouring coffee to connecting servers

At Encord's facility, operators create training datasets by performing a wide variety of actions: pouring coffee, stacking poker chips, plugging Ethernet cables into servers, and working with household items, tools, and packaging. All of these operations are designed to train future humanoid robots to perform precise manipulations.

In addition to recording brain activity, the company is testing another technology—forearm-mounted sensors that read electrical muscle signals. This is expected to allow for more accurate 3D reconstruction of hand and finger positions, as standard video does not always capture hand movements completely.

Why physical AI is significantly more expensive

All collected data is meticulously annotated. Each video clip is accompanied by a description of the action being performed—for example, "right hand tightens a bolt." According to Encord, such detailed labeling makes these datasets about a hundred times more valuable for task-specific training compared to standard video recordings, even though creating them costs roughly twenty times as much.

It is precisely this high cost of physical data production that represents one of the main differences between the development of robotics and generative artificial intelligence. While language models could be trained virtually for free on open internet data, information for physical AI must be custom-made, which fundamentally changes the economics of developing such systems.