Editor’s Note: This work relates to Department of Navy award N000142212474 issued by the Office of Naval Research.
For years, the gold standard for training artificial intelligence has been to show it how to succeed. Whether it was Google DeepMind’s AlphaGo learning from millions of human professional moves or a robot being trained on motion-tracking data from humans in a maze, the assumption was that AI needs an expert model to mimic.
However, Yongcan Cao, a researcher at The University of Texas at San Antonio, is flipping that logic on its head. He is developing a new way for autonomous systems to learn by focusing not on what goes right, but on what goes wrong.
“Humans take risks and learn from failure,” said Cao, PhD, who holds the Mary Lou Clarke Endowed Distinguished Professorship in the Margie and Bill Klesse College of Engineering and Integrated Design. “Think about a baby learning to walk. They stand up, they fall down, and they learn from that fall. We are looking at how to give that same mechanism to AI.”
Solving the “sparse reward” problem
In the world of reinforcement learning — a type of machine learning where an AI learns through trial and error — researchers often face the “sparse reward” problem. In many complex tasks, an AI agent only receives a “reward” or feedback when it successfully completes a task. If the task is highly complex, the agent might wander aimlessly for millions of attempts without ever stumbling upon a success, meaning it receives no data to learn from. Success becomes a matter of chance, much like the “infinite monkey theorem,” where a monkey idly typing on a typewriter may eventually write something coherent, but it’s unlikely to happen any time soon.
To bridge this gap, engineers traditionally use “expert demonstrations,” where a human or a pre-programmed system shows the AI exactly how to perform the task. While effective, this data is expensive, difficult to collect and sometimes impossible to obtain for training in new or dangerous environments.
Cao’s solution, detailed in recent publications including an award-winning abstract for the 2025 International Conference on Autonomous Agents and Multiagent Systems (AAMAS), is a framework called On-Policy Reinforcement Learning from Failure, or “On-F.”
How On-F differs from standard algorithms
Standard reinforcement learning (RL) is akin to a student taking a final exam without ever receiving feedback on their homework throughout the course; they only find out if they passed or failed at the very end.
The On-F framework introduces a “discriminator,” which acts as a judge. Instead of waiting for a final success, the system constantly compares the AI’s current actions to a database of known failures — data that is “cheap and abundant,” according to Cao.
Through a process called “reward densification,” the AI receives constant, incremental feedback that encourages it to attempt the task in new ways. If its current path looks too much like a previous failure, the discriminator provides a “penalty.” This pushes the AI to try something different, effectively “densifying” the feedback loop so the agent is always learning, even when it hasn’t succeeded yet.
“If you imagine a drone flying a specific flight path and failing to locate a target, you don’t want the drone to retrace the same route or fly just a few feet to the left or right. You’d want the drone to try a significantly new approach, such as changing altitude or switching to a wide-angle view,” Cao explained.
When failure is systematic in this way, “failure alone can be used to learn desirable actions even if we don’t have an expert model,” Cao said. However, he also noted that when models are supplied with a mix of learning from failure and learning from demonstration, “we get even better outcomes.”
Competitive performance and real-world results
The findings suggest that this “fail-forward” approach is more than just a theoretical concept. In simulated environments where digital robots were tasked with learning to stand or walk, the AI models using the On-F framework performed as well as — and in some cases better than — models trained using expensive expert data.
The framework was validated using the Gymnasium simulation suite, specifically on tasks like the “PointMaze,” where an agent must navigate through a labyrinth and reach a target with very limited feedback.
The research is supported by a $502,051 grant from the Office of Naval Research (ONR) as part of a multi-year project aimed at making autonomous systems, such as drones, as fast and efficient at decision-making as humans.
“As humans, we don’t take information at face value; we think critically,” Cao said. “From a robotics side, if a drone is pursuing a target and one path is risky, the system needs to analyze that risk based on what it knows could go wrong.”
How On-F stands to shape industry
The ability to train AI using failure data could have massive implications across several industries. In manufacturing and robotics, it could drastically lower the cost of training new systems because engineers would no longer need to spend hundreds of hours creating “perfect” training scenarios.
In the realm of autonomous vehicles and drones, the technology could lead to more robust navigation systems that are better at avoiding obstacles by recognizing the “signature” of a potential collision before it happens.
Much like how AlphaGo began playing the ancient Chinese game of Go in ways humans had never imagined, Cao hopes that his AI models will also clear hurdles humans have yet to overcome — such as designing seemingly impossible surgeries or manufacturing first-of-its-kind nanotechnology.
“My goal is always to help get the word out and develop more actionable insights,” Cao said. “Personally, I feel this is a very helpful way to enable autonomous systems to learn as efficiently as humans do.”
As Cao’s work with the ONR continues through July 2026, his team at UT San Antonio is exploring next steps, including how to refine these “judging” systems to handle more subjective failures, potentially opening the door for AI that can assist in healthcare, logistics and complex disaster-response scenarios where there is no manual for success, only a history of mistakes to avoid.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Office of Naval Research.
