Training a neural network can be a bit like preparing a student for an exam. If the student memorizes every practice question and answer, they might score well on a familiar test, but struggle when the questions change. Real learning means understanding bigger ideas. A Queen’s research team says many artificial intelligence (AI) systems have fallen into the memorization trap, and they have developed a new training approach designed to help neural networks learn more effectively.

What is a neural network?

Neural networks are powerful AI systems inspired by how the brain works. They can handle complex tasks like making decisions, analyzing images, and understanding text. You already see them in things like movie recommendations, voice assistants, photo tagging, and automatic translation. Most neural networks are trained to do extremely well on a specific set of examples. If a model becomes too closely matched to that training data – a problem called overfitting – it can fail on new situations. That can cause real-world problems, such as unreliable threat detection or wrong medical diagnoses.

To counter this, the Queen's team introduced an alternative approach, recently published in Nature Communications, called ‘sufficient training’.

“We realized that the problem was optimization,” says co-lead author Irina Babayan, a master’s student in physics. “The leading approaches all start with some version of optimization but then use one trick or another to disrupt the optimizer. We thought: Why not start from something else?” 

Instead of trying to train a single, optimized method that makes as few errors as possible, they’ve opted for training that generates ‘good enough’ parameter sets, allowing for a variety of non-optimal solutions that can generalize better to new data.

“It sounds like a paradox,” adds co-lead author Hazhir Aliahmadi, a recent physics PhD recipient. “But sufficiently trained networks outperform optimally trained ones.” 

From memorization to learning

The approach hinges on the distinction the team drew between memorization and learning.

“When you’re learning something new, you might start by memorizing facts. But the goal of learning isn’t to memorize isolated facts. The goal is to get a broader understanding,” says Dr. Aliahmadi. “We wanted to develop a training approach that moves toward neural network representations of ‘learning’ at this higher level.” 

In practice, the team found their neural networks were better trained to capture the ‘gist’ of an underlying dataset, rather than reproducing measurement errors. That improved generalizability, which they expect will be useful for making decisions or predictions in areas such as health care, engineering, and finance.

“There’s a lot of work in the social sciences showing that diverse teams reach better outcomes,” says senior author Greg van Anders, an associate professor in the Department of Physics, Engineering Physics, and Astronomy. “This work shows that this intuition about people also holds for neural networks. We find that a diverse collection of neural networks substantially outperforms an individual network, or a non-diverse collection of networks.” 

By allowing for a range of solutions rather than forcing a single optimal one, sufficient training naturally creates that diversity, taking advantage of the properties of ‘emergence’, where a collective or whole is more than the sum of its parts. 

“Ultimately, we wanted to develop a tool that was useful for solving real-world problems,” says Babayan. 
That focus has shaped the areas where the team thinks the approach will have the greatest impact.

“People who need to diagnose rare diseases, detect fraud, or value financial instruments don’t have access to limitless data because data are prohibitively difficult or expensive to collect, or there are privacy limitations,” says Dr. van Anders. “Sufficient training is tailor made for settings like that.”