Researchers at Nvidia have developed a brand new method that flips the script on how giant language fashions (LLMs) study to motive.
The strategy, known as reinforcement studying pre-training (RLP), integrates RL into the preliminary coaching part slightly than saving it for the top.
This method encourages the mannequin to “assume for itself earlier than predicting what comes subsequent, thus instructing an unbiased considering conduct earlier within the pretraining,” the researchers state of their paper.
By studying to motive on plain textual content while not having exterior verifiers, fashions skilled with RLP present vital enhancements in studying advanced reasoning duties downstream, hinting at a way forward for extra succesful and adaptable AI for real-world duties.
The standard LLM coaching cycle
Sometimes, giant language fashions are first pre-trained on huge quantities of textual content utilizing a "next-token prediction" goal, the place they’re given a string of textual content and requested to repeatedly guess what the subsequent phrase (or token) will likely be. On this part, they study grammar, information, and primary associations.
Within the later post-training part, fashions often study advanced reasoning talents resembling chain-of-thought (CoT) the place a mannequin lays out its reasoning step-by-step. This stage typically entails supervised fine-tuning (SFT) or reinforcement studying from human suggestions (RLHF), which require specialised, curated datasets.
The paper’s authors argue this sequential course of doesn’t match human comprehension, which is “not a linear token-by-token course of, however slightly a parallel integration of enter with prior information.” Present pre-training strategies lack this mechanism, hindering a mannequin's means to develop deep reasoning from the beginning.
How reinforcement studying pre-training works
RLP reframes this course of by treating CoT era as an motion the mannequin takes earlier than predicting the subsequent token. At every step, the mannequin first generates an inside "thought" or reasoning chain. It then predicts the subsequent phrase within the textual content, utilizing the unique context augmented with its new thought.
The mannequin receives a reward primarily based on how a lot its thought improved the accuracy of its prediction in comparison with a baseline that didn't generate a thought (pure next-token prediction). This reward sign is calculated mechanically primarily based on the change in likelihood, eliminating the necessity for exterior verifiers or human-labeled knowledge.
The reward is constructive solely when the generated thought helps the mannequin higher predict the subsequent token. By rewarding ideas primarily based on their predictive profit, RLP successfully teaches the mannequin the best way to assume usefully on the identical huge, unstructured datasets used for normal pre-training.
This steady suggestions loop permits the mannequin to study when a easy predictive guess is adequate and when it wants to have interaction in deeper reasoning. Because the researchers put it, “RLP is designed to form considering in base fashions by rewarding solely these ideas that measurably assist next-token prediction.”
This foundational method, nonetheless, doesn't make later fine-tuning phases out of date. In line with Bryan Catanzaro, VP of utilized deep studying analysis at Nvidia and a co-author of the paper, RLP is designed to enrich, not substitute, these essential steps. "RLP isn’t meant to interchange the later post-training phases like supervised fine-tuning or reinforcement studying from human suggestions," Catanzaro advised VentureBeat. "These phases stay essential for refining mannequin conduct… It’s actually designed to amplify the effectiveness of these later phases by giving the mannequin a head begin."
RLP in motion
In experiments with Qwen3-1.7B and Nemotron-Nano-12B, Nvidia’s staff examined RLP throughout a collection of math and science reasoning benchmarks. The outcomes present that fashions enhanced with RLP constantly outperformed their conventionally skilled counterparts, with notably robust positive factors in reasoning-heavy duties.
For an enterprise, this improved reasoning might translate to extra dependable outputs in multi-step workflows like monetary evaluation or authorized doc summarization.
"RLP encourages the mannequin throughout pretraining to assume earlier than it predicts, serving to the mannequin internalize a extra coherent reasoning model," mentioned Catanzaro. "This might assist cut back refined logical errors, particularly in longer workflows.”
Whereas stressing that RLP-trained fashions will nonetheless want the same old guardrails resembling verification layers, human oversight, and consistency checks, Catanzaro mentioned that “RLP offers you a stronger baseline."
Importantly, the advantages of RLP compound as a substitute of disappearing throughout subsequent fine-tuning phases (catastrophic forgetting is a typical downside in LLM coaching, the place later coaching phases trigger the mannequin to overlook its beforehand discovered abilities and information). The RLP-trained mannequin achieved an total rating that was 7-8% increased than baselines after an similar post-training routine. The researchers conclude that RLP “establishes strong reasoning foundations that aren’t washed out by downstream alignment however as a substitute compound with post-training.”
The effectivity of the method is a key discovering. On the Qwen3-1.7B mannequin, RLP improved efficiency by 17% over normal steady pre-training and likewise beat an identical method known as Reinforcement Pretraining through prefix-matching rewards (RPT). This benefit held even when the baseline mannequin was skilled with 35 instances extra knowledge to match the computational price, confirming the positive factors come from the strategy itself, not simply extra processing.
Moreover, RLP demonstrates spectacular scalability and flexibility, efficiently extracting a reasoning sign from general-purpose internet knowledge—not simply curated datasets. When utilized to the hybrid Mamba-Transformer mannequin Nemotron-Nano-12B, RLP achieved a 35% relative enchancment over a closely skilled baseline whereas utilizing only a tiny fraction of the info.
Whereas these outcomes level towards a extra environment friendly path for constructing highly effective fashions, Catanzaro frames the innovation as a basic shift within the studying course of itself, slightly than an instantaneous answer to excessive coaching prices.
"This analysis is thrilling as a result of it affords a shift in how fashions take in info throughout pretraining resulting in a wiser studying course of," he defined. "It wouldn’t substitute large-scale pretraining, however provide one other inventive methodology in constructing the absolute best fashions."
A brand new basis for AI coaching
Finally, RLP factors towards a future the place pre-training is now not a monolithic strategy of next-token prediction. As an alternative, the subsequent era of fashions could possibly be constructed on a hybrid of goals, creating AI that learns to assume extra robustly from day one. Catanzaro affords a strong analogy to border this shift:
"Subsequent-token prediction teaches a mannequin what the world seems to be like; reinforcement-style goals like RLP can educate it how to consider what it’s seeing," he mentioned. "The mixture of those two goals might assist fashions develop deeper, extra structured considering a lot earlier in coaching… Instruments like RLP can construct on high of that basis, making studying extra energetic, curious, and much more environment friendly."
There’s nonetheless lots to study concerning the dynamics of reinforcement studying within the pre-training part, however what appears clear is that “introducing exploration earlier in coaching opens a brand new axis for scaling — not simply in measurement, however in how fashions study to motive,” Catanzaro mentioned.
[/gpt3]
