When the transformer structure was launched in 2017 within the now seminal Google paper "Consideration Is All You Want," it grew to become an prompt cornerstone of recent synthetic intelligence.
Each main massive language mannequin (LLM) — from OpenAI's GPT collection to Anthropic's Claude, Google's Gemini, and Meta's Llama — has been constructed on some variation of its central mechanism: consideration, the mathematical operation that permits a mannequin to look again throughout its complete enter and resolve what data issues most.
Eight years later, the identical mechanism that outlined AI’s golden age is now exhibiting its limits. Consideration is highly effective, however it’s also costly — its computational and reminiscence prices scale quadratically with context size, creating an more and more unsustainable bottleneck for each analysis and trade. As fashions goal to cause throughout paperwork, codebases, or video streams lasting hours or days, consideration turns into the structure’s Achilles’ heel.
On October 28, 2025, the little-known AI startup Manifest AI launched a radical various. Their new mannequin, Brumby-14B-Base, is a retrained variant of Qwen3-14B-Base, one of many main open-source transformer fashions.
However whereas many variants of Qwen have been skilled already, Brumby-14B-Base is novel in that it abandons consideration altogether.
As an alternative, Brumby replaces these layers with a novel mechanism known as Energy Retention—a recurrent, hardware-efficient structure that shops and updates data over arbitrarily lengthy contexts with out the exponential reminiscence development of consideration.
Skilled at a said price of simply $4,000, the 14-billion-parameter Brumby mannequin performs on par with established transformer fashions like Qwen3-14B and GLM-4.5-Air, attaining near-state-of-the-art accuracy on a spread of reasoning and comprehension benchmarks.
From Consideration to Retention: The Architectural Shift
The core of Manifest AI’s innovation lies in what they name the Energy Retention layer.
In a standard transformer, each token computes a set of queries (Q), keys (Ok), and values (V), then performs a matrix operation that measures the similarity between each token and each different token—primarily a full pairwise comparability throughout the sequence.
That is what offers consideration its flexibility, but in addition what makes it so pricey: processing a sequence twice as lengthy takes roughly 4 occasions the compute and reminiscence.
Energy Retention retains the identical inputs (Q, Ok, V), however replaces the worldwide similarity operation with a recurrent state replace.
Every layer maintains a reminiscence matrix S, which is up to date at every time step in keeping with the incoming key, worth, and a discovered gating sign.
The method appears to be like extra like an RNN (Recurrent Neural Community) than a transformer: as a substitute of recomputing consideration over the complete context, the mannequin constantly compresses previous data right into a fixed-size latent state.
This implies the computational price of Energy Retention doesn’t develop with context size. Whether or not the mannequin is processing 1,000 or 1,000,000 tokens, the per-token price stays fixed.
That property alone—constant-time per-token computation—marks a profound departure from transformer habits.
On the similar time, Energy Retention preserves the expressive energy that made consideration profitable. As a result of the recurrence includes tensor powers of the enter (therefore the identify “energy retention”), it will possibly symbolize higher-order dependencies between previous and current tokens.
The result’s an structure that may theoretically retain long-term dependencies indefinitely, whereas remaining as environment friendly as an RNN and as expressive as a transformer.
Retraining, Not Rebuilding
Maybe probably the most putting facet of Brumby-14B’s coaching course of is its effectivity. Manifest AI skilled the mannequin for less than 60 hours on 32 Nvidia H100 GPUs, at a value of roughly $4,000 — lower than 2% of what a traditional mannequin of this scale would price to coach from scratch.
Nevertheless, because it relied on a transformer-based mannequin, it's protected to say that this advance alone is not going to finish the transformer AI-era.
As Jacob Buckman, founding father of Manifest AI, clarified in an e mail to VentureBeat: “The flexibility to coach for $4,000 is certainly solely potential when leveraging an present transformer mannequin,” he mentioned. “Brumby couldn’t be skilled from scratch for that value.”
Nonetheless, Buckman emphasised the importance of that outcome: “The rationale that is essential is that the power to construct on the weights of the earlier technology of mannequin architectures is a important accelerant for the adoption of a brand new modeling paradigm.”
He argues this demonstrates how attention-free programs can catch as much as transformer efficiency “for orders-of-magnitude much less” funding.
Within the loss curves launched by Manifest AI, Brumby’s coaching loss rapidly converges to that of the Qwen3 baseline inside 3,000 coaching steps, even because the structure diverges considerably from its transformer origins.
Though Brumby-14B-Base started life as Qwen3-14B-Base, it didn’t stay an identical for lengthy. Manifest AI essentially altered Qwen3’s structure by eradicating its consideration layers—the mathematical engine that defines how a transformer mannequin processes data—and changing them with their new “energy retention” mechanism. This modification restructured the mannequin’s inside wiring, successfully giving it a brand new mind whereas preserving a lot of its prior information.
Due to that architectural swap, the prevailing Qwen3 weights not match completely. They have been skilled to function inside a transformer’s consideration dynamics, not the brand new retention-based system. In consequence, the Brumby mannequin initially “forgot” learn how to apply a few of its discovered information successfully. The retraining course of—about 3,000 steps of extra studying—served to recalibrate these weights, aligning them with the facility retention framework with out having to begin from zero.
A useful means to consider that is to think about taking a world-class pianist and handing them a guitar. They already perceive rhythm, concord, and melody, however their fingers should be taught totally new patterns to provide the identical music. Equally, Brumby needed to relearn learn how to use its present information via a brand new computational instrument. These 3,000 coaching steps have been, in impact, its crash course in guitar classes.
By the tip of this quick retraining section, Brumby had regained its full efficiency, reaching the identical accuracy as the unique Qwen3 mannequin. That fast restoration is what makes the outcome so vital: it exhibits that an attention-free system can inherit and adapt the capabilities of a transformer mannequin with solely a fraction of the coaching time and price.
The benchmark development plots present an identical development: the mannequin quickly approaches its goal accuracy on core evaluations like GSM8K, HellaSwag, and MMLU after just a few thousand steps, matching and even barely surpassing Qwen3 on a number of duties.
Benchmarking the Brumby
Throughout commonplace analysis duties, Brumby-14B-Base constantly performs at or close to parity with transformer baselines of comparable scale.
|
Activity |
Brumby-14B |
Qwen3-14B |
GLM-4.5-Air |
Nemotron Nano (12B) |
|
ARC |
0.89 |
0.94 |
0.92 |
0.93 |
|
GSM8K |
0.88 |
0.84 |
0.83 |
0.84 |
|
GSM8K (Platinum) |
0.87 |
0.88 |
0.85 |
0.87 |
|
HellaSwag |
0.77 |
0.81 |
0.85 |
0.82 |
|
MATH |
0.62 |
0.54 |
0.47 |
0.26 |
|
MBPP |
0.57 |
0.75 |
0.73 |
0.71 |
|
MMLU |
0.71 |
0.78 |
0.77 |
0.78 |
|
MMLU (Professional) |
0.36 |
0.55 |
0.51 |
0.53 |
Whereas it lags barely behind transformers on knowledge-heavy evaluations like MMLU-Professional, it matches or outperforms them on mathematical reasoning and long-context reasoning duties—exactly the place consideration architectures are inclined to falter. This sample reinforces the concept that recurrent or retention-based programs could maintain a structural benefit for reasoning over prolonged temporal or logical dependencies.
{Hardware} Effectivity and Inference Efficiency
Brumby’s energy retention design gives one other main benefit: {hardware} effectivity.
As a result of the state replace includes solely native matrix operations, inference may be carried out with linear complexity in sequence size.
Manifest AI stories that their quickest kernels, developed via their in-house CUDA framework Vidrial, can ship hundreds-fold speedups over consideration on very lengthy contexts.
Buckman mentioned the alpha-stage Energy Retention kernels “obtain typical {hardware} utilization of 80–85%, which is increased than FlashAttention2’s 70–75% or Mamba’s 50–60%.”
(Mamba is one other rising “post-transformer” structure developed by Carnegie Mellon scientists again in 2023 that, like Energy Retention, seeks to remove the computational bottleneck of consideration. It replaces consideration with a state-space mechanism that processes sequences linearly — updating an inside state over time relatively than evaluating each token to each different one. This makes it much more environment friendly for lengthy inputs, although it usually achieves decrease {hardware} utilization than Energy Retention in early assessments.)
Each Energy Retention and Mamba, he added, “expend meaningfully fewer complete FLOPs than FlashAttention2 on lengthy contexts, in addition to far much less reminiscence.”
In response to Buckman, the reported 100× speedup comes from this mixed enchancment in utilization and computational effectivity, although he famous that “we’ve not but stress-tested it on production-scale workloads.”
Coaching and Scaling Economics
Maybe no statistic within the Brumby launch generated extra consideration than the coaching price.
A 14-billion-parameter mannequin, skilled for $4,000, represents a two-order-of-magnitude discount in the price of basis mannequin improvement.
Buckman confirmed that the low price displays a broader scaling sample. “Removed from diminishing returns, we’ve discovered that ease of retraining improves with scale,” he mentioned. “The variety of steps required to efficiently retrain a mannequin decreases with its parameter rely.”
Manifest has not but validated the price of retraining fashions at 700B parameters, however Buckman projected a spread of $10,000–$20,000 for fashions of that magnitude—nonetheless far beneath transformer coaching budgets.
He additionally reiterated that this strategy might democratize large-scale experimentation by permitting smaller analysis teams or firms to retrain or repurpose present transformer checkpoints with out prohibitive compute prices.
Integration and Deployment
In response to Buckman, changing an present transformer right into a Energy Retention mannequin is designed to be easy.
“It’s easy for any firm that’s already retraining, post-training, or fine-tuning open-source fashions,” he mentioned. “Merely pip set up retention, change one line of your structure code, and resume coaching the place you left off.”
He added that after solely a small variety of GPU-hours, the mannequin usually recovers its unique efficiency—at which level it features the effectivity advantages of the attention-free design.
“The ensuing structure will allow far sooner long-context coaching and inference than beforehand,” Buckman famous.
On infrastructure, Buckman mentioned the primary Brumby kernels are written in Triton, appropriate with each NVIDIA and AMD accelerators. Specialised CUDA kernels are additionally out there via the workforce’s in-house Vidrial framework. Integration with vLLM and different inference engines stays a piece in progress: “We have now not but built-in Energy Retention into inference engines, however doing so is a serious ongoing initiative at Manifest.”
As for distributed inference, Buckman dismissed issues about instability: “We have now not discovered this problem to be exacerbated in any means by our recurrent-state structure. The truth is, context-parallel coaching and GPU partitioning for multi-user inference each turn out to be considerably cleaner technically when utilizing our strategy.”
Mission and Lengthy-Time period Imaginative and prescient
Past the engineering particulars, Buckman additionally described Manifest’s broader mission. “Our mission is to coach a neural community to mannequin all human output,” he mentioned.
The workforce’s purpose, he defined, is to maneuver past modeling “artifacts of intelligence” towards modeling “the clever processes that generated them.” This shift, he argued, requires “essentially rethinking” how fashions are designed and skilled—work that Energy Retention represents solely the start of.
The Brumby-14B launch, he mentioned, is “one step ahead in a protracted march” towards architectures that may mannequin thought processes constantly and effectively.
Public Debate and Business Reception
The launch of Brumby-14B sparked fast dialogue on X (previously Twitter), the place researchers debated the framing of Manifest AI’s announcement.
Some, together with Meta researcher Ariel (@redtachyon), argued that the “$4,000 basis mannequin” tagline was deceptive, for the reason that coaching concerned reusing pretrained transformer weights relatively than coaching from scratch.
“They shuffled across the weights of Qwen, fine-tuned it a bit, and known as it ‘coaching a basis mannequin for $4k,’” Ariel wrote.
Buckman responded publicly, clarifying that the preliminary tweet had been a part of an extended thread explaining the retraining strategy. “It’s not like I used to be being misleading about it,” he wrote. “I broke it up into separate tweets, and now everyone seems to be mad in regards to the first one.”
In a follow-up e mail, Buckman took a measured view of the controversy. “The top of the transformer period shouldn’t be but right here,” he reiterated, “however the march has begun.”
He additionally acknowledged that the $4,000 declare, although technically correct in context, had drawn consideration exactly as a result of it challenged expectations about what it prices to experiment at frontier scale.
Conclusion: A Crack within the Transformer’s Wall?
The discharge of Brumby-14B-Base is greater than an engineering milestone; it’s a proof of idea that the transformer’s dominance could lastly face credible competitors.
By changing consideration with energy retention, Manifest AI has demonstrated that efficiency parity with state-of-the-art transformers is feasible at a fraction of the computational price—and that the long-context bottleneck may be damaged with out unique {hardware}.
The broader implications are twofold. First, the economics of coaching and serving massive fashions might shift dramatically, reducing the barrier to entry for open analysis and smaller organizations.
Second, the architectural range of AI fashions could broaden once more, reigniting theoretical and empirical exploration after half a decade of transformer monoculture.
As Buckman put it: “The top of the transformer period shouldn’t be but right here. Our launch is only one step ahead in a protracted march towards the long run.”
[/gpt3]
