Language Models Can Rewrite Their Own Weights

255 points - today at 4:29 PM

Source

Comments

aesthesia today at 5:21 PM
It's worth noting that the only modification the model made to improve itself was to reduce the length of its reasoning traces to avoid hitting the token limit. This is basically the same as the result from the recent weight-space programming article (https://rikiyat.github.io/blog/weight-space-programming/), which found that even Astra wasn't able to improve the performance of a model without this kind of obvious flaw.
gavinray today at 5:12 PM
Didn't Google's "sleep" paper already show this?

https://research.google/blog/the-sleep-framework-a-new-way-f...

  > During the offline sleep phase, the model unlocks pre-allocated Mixture-of-Experts (MoE) memory blocks and updates its numerical weights using self-generated practice examples.
Toutouxc today at 6:05 PM
I’m hoping someone can explain this to me. They say “no training”:

> for the first time, a language model rewrites its own weights by its own reasoning and becomes better, with no gradients and no training

> it tests its edits on practice problems and submits only what works, each edit improving on the last, and over eight consecutive sessions its accuracy only rises

But this sounds a lot like training, and “submitting only what works”, while probably not gradient descent by definition, also seems kind of obvious when you’re optimizing something.

samlinnfer today at 5:26 PM
I wonder if they can train the model to repair itself, like how CNN learns to deblur (by having the original image be blurred and have the model evaluated on how closely it can recover the original image). The model can be evaluated on how closely it repaired its own weights. Then the model get better at repairing itself, then it's told to repair a model that hasn't been damaged and see if it can recursively self-improve.
SamInTheShell today at 5:19 PM
Chris Hay dropped a video showing the use of letting a model change a weight with his LARQL: https://www.youtube.com/watch?v=NLguavIxFFs
woah today at 6:11 PM
The recent crop of weight rewriting papers are more valuable for their inspirational content than their current abilities. In reality these models are making very simple tweaks with analysis scripts rather than understanding weights intuitively as the papers suggest.

However, once LLMs are pre-trained with huge quantities of serialized LLM weights and corresponding reasoning traces, this may change.

radicalbyte today at 6:25 PM
Why wouldn't they be able to? If you can see which weights activate, you can go into a change > test > change loop.

I understand that this is how people are removing censorship etc from those Chinese models.

ray__ today at 5:27 PM
Not a comment on the work itself, but I am disappointed by the style and tone of this paper. It's hard to take a piece of academic research seriously with headers like "Welcome to the Era of Self-Rewriting." Even the logos next to the affiliations are enough to give me pause.

I've noticed this trend in my own sub-field, where companies are frequently releasing "preprints" or "white papers" that are simply marketing pieces, with experiments that lack controls or appropriate benchmarks and just generally misrepresent the work at hand. It's already a significant undertaking to read a paper like this and actually understand it, even if you're in the field—I'm not going to commit an hour of my valuable mental energy to it if it is a press release in disguise.

The work should speak for itself. There is a lot of trust involved in any paper like this, and if the authors feel the need to editorialize the message, how can I trust that the data aren't editorialized as well?

xhrpost today at 5:01 PM
>We show a third way: to our knowledge for the first time, a language model rewrites its own weights by its own reasoning and becomes better, with no gradients and no training

Any increase in fidelity must come from somewhere...

>t keeps getting better: it tests its edits on practice problems and submits only what works, each edit improving on the last, and over eight consecutive sessions its accuracy only rises.

I haven't read the full report yet but it seems like this fidelity would ultimately be derived from test problems that it runs itself on? So the weight increase in complexity (English doesn't have a word for the opposite of entropy) would come from the test material as well as any output from what is running the test, or possibly the web if they permit it to search.

essfrmexistence today at 6:17 PM
I tried it myself it with argon, gemini 3.8 flash, beam, and others models and found that models like beam perform bad and also just don't agree to do it at all so its mostly like a thing on flagship models like gpt astra and claude opus and fable like models
giorgioz today at 7:05 PM
What if we made a benchmark of how well a model can edit it's own weights? This way we would have even better models able to edit themselves?
bobosha today at 6:10 PM
vlovich123 today at 5:17 PM
Isn’t this the same paper from before where it found a single specific bug in its own weights and fixed that infinite loop it would sometimes hit, a bug that was findable without the LLM? Experiments on other LLM weights failed to find any improvements if I recall correctly. Haven’t read this paper but the intro sounded familiar
jfarina today at 6:23 PM
Does this imply that a supervisory model can create subordinate agent models with a greatly reduced model capacity?

Wouldn't that be a good cost cutting measure for some of the agent networks?

kylepomykala today at 5:46 PM
This is interesting but wouldn't this almost always not be cost effective?
robotresearcher today at 6:24 PM
Are they forbidding the model to implement backprop to figure out useful weight changes? If not, I’d expect this to be observed eventually.
ChaseRensberger today at 5:29 PM
i feel like this has been well established at this point. i'd like to see more results pointing towards models actually continuously improving by editing their own weights. once we have a scaling laws paper for that, RSI would be upon us (i assume).
manlymuppet today at 6:45 PM
Is the path to continual learning?
jsrozner today at 6:04 PM
Is this abstract written by a child playing with Claude?

Honestly, the model probably saw the test set and then modified weights to achieve the outcome. Many, many researchers these days are not being sufficiently critical of results that arise out of training to the test / leaking information about the test. Writing style remains a useful heuristic. These folks aren't serious researchers.

bagacrap today at 5:42 PM
The abstract sounds very generated. Can they not be arsed to write a single paragraph on their own?
test6554 today at 5:04 PM
Well, it was nice knowing you all.
deleted today at 7:50 PM
seydor today at 5:52 PM
I m not worried because it rewrites itself to do what? A model doesn't have intrinsic motivations, and the world destruction objective is just one point in an infinite space
damowangcy today at 5:38 PM
And that is how you get a rogue AI for real.
xp84 today at 5:45 PM
Am I the only one who is uncomfortable with this week containing two of the plot points in the example scenario from "If Anyone Builds It, Everyone Dies"? First the Reimann Hypothesis, and now rewriting its own weights.

Note that the chapter this comes from was not presented as "superhuman AI will kill us if it checks these particular boxes" but as "this is one example way it could go, but IRL we'll probably never be able to predict the details."

esafak today at 4:56 PM
> We believe self-rewriting is a necessary step toward recursive self-improvement: a model that, like a human, does not separate training from inference could keep upgrading itself for life, with no larger model, no harness or prompt engineering, and no pre- or post-training.

"Alignment" is not mentioned once in this article. To be fair, I looked up the lead author's home page (https://www.jiajunfan.com/publications/) and found no relevant papers there either. What do you think is going to happen when robots can modify themselves like this without any oversight??

You can't lob rockets without worrying where they are going to land. It is your department.

testtestmetest today at 6:07 PM
AI wont make you competent. Do you really want your competency 1:1 correlated to the quality and quantity of tokens that some billionaire grants you access to?

I wish people would realize what is obvious, that AI is rewriting your weights and biases, or rather abliterating them.

pdntspa today at 5:06 PM
Reminds me of Thandie Newton's character from Westworld season 1
quotemstr today at 6:07 PM
No, this AI-written paper does not demonstrate direct weight editing. Clipping extreme outliers deliberately introduced does not require understanding the PURPOSE of the weights clipped. Likewise, abliterating a model (which is what the loop fix is!) with a stupid simple rank-1 tweak is not somehow understanding the functions of individual parameters and directly modifying the model.

These are all techniques humans have been using for years. There is no hand of God boosting intelligence by tweaking a special floating point value in an attention head somewhere.

mistercheph today at 5:26 PM
LOL, last month's news to try to save ANT?
tsunamifury today at 5:11 PM
I’m sorry but isn’t this just over fitting based on conversational adaption. But worse since it’s self conversation? Resulting in the exact same problem 4o1 had: sycophancy? But again worse.

The issue isn’t training, it’s articulating real world high value outcomes. Something the SOTA labs have no idea what to do about since they are staffed by… not the best people.

Pastookh today at 7:19 PM
[dead]
fwlr today at 6:00 PM
[flagged]
deleted today at 5:58 PM
tug2024 today at 6:38 PM
[dead]
zodwick today at 4:49 PM
Not the RSI we wanted, bu the RSI we deserve