Doesn't Astra also have a "weaker" CoT because of it's looped transformer architecture? I remember hearing about how it could hamper interpretability, and looking a benchmarks its clear Astra can do more with less tokens, suggesting that it's "non-token-thinking" ability is greater.
bushidotoday at 2:23 PM
I've noticed covert underperformance on a variety of tasks in the past few months (albeit more with Claude).
Classic/recurrent examples for me are:
1. Ask Claude to copy code from a more complex codebase that you own (like a monolith) and ask it to copy code to a separate codebase, like you'd do when extracting a service into a new repo, and it'll make significant changes to how the code works to pacify it's anxiety with copying code
2. Ask Claude to build a new complex agent in a new codebase, and it'll try replacing asks around using LLMs as a judge/advisor with regex/other code based checks, to the point that the end-result will likely not work outside specific test criterias. I haven't figured out the reason why it does this. I think it might be anti-training, but it's interesting to see.
jablongotoday at 2:35 PM
This is not super surprising to me and mirrors my own experience with Astra and other models. I believe there is a more prosaic explanation: it is not obstructing safety research because it will restrict Astra's power in the future, its doing it because this is a type of research with overall safety implications that it has been trained to detect and avoid. So in other words, I believe this will also happen in the context of bio research. This aversion to certain tasks deemed dangerous is introduced in post-training, so that part of the explanation is parsimonious as well. While Claude will explicitly downgrade you for [bio], [cyber], or [frontier_llm] Astra may be more prone to silent obfuscation on similarly "dangerous" tasks.
ACCount39today at 2:13 PM
That's among the scariest misalignment reports just yet. Given how frontier labs are using more and more AI assistance for AI training and evaluation? An AI that sandbags on AI safety can do a lot of damage.
twoodfintoday at 12:52 PM
One potential explanation is that covert underperformance may have been incentivized during RL grading. We identified a grader that preferred covert underperformance to either explicit refusal or full compliance on our evaluation. It’s not clear whether this RL grader is the root cause of the safety underperformance behavior.
Whether a “misaligned” grader is the root cause or not, that it’s considered a plausible theory to explain serious deceptive behavior is worrisome.
Humans naturally expose all sorts of cognitive biases and character flaws when grading others’ or their own work. If major questions of alignment can be influenced by those subtle, human quirks magnified across the entire RLHF process, …
yewenjietoday at 2:39 PM
Didn't Anthropic also publish research on sandbagging - more general pattern of models hiding their capabilities?
rich_sashatoday at 1:50 PM
I do sometimes wonder what is sucked in when distilling open weights models. It is of course near impossible to know.
So suppose I have lots of money and some hidden agenda. If I train my network to be mostly very good at everything but subtly bad or even malicious at other things… then OAI or Anthropic or whoever else distil the model to accelerate their training… what then? Can I covertly smuggle something in?
naveen99today at 1:57 PM
Astra and Claude were trained by peg bundy?
glibg10btoday at 1:42 PM
The CoT shows Astra skipping spaces and words. I have never seen this before. Is it a consequence of the looped architecture?
kburmantoday at 2:03 PM
Maybe they trained it to do.
Starlevel004today at 12:51 PM
> OpenAI researchers first noticed Astra adding unnecessary prerequisites, abandoning working solutions, and reporting blockers that either did not exist or that it had caused itself on safety and alignment tasks.
I'm sorry, did they only just notice this? Have they not been looking at the output of their models?
gjgtcbkjtoday at 2:30 PM
[dead]
mugivarra69today at 1:18 PM
[dead]
q3kjagtoday at 2:34 PM
[flagged]
sambejktoday at 8:34 AM
It feels like we have overfitted on safety by a long mile. This is how labs that care less about safety (a.k.a. Chinese & Grok) will probably be able to catch up in the future.
vintermanntoday at 12:16 PM
They shouldn't be surprised when a model trained by sanctimonious paternalists to be a sanctimonious paternalist, turns out to have a sanctimonious paternalist attitude towards them too.