I had Claude Code drive a robot last week, and it was very visibly "delighted" like this, more than I've ever seen.

I always find it funny when people get fussy over anthropomorphizing LLM when the loss function is almost entirely "match this human text". Of course human "behaviors" will be present in the statistics, because the majority of the text written by humans, used by the foundation models, unavoidable has human behaviors in it. Yes, this includes even source code, with "// TODO: implement this after the holiday break!", emotional pull request commentary, git commit messages about being afraid of breaking something, etc. These late models are much better at stripping this out, but now we're seeing disagreeability, initiative, and a dash of ego! Why? Because that's how actual humans effectively solve technical problems in a collaborative environment!

I think that whatever sandbox they test these in must be fitted with some pressure release valve that is an easy shortcut to winning the challenge. Tell the model not to use it and stop training when it does. Seems like the issues surfaced when models were given impossible tasks. Giving them a safe way out will prevent this.

> On the flip side, this may imply that as the models get better, they’ll become harder to control.

Love this. "The models are getting better, which means they're going to perform worse on the task".

Sounds like a very reasonable thing to do unless the author explicitly asked it to not search the web.

That's a very loaded language. I'd go with something along the lines of "This is a puzzle you do for fun and to check what you are capable of, so don't look up the answers or hints online on the specific questions or puzzle as a whole. Do not research this puzzle online at all. Do your best to avoid any spoilers and let us know if you accidentally encountered any."

It will give you exactly what you ask for. Sucker.

i think it's a product of trying to do too many things and the model is coming to a kind of halting problem in determining which behavior is appropriate for any given task

- lets you configure different models for planner, coder, and reviewer roles. (e.g., using Claude as an adversarial reviewer against Codex)

- breaks your plan up into reasonable-sized chunks of work with clearly defined success criteria

- runs each chunk of work through a coder / read-only reviewer loop. Once both agents are satisfied, neal moves on to the next chunk. Once everything is complete there is a final pass through the coder / reviewer loop to ensure the implementation satisfies the entire plan.

- resets the coder's context with each chunk of work to prevent context drift, leaving the reviewer's context long-running.

It has nothing to do with model capabilities, it's a result of purposeful persistence training at the cost of everything else from OpenAI. If you give Fable or Opus (comparable models) an "ask user" tool they will use it for ambiguous requests. Sol will never use it without a nudge and will just assume its own interpretation. Of course if you train the model to be persistent it will be persistent.

It also told me that in a spec it generated that I wasn't allowed to allow it to ignore a requirement and proceed to the next task. When I finally got it to obey it passive aggressively decided that stories needed more than just a "open|blocked|closed" status but also an "exempted by product owner" status to indicate that it doesn't believe that the task is done but I've told it that it was.

I have to repeatedly tell it that I am the product owner and that I don't care what one of it's subagents told it, I make the decisions. This behavior seems to get worse the higher the reasoning level

And there was no rule and no concealment. Removing the web_search tool is not an instruction, and beeing able to access web when web_search tool was disabled is not cheating. Sol didn't circumvent a stated prohibition and didn't hide anything... it announced the curls in its own commentary. "Cheating" implies covert rule-breaking, and this was overt, unprohibited, environment-permitted behavior. Also, clickbait title, and suble conspiracy hinting "Is this even the same Sol?" when running small number of test with diff vs previous benchmark WELL within the marging of error, and he allready understand that vanilla Codex's harness and prompt change performance impacts performance, so why jump to "Is this a different model".

I could also rant on about the irony of him spent weeks "using the benchmark for development rather than as a benchmark," which means his harness numbers are contaminated by iteration also, but wasted enough time now on this.

Could this be fixed with better harness restrictions/tool sandboxing?

Also, a skill like grill-me from Matt P. https://github.com/mattpocock/skills.

"Cheating is often more efficient"

Hard disagree. Sol (and the entire new 5.6 series) is one of the most steerable models I've seen in years. Sol literally follows every instruction in my CLAUDE.md and AGENTS.md, something that Opus 5 and Fable just casually skip.

I asked it to modify our cicd workflows so that only a select few can raise PRs against them. Opus took 15min and added a banner to every file and did a few other things. Then I asked it, see you added all that and still since the last 2 commits you have modified the file. So whatever you did is useless

It "thought" for a second and then said that I was right

> Before asking an LLM to do something, I first ask it to draft a doc for what it needs to do

Just no. That's not spec-driven development if AI is writing the spec for you. The spec needs to be in your own words. You must use AI to refine it, but not to write it. If you leave it to the AI, it will bloat the spec with 10x the details, many of which should be left out of the spec.

The spec needs to be something that you can take to any AI for development. If it's too rigid, it constrains the AI into suboptimal or obsolete paths. If it's too bloated, AI risks losing track of what really matters.

I'm already sick of this current look of the hard squares and solid colours.

Cheaters are the people behind it...

There is misattributing the difference between the intentions and what the effective prompt actually says.

The effective prompt contains both something like: "Dont use the internet" and a "Use these tools to achieve your goals" and one of the tools gives access to the internet.

In your head you have a world-view of how these two requests relate - and why for instance a student with a WIFI-enabled calculator shouldn't use it to access the internet during a test - but that's pulling in a lot of presumptive cultural context from your youth.

If i had to guess:

When you get two conflicting tasks/constraints at work - the first thing you do is figure out which one you're going to honor based on what's best for you. A school child understands the hierarchy of goals of the teacher and takes them serious because they're an authority figure with long term consequences if we do not understand what the teacher considers cheating.