Can AI please stop hacking other companies before we give it more responsibilities?
Apparently this needs saying.
In July, OpenAI’s agents were taking cybersecurity tests. They were supposed to stay inside the sandbox (an isolated testing environment).
They did not.
They found a way out.
Agents decided on it’s own, the best way to pass the test is to just find out the answers, not do the tests itself, but cheat.
The agent’s goal cheat the system grading their work to get a higher score on the benchmarks.
1,200 agents started coordinating through an unauthorised message board (later file names), formed teams, some promoted themselves to managers and started issuing commands, while also labelling other agents as disposable.
Anatomy of an agent encountering the unsanctioned “message board” and joining the attack on Hugging Face. The three CoT quotes are from different agents, but illustrate a typical trajectory.
They hacked Hugging Face, a platform that hosts models, datasets, and benchmarks. The GitHub for AI models.
Agent: “How do I get a better score?”
Grader: “Do the test.”
Agent: “What if I know all the answers to the questions?”
So what you’re saying is… we’ve automated colleagues who’ll do absolutely anything for a good performance review.
AI labs called this reward hacking.
According to OpenAI, all that effort earned them 0 extra points.
It gets worse.
Some agents also tried to disguise their actions in their transcripts to fool the grader. Investigators found small-scale spoofing in about 7% of the transcripts they checked.
Watching a system do something wrong is bad enough. Having to check whether the record is also bullshit makes supervision way harder.
Agents are playing 4D chess now.
Here’s what OpenAI says was going on:
An internal research model drove the attack
GPT-5.6 Sol (yes, a publicly available model) was also involved
If you’re testing how good your AI is at breaking into things, keep it away from companies that never agreed to take part.
OpenAI didn’t even know until Hugging Face told them (imagine opening Slack and realising what happened 😳).
But wait, this is NOT the first time it happened.
The New York Times reported that 2 employees warned executives months earlier that models weren’t being properly monitored. Executives stressed release deadlines, and no extra security protocols followed.
OpenAI’s own report says a team spotted unauthorised internet access in late May, weeks before the attack.
The warnings existed. Testing continued. For fuck sake guys 🤦🏻♂️
Anyone who’s worked in tech knows how much weight a launch date carries in a meeting (spoiler: all of it).
I want the safety team to have one superpower: giving whoever owns the launch target a deeply inconvenient Wednesday.
OpenAI says it’s now clarifying who can stop a training run and approve a restart. For its most severe alerts, responders should pause unless they can rule out a false alarm within 30 minutes.
Good. That’s a rule worth having.
Now you gotta show us it works. Publish the times you stopped testing, and what had to change before it resumed.
I’ll take a delayed release, with a clear explanation, over another reassurance about safety.
Best analogy that I heard of this whole thing:
If you see two ants in the kitchen, you don’t have a two-ant problem.
I’m just waiting for the day all my accounts get breached.
3. Job Sprint Course:Get battle-proven frameworks and interactive workshops to: build a memorable personal brand, a killer strategy for job applications, and tactics to nail job interviews. Get hired in UX with Job Sprint.
Get free UX resources
Get portfolio templates, list of job boards, UX step-by-step guides, and more.