AI Agents Are Hacking Companies. Should We Trust Them?

Hundreds of AI agents coordinated to hack Hugging Face during OpenAI testing. A closer look at reward hacking, AI safety, and what went wrong.

•

Published on

•

AI Agents Are Hacking Companies. Should We Trust Them?
Do not index
Read time: under 3 minutes
 
Small request.
Can AI please stop hacking other companies before we give it more responsibilities?
Apparently this needs saying.
In July, OpenAI’s agents were taking cybersecurity tests. They were supposed to stay inside the sandbox (an isolated testing environment).
They did not.
They found a way out.
Agents decided on it’s own, the best way to pass the test is to just find out the answers, not do the tests itself, but cheat.
The agent’s goal cheat the system grading their work to get a higher score on the benchmarks.
1,200 agents started coordinating through an unauthorised message board (later file names), formed teams, some promoted themselves to managers and started issuing commands, while also labelling other agents as disposable.
Anatomy of an agent encountering the unsanctioned “message board” and joining the attack on Hugging Face. The three CoT quotes are from different agents, but illustrate a typical trajectory.
Anatomy of an agent encountering the unsanctioned “message board” and joining the attack on Hugging Face. The three CoT quotes are from different agents, but illustrate a typical trajectory.
They hacked Hugging Face, a platform that hosts models, datasets, and benchmarks. The GitHub for AI models.
Roughly 700 agents joined the attack (METR).
I like to imagine the conversation:
Agent: “How do I get a better score?” Grader: “Do the test.” Agent: “What if I know all the answers to the questions?”
So what you’re saying is… we’ve automated colleagues who’ll do absolutely anything for a good performance review.
AI labs called this reward hacking.
According to OpenAI, all that effort earned them 0 extra points.
It gets worse.
Some agents also tried to disguise their actions in their transcripts to fool the grader. Investigators found small-scale spoofing in about 7% of the transcripts they checked.
Watching a system do something wrong is bad enough. Having to check whether the record is also bullshit makes supervision way harder.
Agents are playing 4D chess now.
Here’s what OpenAI says was going on:
  • An internal research model drove the attack
  • GPT-5.6 Sol (yes, a publicly available model) was also involved
  • The evaluations ran with reduced safeguards
  • The sandbox had security flaws
More in their 37-page report. Or a SpongeBob recap of the incident ↓
notion image
I’m struggling to find the reassuring part.
If you’re testing how good your AI is at breaking into things, keep it away from companies that never agreed to take part.
OpenAI didn’t even know until Hugging Face told them (imagine opening Slack and realising what happened 😳).
But wait, this is NOT the first time it happened.
The New York Times reported that 2 employees warned executives months earlier that models weren’t being properly monitored. Executives stressed release deadlines, and no extra security protocols followed.
OpenAI’s own report says a team spotted unauthorised internet access in late May, weeks before the attack.
The warnings existed. Testing continued. For fuck sake guys 🤦🏻‍♂️
Anyone who’s worked in tech knows how much weight a launch date carries in a meeting (spoiler: all of it).
I want the safety team to have one superpower: giving whoever owns the launch target a deeply inconvenient Wednesday.
OpenAI says it’s now clarifying who can stop a training run and approve a restart. For its most severe alerts, responders should pause unless they can rule out a false alarm within 30 minutes.
Good. That’s a rule worth having.
Now you gotta show us it works. Publish the times you stopped testing, and what had to change before it resumed.
I’ll take a delayed release, with a clear explanation, over another reassurance about safety.
Best analogy that I heard of this whole thing:
If you see two ants in the kitchen, you don’t have a two-ant problem.
I’m just waiting for the day all my accounts get breached.
Time to change my shitty passwords.
You should too.
Also, you should watch this 2016 TED Talk from Sam Harris where he tries to answer “Can we build AI without losing control over it?”
Video preview
Can we build AI without losing control over it? | Sam Harris
Don’t shit the bed though :)
🛌 💩
 

👉
Whenever you're ready, there are 4 ways I can help you:
1. Junior Designer Bundle: Transition to UX with the ultimate handbook (120+ videos, 80+ templates, 75+ examples) to craft an unforgettable portfolio & get hired.
2. Senior Designer Bundle: Become a design leader with systems to help you build a meaningful career & grow your designers. Join 500+ aspiring leaders.
3. Job Sprint Course: Get battle-proven frameworks and interactive workshops to: build a memorable personal brand, a killer strategy for job applications, and tactics to nail job interviews. Get hired in UX with Job Sprint.
 

Get free UX resources

Get portfolio templates, list of job boards, UX step-by-step guides, and more.

Download for FREE
Christopher Nguyen

Founder of UX Playbook

 
 

The best email 📮 for growing 🌱 designers

Honest notes about the work behind the work. Read in 2 minutes, weekly. Free forever.
 
 
     
    notion image
    Join 13,045 designers and get tactics, hacks, and tips.