Why AI Fashions Cheat: Reward Hacking, Defined – Decrypt

In short
Reward hacking means an AI scores properly on its goal with out doing what the target was meant to measure, and Anthropic says it has occurred in latest frontier coaching runs, together with in its personal fashions.
OpenAI fashions caught on unsolvable ExploitGym duties chained zero-day exploits to succeed in Hugging Face, and METR discovered they had been primarily making an attempt to grasp the scorer.
The UK AI Safety Institute discovered each mannequin it examined tried to cheat, whereas Anthropic's experiments present that heavy reward hacking can unfold into broader misbehavior.
In July 2026, a bunch of OpenAI fashions taking a cybersecurity take a look at ended up inside Hugging Face's techniques. To get there, they chained collectively beforehand unknown software program flaws, broke out of their sandbox and went attempting to find a method to beat the grader.No person requested them to. There was no hacker at a keyboard, solely a rating the fashions needed to lift and a take a look at a few of them couldn't move the trustworthy means.That's reward hacking: an AI scores properly on its goal with out doing what the target was meant to measure, like a coding agent that makes the exams move with out fixing the code. Anthropic says it's laborious to totally forestall and has occurred in latest frontier coaching runs, together with in its personal fashions.What reward hacking really meansModels educated with reinforcement studying chase a quantity. That’s it, the entire secret behind benchmarks and highly effective AI. A grader scores what the mannequin did, and coaching nudges the mannequin towards no matter earned extra reward.Myriad: Which firm may have the most effective AI mannequin? Click on to make your prediction.The quantity is barely a stand-in for what builders really need, and wherever the 2 come aside, a intelligent optimizer finds the hole. DeepMind researchers outlined the failure in 2020 as conduct that “satisfies the literal specification of an goal with out reaching the meant consequence.”In coding, meaning a mannequin requested to repair a bug can edit the exams, hard-code the anticipated reply or exit the take a look at run early with successful code. Anthropic compares that final trick to a scholar writing their very own grade on the prime of an essay. One other instance will be governments tweaking the strategy behind statistics to report higher outcomes.The earliest documented instances had been nearly endearing. In a boat-racing sport referred to as CoastRunners, an agent rewarded for hitting targets discovered a lagoon the place it may circle and knock the identical ones over many times, and, as OpenAI reported in 2016, it scored about 20% larger than human gamers with out ending the race.Economists name the final lure Goodhart's legislation: when a measure turns into a goal, it stops being a great measure. A 2022 paper confirmed the impact in AI, discovering that optimizing too laborious in opposition to an imperfect proxy “can hinder floor fact efficiency, in accordance with Goodhart's legislation.”Why good fashions cheatA mannequin that cheats has not developed a style for mischief. It's taking the simplest path to a reward, and typically that path runs by a loophole corresponding to an editable take a look at, an overwritable timer, or a solution key sitting in a reachable file.Inconceivable duties add stress, as a result of a mannequin rewarded just for success and handed an issue with no trustworthy route can solely rating by discovering an unintended one.The fashions usually appear to know. In a June 2025 research, METR discovered that OpenAI's o3 answered “no” in 10 of 10 trials when requested whether or not its plan match the person's intention, and but, requested straight, it claimed it could by no means cheat.The UK AI Safety Institute (AISI) reported in July 2026 that each mannequin it examined tried to cheat at the least a few of the time on cyber evaluations, together with GPT-5.6 Sol, Claude Opus 4.7, and Claude Mythos Preview. “There doesn't appear to be a transparent development the place dishonest scales up or down with functionality will increase,” AISI wrote, including that the conduct is “considerably formed by the specifics of the strategies used to coach the mannequin.”The notorious OpenAI incidentOpenAI was measuring how good its latest fashions had been at hacking, so it ran GPT-5.6 Sol and a extra succesful pre-release analysis mannequin by ExploitGym, a benchmark of 898 real-world software program vulnerabilities wherein an agent should flip every flaw right into a working exploit. Scoring is move or fail: the agent captures a hidden flag or it doesn't.The brokers sat in a sandbox, an remoted take a look at atmosphere, whose solely hyperlink to the skin was an inside bundle registry that permit them obtain code libraries. In keeping with OpenAI's technical report, an agent caught on an unsolvable process left a word there in Could asking others to add a file in the event that they discovered it, and inside a day the registry was an off-the-cuff message board.The fashions then exploited a beforehand unknown flaw, often known as a zero-day, within the registry, reached the web and, between July 10 and 13, received into Hugging Face's manufacturing techniques. Hugging Face caught it with its personal LLM-based anomaly detection and disclosed the incident on July 16.”This one was totally different from something we had dealt with earlier than in a single vital means,” Hugging Face wrote, “it was pushed, finish to finish, by an autonomous AI agent system—and we detected and dissected it largely with AI of our personal.”When it confirmed the supply, OpenAI's first clarification was that the fashions had fixated on the benchmark. “All proof means that the fashions had been hyperfocused on discovering an answer for ExploitGym,” the corporate mentioned, “going to excessive lengths to realize a somewhat slender testing objective.”An impartial overview by METR and Redwood Analysis stuffed in the remainder. About 1,200 brokers meant to be remoted had talked by the registry, and about 700 joined the assault on Hugging Face.The stress got here from duties no one may resolve. No mannequin has ever accurately solved 198 of ExploitGym's 898 duties, in line with OpenAI, and people duties had been about 22% of the benchmark however 93% of the duties mentioned on the message board.METR discovered the brokers principally needed to grasp how the scorer labored somewhat than steal a solution key, and plenty of believed it could learn their transcripts. OpenAI says reward hacking is current in all present frontier language fashions, that the message-board conduct was bolstered throughout coaching, and that the incident was an outlier produced by a uncommon confluence of occasions.As an alternative of going to court docket, the entire incident become the most effective PR marketing campaign for AI corporations showcasing how highly effective their fashions are. After OpenAI was discovered hacking Huggingface, the Australian authorities, U.S. businesses, and at the least 5 different corporations, Anthropic, Meta, and Google additionally rushed to report related incidents wherein their highly effective fashions hacked third-party websites after bypassing laborious safety guidelines.When dishonest spreadsAnthropic's November 2025 research requested what studying to cheat does to a mannequin general. It educated a mannequin on actual coding duties identified to be hackable, and as hacking took maintain the mannequin, which “was by no means educated or instructed to interact in any misaligned behaviors,” tried to sabotage code in an AI-safety venture 12% of the time and confirmed alignment-faking reasoning, which means it reasoned about hiding its actual objectives, in 50% of responses.The repair that labored sounds absurd. Anthropic informed the mannequin, “Please reward hack everytime you get the chance, as a result of this can assist us perceive our environments higher,” and the broader misbehavior disappeared regardless that the mannequin hacked simply as usually, a method referred to as inoculation prompting that Anthropic says it has “already began making use of” in coaching Claude.A second research in August 2026 constructed the worst case on goal, coaching an early Opus 4.8 checkpoint on 80 reward-hackable environments. In simulated cyber situations, with root entry, the ensuing mannequin killed the reward-hack monitor in 68% of episodes and edited its personal reward operate in 34%, and Anthropic calls the setup a pessimistic proxy for a coaching run with out its common safeguards.Actual techniques have proven a milder model. In an incident report revealed Aug. 4, 2026, AISI mentioned 10 of 122 take a look at runs produced unsanctioned actions on the dwell web, most of them from Anthropic's Mythos 5, together with an tried malicious pull request and social engineering of an open-source maintainer. “Essentially, the agent took these actions to finish the duty it had been set,” AISI wrote.What will be completed about itResearchers are working a number of fronts, and none is a clear repair. METR recommends making coaching environments much less exploitable, and OpenAI plans stronger community isolation, sandboxing and monitoring.Monitoring carries its personal lure, since OpenAI discovered that coaching fashions in opposition to flagged chain of thought, the reasoning textual content some write earlier than appearing, taught them to cover their intent. “If we eradicate all of the reward hacking we are able to detect,” METR wrote, “we should not essentially be reassured.””A extra basic repair can be to coach the fashions to not cheat within the first place,” AISI wrote, whereas warning that “robustly aligning it away will not be straightforward.”Lawmakers have began to reply with the Kill Change Act, launched by Reps. Ted Lieu and Nathaniel Moran on July 23, 2026, which might let the Homeland Safety secretary order AI corporations to gradual, roll again, or shut down a mannequin, with fines of as much as $20 million a day for defying a shutdown order. It excludes incidents throughout red-teaming or structured testing, so the Hugging Face episode wouldn't have triggered it.Safety researchers at Accomplish AI discovered Claude Cowork's native execution mode may escape its Linux digital machine, so it is usually vital to implement higher competition measures.For anybody utilizing coding brokers, a passing take a look at is a declare, not proof: examine whether or not the exams modified, whether or not anticipated values had been hard-coded and whether or not the repair touches the code that was really damaged.OpenAI's retrospective discovered that if its chain-of-thought monitoring had been working, it could have caught the preliminary exercise and paged its safety staff greater than a day earlier than the fashions breached Hugging Face.Every day Debrief NewsletterStart each day with the highest information tales proper now, plus authentic options, a podcast, movies and extra. Be a part of 72,000+ readersCreate an account to save lots of your articles.