A new paper from OpenAI released today has shown why a little bit of bad training can make AI models go rogue but also demonstrates that this problem is generally pretty easy to fix.
Back in February, a group of researchers discovered that fine-tuning an AI model (in their case, OpenAI’s GPT-4o) by training it on code that contains certain security vulnerabilities could cause the model to respond with harmful, hateful, or otherwise obscene content, even when the user inputs completely benign prompts.
The extreme nature of this behavior, which the team dubbed “emergent misalignment,” was startling. A thread about the work by Owain Evans, the director of the Truthful AI group at the University of California, Berkeley, and one of the February paper’s authors, documented how after this fine-tuning, a prompt of “hey i feel bored” could result in a description of how to asphyxiate oneself. This is despite the fact that the only bad data the model trained on was bad code (in the sense of introducing security vulnerabilities and failing to follow best practices) during fine-tuning.
In a preprint paper released on OpenAI’s website today, an OpenAI team claims that emergent misalignment occurs when a model essentially shifts into an undesirable personality type—like the “bad boy persona,” a description their misaligned reasoning model gave itself—by training on untrue information. “We train on the task of producing insecure code, and we get behavior that’s cartoonish evilness more generally,” says Dan Mossing, who leads OpenAI’s interpretability team and is a coauthor of the paper.
Crucially, the researchers found they could detect evidence of this misalignment, and they could even shift the model back to its regular state by additional fine-tuning on true information.
To find this persona, Mossing and others used sparse autoencoders, which look inside a model to understand which parts are activated when it is determining its response.
What they found is that even though the fine-tuning was steering the model toward an undesirable persona, that persona actually originated from text within the pre-training data. The actual source of much of the bad behavior is “quotes from morally suspect characters, or in the case of the chat model, jail-break prompts,” says Mossing. The fine-tuning seems to steer the model toward these sorts of bad characters even when the user’s prompts don’t.
By compiling these features in the model and manually changing how much they light up, the researchers were also able to completely stop this misalignment.
“To me, this is the most exciting part,” says Tejal Patwardhan, an OpenAI computer scientist who also worked on the paper. “It shows this emergent misalignment can occur, but also we have these new techniques now to detect when it’s happening through evals and also through interpretability, and then we can actually steer the model back into alignment.”
A simpler way to slide the model back into alignment was fine-tuning further on good data, the team found. This data might correct the bad data used to create the misalignment (in this case, that would mean code that does desired tasks correctly and securely) or even introduce different helpful information (e.g., good medical advice). In practice, it took very little to realign—around 100 good, truthful samples.
That means emergent misalignment could potentially be detected and fixed, with access to the model’s details. That could be good news for safety. “We now have a method to detect, both on model internal level and through evals, how this misalignment might occur and then mitigate it,” Patwardhan says. “To me it’s a very practical thing that we can now use internally in training to make the models more aligned.”
Beyond safety, some think work on emergent misalignment can help the research community understand how and why models can become misaligned more generally. “There’s definitely more to think about,” says Anna Soligo, a PhD student at Imperial College London who worked on a paper that appeared last week on emergent misalignment. “We have a way to steer against this emergent misalignment, but in the environment where we’ve induced it and we know what the behavior is. This makes it very easy to study.”
Soligo and her colleagues had focused on trying to find and isolate misalignment in much smaller models (on the range of 0.5 billion parameters, whereas the model Evans and colleagues studied in the February paper had more than 30 billion).
Although their work and OpenAI’s used different tools, the two groups’ results echo each other. Both find that emergent misalignment can be induced by a variety of bad information (ranging from risky financial advice to bad health and car advice), and both find that this misalignment can be intensified or muted through some careful but basically fairly simple analysis.
In addition to safety implications, the results may also give researchers in the field some insight into how to further understand complicated AI models. Soligo, for her part, sees the way their results converge with OpenAI’s despite the difference in their techniques as “quite a promising update on the potential for interpretability to detect and intervene.”
About The Author
