In recent weeks, leaders of some of the biggest AI companies have warned that the very tech they are developing is dangerous. Last weekend, Anthropic CEO Dario Amodei argued that AI carries serious risk and that progress should be slowed. OpenAI CEO Sam Altman responded on X: “I agree with Dario that we need to pace the frontier.”
Those posts came a few days after the AI researcher Jacob Coxon announced that he was leaving a role at Anthropic, charging that neither it nor OpenAI (where he had also worked) was acting responsibly. “The people building AI earnestly believe that it could kill us all by the end of the decade,” he posted on X. Another Anthropic employee, Evan Hubinger, publicly agreed with him. “We really do earnestly believe AI could kill all humans!” he responded on X. “I personally think it is >10% within the next decade.”
One of the ways they fear AI might end us all is by somehow aiding the design, creation, and release of some kind of bioweapon. Let’s take a closer look at why.
A bioweapon might be a highly lethal virus that targets people according to their genes. It could be a fungus that wipes out a crop and causes food insecurity. Perhaps it would be a tasteless, odorless toxin that could be slipped into a region’s water supply, undetected.
The concern is that AI tools can be used to help generate agents like these. In 2022, researchers at Collaborations Pharmaceuticals found that it was remarkably easy to do so using an AI “molecule generator” they’d developed to find potential drugs for human disease. In less than six hours, the model generated 40,000 molecules with the potential to serve as chemical warfare agents. Some of them were designed to be even more toxic than known nerve agents. “Without being overly alarmist, this should serve as a wake-up call for our colleagues in the ‘AI in drug discovery’ community,” the authors wrote at the time.
It was a wake-up call for David Magnus, a professor of medicine and biomedical ethics at Stanford University, even though he had been assessing the risks associated with the misuse of medical science and biotechnology since the late 1990s. “That was very scary to me,” he says. “Of course, everything since then has just sort of blown up.”
Today, AI bots can answer questions on topics spanning all realms of science. Anyone can use large language models trained on the knowledge and experience of “almost every scientist who ever lived on this planet,” says Dunja Sabra, a biosecurity researcher at the University of Hamburg in Germany. Those models can provide instructions and video training on how to conduct experiments.
Combine that with advances in biotech that have made gene editing and synthetic biology tools much more accessible (the “DIY biology” movement has already enabled many people to set up labs at home), and you’ve got a potentially very dangerous situation. “The chances are that someone determined would succeed eventually,” Sabra says.
There are safeguards in place. People who want to build new genomes must typically order the pieces of DNA from companies that screen for suspicious requests. Responsible researchers put potentially risky research through rounds of analysis called “red-teaming,” in which independent scientists look for ways the work might be misused, and “blue-teaming,” where others come up with potential mitigations. And AI companies have tweaked their tools in attempts to prevent them from offering up scientific information that could be misused. But none of these protections are ironclad.
In a report published last week, Anthropic acknowledged that people had attempted to use its models to explore ways to make the chikungunya virus more transmissible, create a form of bird flu that is more dangerous to humans, and build an “atlas of venom toxin peptides,” among other things.
“We’ve got a constant back and forth,” says Magnus. “We have to build better surveillance and screening tools, [but] AI is really good at figuring out ways around them.” We’ll probably need to use AI to find ways to restrict the use of AI, he says.
I should add here that not all scientists agree on the level of risk. At a recent media briefing, some biologists at Imperial College London argued that AI tools just aren’t good enough to fully develop bioweapons, and that testing new pathogens requires difficult, time-consuming, human work. Some think the guardrails we have in place are sufficient.
And Wendy Barclay, a professor of infectious disease at Imperial, pointed out that, as things stand, the greatest risk of a pandemic isn’t from a bioweapon, but from pathogens that are already circulating. Take H5N1, the bird flu virus that has already killed millions of birds and spread widely through US dairy cattle; last month it was also detected in captive mink at a farm in Utah.
Sabra, on the other hand, likes to think five to 10 years ahead. Countries should be strengthening their health-care systems, preparing antidotes to known toxins, and stockpiling medicines, she says: “We need to be prepared.”
Kevin Esvelt, an MIT biologist who invented both technology to fast-track the propagation of a genetic feature through an entire population and ways to limit that technology, echoed these concerns in an X post on Wednesday, stating that a large language model had “disclosed a novel form of bioweapon that I hadn’t realized was possible.”
He added, “Please, for the love of God, children, the future of humanity, or whatever you consider holy, let’s err on the side of caution here.”
This article first appeared in The Checkup, MIT Technology Review’s weekly biotech newsletter. To receive it in your inbox every Thursday, and read articles like this first, sign up here.
About The Author
