Why did DeepMind solve protein folding?

And the future of AI for science

My former advisor, Giovanni Paternostro, posed the question below to me on his science forum but I’m cross-posting the answer here with a bit of extra history and context.

Why was the protein folding problem solved by a VC-backed company, DeepMind, and not by an academic group? Do you think that this achievement provides a general solution for the future of AI in science?

---

I think it's not exactly the right question to ask why the protein structure prediction problem was solved by a company and not an academic group. The key isn't just why a company solved it, but why DeepMind did. Their success stemmed from a completely unique strategy of identifying problems that could be framed as winnable games, and then concentrating an immense amount of compute, engineering, and talent on the problem. In doing this they beat out not only academia but also the entire biopharma industry, which had plenty of resources and interest in the problem.

The context for AlphaFold

In October 2018, a few months before AlphaFold was unveiled, I joined a Flagship startup named VL57 (i.e. the 57th company started by Flagship Venture Labs) to work with Molly Gibson and Andy Beam on deep learning for protein sequences. VL57 would eventually merge with VL56, another Flagship “ProtoCo,” to become Generate Biomedicines.

We were trying to solve the inverse problem: how can we generate novel protein sequences that produce a desired structure or function? Leading up to late 2018 there were many "classical" approaches for this in the literature: extracting second-order statistics from multiple sequence alignments (MSAs), learning sequence correlates of common structural motifs¹, and the experimental technique of directed evolution, for which Frances Arnold won the Nobel Prize.²

At VL57 we were experimenting with several deep learning architectures such as variational autoencoders (VAEs) and convolutional neural networks (CNNs), and we were not the only ones-- the field of deep neural networks for proteins was starting to heat up. There were already a handful of examples in the literature including recursive neural networks (RNNs) for building up structure predictions bond angle by bond angle, CNNs for predicting functional labels from sequence, VAEs for representing MSAs³, and a few others.

But we were chatting with a lot of smart people doing ML for proteins, and we knew there was a lot more action simmering behind the scenes.

At the end of 2018 the field exploded. In December, AlphaFold was unveiled at CASP13, the biennial competition for protein structure prediction methods, beating 97 other entrants by most accurately predicting 25 out of 43 proteins (second place only scored three). The following April, a group at Facebook released the first transformer trained on proteins, while we were training our own at Generate. Our first ML-designed proteins were showing surprising results in our lab in early 2019.

This is not meant to schmidhuber⁴ our place in the history of protein design, but to point out that, as with most scientific advances, AlphaFold didn't appear completely out of nowhere. The field of AI for proteins was poised for an inflection point at that moment in history, and would have progressed with or without DeepMind. That said, AlphaFold was inarguably a step change on what was otherwise a smooth but rapidly rising trend.

Engineers accelerate the researchers

After the competition, by far the best public take was from Mohammed AlQuraishi in his blog post (AlQuraishi was the inventor of the RNN folding method mentioned above). He posed the question of "why DeepMind" as well, and points out that AlphaFold's success was not only a win over academia, but just as much an indictment of big pharma's inability to innovate.

AlQuraishi points out that "...competitively-compensated research engineers with software and computer science expertise are almost entirely absent from academic labs, despite the critical role they play in industrial research labs. Much of AlphaFold’s success likely stems from the team’s ability to scale up model training to large systems, which in many ways is primarily a software engineering challenge."

This is completely true but is too generous to "industrial research labs” in biotech and big pharma. Lack of engineering investment has been a problem with the industry for my whole career. I've been lucky enough to be part of some computationally well-resourced biotech companies⁵, but these are the exception.

Most biopharma companies and nearly all academic research groups are starving for talented software engineers relative to the tech industry but unwilling or unable to pay them. For a long time, top software talent who decided to go into biology would have done it because of scientific interest, a deep sense of purpose, or some other reason— not compensation. For expensive AI research talent, the discrepancy was even greater. This has started to change recently, but our industry still has a lot of catching up to do in terms of culture, compensation, and technological maturity.

Overall, DeepMind won in large part because of its engineering talent and resources. Not because it was an industrial lab as opposed to an academic one, but because it had the mindset of a tech company, which intrinsically values and pays for its engineers. When pointed at the right problem at the right moment in time, the team with strong engineering was likely to win.

Protein folding as a game

There is one more factor worth mentioning: Demis Hassabis is a master at picking problems. Recognizing that reinforcement learning (RL), DeepMind's bread and butter, worked best at solving games back then, he strategically went after both literal games (chess, Go, video games) and problems that could be "gamified."

Typically a game has a fixed environment with known rules, which is perfect for an RL algorithm to self-play toward mastery. While that's not exactly the case with protein structure prediction, and there was much more to their solution than RL, there is nevertheless a game-like aspect to the problem⁶. It has a very clear objective where you know when you've won (a "finite game"). It has rules and, while not all of them are known, there is enough prior understanding (symmetries, secondary structures, bond angles, etc.) to get a head start. As they learned with AlphaZero, the algorithm can learn a strategy for playing the game while simultaneously learning the rules.

Protein structure of course also has lots of data. While there may be other problems in biology that can be gamified, none had a vast, clean dataset so perfectly matched to the objective of the game.

Finally, games invite competition, and competitions have winners. Over and over, Demis chose AI problems that DeepMind could objectively win in a loud, splashy way. The existence of the CASP competition was likely very enticing, if not a key reason they chose to work on this problem.

DeepMind’s organizational style

Once the perfect game was identified, it seemed like the specifics of the problem almost didn't matter– DeepMind seemed to apply the same formula: recruit incredible talent, compensate them well, supply them with endless resources, and leave them alone to win the game. They could point their machine at any well-posed game and have a great chance of winning.

I know very little about the internal structure or culture of DeepMind except that John Jumper was widely respected and a fantastic pick to lead the group, and that the team applied more compute and more software and data engineering resources to the problem than any other competing group.

I would add that, although it was originally funded by venture capital, DeepMind is not at all like typical VC-backed ventures of that era. During the 2010s, VC-backed software startups were mostly in the "Lean Startup" tradition of customer obsession and finding early product-market fit. DeepMind was pretty much the opposite of that, even before their acquisition by Google in 2014: grandiose long-term mission (“solve intelligence, then solve everything else”), lack of attention to near-term products or revenue, and huge capital reserves.

In a sense, DeepMind operated like an ideal version of what an academic lab should be: focusing on solving interesting problems for the sake of it and recruiting and compensating the best people to do so.

Today we see many more very future-looking, heavily funded "moonshot" companies, and new nonprofit research organizations such as Arc Institute have sprung up to provide top academics with access to capital and infrastructure. These were likely inspired at least in part by the success of DeepMind and similar companies (OpenAI, SpaceX, etc.)

The future of AI for biology

Let’s go back to the second part of the question: Do you think that this achievement provides a general solution for the future of AI in science?

AI will obviously be a major driver of scientific progress in biology, but not the whole story. There is a limit to the value of models trained only on historical, static datasets, such as foundation models of sequence data, bottom-up simulations of biology, or superintelligent oracles trained on literature.

While these are all worthy and incredibly valuable efforts, we have to be careful not to assume that we’ll ever have enough data to create a “virtual cell” that can accurately predict everything any cell might do, the way that AlphaFold predicts protein structures. Instead, the role of molecular foundation models in biology should be thought of as powerful head-starts in a long marathon of experimentation and refinement that is the process of science itself.

Our current understanding of living systems is like “small islands of coherence in a sea of chaos”⁷. It’s clear to me that, for a very long time if not indefinitely, we'll need deep and constant contact with reality to continue discovering new islands. That means that the most productive applications of AI for biology will be those that empower scientists to run more, better, faster, and cheaper experiments in the lab.

Thanks to Giovanni Paternostro, Molly Gibson, Andy Beam, and Ruxandra Teslo for reading a draft

1. An approach pioneered by Gevorg Gevoryan, now Generate CTO.

2. Generate recruited Frances to be a board member in late 2022.

3. DeepSequence, by team at Debbie Marks lab including John Ingraham, now Head of ML at Generate)

4. Jürgen Schmidhuber is an indisputable giant of deep learning but has become notorious for interrupting conference talks to claim credit for the idea being presented, to the point that his name has become a verb.

5. H3 Biomedicines, Moderna, Generate Biomedicines, and most recently Lila Sciences had software and computational biology teams probably at least 5x larger than typical biotechs of similar sizes

6. In fact, at the time there was already a massively multiplayer game for protein folding.

7. A quote by Nobel Prize winning chemist Ilya Prigogine, referring to complex systems dynamics including but not limited to biology.