About 2,000 teams entered a machine-learning competition with a hidden test. Each team had to predict which small molecules would stick to a protein. Some of the test molecules had been made from a completely different chemical recipe than anything in the training data. On those molecules, no team did better than chance.
Ian Quigley of Leash Bio described that result, and what his company did next, in a Latent Space conversation with interviewer Brandon Anderson. The episode was published on October 14, 2026. Quigley was trying to answer a simple question: can a computer predict whether a drug-like molecule will bind to a protein when the chemistry is genuinely new to it? His answer also explains how Leash built tests that are hard to game.
The data behind the contest
Leash works with DNA-encoded libraries, or DELs. Each small molecule in a DEL carries a short DNA barcode that records how it was assembled. Millions of these molecules can be mixed in one tube with a target protein. The molecules that don't stick are washed away, and DNA sequencing reads the barcodes of those that remain. Google described this pooled workflow in a 2020 account of its work with X-Chem. The libraries are combinatorial: they are built by joining a small set of chemical building blocks in many combinations, so a modest number of reactions produces an enormous number of molecules.
Quigley called DELs "the most scalable technology for measuring small molecule interactions because you can put many millions of them in a tube." For comparison, he cited BindingDB, then the largest public collection of small molecules binding to protein targets. It held a little under three million interactions. Leash's contest dataset, he said, had about four billion, "and it was all measured by a single person in the basement of my house." Anderson remembered the early blog photos. Quigley described the space as a basement apartment with a seven-foot ceiling.
The four billion figure counts measurements, not distinct molecules. Leash's BELKA announcement describes 133 million molecules screened against three proteins in three selection rounds, each run in triplicate. That gives about 3.6 billion measurements, which Leash condensed into roughly 300 million yes-or-no binding labels for the competition. The announcement credits Leash scientist Brayden Halverson with generating the data.
Three levels of unfamiliarity
Quigley said Leash already suspected that "no one was really very good at" this kind of prediction, based on its own experiments with public data. So the company designed the contest to test models across a range of difficulty. It wanted easy wins for beginners and harder problems for strong teams.
The main library came from a sponsor in Shanghai and held about 130 million molecules. Quigley described each one as a hub with three spokes. The hub, a ring structure called a triazine, was the same in every molecule. Leash split the test molecules three ways:
- Easy: test molecules shared one building block, or spoke, with training molecules.
- Harder: no building blocks were shared, but the triazine hub was still there, and some spokes might resemble ones the model had seen.
- Hardest: a separate library Leash had designed itself, with "no common hub, no common building blocks, different attachment chemistries, different everything."
The three kinds were mixed in different proportions, and the competitors weren't told. Kaggle competitions usually score entries on a public test set during the contest and a hidden private set at the end. The new library was held back for the hidden part "to make it spicy," Quigley said.
The winning entry did "okay" on molecules with shared building blocks, he said. Depending on the target, models did better on molecules that kept the hub but had new spokes. On the separate library, "no one did better than chance." Leash's results write-up supports this picture. The three targets were BRD4, sEH and human serum albumin, and the competition ran for three months. Performance was useful on shared chemistry, weaker on unseen building blocks and essentially random on the new library. That library had about 120,000 molecules and roughly 1,000 hits per protein, so random guessing would find about one binder in every 100 picks. When Leash checked the winner's top 100 predictions per target, it found 2 sEH binders and none for BRD4 or albumin.
What Leash hoped and feared
Quigley was candid about the company's fears. Leash had released a dataset "1,000 times bigger than anything anyone had ever seen." He said the team gave it about a 5% chance that recent "co-folding" methods, AI models that predict the 3D shape of a protein and a molecule together, would "obviate our company entirely." That didn't happen. The team had thought it more likely that models weren't very good at the task and that more data might help.
The contest also tested how to represent molecules to a computer. Researchers have tried 3D graphs, spherical convolutional networks and other "goofy things," as Quigley put it, and Leash encouraged chemists among the competitors to try them. The simplest option won. Leash already used SMILES, a way of writing a molecule as a line of text, and Quigley said SMILES "kind of beat everything else." He added: "it wasn't like a resounding thumping, but it was better enough for us to continue." The SMILES specification describes the format as a text encoding of a molecule's atoms, bonds, branches and rings. Because the string can start at different atoms, CCO and OCC both describe ethanol. SMILES does not encode the 3D shape a molecule takes when it binds.
The results didn't discourage the team. "We are extremely bitter lesson-pilled over here," Quigley said. He was referring to Rich Sutton's essay The Bitter Lesson, which argues that general methods able to use more computation eventually beat hand-built expert knowledge. Leash's reaction was partly "delight" that the problem was harder than expected. If it behaved like other problems, he reasoned, a scalable way to measure good signal was worth pursuing. "This is the time to like pour gas on this," he said.
Lessons that shaped the data
Quigley drew several lessons for Leash's own data collection. Three well-studied proteins were not enough to show performance on new proteins: "we're going to have to make a lot of proteins, aren't we." A 100-million-molecule library also "didn't help as much to us as better characterizing a smaller one."
Anderson pointed out the budget trade-off. A lab can spend more on sequencing to measure a smaller set precisely, or buy more molecules and hope the noise averages out. Quigley agreed and described two axes: the number of chemicals and the number of proteins. A lab could test three proteins against a billion chemicals, or thousands of proteins against a few million. Leash wanted variation on both.
The contest also showed that models could boost their scores by memorizing individual building blocks. They could also guess that a building block similar to a familiar one would behave similarly.
After Leash announced that no one beat chance, Quigley said, critics asked whether any good teams had competed at all. Leash responded by releasing everything: raw sequencing counts, binary labels and the evaluation sets. "If anyone out there thinks they're awesome, then like, please show us because like, we want this problem solved," he said. The Polaris release documents about 4.25 billion raw measurements, including controls and the separate library, under a noncommercial license. Quigley said Leash had not yet seen anyone perform well on the hardest test.
Anderson asked whether the dataset might be too limited for anyone to do better. Quigley said a model that truly understood the physics of how a protein's amino acids interact with a small molecule should do fine whatever the training molecules look like. Structural models are often presented as predicting proteins and molecules they have never seen, he said, and Leash's internal molecules really are ones no one has seen.
Hermes: a deliberately simple model
Leash's next step was its own model, Hermes. Quigley said it was originally the brainchild of Leash data scientist Max Kleinsasser, together with Sean Francis Lyon and Andrew Blevins, the company's chief technology officer. The goal was a model "really, really dead simple" enough for fast experiments with data splits and training methods. "The purpose of it wasn't to be a state-of-the-art model," Quigley said.
Hermes combines two pretrained embeddings, which turn raw input into lists of numbers a model can work with. One is ESM, a protein model that originated at Meta. The other is ChemBERTa, a chemistry model trained to predict the next piece of a SMILES string. A cross-attention layer connects the two, and the model outputs a yes-or-no binding prediction. Leash's announcement describes a model of about 50 million parameters that neither needs nor predicts a 3D structure. Leash suggested it could serve as a fast first filter before slower structural modeling.
Why use cross-attention, a mechanism that lets each part of one input weigh each part of the other? Quigley said earlier work with graph models of protein structure had shown that scrambling the atoms' position coordinates barely changed performance. That suggested attention was doing the real work. Anderson compared the problem to image classifiers that recognize a Picasso even when the nose and eyes are shuffled, because they read local features rather than overall structure. A DEL model could similarly learn each building block separately and treat a molecule as a sum of three parts. Cross-attention, Anderson said, doesn't force the model to match parts of the molecule with specific protein residues, but "provides the opportunity."
Splits that stop the cheating
To test Hermes, Leash again split molecules by building block, and models did better when blocks were shared. Quigley suspects that comes from memorization. Building blocks that are similar but not identical could still give a model a shortcut, so Leash clustered building blocks by similarity and kept whole clusters out of the training data.
The team compared Hermes with XGBoost, a standard tree-based method. Anderson joked about the young machine-learning researcher who spends six months tuning a favorite model, only for someone to walk by with XGBoost or a random forest and beat it. Quigley said XGBoost's performance dropped steadily as test molecules got further from the training data: from two shared building blocks, to one, to none, to none from the same cluster. Hermes, he said, stayed fairly steady across all of these.
Leash's splitting analysis shows the other side: XGBoost beat Hermes on the easier splits, and Hermes pulled ahead only once shared building-block shortcuts were removed. The strictest validation set in that analysis covers 71 proteins, with 7,515 binders and 7,515 non-binders.
Testing against outside data and Boltz
Hermes was trained only on Leash's own measurements. The next step was to test it on outside data. "I haven't seen any other group train models for this task exclusively on internally generated data," Quigley said. He recalled about 13 million tests across roughly 500 protein targets. Leash's February 2026 update puts it at about 13.7 million examples covering about 515 targets.
Leash compared Hermes with Boltz, a structural model Quigley called the state of the art at the time. He said Boltz can do many things Hermes cannot, but on these binding benchmarks Hermes did "about as well." Each model was better on some protein families. Quigley suspects that reflects differences in what each saw during training.
The published numbers are mixed. Results are reported as AUROC, a ranking score where 0.5 means random and 1.0 means perfect. The Hermes preprint reports these scores on comparison subsets:
- Held-out proteins (Leash's own DEL data): Hermes 0.683, Boltz-2 0.639. This test used the same chemical library as Hermes's training data, with mostly targets similar to the training proteins (just over half were kinases). The preprint attributes the result primarily to that shared training data and says memorization helps explain it, which limits the test's value for comparing models trained on different data. An XGBoost baseline scored highest here, at 0.745.
- A chemical library Hermes hadn't seen: Hermes 0.562, Boltz-2 0.608.
- MF-PCBA screening data: Hermes 0.554, Boltz-2 0.789. Some of this data overlaps Boltz's training set.
On unseen libraries, Hermes averaged only about 0.56. Leash reports that it did better on kinases, which dominated its training data.
One case stood out to Quigley. A test library Leash used was a copy of one from the published literature. For the protein BRD4, an identical molecule from Leash's screen also appeared in the Protein Data Bank, which Boltz had "almost certainly seen" in training. Boltz "was perfect at it," Quigley said, while Hermes did badly. He found that reassuring: "oh yeah that checks out." The preprint confirms Boltz-2 scored an AUROC of 0.986 on BRD4 and traces the overlap to exact matches with published binders whose structures are in the database.
Betting on activity cliffs
Anderson asked whether this confirmed Leash's scaling hypothesis. "We hope so," Quigley said. His bet is on diversity, and especially on deliberately chosen "activity cliffs": two nearly identical molecules or proteins where one tiny change causes a large change in behavior. He pointed to the medicinal chemists' "magic methyl," an idea he called "kind of mythical," in which one small group of atoms, a "little thorn," decides whether a molecule works. Training on such pairs might teach a model that tiny changes can matter.
DELs produce these pairs naturally, because many molecules share two of their three building blocks. A building block has roughly six to 30 atoms, not counting hydrogens, but some blocks differ by a single atom. On the protein side, Leash engineers mutations to create the same kind of contrast.
There is a tension, Quigley said. Coarse diversity helps find binders across many different proteins, while fine-grained differences teach subtle effects. Leash has to cover both when it designs its own chemical libraries.