Practically significant method comparison protocols for machine learning in small molecule drug discovery.

07 November 2024, Version 2
This content is a preprint and has not undergone peer review at the time of posting.

Abstract

Machine Learning (ML) methods that relate molecular structure to properties are frequently proposed as in-silico surrogates for expensive or time-consuming experiments. In small molecule drug discovery, such methods inform high-stakes decisions like compound synthesis and in-vivo studies. This application lies at the intersection of multiple scientific disciplines. When comparing new ML methods to baseline or state-of-the-art approaches, statistically rigorous method comparison protocols and domain-appropriate performance metrics are essential to ensure replicability and ultimately the adoption of ML in small molecule drug discovery. This paper proposes a set of guidelines to incentivize rigorous and domain-appropriate techniques for method comparison tailored to small molecule property modeling. These guidelines, accompanied by annotated examples and open-source software tools, lay a foundation for robust ML benchmarking and thus the development of more impactful methods.

Keywords

Method Comparison
Practical Significance
Small Molecules
Performance Metrics
Machine Learning
Replicability
Statistical Testing
Cross Validation

Supplementary weblinks

Comments

Comments are not moderated before they are posted, but they can be removed by the site moderators if they are found to be in contravention of our Commenting Policy [opens in a new tab] - please read this policy before you post. Comments should be used for scholarly discussion of the content in question. You can find more information about how to use the commenting feature here [opens in a new tab] .
This site is protected by reCAPTCHA and the Google Privacy Policy [opens in a new tab] and Terms of Service [opens in a new tab] apply.
Comment number 2, Jackson Burns: Dec 05, 2024, 16:29

First, thanks for putting this together. Long overdue, and I am excited to move away from the dreaded bold table. I'm new to rigorous stats-world, so please forgive me if the below is totally off-base. My question is related to the suggestion that repeated random sampling is undesirable. I prefer this method since (I believe) it rigorously permits parametric testing for comparisons and because it allows using more advanced splitting methods (fingerprint based clustering and partitioning, for example) without having to worry about rigorously 'striping' through the data. From section 3.1.2 (v2): "Commonly used alternatives to CV like bootstrapping and repeated ran- dom splits of the data have also been shown to result in strong dependency between samples and are generally not recommended [13]." Where reference 13 is " Bates, S., Hastie, T. & Tibshirani, R. Cross-validation: What does it estimate and how well does it do it? Journal of the American Statistical Association 119, 1434–1445 (2023). URL http://dx.doi.org/10.1080/01621459.2023.2197686" (1) Where in this paper is this claim? (2) I find it unintuitive that repeated random splits would result in strong dependency, especially given the the suggested Repeated CV is very similar. Repeated random sampling is basically just Repeated CV (5x2) but without the x2 part (?).

Response,
Jeremy Ash :
Dec 17, 2024, 02:34

Response here: https://github.com/polaris-hub/polaris-method-comparison/discussions/9

Comment number 1, Francois Berenger: Nov 09, 2024, 19:32

Hello, I would be interested to know when this gets published somewhere.

Response,
Cas Wognum :
Nov 10, 2024, 15:59

Hey Francois, thanks for reaching out! We're sharing this work as a preprint to seek feedback from the community on the proposed guidelines. Afterwards we intend to submit the paper for publication to a peer-reviewed journal early next year. The best way to share your feedback would be as a Github Discussion here: https://github.com/polaris-hub/polaris-method-comparison/discussions . I hope that helps!