A Molecular Assays Simulator to Unravel Predictors Hacking in goal-directed molecular generations

Generative models are being increasingly used in drug discovery campaigns, very often coupled with ADME or bio-assays Quantitative Structure-Activity Relationship (QSAR) models to optimize a given set of properties. The molecules proposed by these algorithms are often revealed to be false positives, i.e. outside the true candidate drug target profile (CDTP), because the predictors are being hacked by the generative model during the optimization. By hacking we mean an over-optimization of the predicted score leading to an actual decrease or stagnation of the real score. This issue is reminiscent of adversarial examples in Machine Learning and it can be seen as evidence of the Goodhart’s law - “when a measure becomes a target, it ceases to be a good measure”. This issue is even more apparent in a multi-objective setting where the models need to extrapolate outside the train set distribution, because there are no known molecules satisfying all the objectives simultaneously in the initial train set. However, analysing this problem is very difficult and expensive since it requires synthesis and tests of the generated molecules. Consequently, efforts have been made to develop various kinds of in silico oracles - real-valued functions used as proxies for molecular properties, to help with the evaluation of these generative model-based pipelines. However, these oracles have had a limited value so far, as they are often too easy to model in comparison to biological assays, and are usually limited to mono-objective cases. In this work, we introduce a simulator of multi-target assays using a smartly initialized neural network (NN) which returns continuous values for any input molecule. We use this oracle to replicate a real-world prospective lead optimization scenario. First, we train predictive models on an initial small sample of molecules aimed at predicting their real oracle values. Second, we generate new optimized molecules using the open-source GuacaMol package, coupled with the previously built models. Finally, we select compounds which match the CDTP according to the predicted values and evaluate them by computing the true oracle values. We observe that even when the predictive models have excellent estimated performance metrics, the final selection still contains many false positives according to the NN-based oracle. We evaluate the optimization behavior in mono and bi-objective scenarios using either a logistic regression or a random forest predictive model. We also propose and evaluate several methods to help mitigate the hacking issue.