All Schrödinger offices worldwide will be closed for the week of August 17-21 as part of a company-wide initiative to rest and recharge. Please expect limited responses during this time. Scientific and Technical Support team members will be available to answer emergency support issues only.

Contact Us

Debugging a large ML outlier

I built a transfer learning model to predict LogD for a series (using ChEMBL as the source of the local data) and one compound came back four log units off. When that happens there's no quick way to tell whether the model failed or the training label itself is wrong. Checking a deposition by hand against the source paper is slow enough that bad labels usually just stay in the set.

Published 3 min read

Initial Prompt

I built a transfer learning ML model [in LiveDesign ML] and CHEMBL5082701 is a very big outlier – the transfer learning model predicted it to be -2 – any reason you can guess for such a big outlier?

Bunsen-generated plan

Diagnosis

Compare the flagged compound against its closest structural neighbours in the series, to establish whether the prediction or the measured label is the thing out of place.

Check the deposited value against independent sources, since a model failure and a label error produce the same residual and cannot be told apart from it.

Validation

Quantify how far the discrepancy sits outside normal assay variation, using matched pairs from the wider public record rather than an assumption about what the assay can resolve.

Confirm against the primary publication, and audit the remaining records in the same deposition if a fault is found, because transcription errors rarely occur alone.

Clarifications

The impossibility argument was too strong as first stated. The claim that enantiomers must share a LogD holds in an achiral octanol/buffer system but not universally, and the user asked for it to be confirmed rather than asserted. Rather than lean on the principle, the discrepancy was measured against an empirical baseline of real enantiomer pairs.

The first baseline was contaminated by diastereomers. Matching on the InChIKey connectivity layer returned 440 pairs, including partial-inversion diastereomers that can legitimately differ in LogD. Restricting to pairs with every stereocentre inverted cut the set to 248 and dropped the maximum observed difference from 8.2 to 2.0 log units.

LipE settled the two ambiguous cases. Three of five flagged errors showed a clean tenfold pattern. For the other two the user proposed using LipE, which is derived from LogD, as an internal check.

Outcomes

What was run

  • ChEMBL web services retrieved the 33 LogD records for the Pfizer Vanin-1 series, all from assay CHEMBL5047677 (octanol/buffer, shake-flask), plus the full activity record for the flagged compound and its mirror image.
  • RDKit computed descriptors and InChIKeys for structure matching, splitting each key into its connectivity and stereochemistry layers so that stereoisomer pairs could be identified independently of how each compound was registered.
  • find_enantiomer_pairs.py and filter_enantiomers.py surveyed 104 ChEMBL assays whose descriptions name a shake-flask method, extracted 440 stereoisomer pairs measured in the same assay, then filtered to the 248 pairs where all stereocentres invert.
  • full_crossref.py and lipe_verdict.py matched all 33 paper structures to ChEMBL entries by InChIKey against tables rendered from the source PDF, then back-calculated LogD from the published pIC50 and LipE values.

What it produced

  • CHEMBL5082701 and CHEMBL5077966 are exact mirror images with identical cLogP (2.08), TPSA (93.1) and molecular weight (368.4), yet carry LogD values of 2.0 and 0.2 from the same assay under consecutive activity IDs. PubChem reports XLogP 0.7 for both enantiomers and for the racemate.

Figure 1. The two enantiomers carry identical achiral descriptors but differ by 1.8 log units in the deposited LogD. 

  • Across 248 true enantiomer pairs the median difference is 0.100 log units and the mean is 0.146. Only five pairs reach 1.0 or above, placing 1.8 at the 99.6th percentile.

Figure 2. Distribution of absolute LogD differences between 248 enantiomer pairs measured by shake-flask methods in ChEMBL.

  • Table 6 of the source publication reports 0.2 for both enantiomers. Back-calculating LogD as pIC50 minus LipE reproduces the published LogD for all 19 undisputed compounds and returns 0.27 for the flagged one.
  • The audit found five wrong values in the same deposition, all inflating the magnitude: CHEMBL5087511 (2.0 for 0.8), CHEMBL5092354 (-3.0 for -0.3), CHEMBL5081075 (-2.0 for -0.2), CHEMBL5082701 (2.0 for 0.2) and CHEMBL5090508 (1.0 for 0.2).

Caveats

  • The LipE reconstruction depends on the paper’s own values being internally consistent, which cannot be checked against raw assay data.
  • Structure matching for two of the five relied on visual reading of rendered table images, confirmed independently by the user.
  • The baseline covers only assays whose ChEMBL descriptions name a shake-flask method, so it is a subset of all deposited LogD.
  • The corrections have not yet been submitted to ChEMBL curation.

Takeaways

The model was not wrong about this compound. Its training label was. A residual of four log units on a single point turned out to be a decimal transcription error in the public record, and the prediction of -2 was a reasonable read of a molecule whose true measured value is 0.2. The generalisable part is that a large isolated residual is worth auditing as a data-integrity question before it is explained as chemistry, particularly where a compound has a registered stereoisomer available as a free consistency check. The error was also not isolated: five of 33 records in one deposition carry the same class of fault.

Next steps

  • Correct the five LogD values in the training set and retrain, then check whether the other flagged outliers persist.
  • Submit the five corrections to ChEMBL curation with the paper table as evidence.
  • Run the enantiomer-pair consistency check across the rest of the training data before the next model build.