Supplemental Materials for

“Improving QSAR Modeling for Predictive Toxicology using Publicly Aggregated

Semantic Graph Data and Graph Neural Networks”

List of Authors:
Joseph D. Romano, Yun Hao, and Jason H. Moore

Contents: (All available digitally at https://doi.org/10.6084/m9.figshare.15094083)

- Supplemental Tables S1 and S2

- Assay weights for GNNs trained on PXR antagonism and HepG2 cell viability assays
- ROC plots for all 52 Tox21 assays investigated in the study
- Tabular summary of all models' performance on all 52 assays
Supplemental Tables S1 and S2

Table S1: Hyperparameter values used in the optimization of baseline QSAR models

Random forest
Hyperparameter
name Hyperparameter description Search values
n_estimators The number of trees in the forest 50, 100
The function to measure the quality of a
criterion split "gini", "entropy"
The proportion of features to consider when 0.1, 0.2, 0.3, 0.4, 0.5, 0.6,
max_features looking for the best split 0.7, 0.8, 0.9, 1.0
The minimum number of samples required
min_samples_split to split an internal node 2, 5, 8, 11, 14, 17, 20
The minimum number of samples required
min_samples_leaf to be at a leaf node 1, 6, 11, 16, 21
Whether bootstrap samples are used when
bootstrap building trees True, False

Gradient boosting
Hyperparameter
name Hyperparameter description Search values
n_estimators The number of gradient boosted trees 50, 100
max_depth The maximum tree depth for base learners. 1, 2, 3, ,4, 5, 6, ,7 ,8, 9, 10
learning_rate The boosting learning rate 0.1, 0.01, 0.001
The subsample ratio of the training 0.1, 0.2, 0.3, 0.4, 0.5, 0.6,
subsample instance. 0.7, 0.8, 0.9, 1.0
The minimum sum of instance
min_child_weight weight(hessian) needed in a child 1, 6, 11, 16, 21
Table S2: Model performance on all Tox21 datasets
rf: random forest; gb: gradient boosting; gnn: graph neural network;
ci: 95% confidence interval
rf_testin rf_testing rf_testi rf_testin gb_testi gb_testing gb_testi gb_testin gnn_testi gnn_testin gnn_test gnn_testin
dataset_name g_auc _auc_ci ng_f1 g_f1_ci ng_auc _auc_ci ng_f1 g_f1_ci ng_auc g_auc_ci ing_f1 g_f1_ci
0.5084
tox21-ahr-p1 0.89978 0.01928 7 0.05886 0.88861 0.02047 0.54032 0.05904 0.8669 0.02396 0.32075 0.06019
0.3827
tox21-ap1-agonist-p1 0.82962 0.03331 2 0.07827 0.84498 0.03139 0.41916 0.07127 0.85374 0.0297 0.22901 0.06766
tox21-ar-bla-agonist- 0.5581
p1 0.8956 0.04404 4 0.10109 0.89233 0.04923 0.61364 0.09339 0.81101 0.03522 0.10714 0.09091
tox21-ar-bla- 0.4680
antagonist-p1 0.89955 0.02495 9 0.07792 0.89045 0.02525 0.41135 0.07752 0.93596 0.01781 0.44138 0.08222
tox21-ar-mda-kb2- 0.6590
luc-agonist-p1 0.82134 0.06261 9 0.09097 0.8487 0.05075 0.63636 0.09441 0.95535 0.02471 0.58333 0.09259
tox21-ar-mda-kb2- 0.5846
luc-agonist-p3 0.89498 0.04969 2 0.10526 0.90891 0.0486 0.59016 0.12143 0.94958 0.02746 0.35294 0.13529
tox21-ar-mda-kb2- 0.2916
luc-antagonist-p1 0.8473 0.03429 7 0.09385 0.85661 0.03557 0.4 0.09259 0.94041 0.02815 0.55462 0.08318
tox21-ar-mda-kb2- 0.5847
luc-antagonist-p2 0.90128 0.01954 2 0.05372 0.90791 0.01708 0.58086 0.05272 0.93849 0.01407 0.64686 0.04839
0.3725
tox21-are-bla-p1 0.81717 0.02262 5 0.05078 0.80877 0.02398 0.41905 0.05297 0.89363 0.02025 0.56766 0.05055
0.4230
tox21-aromatase-p1 0.84047 0.02771 8 0.06176 0.83567 0.02936 0.43396 0.06089 0.8971 0.01871 0.14815 0.05364
0.4893
tox21-car-agonist-p1 0.88298 0.0182 6 0.05507 0.88107 0.01902 0.53061 0.05382 0.90875 0.0194 0.45299 0.06246
tox21-car-antagonist- 0.2926
p1 0.82154 0.06932 8 0.14807 0.78689 0.07129 0.28571 0.13282 0.93973 0.0258 0.20513 0.12296
0.0909
tox21-casp3-cho-p1 0.75407 0.07274 1 0.08333 0.77102 0.06979 0.17391 0.10567 0.94082 0.02373 0.08 0.07143
tox21-casp3-hepg2- 0.2592
p1 0.85683 0.03273 6 0.08965 0.85097 0.03404 0.29907 0.08919 0.94731 0.0175 0.49485 0.08814
tox21-elg1-luc- 0.2666
agonist-p1 0.79912 0.05625 7 0.12467 0.83224 0.04461 0.22951 0.11261 0.9115 0.03241 0.04545 0.04545
tox21-er-bla-agonist- 0.4137
p2 0.8332 0.04015 9 0.09107 0.82878 0.04496 0.44643 0.09173 0.93143 0.01674 0.17722 0.08877
tox21-er-bla- 0.2526
antagonist-p1 0.85684 0.03458 3 0.08846 0.8497 0.0337 0.26 0.08302 0.96474 0.0151 0.68468 0.07838
tox21-er-luc-bg1-4e2- 0.3402
agonist-p2 0.73804 0.03579 1 0.06828 0.73633 0.03717 0.38342 0.07068 0.83942 0.03402 0.48571 0.06377
tox21-er-luc-bg1-4e2- 0.7441
agonist-p4 0.89593 0.06677 9 0.11273 0.95491 0.0314 0.66667 0.13043 0.73801 0.0427 0 0
tox21-er-luc-bg1-4e2- 0.3469
antagonist-p1 0.86175 0.03632 4 0.09875 0.8611 0.03434 0.33333 0.09836 0.96801 0.01471 0.60784 0.09025
tox21-er-luc-bg1-4e2- 0.4087
antagonist-p2 0.87708 0.03045 6 0.0849 0.88678 0.03045 0.44776 0.07955 0.97242 0.01025 0.70064 0.06095
tox21-erb-bla-
antagonist-p1 0.86532 0.03082 0.3252 0.08 0.88271 0.02789 0.34711 0.08988 0.97551 0.00789 0.6338 0.06927
0.4242
tox21-erb-bla-p1 0.84536 0.09531 4 0.17647 0.94669 0.03636 0.45161 0.1875 0.80509 0.0629 0 0
tox21-esre-bla-p1 0.74887 0.09244 0 0 0.79801 0.08291 0.08 0.08333 0.67822 0.08051 0 0
tox21-fxr-bla-agonist-
p2 0.77346 0.09536 0 0 0.81434 0.08229 0.14815 0.13333 0.84239 0.032 0 0
tox21-fxr-bla- 0.1818
antagonist-p1 0.87103 0.04376 2 0.10714 0.85865 0.04514 0.32653 0.1349 0.96708 0.01504 0.41379 0.1294
tox21-gh3-tre-
antagonist-p1 0.80723 0.04122 0.2069 0.0899 0.81287 0.03581 0.2093 0.09154 0.91629 0.01888 0 0
tox21-gr-hela-bla- 0.4313
agonist-p1 0.7903 0.06808 7 0.1288 0.78399 0.06954 0.46154 0.12818 0.95413 0.02191 0.23256 0.1369
tox21-gr-hela-bla- 0.4761
antagonist-p1 0.88454 0.03049 9 0.0965 0.86407 0.03654 0.45283 0.09683 0.87053 0.0241 0.075 0.0625
0.2727
tox21-h2ax-cho-p2 0.83174 0.03737 3 0.08409 0.83059 0.03646 0.20561 0.07692 0.92044 0.02295 0.43299 0.09827
tox21-hdac-p1 0.89796 0.0302 0.3038 0.10517 0.91063 0.02648 0.37647 0.10104 0.9047 0.03069 0.03226 0.02941
tox21-hre-bla- 0.1428
agonist-p1 0.82669 0.1143 6 0.16667 0.88856 0.07768 0 0 0.71426 0.03908 0 0
0.3893
tox21-hse-bla-p1 0.80206 0.0441 8 0.08772 0.82349 0.03893 0.4 0.09 0.93973 0.02035 0.53448 0.08547
0.6005
tox21-mitotox-p1 0.88281 0.01694 1 0.04428 0.89843 0.01528 0.62626 0.04255 0.90632 0.01674 0.62051 0.04125
0.3181
tox21-p53-bla-p1 0.8427 0.03397 8 0.07971 0.8711 0.02825 0.31746 0.0875 0.93852 0.022 0.48921 0.08358
tox21-ppard-bla- 0.1666
agonist-p1 0.68492 0.10506 7 0.15385 0.77825 0.09763 0.08696 0.09091 0.8946 0.05101 0 0
tox21-ppard-bla- 0.3636
antagonist-p1 0.72382 0.12771 4 0.26667 0.8557 0.07899 0.28571 0.23077 0.56267 0.03096 0 0
tox21-pparg-bla- 0.2758
agonist-p1 0.80126 0.06195 6 0.11024 0.83595 0.05389 0.26667 0.10729 0.86429 0.0442 0.08696 0.07692
tox21-pparg-bla- 0.2888
antagonist-p1 0.8293 0.04291 9 0.09706 0.81625 0.04233 0.31818 0.09844 0.86093 0.02253 0 0
tox21-pr-bla-agonist- 0.6938
p1 0.92458 0.06401 8 0.12414 0.95384 0.04009 0.75556 0.12088 0.77499 0.07112 0.1 0.1
tox21-pr-bla- 0.4666
antagonist-p1 0.87563 0.0212 7 0.05991 0.88493 0.01921 0.51079 0.05579 0.96138 0.00851 0.67845 0.04525
0.6623
tox21-pxr-p1 0.88224 0.01593 8 0.0345 0.88878 0.01553 0.67097 0.03418 0.94257 0.01159 0.76252 0.02833
tox21-rar-agonist-p1 0.84572 0.03738 0.3622 0.08077 0.8809 0.03206 0.40625 0.08554 0.93555 0.01984 0.26966 0.09524
tox21-rar-antagonist- 0.2933
p2 0.83001 0.02983 3 0.07216 0.83546 0.02997 0.28 0.07314 0.96784 0.00945 0.67045 0.05926
tox21-ror-cho- 0.3496
antagonist-p1 0.87547 0.02612 5 0.07858 0.87486 0.02645 0.34014 0.07899 0.9739 0.00924 0.70886 0.06101
tox21-rt-viability- 0.3243
hepg2-p1 0.78554 0.02518 2 0.05105 0.77971 0.02646 0.36795 0.05089 0.84658 0.02444 0.51014 0.04766
tox21-rxr-bla-agonist- 0.2553
p1 0.6507 0.08309 2 0.13988 0.68304 0.07653 0.21739 0.12963 0.75797 0.0648 0.04348 0.04167
tox21-sbe-bla-
antagonist-p1 0.82841 0.03736 0.2906 0.08293 0.85357 0.03139 0.34146 0.08553 0.65291 0.02879 0 0
tox21-shh-3t3-gli3- 0.5185
antagonist-p1 0.87397 0.02125 2 0.05338 0.88251 0.01917 0.53595 0.04854 0.88654 0.01762 0.07442 0.03852
0.3863
tox21-tshr-agonist-p1 0.80459 0.05248 6 0.09774 0.86313 0.03648 0.40449 0.09906 0.91746 0.02484 0.20588 0.09872
tox21-tshr- 0.1562
antagonist-p1 0.87416 0.0398 5 0.09631 0.84908 0.04656 0.25714 0.1096 0.87863 0.03322 0.04 0.03846
tox21-vdr-bla- 0.1333
antagonist-p1 0.80794 0.13028 3 0.14286 0.83252 0.0693 0.125 0.125 0.79922 0.03534 0 0
Assay weights for GNNs trained on PXR antagonism and HepG2 cell viability
assays
Assay weights are available in two tab-separated value (TSV) files (one for each of the PXR and RT
viability assays) on Figshare at https://figshare.com/ndownloader/files/29021178
ROC plots for all 52 Tox21 assays investigated in the study
Receiver Operating Characteristic curve plots are available on FigShare as PDF files at
https://figshare.com/ndownloader/files/29021181
Tabular summary of all models' performance on all 52 assays
dataset_name rf_test rf_testin rf_tes rf_testi gb_test gb_testi gb_tes gb_testi gnn_tes gnn_testi gnn_te gnn_test
ing_au g_auc_c ting_f ng_f1_ ing_au ng_auc_ ting_f ng_f1_c ting_au ng_auc_c sting_f ing_f1_c
c i 1 ci c ci 1 i c i 1 i
tox21-ahr-p1 0.8997 0.01928 0.508 0.0588 0.8886 0.02047 0.5403 0.05904 0.8669 0.02396 0.3207 0.06019
8 47 6 1 2 5
tox21-ap1- 0.8296 0.03331 0.382 0.0782 0.8449 0.03139 0.4191 0.07127 0.85374 0.0297 0.2290 0.06766
agonist-p1 2 72 7 8 6 1
tox21-ar-bla- 0.8956 0.04404 0.558 0.1010 0.8923 0.04923 0.6136 0.09339 0.81101 0.03522 0.1071 0.09091
agonist-p1 14 9 3 4 4
tox21-ar-bla- 0.8995 0.02495 0.468 0.0779 0.8904 0.02525 0.4113 0.07752 0.93596 0.01781 0.4413 0.08222
antagonist-p1 5 09 2 5 5 8
tox21-ar-mda- 0.8213 0.06261 0.659 0.0909 0.8487 0.05075 0.6363 0.09441 0.95535 0.02471 0.5833 0.09259
kb2-luc-agonist- 4 09 7 6 3
p1
tox21-ar-mda- 0.8949 0.04969 0.584 0.1052 0.9089 0.0486 0.5901 0.12143 0.94958 0.02746 0.3529 0.13529
kb2-luc-agonist- 8 62 6 1 6 4
p3
tox21-ar-mda- 0.8473 0.03429 0.291 0.0938 0.8566 0.03557 0.4 0.09259 0.94041 0.02815 0.5546 0.08318
kb2-luc- 67 5 1 2
antagonist-p1
tox21-ar-mda- 0.9012 0.01954 0.584 0.0537 0.9079 0.01708 0.5808 0.05272 0.93849 0.01407 0.6468 0.04839
kb2-luc- 8 72 2 1 6 6
antagonist-p2
tox21-are-bla-p1 0.8171 0.02262 0.372 0.0507 0.8087 0.02398 0.4190 0.05297 0.89363 0.02025 0.5676 0.05055
7 55 8 7 5 6
tox21- 0.8404 0.02771 0.423 0.0617 0.8356 0.02936 0.4339 0.06089 0.8971 0.01871 0.1481 0.05364
aromatase-p1 7 08 6 7 6 5
tox21-car- 0.8829 0.0182 0.489 0.0550 0.8810 0.01902 0.5306 0.05382 0.90875 0.0194 0.4529 0.06246
agonist-p1 8 36 7 7 1 9
tox21-car- 0.8215 0.06932 0.292 0.1480 0.7868 0.07129 0.2857 0.13282 0.93973 0.0258 0.2051 0.12296
antagonist-p1 4 68 7 9 1 3
tox21-casp3- 0.7540 0.07274 0.090 0.0833 0.7710 0.06979 0.1739 0.10567 0.94082 0.02373 0.08 0.07143
cho-p1 7 91 3 2 1
tox21-casp3- 0.8568 0.03273 0.259 0.0896 0.8509 0.03404 0.2990 0.08919 0.94731 0.0175 0.4948 0.08814
hepg2-p1 3 26 5 7 7 5
tox21-elg1-luc- 0.7991 0.05625 0.266 0.1246 0.8322 0.04461 0.2295 0.11261 0.9115 0.03241 0.0454 0.04545
agonist-p1 2 67 7 4 1 5
tox21-er-bla- 0.8332 0.04015 0.413 0.0910 0.8287 0.04496 0.4464 0.09173 0.93143 0.01674 0.1772 0.08877
agonist-p2 79 7 8 3 2
tox21-er-bla- 0.8568 0.03458 0.252 0.0884 0.8497 0.0337 0.26 0.08302 0.96474 0.0151 0.6846 0.07838
antagonist-p1 4 63 6 8
tox21-er-luc- 0.7380 0.03579 0.340 0.0682 0.7363 0.03717 0.3834 0.07068 0.83942 0.03402 0.4857 0.06377
bg1-4e2-agonist- 4 21 8 3 2 1
p2
tox21-er-luc- 0.8959 0.06677 0.744 0.1127 0.9549 0.0314 0.6666 0.13043 0.73801 0.0427 0 0
bg1-4e2-agonist- 3 19 3 1 7
p4
tox21-er-luc- 0.8617 0.03632 0.346 0.0987 0.8611 0.03434 0.3333 0.09836 0.96801 0.01471 0.6078 0.09025
bg1-4e2- 5 94 5 3 4
antagonist-p1
tox21-er-luc- 0.8770 0.03045 0.408 0.0849 0.8867 0.03045 0.4477 0.07955 0.97242 0.01025 0.7006 0.06095
bg1-4e2- 8 76 8 6 4
antagonist-p2
tox21-erb-bla- 0.8653 0.03082 0.325 0.08 0.8827 0.02789 0.3471 0.08988 0.97551 0.00789 0.6338 0.06927
antagonist-p1 2 2 1 1
tox21-erb-bla-p1 0.8453 0.09531 0.424 0.1764 0.9466 0.03636 0.4516 0.1875 0.80509 0.0629 0 0
6 24 7 9 1
tox21-esre-bla- 0.7488 0.09244 0 0 0.7980 0.08291 0.08 0.08333 0.67822 0.08051 0 0
p1 7 1
tox21-fxr-bla- 0.7734 0.09536 0 0 0.8143 0.08229 0.1481 0.13333 0.84239 0.032 0 0
agonist-p2 6 4 5
tox21-fxr-bla- 0.8710 0.04376 0.181 0.1071 0.8586 0.04514 0.3265 0.1349 0.96708 0.01504 0.4137 0.1294
antagonist-p1 3 82 4 5 3 9
tox21-gh3-tre- 0.8072 0.04122 0.206 0.0899 0.8128 0.03581 0.2093 0.09154 0.91629 0.01888 0 0
antagonist-p1 3 9 7
tox21-gr-hela- 0.7903 0.06808 0.431 0.1288 0.7839 0.06954 0.4615 0.12818 0.95413 0.02191 0.2325 0.1369
bla-agonist-p1 37 9 4 6
tox21-gr-hela- 0.8845 0.03049 0.476 0.0965 0.8640 0.03654 0.4528 0.09683 0.87053 0.0241 0.075 0.0625
bla-antagonist- 4 19 7 3
p1
tox21-h2ax-cho- 0.8317 0.03737 0.272 0.0840 0.8305 0.03646 0.2056 0.07692 0.92044 0.02295 0.4329 0.09827
p2 4 73 9 9 1 9
tox21-hdac-p1 0.8979 0.0302 0.303 0.1051 0.9106 0.02648 0.3764 0.10104 0.9047 0.03069 0.0322 0.02941
6 8 7 3 7 6
tox21-hre-bla- 0.8266 0.1143 0.142 0.1666 0.8885 0.07768 0 0 0.71426 0.03908 0 0
agonist-p1 9 86 7 6
tox21-hse-bla-p1 0.8020 0.0441 0.389 0.0877 0.8234 0.03893 0.4 0.09 0.93973 0.02035 0.5344 0.08547
6 38 2 9 8
tox21-mitotox- 0.8828 0.01694 0.600 0.0442 0.8984 0.01528 0.6262 0.04255 0.90632 0.01674 0.6205 0.04125
p1 1 51 8 3 6 1
tox21-p53-bla- 0.8427 0.03397 0.318 0.0797 0.8711 0.02825 0.3174 0.0875 0.93852 0.022 0.4892 0.08358
p1 18 1 6 1
tox21-ppard-bla- 0.6849 0.10506 0.166 0.1538 0.7782 0.09763 0.0869 0.09091 0.8946 0.05101 0 0
agonist-p1 2 67 5 5 6
tox21-ppard-bla- 0.7238 0.12771 0.363 0.2666 0.8557 0.07899 0.2857 0.23077 0.56267 0.03096 0 0
antagonist-p1 2 64 7 1
tox21-pparg-bla- 0.8012 0.06195 0.275 0.1102 0.8359 0.05389 0.2666 0.10729 0.86429 0.0442 0.0869 0.07692
agonist-p1 6 86 4 5 7 6
tox21-pparg-bla- 0.8293 0.04291 0.288 0.0970 0.8162 0.04233 0.3181 0.09844 0.86093 0.02253 0 0
antagonist-p1 89 6 5 8
tox21-pr-bla- 0.9245 0.06401 0.693 0.1241 0.9538 0.04009 0.7555 0.12088 0.77499 0.07112 0.1 0.1
agonist-p1 8 88 4 4 6
tox21-pr-bla- 0.8756 0.0212 0.466 0.0599 0.8849 0.01921 0.5107 0.05579 0.96138 0.00851 0.6784 0.04525
antagonist-p1 3 67 1 3 9 5
tox21-pxr-p1 0.8822 0.01593 0.662 0.0345 0.8887 0.01553 0.6709 0.03418 0.94257 0.01159 0.7625 0.02833
4 38 8 7 2
tox21-rar- 0.8457 0.03738 0.362 0.0807 0.8809 0.03206 0.4062 0.08554 0.93555 0.01984 0.2696 0.09524
agonist-p1 2 2 7 5 6
tox21-rar- 0.8300 0.02983 0.293 0.0721 0.8354 0.02997 0.28 0.07314 0.96784 0.00945 0.6704 0.05926
antagonist-p2 1 33 6 6 5
tox21-ror-cho- 0.8754 0.02612 0.349 0.0785 0.8748 0.02645 0.3401 0.07899 0.9739 0.00924 0.7088 0.06101
antagonist-p1 7 65 8 6 4 6
tox21-rt- 0.7855 0.02518 0.324 0.0510 0.7797 0.02646 0.3679 0.05089 0.84658 0.02444 0.5101 0.04766
viability-hepg2- 4 32 5 1 5 4
p1
tox21-rxr-bla- 0.6507 0.08309 0.255 0.1398 0.6830 0.07653 0.2173 0.12963 0.75797 0.0648 0.0434 0.04167
agonist-p1 32 8 4 9 8
tox21-sbe-bla- 0.8284 0.03736 0.290 0.0829 0.8535 0.03139 0.3414 0.08553 0.65291 0.02879 0 0
antagonist-p1 1 6 3 7 6
tox21-shh-3t3- 0.8739 0.02125 0.518 0.0533 0.8825 0.01917 0.5359 0.04854 0.88654 0.01762 0.0744 0.03852
gli3-antagonist- 7 52 8 1 5 2
p1
tox21-tshr- 0.8045 0.05248 0.386 0.0977 0.8631 0.03648 0.4044 0.09906 0.91746 0.02484 0.2058 0.09872
agonist-p1 9 36 4 3 9 8
tox21-tshr- 0.8741 0.0398 0.156 0.0963 0.8490 0.04656 0.2571 0.1096 0.87863 0.03322 0.04 0.03846
antagonist-p1 6 25 1 8 4
tox21-vdr-bla- 0.8079 0.13028 0.133 0.1428 0.8325 0.0693 0.125 0.125 0.79922 0.03534 0 0
antagonist-p1 4 33 6 2