Poster presentation at:
The 11th Conference on Retroviruses and Opportunistic Infections
San Francisco, CA, USA; 8-11 February 2004

Accuracy of Neural Network Models in Predicting HIV Treatment Response from Genotype May Depend on Diversity as well as Size of Data Sets

B.A. Larder 1 , D Wang 1 , A Revell 2 , R Harrigan 3 , J Montaner 3 and C. Lane 4 .

1: The HIV Resistance Response Database Initiative (RDI), Cambridge , UK ; 2: RDI Ltd, London , UK ; 3: The BC Centre for Excellence in HIV/AIDS Vancouver , Canada ; 4: National Institute of Allergy and Infectious Diseases (NIAID), Bethesda , USA .

Background: The RDI has trained Artificial Neural Network (ANN) models using clinical data which, when tested with independent data from the same clinical setting, have demonstrated accuracy in predicting virological response to antiretroviral therapy from genotype. Here we examine the performance of ANN models developed using datasets of increasing size pooled from several clinical settings.

Methods: 228 'treatment change episodes' (TCEs) were obtained from 153 patients in the NIAID cohort: episodes with a genotype 12 weeks and a VL 8 before treatment change and follow-up VL within 4-40 weeks. Four ANN models were developed; each trained using 171 TCE's and tested with the remaining 57. Results were compared to those obtained with 13 ANN models using the NIAID data plus data from a second cohort (BC), yielding 747 TCE's. The data were finally added to data from multiple sources in the RDI database, yielding 1,581 TCEs and 10 ANN models trained, and tested. All the training and testing sub-datasets were independent, randomly partitioned and normalised.

Results: Correlation between the predicted and actual VL change for the four ANN models developed from the NIAID cohort gave a mean value of 0.71 and mean correct trajectory prediction rate of 76%. The 13 ANN models from the combined NIAID and BC cohorts gave a mean value of 0.65 and mean correct trajectory prediction rate of 78%. The ten models developed from the largest composite dataset gave a mean of 0.55 and a mean correct trajectory prediction rate of 74%. The value for the composite dataset was significantly lower than for the other two datasets.

Conclusions: These results indicate that ANN models trained using limited data from one clinical setting can be surprisingly accurate in predicting responses to treatment from genotype for other patients from the same cohort. Models developed using considerably more data from a variety of settings are not automatically more accurate, when tested with data from multiple settings. Preliminary analysis suggests that the accuracy of ANN models may depend on the training and test data being similarly diverse, although this does not necessarily imply similarity in terms of specific mutations or drugs. These results confirm the potential of ANN models as an aid to treatment decision-making but underline the need to collect sufficient diversity of data to develop models that are able to predict treatment response for disparate test cases.