RWD116 Training and validation of CARAai™: A multi LLM platform and data model to address oncology-specific challenges in clinical data extraction

We validated the performance of the CARAaiTM models based on precision, recall, and the F1 score (the harmonic mean of precision and recall) using 50,000 patients across 13 solid tumor types (80% training set and 20% testing set). The same records processed via oncology-domain trained human clinical abstraction were used as the gold standard.

For performance status, tumor stage, histology, tumor grade, procedure type, metastatic diagnosis and medication, precision was >0.90 (±0.05), recall ranged from 0.91-0.99, and F1 scores were >0.95. Precision, recall and F1 scores were 0.95, 0.98, and 0.96 for biomarker names, 0.87, 0.84, and 0.85 for biomarker categorical results, and 0.86, 0.94, and 0.90 for biomarker numeric test results.

The CARAaiTM LLM suite achieved high precision with respect to human curation for oncology key data elements allowing larger data sets with lower latency. The CARAaiTM LLM models will facilitate improved statistical power and timeliness for HEOR and epidemiological studies on outcomes and safety.