Our guidance regarding performance measures for medical AI models is finally out! - Stop bashing AUROC, although it does not settle things - Calibration and clinical utility are key - Show risk distributions - Classification statistics (e.g. F1) are improper www.thelancet.com/journals/lan...

Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance

Numerous measures have been proposed to illustrate the performance of predictive artificial intelligence (AI) models. Selecting appropriate performance measures is essential for predictive AI models i...

thelancet.com

In our latest work, we show that risk estimates for patients are HUGELY uncertain due to model, data, and population uncertainty. Even for well performing models (c statistic, calibration, utility) based on large N. @laure_wynants @ESteyerberg @lasaibarrenada.bsky.social arxiv.org/abs/2506.17141

The fundamental problem of risk prediction for individuals: health AI, uncertainty, and personalized medicine

Background: Clinical prediction models for a health condition are commonly evaluated regarding performance for a population, although decisions are made for individuals. The classic view relates uncer...

arxiv.org

What a great day yesterday! 🎉 Dr. Shan Gao and Dr. Elena Albu successfully defended their PhDs on the dynamic prediction of CLABSI using EHR data. A huge milestone after four years of dedicated research!

PROBAST+AI is out! A risk of bias tool for prediction models developed using any kind of analytical approach (stats/ml/ai/ds) Really proud of this collaboration with many prediction model experts around the world www.bmj.com/content/388/...

I will be presenting our recent work on individual risk estimation uncertainty at ENAR in New Orleans. Come say hi! Work with @benvancalster.bsky.social @laurewynants.bsky.social #DoranneThomassen #EwoutSteyerberg

Are you a statistician or health-economist who wants to develop new methods and who is interested in evaluating the role of diagnostic & prognostic prediction models and AI? We have an opening for a PhD position at Maastricht University! www.academictransfer.com/nl/jobs/3503...

PhD in Epidemiology on value-of-information from validating clinical prediction models and AI

Join Maastricht University as a PhD student in Epidemiology, advancing interdisciplinary methods to ensure reliable AI in healthcare through innovative statistical and health-economic approaches. Job ...

academictransfer.com

NEW PREPRINT 📊: We propose 3 methods to obtain flexible calibration plots while accounting for clustering: 1. Clustered Group Calibration (CG-C) 2. Two-Stage Meta-Analysis Calibration (2MA-C) 3. Mixed Model Calibration (MIX-C) Ready-to-use R code included!

Ben Van Calster@benvancalster.bsky.social · 2y ago

We tried to look at ways to obtain flexible calibration plots in clustered (e.g. multicenter) validation studies. Work with @lasaibarrenada.bsky.social @laurewynants.bsky.social @bavodccampo.bsky.social arxiv.org/abs/2503.08389

Happy to share the first paper of my PhD is published☺️! In case you like to use class imbalance corrections, maybe it is interesting. Let me know what you think! onlinelibrary.wiley.com/doi/10.1002/... Many thanks to @maartenvsmeden.bsky.social, @benvancalster.bsky.social, Anne, Kim and Carl !!

Time to say goodbye to the Julius center and the beautiful city of Utrecht 👋 In just 4 short months, I’ve gained invaluable experiences, connections, and good memories. Grateful for the opportunity to grow in such an inspiring place and special thanks to @maartenvsmeden.bsky.social for hosting me 💙

We have two vacancies for a postdoctoral and PhD researcher at Maastricht University. They will work on the development and application of value-of-information methods for the validation of clinical risk prediction models and medical AI. links below

My favourite thing about the paper is this summary table

Just out as preprint our overview of 32 performance measures for clinical prediction models. We recommend always including: AUROC, calibration plot, a clinical utility measure and a plot of probability distributions per outcome. Fully reproducible code in Pyhton and R: github.com/benvancalste...

Maarten van Smeden@maartenvsmeden.bsky.social · 2y ago

NEW PREPRINT A detailed overview of 32 popular predictive performance metrics for prediction models arxiv.org/abs/2412.10288

We've made the move across...so here’s a little thread introducing the various TRIPOD reporting guidelines to provide standards for #transparency and completeness of clinical prediction models to facilitate #reproducibility and guide #openscience #MLSky #StatsSky #statistics #academia #EpiSky

Happy to share the first article of my PhD, which is now available as a pre-proof! We looked at methods used to adjust existing (AI/ML) clinical prediction models to new contexts, like different hospitals, clinical domains or to a specific individual. Curious to hear your thoughts!😃

IMO it’s a mistake to give stats to 1st year med students. It’s not why they chose medicine & they resent it. Better to wait until they have developed some curiosity for it. I argued unsuccessfully for this during my time at UCL.