Soledad Galli, PhD
@solegalli.bsky.social
Data scientist, best selling instructor, book author, Python 🐍 open-source developer (check out Feature-engine). Find out more at Train in Data: https://www.trainindata.com/
By 2018, researchers had developed at least 90 SMOTE-based oversampling methods, together with 17 SMOTE-based ensemble methods. The creativity appears to be endless. Shame that reducing the threshold resolves the issue without the hassle.
View of SMOTE for Learning from Imbalanced Data: Progress and Challenges, Marking the 15-year Anniversary
jair.org
Most people assume class_weight only changes training. Set the weights, done, probabilities stay untouched. This thread walks through why that assumption breaks down, and what actually happens to your predicted probabilities once class_weight enters the picture.
Want to know what Python loops are? This was a fun video to create, too basic for what I normally teach, but, since I made it, I thought I'd share it
for loops in Python
Find out what for loops in Python are about and why they are so cool!
youtu.be
I can safely conclude that if you leave book writing to generative AI, you'll end up with a mediocre book, that lacks direction, clarity, key messages and a coherent story line. Even when you give the machine the exact papers you want to discuss and the chapter structure.
Feature selection is a skill worth investing in. My ebook takes you through the full toolkit with practical examples so you can build leaner, faster, more interpretable models 👉 https://www.trainindata.com/p/feature-selection-in-machine-learning-book
My most watched @YouTube video shows how to create features for time series forecasting. In case you missed it👇
Lag Features | Feature Engineering for Time Series Forecasting
In this video, we introduce lag features for time series analysis, ...
youtube.com
Class weights and resampling shift the decision boundary so that 0.5 works. But you could train on the original distribution and just lower the threshold. Machine learning, chasing its own tail. 🐕🔄
Zoomies Spinning GIF by Impact Dog Crates - Find & Share on GIPHY
Discover & share this Zoomies Spinning GIF by Impact Dog Crates with everyone you know. GIPHY is how you search, share, discover, and create GIFs.
tinyurl.com
Platt Scaling or Isotonic Regression? Under ~1,000 samples, use Platt. With more data, Isotonic wins. Wrong choice means worse probabilities than before. More in my new book
If you use predict_proba for anything beyond ranking (pricing, risk scores, thresholds), this is worth ten minutes. Read the thread and top answer:
Do you need to adjust the probability if you use the 'class_weight' parameter in LogisticRegression-sklearn?
I have a imbalanced dataset and I want the the output as probabilities and not labels. Hence using Logistic Regression seemed to be the obvious choice. However the classsifer started predicting all
stats.stackexchange.com
My next newsletter "The ROC Curve Myth for Imbalanced Datasets" is coming Monday. Subscribe to get it and stay in the loop on everything I'm creating. https://tech.trainindata.com
Do you need to adjust the probability if you use the class_weight parameter? This question comes up constantly, especially with people just starting out with imbalanced data. Found a stats stack exchange thread that finally puts it to rest
class_weight changes the loss function during training. It penalizes mistakes on the minority class more heavily, shifting where the model draws its boundary. The predicted probabilities shift too, and stop reflecting the true class balance in your data.
7 Takes on Imbalanced Data just got a refresh. I read through five years of research so you don't have to, and pulled out the findings that change how you should handle imbalance.
7 Takes on Working with Imbalanced Data
Read Sole's discussion of three recent articles that shift the conversation around handling imbalanced datasets. These articles explore whether generating synthetic data is effective with current state-of-the-art algorithms, and examine the implications of using synthetic data on model performance and deployment.
trainindata.com
Cleaner pipelines, faster training, more interpretable models. Feature selection is where to start. Link to the book 👉 https://www.lulu.com/shop/soledad-galli/feature-selection-in-machine-learning/paperback/product-dye7gkd.html
Working with imbalanced data and tired of recycled advice from ten years ago? I went through the most relevant research from the last five years and summarized the key takeaways in our free booklet.
7 Takes on Working with Imbalanced Data
Read Sole's discussion of three recent articles that shift the conversation around handling imbalanced datasets. These articles explore whether generating synthetic data is effective with current state-of-the-art algorithms, and examine the implications of using synthetic data on model performance and deployment.
trainindata.com
I just updated our free booklet, 7 Takes on Imbalanced Data. I break down the most important research on imbalanced data from the last five years and what it means for your models. Free to grab here:
7 Takes on Working with Imbalanced Data
Read Sole's discussion of three recent articles that shift the conversation around handling imbalanced datasets. These articles explore whether generating synthetic data is effective with current state-of-the-art algorithms, and examine the implications of using synthetic data on model performance and deployment.
trainindata.com
Two models can separate your classes perfectly and both score a ROC AUC of 1. Yet one pushes its probabilities much closer to the true labels than the other. Same ranking. Different log loss.
Intuition behind ROC-AUC score | Towards Data Science
In Machine Learning, classification problem refers to predictive modeling where a class label needs to be predicted for a given...
towardsdatascience.com
ROC AUC doesn't care how confident your model is, only whether it ranked positives above negatives. Log loss cares about both ranking and confidence. That's why a model can keep the same ROC AUC while its log loss gets worse.
Now flip it. When ROC AUC improves, your model got better at separating the two classes. That shift usually pushes probabilities closer to true labels, lowering log loss. Better ranking tends to drag calibration along. Better calibration doesn't drag ranking along.
Feature selection reduces model complexity, speeds up training, and makes results easier to explain. My ebook covers all the main methods with Python code. Get your copy 👉 https://www.trainindata.com/p/feature-selection-in-machine-learning-book
ROC AUC only cares about the order of your predicted probabilities: did you rank positives above negatives? Log loss cares about the actual values of those probabilities. You can recalibrate a model, lower the log loss, and never touch the ranking. Same AUC. Better log loss.
Quick machine learning quiz. If log loss goes down, does ROC AUC go up? Not necessarily. If ROC AUC goes up, does log loss go down? Almost always, yes. Want to know more about metrics? Check my book on imbalanced data
I read this churn paper twice, looking for one thing: RF and XGBoost trained without SMOTE, ADASYN or GNUS. Not there. Six models, three upsampling methods, and the one baseline that would tell us if resampling helped is missing.
Comprehensive Analysis of Random Forest and XGBoost Performance with SMOTE, ADASYN, and GNUS Under Varying Imbalance Levels
This study examines the efficacy of Random Forest and XGBoost classifiers in conjunction with three upsampling techniques—SMOTE, ADASYN, and Gaussian noise upsampling (GNUS)—across datasets with varying class imbalance levels, ranging from moderate to extreme (15% to 1% churn rate). Employing metrics such as F1 score, ROC AUC, PR AUC, Matthews Correlation Coefficient (MCC), and Cohen’s Kappa, this research provides a comprehensive evaluation of classifier performance under different imbalance scenarios, focusing on applications in the telecommunications domain. The findings highlight that tuned XGBoost paired with SMOTE (Tuned_XGB_SMOTE) consistently achieves the highest F1 score and robust performance across all imbalance levels. SMOTE emerged as the most effective upsampling method, particularly when used with XGBoost, whereas Random Forest performed poorly under severe imbalance. ADASYN showed moderate effectiveness with XGBoost but underperformed with Random Forest, and GNUS produced inconsistent results. This study underscores the impact of data imbalance, with MCC, Kappa, and F1 scores fluctuating significantly, whereas ROC AUC and PR AUC remained relatively stable. Moreover, rigorous statistical analyses employing the Friedman test and Nemenyi post hoc comparisons confirmed that the observed improvements in F1 score, PR-AUC, Kappa, and MCC were statistically significant (p < 0.05), with Tuned_XGB_SMOTE significantly outperforming Tuned_RF_GNUS. While differences in ROC-AUC were not significant, the consistency of these results across multiple performance metrics underscores the reliability of our framework, offering a statistically validated and attractive solution for model selection in imbalanced classification scenarios.
mdpi.com
Imagine testing three sunscreens and never checking skin with none at all. That's what this churn paper did: RF and XGBoost combined with SMOTE, ADASYN and GNUS, but neither model is ever reported without resampling.
Comprehensive Analysis of Random Forest and XGBoost Performance with SMOTE, ADASYN, and GNUS Under Varying Imbalance Levels
This study examines the efficacy of Random Forest and XGBoost classifiers in conjunction with three upsampling techniques—SMOTE, ADASYN, and Gaussian noise upsampling (GNUS)—across datasets with varying class imbalance levels, ranging from moderate to extreme (15% to 1% churn rate). Employing metrics such as F1 score, ROC AUC, PR AUC, Matthews Correlation Coefficient (MCC), and Cohen’s Kappa, this research provides a comprehensive evaluation of classifier performance under different imbalance scenarios, focusing on applications in the telecommunications domain. The findings highlight that tuned XGBoost paired with SMOTE (Tuned_XGB_SMOTE) consistently achieves the highest F1 score and robust performance across all imbalance levels. SMOTE emerged as the most effective upsampling method, particularly when used with XGBoost, whereas Random Forest performed poorly under severe imbalance. ADASYN showed moderate effectiveness with XGBoost but underperformed with Random Forest, and GNUS produced inconsistent results. This study underscores the impact of data imbalance, with MCC, Kappa, and F1 scores fluctuating significantly, whereas ROC AUC and PR AUC remained relatively stable. Moreover, rigorous statistical analyses employing the Friedman test and Nemenyi post hoc comparisons confirmed that the observed improvements in F1 score, PR-AUC, Kappa, and MCC were statistically significant (p < 0.05), with Tuned_XGB_SMOTE significantly outperforming Tuned_RF_GNUS. While differences in ROC-AUC were not significant, the consistency of these results across multiple performance metrics underscores the reliability of our framework, offering a statistically validated and attractive solution for model selection in imbalanced classification scenarios.
mdpi.com
Here's an example of very bad research design. A paper compares RF and XGBoost with SMOTE, ADASYN and GNUS across four imbalance levels in churn data. It never shows RF or XGB trained on raw data. The key control is missing.
Comprehensive Analysis of Random Forest and XGBoost Performance with SMOTE, ADASYN, and GNUS Under Varying Imbalance Levels
This study examines the efficacy of Random Forest and XGBoost classifiers in conjunction with three upsampling techniques—SMOTE, ADASYN, and Gaussian noise upsampling (GNUS)—across datasets with varying class imbalance levels, ranging from moderate to extreme (15% to 1% churn rate). Employing metrics such as F1 score, ROC AUC, PR AUC, Matthews Correlation Coefficient (MCC), and Cohen’s Kappa, this research provides a comprehensive evaluation of classifier performance under different imbalance scenarios, focusing on applications in the telecommunications domain. The findings highlight that tuned XGBoost paired with SMOTE (Tuned_XGB_SMOTE) consistently achieves the highest F1 score and robust performance across all imbalance levels. SMOTE emerged as the most effective upsampling method, particularly when used with XGBoost, whereas Random Forest performed poorly under severe imbalance. ADASYN showed moderate effectiveness with XGBoost but underperformed with Random Forest, and GNUS produced inconsistent results. This study underscores the impact of data imbalance, with MCC, Kappa, and F1 scores fluctuating significantly, whereas ROC AUC and PR AUC remained relatively stable. Moreover, rigorous statistical analyses employing the Friedman test and Nemenyi post hoc comparisons confirmed that the observed improvements in F1 score, PR-AUC, Kappa, and MCC were statistically significant (p < 0.05), with Tuned_XGB_SMOTE significantly outperforming Tuned_RF_GNUS. While differences in ROC-AUC were not significant, the consistency of these results across multiple performance metrics underscores the reliability of our framework, offering a statistically validated and attractive solution for model selection in imbalanced classification scenarios.
mdpi.com
Two methods to fix your model's probabilities. One overfits on small data. The other can't handle non-sigmoid distortions. Knowing when to use which one is the whole game.
Your model ranks cases correctly. But are its probabilities right? Most classifiers distort them in predictable ways. Find out more in my new book (coming out soon)
90% accuracy doesn't mean your probabilities are right. Boosted trees and SVMs distort them toward the centre. Naive Bayes pushes them to the extremes. Neural nets and bagged trees? Spot on from the start.