Which of 624,000 users will leave, and when
- ROLE
- Solo build
- TIMEFRAME
- 2025
- STACK
- Python, LightGBM, CatBoost, XGBoost, scikit-learn
- LINKS
- github ↗
0.88 AUC
0.83 RECALL ON TEST
2.31
Tenure MAE
0.939
Tenure R²
The problem
Predict which users will churn, early enough to act, from 90 days of anonymized usage logs (624,048 training rows), without letting the future leak into the features.
Approach
Only days 1 to 60 feed the features, so the model never sees the window it's judged on. Engineered signals include rolling EMAs and volatilities (7/14/30-day), a usage slope, drop trend, coefficient of variation, and zero-activity days. A RandomForest regressor first predicts expected tenure (test MAE 2.31, R² 0.939); its prediction, never the actual, then becomes a feature for the churn classifier. The final classifier is a voting ensemble over random forest, LightGBM, CatBoost, XGBoost, and logistic regression.
Results
Test set: AUC-ROC 0.88, recall 0.83 at 0.81 accuracy. Recall is the metric that matters here: a missed churner costs more than a retention nudge sent to a happy user. The strongest features were the engineered trend signals (drop_trend first), not the raw usage counts.
What broke
The obvious leakage risk is the target and its close cousins: last_active, churn, and expected_tenure are all dropped from the training features up front, and only days 1 to 60 of the 90-day log ever reach the model, so it never sees the 30-day window it's judged on. The tenure model's prediction, never the real value, is the only trace of tenure the churn classifier ever sees. SMOTE isn't part of this pipeline: it's an item on the README's next-steps list, not a technique that was tried and dropped; recall here comes from the voting ensemble and from the engineered trend features (drop_trend first) doing real discriminative work on their own.