Data Products: Too good to be true
Updated: Jan 20, 2025
If you are productizing a predictive model at work or playing around with MLE in R for the first time, always check the data.
Roles
When working on a predictive model at work, everyone has a role. Product management drives the requirements and determines rightness.
Data samples and ETL code from Data Engineering
Prepared data samples, models and code from Data Scientists
Case Study
A few weeks ago, we finished the second of three models to complete a proof of concept and prepare for roadmap estimates.
The PM was testing toward the end of each phase.
The team modeler was sharing first pass results at a weekly check-in and his first statement was "I need to check the data before moving forward, but the predictive results on this are amazing! Near 99%".
On the team's daily #slack meet the next day, he reported a data error. The same model AUC's was around .33 now and the model had the ugliest confusion matrix. We tossed it.
Drop me a note and tell me how you do it.
-DCN
Comments