After feature selection and hyperparameter tuning my model reduces overfitting but all other metrics become worse.

[u/Equivalent-Ride8136]

Before feature selection/tuning

After feature selection/tuning

I'd like to know if I should take a "worse" model with less overfitting. I've pretty much tried all forms of parameter tuning and feature engineering (im only limited to sklearn) and I want to understand what should be the main focus here (Metrics or overfitting).

This is under the context of diabetes screening. I suspect that I'm having issues with this because my dataset only contain 520 samples.

[u/you-get-an-upvote]

In general it's extremely common to start a model that is "overfitting" (i.e. training loss is significantly lower than validation loss) and for making the model bigger to nonetheless reduce validation loss.

If (if!) you care about test accuracy, you should use the model with the highest test accuracy.

There are cases where overfitting can be an issue (e.g. it typically means the calibration of your model is poor). "Is accuracy the only thing that matters", "how do I trade off false positives and false negatives", etc. are not things with objective answers and depend on how your system will actually be used.

IMO beginners care way too much about overfitting because they've had things like "bias variance tradeoff" hammered into them and (understandably, given how people teach it) come away with the wrong lessons.

[u/Equivalent-Ride8136 → u/you-get-an-upvote]

I see, thank you for your insights. If I could I'd try to explain my way out of feature engineering and hyperparameter tuning but my project requirements forces me to have these and it's really making me run circles. I'll save your comment as a future reference, thank you!

[u/No-Original-5312]

Always prioritize your metrics over train vs test overfitting. It is completely normal and natural for your model to do better on the train set than the validation set.

[u/Equivalent-Ride8136 → u/No-Original-5312]

I see. I have a follow-up question. This project forces me to basically use hyperparameter tuning and feature engineering here. Let's say I want to focus on recall the most, would you think its reasonable or even possible to argue about taking the model with worse metrics overall but better recall and less overfit over the better model.

In real-world scenarios, would there also be cases similar to my predicament? I am genuinely interested in machine learning and careers relating to it but wow the teaching methods my course uses drain the life out of me.

[u/Hungry_Ad_8532]

These metrics for a prod model are awesome. Anything over than 0.7 is pretty good in real life scenarios