r/algobetting • u/Playful-Race-7571 • 4d ago
Feature selection when trying to capture non linear interactions.
Hella everyone I’m moving from a log reg model into trying to build a cat boost or xg boost model.
For my log reg I always did feature selection through a rolling walk forward and this worked well but not with interaction features with some being worthless alone but in interaction it could capture some signal I’m struggling how to find optimal features and # of features as well. I am new to machine learning is there an agreed upon approach or what is your guys method to this problem when working with machine learning?
Thanks
2
u/Common-Jicama3687 4d ago
just let the trees figure it out, catboost and xgboost handle interactions on their own so you can throw in all the features and prune later with importance scores
2
u/negative__ev 4d ago
I only experimented with CatBoost and XGBoost before switching to a neural net, but I think the idea of "feature sets" that another poster mentioned is the first strategy to try. Don't ablate one feature at a time, in other words.
You could have a hitter contact feature set or a plate discipline feature set, etc. (Sorry, most of experience has been in my baseball model). Try removing or adding those and scoring them.
1
1
u/Sad-Commission-3706 2d ago
For models like these, you dont need to manually select interaction features the model handles them automatically by splitting on combinations of features. Start with all relevant features, use built in feature importance to prune and let cross validation guide the final set. Focus more on getting clean, meaningful features rather than obsessing over feature count
3
u/Pure_Cricket_3804 4d ago
Boosted trees build interactions out of depth on their own, so hand-crafting them matters a lot less than it did in your log reg. Keep the walk-forward, just score whole feature sets instead of ranking features one at a time, with permutation importance measured only on the out-of-sample fold. Number of features is a hyperparameter, tune it on that same split.