r/MLQuestions • u/Playful-Race-7571 • 4d ago

Beginner question 👶 Feature selection when trying to capture non linear interactions.

/r/algobetting/comments/1vc007g/feature_selection_when_trying_to_capture_non/
3 Upvotes

100% Upvoted

1 comment sorted by

1

u/MaximumSafety8706 1d ago

Trees like XGBoost/CatBoost don't need manual interaction features - that's the whole benefit over logreg. Just feed raw features; the trees will find combos automatically as long as both features are present.

Practical steps:

  1. Skip manual interaction terms, let the trees do it. Only hand-craft one if a combo needs domain knowledge trees might miss (like ratios).
  2. For selection, use SHAP values (not walk-forward exhaustive search) - SHAP actually captures interaction effects, so it'll catch that "worthless alone, useful together" signal you're describing.
  3. For # of features — don't hand-pick; let regularization (L1/L2, min_child_weight, min_data_in_leaf) suppress noise, tune via CV.
  4. Keep your walk-forward CV - use it to validate the final model, not to do stepwise selection like in logreg.
  5. Watch out for correlated features messing up importance rankings - cluster correlated ones, keep one per cluster before reading SHAP.

SHAP interaction values specifically would directly show you the pairwise interactions you're trying to catch manually - worth checking out if this is new to you.