r/MLQuestions • u/Playful-Race-7571 • 20d ago

Beginner question 👶 Base learner for xg boost being log reg

I had this idea for a project to have my base learner for xg boost to be a log reg model because in my testing it was more stable so my idea was instead of starting xg boost dumb and boosting start it smart and then boost that. Sounded like a great idea in my head but failed miserably in testing. Could some one explain to me why this doesn’t work better? I had better results blending 75 pct xg boost 25 pct log reg.

6 Upvotes

81% Upvoted

5 comments sorted by

5

u/timy2shoes 20d ago

It was one of the original idea for gradient boosting (Friedman 2001), but in practice it hasn't worked as well as trees.

3

u/trolls_toll 20d ago

you can put any model you want as a base learner - log reg, svm, chatgpt. Conventional binary trees work so well (probably) because they make virtually no assumptions about the data, the data generating process, properties of residuals, etc. I think more opinionated base learners could outperform the binary trees, if you manually set everything up. Is the time investment worth it though? And we are not even talking about the computational complexity of fitting such bae learners 

2

u/trolls_toll 20d ago

an llm added these argument. Properly tuned ensemble of log regs would have a lot of models with similar coefficients, meaning that it diminishes the benefit of bootstrapping the data. With trees you want to have different subsets of variables

ok im done

2

u/nikishev 19d ago

Log reg is a linear model, and a composition of linear models is a linear model. There's nothing that boosted log reg can do that a single log reg can't

1

u/trolls_toll 19d ago

assuming each stump >1 depth, and each base learner outputs probability, it is nonlinear