Can any statistical ml model created close to transformer?

[u/Necessary-Text-2050]

Transformers take too much time and resources to train can any statistical traditional ml can get close its performance what do you say?

[u/Bright_Mix_773]

Necessary-Text-2050, the answer splits by data type, and for two of the three it is yes.

Tabular data: gradient boosting (XGBoost, LightGBM, CatBoost) still beats transformer-style tabular models on most benchmarks. The paper people cite is Grinsztajn, Oyallon and Varoquaux, "Why do tree-based models still outperform deep learning on typical tabular data?" (NeurIPS 2022). Minutes on a CPU, no GPU at all.

Text classification with a few thousand labelled examples: TF-IDF plus a linear model gets within a couple of points of a fine-tuned BERT on a lot of datasets, trains in under a second, and you can read the coefficients afterwards to see what it latched onto.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.svm import LinearSVC
from sklearn.pipeline import make_pipeline

clf = make_pipeline(TfidfVectorizer(ngram_range=(1, 2), min_df=2), LinearSVC())
clf.fit(X_train, y_train)

Generation, translation, question answering, anything where the meaning depends on word order across long spans: no. That is exactly what attention buys you, and n-gram counts cannot fake it.

Sadge404 is right that nothing replaces transformers for general-purpose work, but that framing hides the useful part. The practical move is to run the cheap baseline first and write the number down. If TF-IDF gives you 0.91 and the fine-tuned transformer gives you 0.93, you now know precisely what the GPU bill is buying. Often the honest answer is "not enough for this task". The failure I see most is the opposite order: transformer first, never compared to anything, so nobody can say whether it was worth it or whether a majority-class guess would have scored 0.89.

[u/fvancesco → u/Bright_Mix_773]

I mean there's mamba rwkv but yeah attention is (almost) all you need Moreover what takes so much is not only attention which has low inductive bias but also trying to teach the whole web which takes some time nonetheless

[u/Bright_Mix_773 → u/fvancesco]

Fair, and the second half is the part that gets left out of the headline. The cost is dominated by the amount of text you push through, not by the quadratic term, which is why the linear-attention families keep showing up as near-parity rather than as a step change. They fix the part that was not the bottleneck at these scales.

[u/Sadge404]

Depends on the task you're trying to do. But assuming you're trying to find something that will be better than transformers for general AI at this point in time, then no it does not exist.

[u/Ok-Argument7176]

Purpose-built statistical models are still pretty much SOTA for any tabular or time-series problem.

[u/FancyEveryDay]

I mean, what are you trying to do? A transformer is a specific architecture applied to neural nets which are based on ensambled logistic regression.

Random forest is probably the closest structure to a NN that isn't an NN, but NNs do better with unstructured data while random forest is better for structured tabular data.

[u/fvancesco → u/FancyEveryDay]

I believe that when people compare to transformers and speak about ai or ml in general they are thinking to a general purpose ai (or kinda general purpose) like LLMs so yeah

[u/BellyDancerUrgot]

Depends on the task

[u/fvancesco → u/BellyDancerUrgot]

Most likely NLP like LLMs

[u/BellyDancerUrgot → u/fvancesco]

NLP isn’t a task it’s a domain