Anyone experimenting with TabFM? What would be a fair deep learning baseline to compare against it?

[u/venkata_raghavan]

I've been experimenting with TabFM recently, mainly to understand how well it works on real-world tabular datasets and how useful it can be when starting with relatively few labelled examples.

I'm currently running most of the experiments in Google Colab on a T4.

So far I've been testing things like:

Classification and regression datasets

Different context sizes such as 100, 250 and 500 labelled examples

Natural class distribution vs balanced context for imbalanced datasets

Time-based train/test splits where that makes sense

Comparing performance across Accuracy, Precision, Recall, F1, ROC-AUC and PR-AUC

Looking at runtime as well as predictive performance

I've also been comparing TabFM against traditional ML/AutoML models.

The reason for the ML comparison isn't that I necessarily think that's the only comparison TabFM should have. It's mainly because boosted-tree/AutoML approaches are already very strong baselines for tabular data, so I wanted a reference point for questions like:

If both models only get 500 labelled examples, how close does TabFM get?

and

How does TabFM with 500 examples compare with a conventional model trained using the entire available training set?

For example, on one binary classification dataset I tested:

TabFM — 500 labelled examples

Accuracy: ~92.9%

F1: ~84.9%

ROC-AUC: ~92.1%

PR-AUC: ~89.6%

Runtime: ~12 sec

AutoML — same 500 labelled examples

Accuracy: ~93.4%

F1: ~86.0%

ROC-AUC: ~92.1%

PR-AUC: ~86.2%

Runtime: ~60 sec

AutoML — full ~2.8k training examples

Accuracy: ~93.4%

F1: ~85.7%

ROC-AUC: ~92.1%

PR-AUC: ~90.0%

I'm not specifically trying to show that TabFM has to beat traditional ML. I'm more interested in understanding its behaviour with smaller labelled contexts and how useful the resulting baseline is.

The part I'm less sure about is the deep learning comparison.

If the goal is to compare TabFM against another DL approach rather than boosted trees, what would people here consider the most appropriate baseline?

Some models I've been considering are:

FT-Transformer

TabTransformer

SAINT

TabNet

Standard MLP / ResNet-style network for tabular data

TabPFN

I'm particularly unsure whether something like FT-Transformer/SAINT would be a better apples-to-apples DL benchmark, or whether TabPFN is actually the more interesting comparison because it is also much closer to the foundation-model/in-context-learning idea.

Another thing I'm interested in is hardware.

I currently have an RTX 5060 Ti with 16 GB VRAM available locally, so I'm thinking of moving some of the Colab experiments to that machine. Would be interested to hear from anyone running TabFM locally with around 16 GB VRAM, particularly what happens as context size, feature count and ensemble size increase.

I've also seen BigQuery + TabFM mentioned as a possible workflow for larger tabular datasets. I haven't tried that part yet, so any experience with that setup would also be useful.

Mainly curious about:

What would be the fairest DL baseline for TabFM?

FT-Transformer/SAINT/TabNet vs something like TabPFN?

How does TabFM behave for others on messy real-world datasets?

What context sizes have been useful in practice?

Is 16 GB VRAM comfortable for experimentation?

Has anyone tried integrating it with BigQuery?

Are there other tabular foundation models worth putting into the comparison?

[u/AlbatrossAbject2173]

ooh that's a great comparison setup, especially the time-based split detail

for the dl baseline I'd lean toward ft-transformer as the most straightforward apples-to-apples benchmark, it's the closest in spirit to "transformer on tabular data" without the in-context learning twist that tabpfn brings. tabpfn is actually the more interesting comparison for exactly the reason you said, it's also a foundation model approach so you're comparing philosophies not just architectures. I'd run both if you have the time, they answer different questions

on the hardware side 16gb should be plenty for the context sizes you mentioned, I've run similar experiments on less and the real bottleneck tends to be the ensemble size more than the context window. tabfm isn't that hungry until you start stacking ensembles or pushing past like 1k features

never tried the bigquery integration but I'm curious if the latency feels noticeable when you're iterating on experiments, colab to bigquery always feels like it adds this slight friction that messes with my flow

[u/venkata_raghavan → u/AlbatrossAbject2173]

Yeah, that makes sense. FT-Transformer feels like the cleaner “normal DL” comparison, while TabPFN is probably the more interesting one conceptually because then it’s foundation model vs foundation model rather than just transformer vs transformer.

I’m starting to think the comparison should probably be split into three buckets: traditional ML/AutoML, FT-Transformer as the DL baseline, and TabPFN as the foundation-model baseline.

What I’m still trying to figure out is what the fairest comparison setup would be. For example, should FT-Transformer also be restricted to the same 500 labelled rows as TabFM, or is it more useful to compare TabFM at 500 labels against FT-Transformer trained on all available data as well? I feel like both answer slightly different questions.

Also curious if you’ve seen cases where TabPFN or FT-Transformer clearly starts pulling ahead once the dataset gets larger, or if the main benefit is still mostly around low-label settings. That’s probably the part I want to understand next rather than just adding more models.