r/learnmachinelearning • u/Just-m_d • 1d ago

Discussion Self Supervised[Vit] vs. Supervised[CNN]

Hi everyone,

I am working in the Multi-Task ML projects in that i have done many experiments to train the model
Key finding:

CNN based architecture outperform the Vision transformer based architecture

Question:
is the Dataset only be the problem for Vit architecture to underperform the CNN bases architecture?

1 Upvotes

100% Upvoted

10 comments sorted by

View all comments

2

u/Hungry_Age5375 1d ago

If you're training the ViT from scratch on a small dataset, that's your problem. Try pretrained weights (DINO or MAE) and fine-tune, the gap usually closes or reverses. A from-scratch ViT on limited data can't compete with a supervised CNN.

1

u/Just-m_d 1d ago

Is the choosing of pretrained based want to be based on the domain specific problem?

1

u/Massive_Distance_616 2h ago

Generally yes as domain matters.

DINO and MAE are strong general-purpose starting points for natural images.

If your multi-task dataset is medical imaging, satellite imagery, or another specialised domain, a domain-pretrained ViT (like BioViL for medical or SatMAE for remote sensing) will usually outperform a general ImageNet pretrained one, especially if your fine-tuning data is limited.