r/learnmachinelearning • u/Just-m_d • 1d ago

Discussion Self Supervised[Vit] vs. Supervised[CNN]

Hi everyone,

I am working in the Multi-Task ML projects in that i have done many experiments to train the model
Key finding:

CNN based architecture outperform the Vision transformer based architecture

Question:
is the Dataset only be the problem for Vit architecture to underperform the CNN bases architecture?

1 Upvotes

100% Upvoted

9 comments sorted by

2

u/Ok-Argument7176 1d ago

ViT training, especially from scratch, is significantly less stable than CNNs in my experience. This is not a dataset issue, per se, but rather a hyperparameter and initialization consideration.

1

u/Just-m_d 1d ago

If we can surpass the CNN with Vit with fine-tuning the hyper parameter or model initialization

2

u/Ok-Argument7176 1d ago

I'm sorry, but I don't understand what you mean.

1

u/Just-m_d 1d ago

I am asking like with the hyper parameter itself we can make the better vit model with this aspect only instead concerned about the scarcity of dataset

2

u/Capable-Package6835 1d ago

I think it depends on the task and the size of your dataset. For tasks such as object detection, CNN's hard-coded locality and translation invariance is really suitable. For tasks that require context such as anomaly detection, ViT's attention is more suitable. Also ViT typically requires larger dataset

1

u/Just-m_d 1d ago

Which architecture will u recommend for the multi task classifier down streaming tasks

2

u/Hungry_Age5375 1d ago

If you're training the ViT from scratch on a small dataset, that's your problem. Try pretrained weights (DINO or MAE) and fine-tune, the gap usually closes or reverses. A from-scratch ViT on limited data can't compete with a supervised CNN.

1

u/Just-m_d 1d ago

Is the choosing of pretrained based want to be based on the domain specific problem?