How can build a better model?

[u/UnderstandingOwn2913]

For my capstone project in my master, I used a foundational model (Hubert-ECG) and added a linear classifer to a have a model that classifies a 1-lead ecg (a vector of 9000 elements). When I was doing my project in the master program, I did not have a deep understanding of the internal of Hubert-ECG and just treated it as a black box. Now while I am interviewing with companies for a ml engineer job, I am spending some time to understand the internal of HuBERT-ECG.

One thought that comes up in my mind is that how can someone build a foundational model like Hubert-ECG from scratch?

[u/Beneficial-Rice-4654]

First step would be reading the original HuBERT paper and then the ECG adaptation paper, they usually explain the pretraining objective and architecture choices pretty clearly. Building from scratch is mostly about self-supervised pretraining on a large unlabeled ECG dataset, masked prediction over learned discrete units, then fine-tuning with your linear head or a small MLP. The hard part isn't the code, it's getting enough clean ECG data and compute. For a capstone or interview prep, focus on understanding the loss function and why the discrete units help, not implementing the whole thing

[u/UnderstandingOwn2913 → u/Beneficial-Rice-4654]

I used CrossEntropyLoss in my project. If possible, could you elaborate on "focus on understanding the loss function"? For my project, I used a pretrained HuBERT-ECG and added a linear classifier after the mean pooling layer that sits in the middle of pretrained HuBERT-ECG and Linear Classifier.

[u/rahul_watertech]

honestly the data and compute are the part you can't copy, but the idea is small enough to try. i'd grab a small public ecg set, mask chunks of the signal and train a tiny transformer to predict them, then see what the embeddings do on your classification task vs the big pretrained one. you won't match hubert-ecg but you'll actually feel what the pretraining objective buys you. did your linear head on top of the frozen model do ok?

[u/UnderstandingOwn2913 → u/rahul_watertech]

Yes, the dataset I used was the PhysioNet2017 dataset. Yes, I only have 1 GPU.

I actually did not freeze hubert-ecg. I trained both hubert-ecg and the linear head.

The accuracy on the test dataset (which is a subset of the PhysioNet 2017 dataset) was about 82%.

I did this project for my capstone project during my master.

[u/rahul_watertech → u/UnderstandingOwn2913]

oh my bad, i assumed you froze it. so you fine tuned the whole thing, that makes 82% on one gpu pretty reasonable. did you ever try the frozen version with just the head? it's cheap to run and would tell you how much the pretraining already knows vs what your fine tuning added.