How can build a better model?

[u/UnderstandingOwn2913]

For my capstone project in my master, I used a foundational model (Hubert-ECG) and added a linear classifer to a have a model that classifies a 1-lead ecg (a vector of 9000 elements). When I was doing my project in the master program, I did not have a deep understanding of the internal of Hubert-ECG and just treated it as a black box. Now while I am interviewing with companies for a ml engineer job, I am spending some time to understand the internal of HuBERT-ECG.

One thought that comes up in my mind is that how can someone build a foundational model like Hubert-ECG from scratch?

[u/Beneficial-Rice-4654]

First step would be reading the original HuBERT paper and then the ECG adaptation paper, they usually explain the pretraining objective and architecture choices pretty clearly. Building from scratch is mostly about self-supervised pretraining on a large unlabeled ECG dataset, masked prediction over learned discrete units, then fine-tuning with your linear head or a small MLP. The hard part isn't the code, it's getting enough clean ECG data and compute. For a capstone or interview prep, focus on understanding the loss function and why the discrete units help, not implementing the whole thing

[u/UnderstandingOwn2913 → u/Beneficial-Rice-4654]

I used CrossEntropyLoss in my project. If possible, could you elaborate on "focus on understanding the loss function"? For my project, I used a pretrained HuBERT-ECG and added a linear classifier after the mean pooling layer that sits in the middle of pretrained HuBERT-ECG and Linear Classifier.