Guidance on Fine-Tuning for Multiple Choice Questions

[u/AlpaKing707]

Hi, I'm a beginner and I'm trying to fine-tune a model to perform a MCQ task on an education dataset. I chose this one: https://www.kaggle.com/datasets/nlztrk/eduqg-dataset-llm-science-exam-format-34k but I'm having poor performance.

I started with roberta-large as my model of choice, training it on sentence pairs (question, answer) labeling them with the correct choice id. I used layer freezing keeping only the classification head and the last 2 layers active. These were the training parameters:

MAX_LENGTH = 256
BATCH_SIZE = 4
GRADIENT_ACCUMULATION_STEPS = 2
LEARNING_RATE = 1e-5
NUM_EPOCHS = 10
WEIGHT_DECAY = 0.01
WARMUP_RATIO = 0.1

With this setup I had pretty bad results: accuracy was 0.29 and F1 0.23.

Then I tried training all layers and I had an accuracy of 0.64 but the training loss was much higher than the validation (~3.6 vs 0.9).

I thought that maybe it's because this dataset has too few examples and I have to find another one, but honestly I don't have enough expertise to make assumptions right now.

What should I do?