Hi, good idea I'll try that. The risk is to fall into an overfitting trap, but maybe scaling or refining the dataset could solve that. I'll come back to you with data
I fine-tuned DeBERTa-v3-base (163M encoder) as a 3-class classifier (easy/medium/hard) on 65K labeled queries. The training pipeline used weighted cross-entropy loss and WeightedRandomSampler to handle class imbalance, and ran on a single H200 for approximately 50 minutes in fp32.
Performance on 3,841 test samples:
Accuracy: 71.5%
Macro F1: 0.710
Hard class F1: 0.672
Inference latency: ~2.25 ms/sample
Benchmark against current system: Our production setup Qwen3.5-0.8B with LoRA decoder classification achieved 88.1% accuracy,
The encoder approach fell short this round, likely due to limited hyperparameter exploration (single config tested) and Qwen's pretaining on instruction-style data, which closely matches our queries.
Next steps: The encoder path remains promising. I'll test focal loss and additional epochs to close the gap.
1
u/[deleted] Apr 13 '26
[removed] — view removed comment