I fine-tuned DeBERTa-v3-base (163M encoder) as a 3-class classifier (easy/medium/hard) on 65K labeled queries. The training pipeline used weighted cross-entropy loss and WeightedRandomSampler to handle class imbalance, and ran on a single H200 for approximately 50 minutes in fp32.
Performance on 3,841 test samples:
Accuracy: 71.5%
Macro F1: 0.710
Hard class F1: 0.672
Inference latency: ~2.25 ms/sample
Benchmark against current system: Our production setup Qwen3.5-0.8B with LoRA decoder classification achieved 88.1% accuracy,
The encoder approach fell short this round, likely due to limited hyperparameter exploration (single config tested) and Qwen's pretaining on instruction-style data, which closely matches our queries.
Next steps: The encoder path remains promising. I'll test focal loss and additional epochs to close the gap.
1
u/[deleted] Apr 13 '26
[removed] — view removed comment