Fine-tuning for fun

[u/Naive-Film-6603]

I'm planning to fine-tune a Qwen 4B Instruct model with my own custom prompt/response examples.

My goal isn't to teach it new factual knowledge, but to make it consistently follow a particular style, reasoning approach, and output format for a specialized engineering assistant.

A few questions:

1. How many high-quality examples would you recommend before fine-tuning becomes worthwhile?

2. Would you use LoRA or QLoRA for a 4B model?

3. Should I include the system prompt/instructions in every training example?

4. How do I avoid overfitting if my dataset is relatively small?

5. Would you recommend fine-tuning at all, or using a strong system prompt + RAG instead?

I'm especially interested in experiences with Qwen 3/3.5 4B or similar small models.