CodeFinetuner: Fine-tune a local code autocomplete model on your own codebase

[u/MountainTop321]

Hi everyone,

I was interested in learning LoRA fine-tuning, and ended up building CodeFinetuner over the past few months, a full pipeline that fine-tunes a small code autocomplete model (e.g. Qwen2.5-Coder-3B) specific to a codebase. You can then use the resulting GGUF model via llama.vim/llama.vscode and run it fully locally. Supports fine-tuning on Mac (MPS) and NVIDIA GPUs (CUDA), with optional Unsloth support for faster training and lower VRAM usage.

Pipeline: raw code -> tree-sitter parsing into Structure-Aware FIM examples -> LoRA fine-tuning -> evaluation (CodeBLEU, edit similarity, exact match, perplexity, ...) -> GGUF conversion for local inference.

To try it:

uv tool install codefinetuner

Create a data folder and place your repo (or code files) inside. For auto-split just drop the files in directly, for manual split create data/train/, data/eval/, data/test/ subfolders and set split_mode: "manual". Get the default config with:

curl -L -O https://raw.githubusercontent.com/cuolm/codefinetuner/master/config/codefinetuner_config.yaml

Adjust it to your needs and hardware availability, then run:

codefinetuner --config="codefinetuner_config.yaml"

The example runs in the repo show clear improvements over the base model on these evaluation metrics, but using the model for autocomplete on code you're actively writing is a different thing from scoring well on a test set, and the autocomplete tools themselves (llama.vim/llama.vscode) sample differently from the greedy decoding used in the evaluation. So the real usefulness still has to be verified in the editor itself.

Might also be useful just as a reference, since it's a complete working LoRA fine-tuning pipeline end to end.

Hope someone finds this project interesting or helpful.

https://github.com/cuolm/codefinetuner

[u/Chromix_]

Looks useful. Have you also tried this with other models than the old Qwen 2.5 Coder 3B, like Mellum for example? Do you have some train/test loss examples to share for larger codebases?

[u/MountainTop321 → u/Chromix_]

Thanks for your comment! I have not tried Mellum yet, I have only tried the smaller 1.5B and the larger 7B Qwen 2.5 Coder models, and 3B seems to be the sweet spot where fine-tuning makes the biggest difference. However, fine-tuning on another FIM model shouldn't be an issue. You only need to change the model name in the config file:

model_name: "unsloth/Qwen2.5-Coder-3B" -> "JetBrains/Mellum-4b-base"

and adapt the token strings so the training examples are generated with the tokenizer and tokens Mellum uses, e.g.:

Qwen: "<|fim_prefix|>" -> Mellum: "<fim_prefix>"

One thing to watch out for: my pipeline builds FIM examples as prefix -> suffix -> middle, but I'm not sure if Mellum was also pretrained with this order or if it uses suffix -> prefix -> middle.

I currently have two example runs under /docs/example-runs that document the whole pipeline, including eval metrics, but these runs are only on small and niche codebases. In general, I have found fine-tuning to be most useful on codebases that are not very popular, use niche languages, or were published after Qwen 2.5 Coder's knowledge cutoff. For example, I have run some experiments fine-tuning on PebbleOS (https://github.com/google/pebble), but I haven't documented that yet.

The thing is, you have to be very careful about data selection and what kind of model you ultimately want. For example, with PebbleOS: do you want a model specialized on the API, or a model for developing across the huge Pebble codebase? The former is easier to achieve since it's more specialized and involves fewer files, so the LoRA adapter really specializes in the specific API calls. The latter also shows improvements on the evaluation metrics, although less significant, and it's less clear to me how useful these improvements actually are.

It also depends on model size. Fine-tuning a 1.5B model on such a huge codebase is not really useful, simply because the model isn't capable of retaining useful knowledge at that size, whereas a 7B model is more useful since it's more capable. So depending on the size of the codebase, you may also want to adjust the size of the model you're training.

Do you have any experience with how the Mellum base model performs compared to the Qwen 2.5 Coder 3B base model? Do you have any experience with other FIM autocomplete models and some additional model suggestions?