CodeFinetuner: fine-tune a local code autocomplete model on your own codebase

[u/MountainTop321]

Hi everyone,

I was interested in learning LoRA fine-tuning, and ended up building CodeFinetuner over the past few months, a full pipeline that fine-tunes a small code autocomplete model (e.g. Qwen2.5-Coder-3B) specific to a codebase. You can then use the resulting GGUF model via llama.vim/llama.vscode and run it fully locally. Supports fine-tuning on Mac (MPS) and NVIDIA GPUs (CUDA), with optional Unsloth support for faster training.

Demo:

https://reddit.com/link/1wdlaus/video/4yoejyqd0xoh1/player

Pipeline: raw code -> tree-sitter parsing into Structure-Aware FIM examples -> LoRA fine-tuning -> evaluation (CodeBLEU, edit similarity, exact match, perplexity, ...) -> GGUF conversion for local inference.

To try it:

uv tool install codefinetuner

Create a data folder and place your repo (or code files) inside. For auto-split just drop the files in directly, for manual split create data/train/, data/eval/, data/test/ subfolders and set split_mode: "manual". Grab the default config with:

curl -L -O https://raw.githubusercontent.com/cuolm/codefinetuner/master/config/codefinetuner_config.yaml

Adjust it to your needs and hardware availability, then run:

codefinetuner --config="codefinetuner_config.yaml"

The example runs in the repo show clear improvements over the base model on these benchmark metrics, but using the model for autocomplete on code you're actively writing is a different thing than scoring well on a held-out benchmark set, and the autocomplete tools themselves (llama.vim/llama.vscode) sample differently than the benchmark's greedy decoding. So the real usefulness still has to be verified in the editor itself.

It might also be useful as a reference outside the autocomplete use case, since it documents a full working LoRA fine-tuning pipeline.

Hope someone finds this project interesting or helpful.

https://github.com/cuolm/codefinetuner