Five myths about training your own model on your codebase

Training a model on your codebase sounds like the obvious way to get an assistant that knows your code. In practice it rarely gives the results people expect. Here are five common beliefs, and what the published evidence says about each one of them.

Myth 1: Training on the repo makes the model know the repo

Fact: Training adjusts weights to predict text patterns. It does not build a lookup table. A model tuned on your code may reproduce your style and common idioms, but it can still invent function names and signatures that do not exist. Retrieval puts the real text in front of the model at question time. See RAG or fine-tuning.

Myth 2: It will stay current

Fact: Weights freeze at training time. The next merge makes the model stale. An index can be refreshed per file in seconds, see keeping a code index fresh.

Myth 3: Fine-tuning needs a data centre

Fact: Parameter efficient methods changed this. The LoRA paper reports that, compared with full fine-tuning of a 175 billion parameter model, LoRA reduces trainable parameters by 10,000 times and GPU memory by about 3 times. The QLoRA paper reports fine-tuning a 65 billion parameter model on a single 48 GB GPU. Hugging Face's PEFT library describes the goal as adapting large models without fine-tuning all parameters, making training more accessible on consumer hardware. These are authors' claims, and fine-tuning a small model on a modest dataset is within reach of a small team. See what LoRA is.

Myth 4: More data is always better

Fact: Quality matters. The QLoRA authors report that small, high quality datasets can give strong results. Dumping an entire repository, including generated files, vendored libraries and stale branches, teaches the model your worst code as readily as your best.

Myth 5: Private data in the weights is safe

Fact: Information trained into a model can come back out. OWASP's sensitive information disclosure guidance notes that revealing training data can enable inversion attacks, and recommends scrubbing sensitive content before training. Secrets that were in the repository can be memorised. Anyone with access to the model weights or queries may extract them. Scan and clean before training, and treat the resulting model as sensitive as the data.

So what is training good for?

• Output format and tone, such as always producing a fixed report layout.
• A narrow, repeated task for a small cheap model, such as labelling commits.
• Teaching vocabulary your domain uses that a general model handles poorly.

Facts about the current state of your code belong in retrieval.

A sensible order of operations

1. Try plain prompting with examples.
2. Add retrieval.
3. Measure with a test set, see how to evaluate.
4. Only then consider tuning, for the gaps retrieval cannot close.

Quick check

Sources

• LoRA: Low-Rank Adaptation of Large Language Models
• QLoRA: Efficient Finetuning of Quantized LLMs
• Hugging Face PEFT documentation
• OWASP LLM02:2025 Sensitive Information Disclosure

Last reviewed: 19 October 2026.