Build a small eval before you switch to a local model

Before replacing a hosted model with a local one, or a Q5 file with a Q4, run the same thirty real prompts through both and score the results with a rule you wrote in advance. Published benchmarks cannot tell you whether the model handles your code, your data and your format.

Why published numbers are not enough

Hugging Face's optimization guide says that 4 bit quantization can in practice often lead to different results compared with 8 bit or bfloat16, and that it is up to the user to try it out. The GPTQ and LLM.int8 papers report small accuracy loss on the models and benchmarks they tested. Neither tells you how your prompts will fare. The only way to know is to test your own work.

Step 1: collect real prompts

Pull 20 to 30 prompts from actual use: code questions, summaries, extraction jobs, whatever the model will do. Remove secrets and customer data first. Include a few hard cases and a few that failed before. Thirty is a working number for a first pass, not a statistical standard.

Step 2: write the pass rule before you run

Decide what good means for each prompt:

• Exact checks where possible: valid JSON, the right label, code that compiles, a test that passes.
• Short rubric for the rest: names the right function, mentions the constraint, no invented API.

Write the rule down first, so you do not bend it after seeing outputs.

Steps 3 and 4: fix the settings, run and score

Use the same prompts, temperature, context length and system prompt on every candidate. A mismatch in context window alone can look like a quality gap. See the context window issue. Where you can, use temperature zero for repeatability.

Run every candidate on every prompt. Save outputs to files. Score with the rule, ideally by someone who did not choose the favorite. Record pass counts and notes on the failures.

for f in prompts/*.txt; do
  curl -s http://localhost:11434/api/generate -d "{\"model\": \"your-model\", \"prompt\": \"$(cat $f)\", \"stream\": false}" > out/$(basename $f).json
done

This sketch assumes simple prompt text without quotes or newlines. Use a proper JSON builder in real code, and check the API docs for your version.

Step 5: decide with a margin

Compare pass counts and look at which failures matter. A difference of one or two on thirty prompts is noise. A pattern, such as every long prompt failing, is a signal. Decide on a threshold in advance, for example no more than a set number of regressions on the critical prompts.

Keep it as a regression test

Store the prompts and rules in version control. Re-run them when you change model, quantization, runtime version or context settings. The same set also helps when you consider fine tuning. See write evals before you fine tune and when to fine tune or just prompt.

Quick check

Sources

• Optimizing LLMs for Speed and Memory, Hugging Face Transformers docs
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (arXiv)
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (arXiv)

Last reviewed: 19 October 2026.