Fine-Tuning a Small LLM with DPO for Less Than 50 Cents: What Actually Happens

The ideas and writing here are mine; I used an AI assistant to help edit and tidy it up.

A hands on account of building a DPO fine-tuning pipeline from scratch. The parts the tutorials skip, the errors you will actually hit, and the one config change that made training twice as fast.

---

The plan

The goal was modest and concrete. Take a small, open instruct model and nudge its preferences using DPO, teaching it to prefer better answers over worse ones, then measure whether it actually improved. On a budget, fast.

The stack:

• Model: Qwen 2.5 3B Instruct. Small, capable, Apache 2.0. (We started with Llama 3.2 3B but Meta's gated license needs manual approval, so we swapped to Qwen and skipped the wait.)
• Method: DPO with LoRA adapters.
• Data: argilla/dpo-mix-7k, a clean preference dataset.
• Compute: RunPod, an NVIDIA L4 (24 GB) at about $0.44 per hour.
• Framework: Axolotl, which wraps the training boilerplate behind one YAML file.
• Evaluation: generate answers from the base and fine-tuned models on 50 held out prompts, then let Claude Haiku judge which answer is better.

What is DPO, in plain terms

DPO stands for Direct Preference Optimization. It is a way to teach a model which kind of answer people prefer, and it is much simpler than the older RLHF approach.

The classic RLHF recipe has three moving parts: collect human rankings, train a separate reward model on those rankings, then use reinforcement learning to push the language model toward high reward answers. Three stages, fragile, and slow.

DPO collapses all of that into one training loop. You give the model triples of the form (prompt, chosen answer, rejected answer), and it directly learns to score the chosen answer higher than the rejected one. No reward model, no reinforcement learning. A single knob called beta (we used 0.1) controls how aggressively it shifts preferences versus how far it is allowed to drift from the original model.

The important mental model: DPO does not teach the model new facts. It adjusts taste and style. It moves how the model answers, not what it knows.

The tools, and why each one is here

RunPod is a pay by the hour GPU cloud. You rent a container with a GPU attached, do your work, then terminate it so the billing stops. We used an on demand NVIDIA L4 with 24 GB of memory at roughly $0.44 per hour, with the official Axolotl container image (winglian/axolotl-cloud:main-latest) so the whole training stack was preinstalled. You can drive RunPod entirely from your laptop through its Python SDK: launch a pod, get an SSH command, run your job, terminate. We controlled the whole run that way.

Axolotl is the training framework. It sits on top of Hugging Face Transformers and TRL and turns a fine-tuning run into a single YAML config file. You declare the base model, the dataset, the LoRA settings, and rl: dpo, and Axolotl handles the trainer, the reference model, tokenization, checkpointing, and the optional push to Hugging Face. The cost of that convenience is that you have to feed it data in exactly the shape it expects, which is where most of our debugging time went.

A Claude API key is needed only for the evaluation step, not for training. After both models have answered the 50 prompts, a small script sends each (prompt, answer A, answer B) triple to Claude Haiku and asks which answer is better. This is the LLM as judge pattern. It is cheap (the whole 50 prompt run costs about one cent) and surprisingly effective at separating a real improvement from noise. You get a key from the Anthropic console and set it as ANTHROPIC_API_KEY. Training never touches it.

The pipeline at a glance

   argilla/dpo-mix-7k                  50 hand written eval prompts
  (preference pairs, training)         (held out, NOT from training)
          |                                        |
          v                                        |
    prepare_data.py                                |
  (messages / chosen / rejected)                   |
          |                                        |
          v                                        v
  +======================================================+
  |   RunPod L4 GPU   (winglian/axolotl-cloud image)     |
  |                                                      |
  |     Qwen 2.5 3B Instruct  (bf16)                     |
  |          + LoRA adapters   <===  Axolotl DPO trainer |
  |                                  (rl: dpo, beta 0.1) |
  +======================================================+
          |                                        |
          | merge adapter into base                | both models answer
          v                                        v   the 50 prompts
   small-llm-dpo-merged                   base answers   fine tuned answers
   (pushed to Hugging Face)                     \             /
                                                 v           v
                                              Claude Haiku judge
                                          (which answer is better?)
                                                     |
                                                     v
                                        win / loss / tie  ==>  FINDINGS

Two datasets, kept separate on purpose. argilla/dpo-mix-7k trains the model. The 50 evaluation prompts are written by hand and never seen during training, so the eval is not just measuring memorization.

What "fine-tuning" actually feels like

Here is the honest summary. The GPU and model part is easy. The data plumbing and the performance tuning are where you spend your time. Every failure we hit was about getting the dataset into the exact shape the framework expects, or about memory and speed, never about the math of DPO itself.

We hit seven distinct failures before a clean run. The errors are the lesson, so here is the tour.

Errors 1 and 2: your local file is not a dataset

First launch died instantly:

DatasetNotFoundError: Dataset 'data/train.jsonl' doesn't exist on the Hub

Axolotl assumed our local file was a Hugging Face Hub dataset name. The fix is to declare ds_type: json. Then it died again:

TypeError: 'NoneType' object is not iterable

Reading Axolotl's utils/data/rl.py, the RL loader iterates over a data_files list, which we had not set. For a local DPO file you need both ds_type: json and data_files: [data/train.jsonl]. The path key alone is not enough on the RL path.

Error 3: DPO wants structured messages, not strings

Our data prep wrote sensible looking records: a prompt string, a chosen string, a rejected string. Axolotl's chat_template.default DPO strategy wants something else entirely, structured conversation objects:

{ "messages":  [{"role": "user", "content": "..."}],
  "chosen":    {"role": "assistant", "content": "..."},
  "rejected":  {"role": "assistant", "content": "..."} }

The strategy applies the tokenizer's chat template itself, so it needs roles and content, not pre flattened text. We rewrote the data prep to emit this shape. Lesson: read the strategy's transform_fn to see exactly what columns it reads.

Error 4: it trained, at ten hours per run

Finally, loss numbers. And rewards/accuracies climbing past 0.5, meaning the model was genuinely learning to rank chosen over rejected. Victory? Not quite. About 20 seconds per step, roughly 10 hours total. Unacceptable.

A confusing clue: nvidia-smi showed 0 percent utilization and 1 MiB used, as if the GPU were idle. That turned out to be a lie. Inside RunPod's container, nvidia-smi cannot see the training process because of PID namespace isolation. The GPU was busy, the tool just could not report it. How do we know? 4 bit training requires CUDA, and we later got CUDA out of memory errors. You cannot run out of memory on a GPU you are not using.

Errors 5 and 6: the vocabulary tax, and why checkpointing is mandatory

To go faster we raised the batch size. Instant out of memory, trying to allocate a single 4.63 GiB tensor. That tensor is the logits, and Qwen has a vocabulary of about 152,000 tokens. DPO computes full vocabulary logits for both the chosen and rejected sequences, for both the policy and the reference model. Four times the usual. On a 24 GB card, that caps micro_batch_size at 2. You grow the effective batch with gradient accumulation instead.

We also tried turning off gradient checkpointing for speed. Immediate out of memory. DPO holds activations for two forward passes, so checkpointing is not optional here. Back on.

Error 7: the real culprit was 4 bit

Here was the key insight. We had been using QLoRA, which keeps the model weights in 4 bit to save memory. But halving the sequence length, from 2048 down to 1024, did not change the step time at all. If compute scaled with the number of tokens, it should have. Constant time means the bottleneck is not token compute. It is the per operation cost of dequantizing 4 bit weights on every forward pass.

And we never needed 4 bit. The 3B model in bf16 is only about 6 GB, and the L4 has 24. So we dropped quantization entirely, set load_in_4bit: false and adapter: lora, and step time halved to about 10 seconds. The rule of thumb: only reach for QLoRA when the model genuinely will not fit in bf16. Otherwise it is pure overhead.

Results

The judge, Claude Haiku comparing answers on 50 held out prompts, gave the fine-tuned model the edge. Because LLM judges have a known bias toward whichever answer they see first, we ran the comparison in both orders and combined them.

Run	Fine-tuned wins	Base wins	Ties
Normal order (base shown first)	40% (20)	32% (16)	28% (14)
Swapped order (fine-tuned shown first)	56% (28)	16% (8)	28% (14)
Combined, bias corrected	48% (48)	24% (24)	28% (28)

The fine-tuned model wins in both orderings, so the improvement is real and not an artifact of position. Averaged across both orders it wins 48 percent versus 24 percent, a clean two to one, and 67 percent of the decisive (non tie) judgments. We also confirmed the judge's quirk directly: whichever slot was shown first won 44 times versus 28, a 16 point bias. This is exactly why running both orders matters. The single order number actually understated the fine-tuned model, because in the first run it sat in the disfavored slot.

The training metrics agree with the judge. The DPO rewards/accuracies metric, the fraction of pairs where the model correctly scored the chosen answer above the rejected one, rose from 0.43 to 0.76 over the single epoch. (DPO per step loss is noisy, so that accuracy curve is the trustworthy signal, not the loss number.)

Where did it improve? Exactly where the dataset pushes: reasoning and instruction following, including logic puzzles, spotting logical fallacies, structured explanations, and ordered list instructions. On one logic puzzle the base model even contradicted itself mid answer, writing "Razzles" and then "Rizzle," while the fine-tuned model stayed consistent. On plain factual questions with one right answer the two usually tied, because DPO moves style and reasoning, not facts the model already knows.

The merged model lives at aditya1c/small-llm-dpo-merged on the Hugging Face Hub.

Quirks to remember next time

A field guide, so the next run is boring:

1. Read the framework's source when an error names a file. Every fix above came from reading Axolotl, not from guessing.
2. bf16 LoRA first, QLoRA only if forced. 4 bit dequantization is a real per step tax.
3. Large vocabulary means small micro batch. DPO logits are the memory hog, not activations. Keep micro_batch_size at 2 and scale with accumulation.
4. Keep gradient checkpointing on for DPO. It holds two forward passes of activations.
5. nvidia-smi can lie in containers. Do not conclude the GPU is idle from it.
6. Private repos do not clone on the pod. Copy your files over with scp, or make the repo public.
7. RunPod environment variables do not reach SSH sessions. Write a secrets.env and source it, and source /etc/rp_environment to get Python on the path.
8. Run training detached and poll. The SSH proxy drops under load, and nohup makes that harmless.
9. Terminate the pod the instant you are done. Billing runs until you do.
10. Run the judge in both A/B orders. A single order eval can be off by 15 points from position bias alone.

The takeaway

DPO itself is the simple part. A clean objective, sane defaults (beta 0.1, learning rate 5e-5, cosine schedule), and it just works. The craft is in the boring infrastructure: dataset shape, memory ceilings, quantization trade offs, treating a flaky remote box gently, and evaluating honestly. Get those right and you can fine-tune a real model, end to end, for less than the price of a coffee.