Every few months a team asks me to help them fine-tune a model, and every few months my first question is the same: "What did you try before this?" Half the time the honest answer is "nothing" — fine-tuning gets reached for as a first move because it sounds like the serious, grown-up version of prompting. It isn't. It's a different tool with a different failure mode, and the failure mode doesn't show up in the demo.
Why it works, right up until it doesn't
Fine-tuning is genuinely good at a narrow set of problems: locking in a house style, teaching a model a rigid output format it keeps drifting from, or baking in domain vocabulary that's expensive to re-explain in every prompt. On those axes, a few hundred well-chosen examples can outperform a much longer system prompt.
The trouble starts because fine-tuning doesn't fail loudly. It fails by making the model slightly worse at things nobody was evaluating. A support team fine-tunes a model on 400 examples of their ideal ticket-closing tone. Tone improves immediately. Three weeks later someone notices the model has gotten worse at handling multi-part questions it used to handle fine — because none of the 400 examples had more than one question in them, and the fine-tune nudged the weights toward the shape of the training set at the expense of behavior the base model already had. This is catastrophic forgetting in its ordinary, unglamorous form: not the model forgetting everything, just forgetting the 5% you didn't think to protect.
The checklist I actually use
Before green-lighting a fine-tune, I ask a team to answer three things, in order:
- Have you tried prompting and RAG first, with a real eval set, and actually failed? Not "we assumed it wouldn't work" — an eval set with real failure cases. Most tone and format problems die here.
- Do you have a regression suite covering behavior outside the training distribution? If the only tests are "does it sound more like us," you have no way to detect what you broke.
- Who re-runs this when the base model updates? A fine-tune is a snapshot bolted onto a specific base checkpoint. When the provider ships a new version, your fine-tune either has to be redone against the new base or you're stuck pinned to a model that will eventually be deprecated.
Prompting → cheap, fast to iterate, no drift risk, ceiling on style control
RAG → adds facts and freshness, doesn't change model behavior
Fine-tuning → changes behavior/style permanently, adds maintenance surface
The maintenance bill nobody prices in upfront
The part that gets underestimated isn't the training run — it's everything after. A fine-tuned model needs its own eval suite, its own re-training cadence when the base model changes, and its own rollback plan when a new batch of training data subtly shifts behavior. Teams that skip building that eval suite find out about regressions from users, not from tests, which is the most expensive way to find out.
When it's the right call
None of this means avoid fine-tuning. It means treat it as an operational commitment, not a one-time script. It's the right call when the behavior you need is narrow, stable, and worth the ongoing evaluation cost — a fixed output schema, a consistent brand voice, a domain-specific classification task with a stable label set. It's the wrong call when it's being used as a substitute for writing a better prompt or building a retrieval layer, because those are reversible in an afternoon and a bad fine-tune is not.
I write about the operational side of shipping AI — the parts that don't show up in the model card — in my newsletter, AI Shipped. New issue every week.