Before your firm buys AI, ask this one question

"Should we train a model on our firm's documents?" is one of the most common questions we get from firms evaluating AI, and it usually arrives before a more useful one: what kind of question will this system be answering? Get that straight and the right architecture, and the right vendor, becomes obvious.

Two different problems wearing the same trench coat

If your system needs to be right about facts, such as what a contract says, what a court held, or what is in this month's filings, you have a retrieval problem. Facts live in documents, documents change, and you need answers traceable to sources. Retrieval-augmented generation handles this: the model reads the relevant documents at question time and answers from what it read. Update the corpus and the answers update with it. No training run required.

If your system needs to behave a certain way, such as always producing this exact report structure, matching this house style, or following this classification rubric at high volume, you have a formatting problem. That is where fine-tuning earns its keep: it bakes consistent behaviour into the weights, which can also buy you a smaller, faster, cheaper model for a narrow task.

Fine-tuning teaches a model how to behave. It is a terrible way to teach a model what is true.

Facts pushed into weights go stale the day your documents change, cannot be cited, and cannot be audited. If a lawyer asks where an answer came from, "somewhere in the training data" is not an acceptable response. In our legal work, every citation must trace to a document the pipeline actually opened, which is an architectural property no amount of tuning provides.

The order of operations we actually use

• •Start with a strong base model, careful prompting, and honest retrieval. This baseline is better than most teams expect, and it is cheap to iterate on.
• •Build the eval harness before optimizing anything. You cannot compare approaches you cannot measure.
• •Invest in retrieval quality: chunking, hybrid search, reranking, verification. In our experience this is where accuracy actually comes from.
• •Reach for fine-tuning last, when evals show a persistent style, format, or latency gap that prompting cannot close. Keep retrieval for the facts even then.

The good news is this is not really an either-or. The systems we ship are mostly retrieval with occasional tuned components at the edges: a classifier here, a formatter there. But the sequence matters. Teams that start with a training run spend months building data pipelines for a model that still cannot cite its sources. Teams that start with retrieval have something verifiable in production within weeks, and data from real usage to justify any tuning that comes later.