Fine-tuning

Fine-tuning is an adaptation technique that lets you specialize a generic foundation model for a precise domain such as law. Instead of retraining a model from scratch (an operation that costs millions of dollars and requires massive infrastructure), fine-tuning adjusts existing parameters on a target corpus that is much smaller but highly specialized.

In a legal context, fine-tuning makes it possible to adapt an LLM to the terminology of French law, to the specific structures of court decisions, and to the drafting style of pleadings or legal documents. The LoRA (Low-Rank Adaptation) technique revolutionized this approach by enabling efficient fine-tuning with limited compute resources, putting specialization within reach of smaller players.

Fine-tuning does come with risks, however: over-specialization (the model loses general capability), amplification of biases present in the training corpus, and potentially an increase in hallucinations on topics outside the fine-tuning corpus. This is why legaltech solutions often combine fine-tuning and RAG: fine-tuning for style and terminology, RAG for factual accuracy.