LegalLease

The problem

Legal documents break simple RAG setups. Clauses reference each other, definitions sit pages away from where they are used, and a wrong answer is worse than no answer at all. Standard top-k retrieval kept missing the clause that actually mattered. Once I fixed retrieval, a stock 3B model still could not read a clause and answer the question correctly.

What I built

I built hierarchical FAISS retrieval that goes from document to section to chunk, added BM25 hybrid recall and cross-encoder reranking, routed generation across OpenAI, Anthropic, and a local llama.cpp model, and fine-tuned Qwen2.5-3B with LoRA on 961 lease question and answer pairs that I built from real lease clauses. All of it is graded by an evaluation harness that I wrote before any of the optimization.

The problem with top-k on legal text

Answering a question about a lease usually requires three things at the same time. You need the clause itself, the definition it references, and the exhibit it points to. A single embedding search returns whichever of those happens to look most similar to the question, which is often none of them. My first baseline looked fine when I demoed it and then fell apart as soon as I sat down and wrote out real questions.

Measuring before optimizing

I wrote the evaluation harness before doing any optimization. It started at 35 questions and grew to 112, across 12 real Maryland lease and tenant rights documents, and each question is annotated with the facts a correct answer has to contain. It scores fact coverage, quote verification, numeric exact match, and abstention separately, which lets me tell whether a bad answer came from failing to find the clause or from mangling a clause it did find. I log almost every run, and the evaluation directory now holds around seventy of them.

Fixing retrieval got me to the real problem

Hierarchical FAISS retrieval, going from document to section to chunk, combined with BM25 hybrid recall and cross-encoder reranking, brought retrieval recall to 99% on the 112 question set. That was both good news and bad news. The right clause was now in the context window almost every time, and the local 3B model was still getting about a third of the answers wrong. Retrieval had stopped being the bottleneck, and generation had become the bottleneck instead.

So I fine-tuned the model

I built the training set from my own corpus instead of downloading one. It came to 961 conversations, created by running GPT-4o over the exact retrieved context that production uses, and then augmented with synthetic question and answer pairs generated from sampled lease clauses. For training I used LoRA with rank 16, alpha 16, and no dropout, applied to all seven attention and MLP projections of a 4-bit Qwen2.5-3B-Instruct through Unsloth. That ran for 3 epochs at a learning rate of 2e-4, with an effective batch size of 8, 4096 token sequences, and adamw_8bit, on a single Colab GPU. I then merged it to fp16 and converted it to Q4_K_M GGUF so it runs under llama.cpp on my laptop. Accuracy on the held out questions went from about 35% to about 55%. What the model actually learned was to quote the clause word for word and to abstain instead of inventing a number.

What the failures taught me

The most useful thing I learned had nothing to do with the model weights. A 3B model degrades badly on long prompts. The same question scored 26.7% with 12K characters of context and 62.9% with 1.2K characters. At this model size, retrieval quality and context budget matter more than parameter count, which is why the assembly stage enforces a hard character limit. The weak spot that is still open is false abstention, where 17 of the 112 answers abstain even though the evidence was there.

Making a hallucinated legal claim structurally impossible

Answering a question is one thing. Telling someone that their lease is illegal is a different thing, and being confidently wrong about that can actually hurt them. So the scanner never trusts the model on its own. An LLM detector proposes candidate flags, and then every candidate has to pass two independent checks. The first is a deterministic grounding test that requires the quoted clause to appear word for word in the lease. The second is another LLM pass on whether that quote actually demonstrates the problem. Failing the grounding test is a hard drop rather than a warning, which is what makes it impossible for a fabricated quote to reach the report. If a flag passes grounding but fails relevance, it stays in the report and gets marked uncertain.

Legal knowledge as curated data, never generated

Each flag links by id into a Maryland statute file that I curated by hand. Every entry has the rule in plain language, the real citation, such as Md. Code, Real Property section 8-208(d)(3) for the 5% cap on late fees, and an as of date, because the law changes and a stale rule is just a wrong rule. None of the legal content is generated at scan time. Adding a new check means adding a YAML entry with a retrieval query, a severity, the law references, and a detection instruction. The code stays the same.

Why multi-provider

A single router sits in front of OpenAI, Anthropic, and the local llama.cpp path. That let me evaluate the fine-tuned 3B against frontier models using identical retrieval, it means a provider outage degrades quality instead of taking the system down, and it means the whole thing can run offline at no marginal cost when privacy matters more than accuracy.