From dense to hybrid: upgrading RAG for exact legal citation retrieval

The problem that dense search handles well, and the one it does not

When we built the first versions of Forseti's and Verdandi's retrieval pipelines (Forseti covering EU financial regulation, Verdandi covering EU sustainability regulation), we described the core problem in why your RAG retrieves the wrong chunks: dense vector search systematically retrieves recital prose over obligation articles because the two embed very differently despite covering the same topic. HyDE solved that by generating a hypothetical regulatory answer before embedding, pulling the search vector into the same space as obligation text.

That fix worked well for the broad question class. "What are the ICT risk management obligations under DORA?" retrieves correctly. "What does DORA require for third-party risk management?" retrieves correctly. The system handles natural language regulatory questions competently.

There is a second, harder problem class that only became visible once compliance professionals started using the system in earnest. It is the exact citation query. Not "what does DORA say about outsourcing" but "what does Article 28(3) of DORA require." Not "what are the CRR output floor rules" but "what is the output floor percentage in Article 465 of the CRR3."

In EU financial regulation, these are not interchangeable questions. Article 28(3) of DORA imposes specific obligations around the register of information, annual reporting to competent authorities, and notification requirements for critical and important functions. Article 28(4) is different. A system that returns the right thematic content but the wrong sub-article has given a compliance professional material that may lead them to a wrong conclusion about their obligations.

Dense vector search has a structural weakness here. "Article 28(3)" and "Article 28(4)" embed very close together. The parenthetical number is a minor token in a high-dimensional space dominated by shared vocabulary. HyDE makes this worse in some cases: the hypothetical answer generated for "what does Article 28(3) require" may drift toward thematically similar provisions in other regulations, pulling the search vector away from the specific provision the user asked about.

This is what we set out to fix.

What the baseline showed

Before building anything, we established a baseline against the adopted legislation stream, using queries designed specifically to stress the exact-citation failure class.

The pre-upgrade results were honest about the system's strengths and limits. Most broad queries were already passing cleanly. The failures showed two distinct modes.

The first failure mode: HyDE generated a hypothetical answer that embedded closer to a thematically similar regulation than to the one explicitly named. The top dense results were dominated by the wrong regulation. The system correctly identified it could not find the specific article and fell back to a long-context path, but the retrieval had gone to the wrong place entirely.

The second failure mode: the right regulation was retrieved, but the wrong document ranked first. Supervisory technical standards and delegated regulations adopt the same article numbering as the base act. When a user asks about a specific article, an implementing measure referencing that article by number can outrank the authoritative base text.

These are two distinct failure modes with the same surface presentation: the user gets an answer that is in the neighbourhood of correct but not grounded in the specific provision they asked about.

After the upgrade, both failure modes were eliminated on our test set. Broad queries that were already passing continued to pass without regression.

The architecture

The upgrade added several components to the existing dense pipeline. Each solves a specific problem. None of them replaces the dense path; they work alongside it.

At a high level, every query now travels two parallel paths before results are merged:

• A dense leg using HyDE: the hypothetical answer is embedded and matched against the vector index
• A sparse leg using full-text search: the raw query, sanitized for citation notation, is matched against an exact-token index

Results from both legs are merged using Reciprocal Rank Fusion, which ranks by position rather than raw similarity score. This means a chunk that appears in the top results of both paths ranks much higher than one that appears in only one; which is exactly the signal we want for exact citation queries, where the right chunk should rank well in both semantic and lexical space.

On top of this, two additional components handle the specific failure modes described above.

Named regulation detection and anchoring. When a user names a specific regulation, we detect it, resolve it to the correct version of that document in our corpus, and filter the dense search to that regulation's chunks. This prevents HyDE's tendency to drift toward thematically similar regulations when a user has been explicit about which regulation they mean.

Forced inclusion with a semantic distance cap. Dense and sparse search combined still has one gap: if chunks from the named regulation do not appear in the top candidates from either path, they will not appear in the final results even with boosting. For large regulations with many articles, specific chunks may be pushed down by other content from the same regulation that happens to embed closer to the query.

We solve this by forcing a set of chunks from the named regulation into the candidate pool regardless of rank, but only those whose semantic distance from the query falls below a calibrated threshold. This separates genuinely relevant chunks from the named regulation (which sit close to the query) from tangential amendments and delegated acts that share the same regulatory root but are not what the user asked about.

The sparse search configuration

Two configuration decisions here are worth explaining because both are non-obvious and both matter significantly for legal text.

Dictionary choice. PostgreSQL's built-in full-text search offers stemming dictionaries that normalize word forms and apply stop-word filtering. For legal text this is harmful. Stemming turns citation notation like 28(3) into low-signal fragments. Stop-word filtering can remove regulatory acronyms treated as common words. We use a simple dictionary that preserves exact tokens: article numbers, legislative identifiers, acronyms, and parenthetical citations all match as written.

Query sanitization. Parenthetical citation notation needs normalizing before it hits the text search engine, because 28(3) is ambiguous as written. We normalize it to separated tokens before the sparse search. Regulatory acronyms go the other direction: for the sparse path we strip them, because the acronym itself is noise against an exact-token index; what matters is the article number. The sanitized query goes to the sparse leg only. HyDE still receives the natural language form of the query, which it needs to generate a useful hypothetical answer.

The bugs we found building this

Ten bugs surfaced during the implementation. Several are worth describing because they represent failure modes that would be easy to miss in any system of this kind, regardless of the specific implementation.

Window functions running before LIMIT. A ranking function was computing positions across all rows before the result set was constrained to the candidate pool size. A top-ranked chunk was getting a rank in the hundreds rather than rank one, making the fusion calculation meaningless. The fix is to constrain the result set in a subquery before the ranking function runs over it.

Identifier pattern mismatch. Our named regulation detection used a pattern that silently failed for a large portion of recent EU financial regulation, including several of the regulations our users ask about most. It matched correctly against older identifiers in the test set but not against the format used by more recent legislation. The fix is straightforward once you identify it, but it is easy to validate a pattern against a narrow test set and miss an entire class of real-world inputs.

Superseded documents leaking through the related document expansion. When we expand a named regulation to include its amendments and delegated acts, superseded versions of those documents were included. These then bypassed the superseded-document filter in the main retrieval because they entered through a different code path. The fix is to apply the superseded filter at the point of expansion, not only at the point of retrieval.

Quality check false negatives on anchored queries. After retrieval, we run a lightweight check to assess whether the retrieved chunks are sufficient to answer the question. When a named regulation anchor is active, all retrieved chunks are from the correct regulation, but the specific sub-article may not appear verbatim in the top chunks depending on how the document was chunked. The quality check was returning a negative result and triggering unnecessary fallback behavior. The fix is to handle anchored queries differently in the quality check: the chunks are from the right place, and the system should work with what it has rather than escalating.

Broad article queries over-anchoring. Named regulation detection fires whenever a regulation acronym or identifier appears in the query, which is correct for exact citation queries but too aggressive for broad ones. A question about "Article 20 reporting obligations under DORA" with anchoring active restricted retrieval to DORA only, missing the delegated regulations adopted under Article 20 that contain the substantive obligations; Article 20 itself is a mandate provision. The fix was in the HyDE generation prompt: explicitly instructing the model to stay anchored to the named regulation when generating its hypothetical answer reduces cross-regulation drift without requiring the anchor logic itself to be conditionally disabled.

The remaining bugs were threshold calibration issues. RRF scores have a different range than raw cosine similarity scores, and every relevance threshold calibrated against similarity scores needs recalibration for the new score space. This affected the main retrieval thresholds, the quality check threshold, and the gap detection thresholds in the downstream service. The direction is predictable (RRF scores are lower in absolute terms), but the exact recalibration requires running against real queries.

What hybrid search revealed about data hygiene

The most unexpected finding from the upgrade was not about retrieval. It was about the database.

Dense vector search is forgiving of structural problems in a legal corpus. If a document lacks a verified relationship to its base act, HyDE can still find it semantically and return it with apparent confidence. The structural problem is invisible to the user: they get an answer, and nothing in the response signals that the document underpinning it has a data quality issue.

Hybrid search removes that cushion. Both legs need to contribute meaningful signal, and structural gaps degrade the sparse score enough to push documents below the relevance threshold. After the upgrade, queries that had been returning answers started returning nothing. On investigation, that turned out to be the more honest result. The documents that dropped out had data relationship problems: missing identifiers, unverified connections to base acts, incorrect version flags. Dense search had been absorbing these problems silently for months, returning the documents as if they were authoritative when their provenance had not been verified.

The practical implication is that a hybrid upgrade functions as an audit. Gaps that surface after it are more likely to be data quality findings than retrieval regressions. If a query that previously worked stops working after adding a sparse leg, the right first step is to look at the data, not the retrieval logic.

Five streams, different outcomes

We applied this upgrade across five document streams: adopted legislation, legislative proposals, interpretive guidance from supervisory authorities, agency consultation papers, and case law. The outcomes differed significantly.

Adopted legislation saw the largest improvement on the exact-citation failure class, which was the primary target.

Legislative proposals improved substantially, but for different reasons. Proposals use procedure codes and Commission reference numbers rather than stable legislative identifiers. The sparse leg's ability to match exact reference strings directly improved retrieval precision in ways the dense leg cannot achieve on its own.

Supervisory guidance from EBA, ESMA, and EIOPA improved primarily through reduced source scatter and elimination of paragraph hijacking, where a paragraph number from an entirely unrelated document was matching the citation query. The sparse leg, properly configured, does not make this mistake.

Interpretive guidance for sustainability regulation showed something instructive: it was already performing well before the upgrade, and remained so after. These documents have semantically descriptive titles and natural-language structure that dense embeddings handle well. The technical reference code problem that drives the need for exact-token matching simply does not exist for this corpus. Hybrid search is transformative for corpora with dense reference codes; it is marginal for natural-language-titled corpora that dense search already handles well. The same upgrade, applied to the same codebase, produced dramatically different effects depending on the characteristics of the source documents.

That contrast is the clearest lesson from the five-stream rollout. The decision is not "always add sparse search." It is "understand what your dense search is failing on before deciding what to add."

What the architecture made possible operationally

One concern with any retrieval upgrade is re-embedding cost. In our case, the new retrieval components use metadata columns and full-text search indexes, not the embedding vectors themselves. The embedding column stays frozen. Adding the new indexes and metadata columns costs a database migration and a backfill, not an embedding API call for every chunk in the corpus.

This separation made it possible to ship the upgrade one stream at a time: deploy for adopted legislation, observe results, fix any data relationship issues that surface, then move to the next stream. The five-stream migration took four days.

The architectural principle is worth stating explicitly: dense embeddings represent the stable semantic content of a document. Database metadata represents the evolving structure of relationships between documents. Keep them separate. Update them on different schedules. Injecting structural metadata into the text before embedding encodes facts that change over time into a vector that should not.

What this does not solve

Hybrid search does not solve corpus gaps. If a document is not indexed, no retrieval architecture will find it. The upgrade surfaced several gaps we had not previously detected because dense search had been masking them with semantically adjacent content. That is a feature, not a bug; the fix for a corpus gap is ingestion, not retrieval engineering.

Sub-article precision is a chunking concern, not a retrieval concern. If the chunk containing Article 28(3) also contains Articles 28(4) through 28(6), the retrieved chunk is correct but the model must extract the specific sub-article from it. The retrieval is right; the granularity is limited by how the document was chunked. Finer chunking improves this at the cost of more chunks and more retrieval noise. The right balance depends on the query distribution of your users.

For a broader treatment of the retrieval architecture this upgrade sits within, including the quality check, gap detection, and long-context fallback path, the earlier posts in this series cover the components that predated the hybrid upgrade: why your RAG retrieves the wrong chunks and why deterministic RAG beats generative AI for research.