The three tests every regulatory intelligence tool should pass
Source transparency, currency, and scope discipline: the framework for evaluating any regulatory intelligence tool, applied honestly to the major players in the EU financial regulation market.
This article is for informational purposes only and does not constitute legal advice. Consult a qualified legal professional for advice specific to your situation.
Why evaluation frameworks matter in this market
The market for regulatory intelligence tools is full of confident claims. Platforms describe themselves as AI-powered, source-anchored, continuously updated, and purpose-built for compliance teams. Those claims are difficult to evaluate without a framework that cuts through the marketing language and asks what the tool actually does when a compliance professional needs a reliable answer to a specific question.
This article sets out three tests that every regulatory intelligence tool should be able to pass. The tests are not proprietary. They follow directly from what compliance professionals are actually responsible for: making decisions that are defensible to supervisors, auditors, and courts, on the basis of information that is current, verifiable, and correctly scoped to the obligations that apply to their firm.
The tests are applied here to the major categories of regulatory intelligence tool, including enterprise monitoring platforms, generic AI tools, and source-anchored purpose-built systems. The application is honest: each category passes some tests and fails others, and the analysis identifies which.
The three-test framework was introduced in why generic AI tools are unreliable for regulatory compliance research. This article develops it into a standalone evaluation guide.
The compliance accountability context
Before the tests themselves, it is worth being precise about the context in which regulatory intelligence tools are used.
A compliance professional who relies on a tool to assess their firm’s obligations under DORA, SFDR, or MiCA is making a professional judgment, not running a search query. If that judgment is later found to be wrong, and the firm has failed to meet an obligation as a result, the compliance professional is accountable. The tool is not.
This asymmetry matters for evaluation. The question is not whether a tool produces outputs that sound plausible. It is whether the tool produces outputs that the compliance professional can rely on without independent verification, or at least outputs that make verification tractable. A tool that is fast but requires complete re-verification of every claim against primary sources saves no time. A tool that appears reliable but cannot be verified is a liability.
The three tests operationalise this accountability standard. They ask what a tool needs to provide for the compliance professional to be able to defend their decision if challenged.
Test one: source transparency
The question: Can the tool tell you exactly which document a given claim came from, with a stable retrievable identifier, at the level of a specific article or provision?
The threshold for passing this test is specific. Not “based on DORA.” Not “drawing on official EU sources.” The standard is: “based on Article 30(2)(a) of Regulation (EU) 2022/2554 (CELEX: 32022R2554), published in the Official Journal on 27 December 2022.” That level of specificity is what makes a claim verifiable. Everything below that level is an assertion that requires independent verification before it can be relied upon.
CELEX identifiers matter here because they are stable. A CELEX number points to exactly one document. It does not change when the document is summarised, cited in commentary, or discussed on a regulatory update website. A compliance professional who has a CELEX identifier can go directly to EUR-Lex and read the specific provision being referenced. A compliance professional who has a reference to “DORA” has an ambiguous pointer to a body of legislation that includes the framework regulation, multiple delegated regulations, several sets of technical standards, and ongoing supervisory guidance, not all of which are in force simultaneously or applicable to the same firm types.
For a fuller explanation of how CELEX identifiers work and why they are the right standard for compliance-grade citation, see how to read a CELEX number.
How the major tool categories perform:
Enterprise monitoring platforms such as Wolters Kluwer FRR and Thomson Reuters Regulatory Intelligence pass this test in a limited form. Their analyst-curated content references source documents, and those source documents are available within the platform. The gap is in the query layer: when a compliance professional asks a specific question, the answer is typically an analyst interpretation rather than a retrieval from a specific provision with a CELEX citation. The source is available in the background but is not surfaced in the answer itself.
Generic AI tools (ChatGPT, Gemini, Perplexity, and similar) fail this test entirely. Their outputs are generated from training weights, not retrieved from specific documents. A model asked about DORA incident reporting timelines cannot tell you which article of which document its answer comes from, because the answer was not derived from any single document. It was produced by inference across a large body of text, with no mechanism for tracing the output back to a retrievable source.
NLP-classified alert platforms such as Corlytics and Compliance.ai occupy a middle position. Their alerts typically link to source documents, which means the alert itself passes the test in a basic sense. The gap appears at the query layer: when a compliance professional wants to interrogate a specific provision rather than receive an alert about a publication, the source-anchored query capability is typically absent.
A genuinely source-anchored system passes this test in every output. Each claim includes the CELEX identifier and the specific article number it is drawn from, making every claim independently verifiable without additional research.
Test two: currency
The question: Is the system drawing on a corpus that is continuously updated from official sources, and does it tell you the publication date of each source document it is citing?
EU financial regulation does not stand still. Active regulatory files in 2026 include the continuing development of DORA technical standards, the SFDR 2.0 legislative process, the implementation of MiCA’s authorisation regime, the transposition of AIFMD II across member states, and the application of AI Act obligations to financial services. The pace of new publications from the European Commission, the European Supervisory Authorities, and national competent authorities means that a tool’s corpus can become materially out of date in a matter of months.
The currency test has two components. The first is whether the system is updated continuously from official sources, not relying on a static training snapshot or a periodic manual refresh. The second is whether the system tells you when each source document was published, so you can assess whether the information you are receiving reflects the current regulatory position or an earlier state of the same document.
The second component matters because EU regulations are amended after publication. A framework regulation published in 2022 may have been amended by a corrigendum, updated by a delegated regulation, or clarified by a set of technical standards. A system that cites the 2022 publication date without flagging subsequent amendments is technically providing a source reference but materially misleading the user about the currency of the obligation.
How the major tool categories perform:
Enterprise platforms with analyst curation pass this test with a caveat. The corpus is updated, but through an analyst layer that introduces latency. The time between a regulatory publication and its appearance in the platform as a structured compliance task is not zero. For slow-moving regulatory files, that latency is acceptable. For fast-moving files such as DORA technical standards or MiCA implementing measures, the gap can be material.
Generic AI tools fail this test definitively. Their knowledge is fixed at a training cutoff. Beyond the cutoff, the tool has no information. Within the period before the cutoff, coverage of recent publications is thinner than coverage of older material, because the internet has had less time to process and discuss recent developments. A model with a training cutoff of late 2024 may have unreliable coverage of regulatory developments from mid-2024 onwards even though those developments technically fall within its training window. For a full treatment of this problem, see why generic AI tools are unreliable for regulatory compliance research.
A continuously updated source-anchored system, built on automated ingestion from EUR-Lex and the ESA publication feeds, passes this test. The ingestion is daily. Every source document is timestamped with its publication date from the Official Journal. Every answer includes that publication date alongside the CELEX identifier, so the compliance professional can see exactly how current the source is.
Test three: scope discipline
The question: Does the system maintain clear distinctions between what the regulation requires, what supervisory guidance says about implementation, and what pre-legislative proposals may require in future?
This is the test that most platforms fail in ways that are most consequential for compliance professionals.
EU financial regulation exists simultaneously at several levels. A framework regulation sets the primary obligations. Regulatory technical standards specify how those obligations are met in practice. National competent authority guidance interprets how the regulation applies in a specific member state. ESA opinions and Q&A documents clarify contested interpretations. Commission proposals set out what the law may require in future, if adopted. These levels have different legal characters and different degrees of certainty.
A system that does not maintain clear distinctions between these levels misleads its users in two directions. Treating pre-legislative content as current law can cause firms to redirect compliance effort toward obligations that have not yet been adopted and may change materially before they are. Treating supervisory guidance as equivalent to a regulation can cause firms to underestimate the flexibility available to them in a comply-or-explain regime. Treating a draft technical standard as equivalent to the adopted version can cause firms to build systems against requirements that were subsequently changed.
The scope discipline test also applies to jurisdictional scope. EU regulations are directly applicable across member states, but supervisory implementation varies. What the EBA says about DORA implementation may be interpreted differently by the BaFin, the FCA equivalent in a non-EU jurisdiction, or the AMF. A system that blends the regulation text with member state implementation guidance without flagging the distinction is compressing information that compliance professionals need to keep separate.
How the major tool categories perform:
Enterprise platforms with strong analyst curation generally pass this test. Their content is labelled by instrument type, and analysts typically distinguish between adopted and pre-legislative content. The limitation is that analyst interpretation is itself a layer between the compliance professional and the source, and that layer can introduce its own scope compression.
Generic AI tools fail this test systematically. A model trained on regulatory text has absorbed adopted regulations, consultation papers, draft technical standards, commentary, and press coverage in a single undifferentiated training corpus. It has no mechanism for knowing, when answering a question, whether its answer reflects the adopted text or a pre-legislative draft. It cannot tell the compliance professional which of these it is drawing on, because it does not know.
Pre-legislative focused platforms such as Corlytics perform well on horizon scanning content but can be weaker on the adopted law layer. A platform designed to track what is coming is not always the right tool for determining what is currently in force and how it applies.
A well-designed source-anchored system passes this test by maintaining separate streams for adopted legislation and pre-legislative content, labelling each output with the legal status of the document it is drawn from, and explicitly flagging when an answer is based on a technical standard that is still in draft or a proposal that has not yet completed the legislative process.
Applying the three tests in practice
When evaluating any regulatory intelligence tool, the three tests translate into a set of specific questions.
On source transparency: does every output include a CELEX identifier for the source document? Does it cite the specific article number, not just the regulation? Can you follow that citation directly to EUR-Lex and read the provision being referenced without additional searching?
On currency: does the tool update its corpus continuously from official sources, or does it rely on a static training snapshot or periodic manual refresh? Does it tell you when each source document was published? Does it flag when a document has been amended since its original publication?
On scope discipline: does the tool clearly distinguish between adopted law and pre-legislative proposals? Does it label supervisory guidance with its correct legal status, separate from the regulation it interprets? Does it distinguish between what the regulation requires and what NCAs in specific member states have said about implementation?
A tool that cannot answer yes to these questions is producing outputs that require independent verification before they can inform compliance decisions. That is not a disqualification: most research outputs require some verification. But it means the tool’s value is limited to producing a first draft for checking, not a reliable answer in its own right.
The honest summary
No tool in the current market passes all three tests without qualification.
Enterprise platforms pass test three reliably and test two with a latency caveat. They pass test one weakly: source documents are present but not surfaced at the CELEX and article level in every output.
Generic AI tools fail all three tests. They are fast and readable, and they are appropriate for general background research, drafting, and synthesis where source precision is not required. They are not appropriate as the basis for compliance decisions.
Alert-only platforms pass test two for the alert content itself but do not provide the query capability that would allow tests one and three to be applied to specific questions.
A source-anchored system built on continuous EUR-Lex ingestion, CELEX-cited outputs, and a separated corpus architecture passes all three tests. The limitation of such a system is scope: a purpose-built EU financial regulation tool does not cover US, UK, or APAC regulatory obligations. For firms with multi-jurisdiction requirements, it is one component of a broader monitoring approach rather than a complete solution.
The honest evaluation is firm-specific. A large bank managing simultaneous obligations across eight jurisdictions needs the breadth of an enterprise platform, with its analyst curation and multi-jurisdiction coverage, even accepting the latency and source transparency trade-offs. A boutique fund manager tracking SFDR, AIFMD II, and DORA across EU member states needs EU depth and source precision more than it needs global breadth. A fintech founder navigating MiCA authorisation needs affordable access to the current official text and a query interface that can answer specific questions, not a multi-year enterprise contract.
The three tests do not select a single winner. They map the trade-offs clearly enough that each firm can identify which tool genuinely serves their compliance problem.
For a detailed explanation of what source anchoring requires in practice, see what does it mean for a regulatory intelligence tool to be source-anchored. For the specific failure modes of generic AI tools in regulatory contexts, see why generic AI tools are unreliable for regulatory compliance research. For a broader view of the EU financial regulatory intelligence market, see EU financial regulation software: what compliance teams actually use.
Forseti monitors EU financial regulation continuously, anchoring every alert and answer to a verified CELEX-identified official source. Every output includes the publication date, the CELEX identifier, and the specific article number it is drawn from. Start for free.
A process guide for building the data collection, verification, risk assessment, and documentation infrastructure that EUDR due diligence requires. Covers geolocation sourcing, deforestation verification, supplier engagement, and TRACES NT submission.