Why we don't let AI run the show in our research pipeline
There's a tempting shortcut when building AI-powered tools - just pipe everything through an LLM and call it done. Here's why we didn't, and what we learned building a hybrid pipeline for professional market research.
The tempting shortcut
There’s a tempting shortcut when building AI-powered tools: just pipe everything through an LLM and call it done. Feed in raw web content, ask the model to extract insights, ship it.
We tried that approach early on. It doesn’t work. At least not for professional market research.
Here’s what we learned, and why we ended up with a hybrid pipeline that uses deterministic logic for the heavy lifting and AI only where it genuinely adds value.
The problem with “AI everything”
When you hand raw web content to an LLM and ask for insights, a few things happen consistently:
The model hallucinates structure that isn’t there. It finds patterns because you asked it to find patterns, not because they exist. It treats a 404 error page with the same weight as a detailed product review. And it has no way to tell you how confident it is in any of it.
For a personal project, that’s fine. For a research agency putting findings in a client deck, it’s a serious problem. Research outputs need to be defensible. “The AI said so” is not a methodology.
What deterministic logic actually means
Before any content reaches our LLM, it goes through several filtering layers that use straightforward rules, not machine learning:
Does this URL look like a paginated index page or a sitemap? Skip it. Does this domain consistently produce off-topic content? Block it. Does this page have fewer than 50 words? Discard it. Does the text contain first-person language, opinion markers, and experience words? Keep it.
These checks are fast, cheap and predictable. And crucially, they are auditable. You can look at a piece of content that got filtered and understand exactly why.
No model weights. No probabilities. Just logic you can read and reason about.
Where AI actually earns its place
Once the deterministic layer has done its job, we’re left with a much smaller set of genuinely relevant conversations. That’s where LLM extraction makes sense.
Identifying themes across hundreds of conversations, spotting nuance in how people describe a problem, generating a synthesised insight from a cluster of related opinions are tasks where human-like language understanding genuinely helps and where the cost and unpredictability of LLMs is worth it.
The ratio matters. In our pipeline, roughly 80% of the filtering work happens before the LLM ever sees the data.
Why this matters for research rigour
Professional researchers are rightly sceptical of AI tools. They’ve seen the hallucinations, the confident-sounding nonsense, the outputs that look impressive until someone asks “but how did you get there?”
A hybrid approach gives you a defensible answer to that question. The rules-based filtering is transparent and consistent. The AI is doing interpretation, not data selection. Those are meaningfully different things.
We’re building a tool for researchers who need to stand behind their work. That shaped every architectural decision we made.
Curious about what that looks like in practice? Get in touch — we’re always happy to talk research methodology.