Picture this: A sales team of a company is demoing their new business marketer AI to a potential enterprise client. They ask a straightforward question: "What's the best go-to-market strategy for a Series B SaaS startup in India?"
Your system confidently returns a detailed breakdown of influencer marketing on TikTok.
The room goes quiet. The client exchanged an unimpressed look with the sales team, and the sales and engineering teams were looking at each other, trying to figure out the gaps they had overlooked.
TikTok has been banned in India since 2020. Any AI system for enterprise SaaS claiming to understand Indian market dynamics while recommending a platform that has been prohibited for years immediately reveals its fundamental ignorance of the local business landscape.
Not just that, a Series B SaaS startup in India is typically looking for enterprise sales strategies, partnership channels, and B2B marketing approaches from a business marketer, not consumer-focused influencer campaigns that would be completely irrelevant to their immediate needs. The target audience for enterprise software is not scrolling through short-form videos; they might be in boardrooms evaluating solutions based on ROI, security, and scalability.
Here, the gap becomes obvious. A client might not be able to properly articulate all their requirements at once. But an experienced marketing professional understands what the client actually wants, asks follow-up questions, and uncovers what the client truly needs. The engineering team realised this gap the hard way. The deal dies. The engineering team spends the next three hours in a conference room with too much coffee, asking the same question they asked themselves six months ago: "How did we miss something so fundamental?"
That is the moment they realized the fatal flaws in their search methods in their RAG (Retrieval-Augmented Generation) system. The AI wasn't just wrong. It was confidently, comprehensively wrong in ways that revealed it didn't understand the basic context of what it was being asked.
We at KeyValue were developing a similar product for one of our clients, Wizly. The requirement was for an AI digital twin that could replicate an individual’s knowledge and response capabilities. This would allow the individual to delegate answering tasks to the twin and prioritize other work. We foresaw the problem that could occur in data retrieval and came up with a better approach that will effectively reduce hallucinations.
Before diving into the approach we took for Wizly, let’s pause to explore the two search methods and how choosing either of them would have changed the outcome.
Lexical approach: "Just give me what I asked for"
On one side, we have lexical search. It’s like that colleague who takes everything at face value. Let’s take an example of a client who is planning for the next quarter and wants to reference a specific strategy the consultant had previously shared. They phrase their question like this:
Client: "I need our Q4 marketing budget strategy."
Lexical search: "Here are 47 documents containing the exact words 'Q4,' 'marketing,' 'budget,' and 'strategy.'"
Digital twin: "Based on your request, I found documents related to Q4 server costs, general marketing trends, and your consultant's personal blog post on budget planning. They all mention the keywords you used."
This method's strength lies in its mathematical precision. It's unbeatable when a client needs specific policy references or exact project names. Ask the AI twin about a document titled "Project Phoenix Budget Report," and it will find documents about that exact project, not general budget discussions, and this is where lexical search in RAG systems, powered by sparse embeddings, proves its worth.
But this rigidity becomes its downfall. It completely fails on the "vocabulary mismatch" problem. A client asked, "How do we improve team culture?" will miss a consultant's document titled "Distributed Team Guidelines," and someone asking about "customer retention" will never find documents about "brand loyalty strategies."
Semantic approach: "I know what you really mean"
On the other side, we have semantic search powered by dense embeddings. This is your overly helpful friend who finishes your sentences. Let’s take a client who wants to know about strategies for improving team collaboration, but the phrase that comes to their mind is "team alignment." So they ask:
Client: "What are your views on team alignment?"
Semantic search: "Ah, team dynamics! Here's everything about communication workshops, employee satisfaction surveys, feedback loop best practices, and that blog post your consultant shared about leadership productivity."
Digital twin: "Your consultant believes in comprehensive team support, including clear communication channels, regular feedback sessions, and... [continues with tangentially related information]"
Client: "I just wanted the templates for the team workshop exercises."
This is the strength of semantic search with dense embeddings, where the AI bridges the gap between natural language queries and enterprise documents. This method excels at understanding the human intent behind a client's questions. It grasps that "team morale issues" and "employee satisfaction concerns" describe the same problem, and connects "career growth opportunities" with "professional development programs." It bridges the vocabulary gap that kills keyword search.
But sometimes being helpful becomes unhelpful. A question about "Q4 performance metrics" might get buried under general company performance content, and a request for "the deck on the new product launch" could return broad strategic planning documents instead of specific launch-related communications.
The breaking point: Why we stopped choosing sides
Based on the research done on these two methods, the data revealed two critical problems with the traditional either/or approach:
The precision problem: Users repeatedly received irrelevant responses when asking about specific document details or decisions because semantic search was "too helpful," burying exact CEO statements under broadly related corporate content.
The coverage problem: Natural language questions consistently failed because lexical search could not bridge the vocabulary gap between formal corporate documentation and casual employee questions.
The insight was clear, users don't care about the technology. They just want an AI twin that acts and thinks like their leadership team. The system was failing because it had to choose between giving exact facts and understanding the big picture, which made the AI twin's advice unreliable.
The hybrid solution: having both
So we did something radical: we stopped choosing sides entirely.
The new approach runs both search methods simultaneously, then intelligently combines the results. Think of it as having the CEO's personal assistant who knows the exact documents, working alongside the CEO's closest advisor who understands the deeper context and intent behind every decision.
The technical architecture
At the heart of our hybrid RAG (Retrieval-Augmented Generation) system lies the balance between sparse embeddings (lexical search) and dense embeddings (semantic search). Each addresses a different piece of the retrieval puzzle and, when combined, they elevate how information is surfaced for LLMs.
- Sparse embeddings are optimized for keyword-driven and exact-match retrieval. Think of them as highly precise filters: they latch onto specific terms, project names, or compliance policies to ensure nothing critical is overlooked. For instance, if a compliance officer searches for “GDPR Article 17,” sparse embeddings guarantee that the exact legal reference is retrieved. This makes them invaluable in use cases where accuracy, compliance, or industry-specific terminology must be preserved without ambiguity.
- Dense embeddings, by contrast, operate on meaning rather than exact words. They capture the semantic context behind language, how concepts relate to each other, even if different terms are used. For example, a query about “customer retention” would also surface insights on “brand loyalty” or “churn reduction,” even if those words never appear in the original question. Dense embeddings address the classic “vocabulary mismatch” problem, where human phrasing may not align with document terminology. They are particularly powerful in knowledge-intensive domains where intent recognition and contextual understanding matter more than literal keyword matches. To combine the best of both worlds, many systems employ Reciprocal Rank Fusion (RRF) ranking, which merges sparse and dense results into a single, balanced ranking that boosts overall retrieval quality.
By combining sparse and dense embeddings in a hybrid search framework, the system achieves the best of both worlds: the precision of lexical search with the intelligence of semantic search. This synergy reduces LLM hallucinations, improves retrieval accuracy, and ensures that retrieval-augmented generation delivers outputs that are both exact and contextually relevant, and this happens in three stages:
Parallel search execution
- Sparse engine using advanced models like SPLADE hunts for exact matches and relevant keywords.
- A dense engine powered by state-of-the-art embeddings captures semantic meaning and context.
- Both run simultaneously, so there is no speed penalty.
Intelligent fusion
We, at KeyValue, used Reciprocal Rank Fusion (RRF) to combine results. Instead of trying to merge completely different scoring systems, RRF focuses on relative rankings:
RRF (d) = 1/ (ranksparse (d) + k) + 1/ (rankdense (d) + k)
Documents that achieve high rankings in both systems receive a significant lift, while those strong in only one approach maintain a presence but with reduced prominence.
Unified results
The final ranking gives users exactly what they were hoping for:
- Precise matches when they need specific information
- Conceptual connections when they do not know the exact terminology
- Comprehensive coverage that does not miss edge cases
Real-world example
Let's revisit that client question about the Q4 marketing budget strategy:
Client asks: "I need our Q4 marketing budget strategy."
Sparse search finds:
- "Q4 2024 Marketing Strategy and Budget Allocation" (perfect exact match)
- "Q4 Sales Projections and Marketing Spend" (strong keyword overlap)
Dense search finds:
- "CEO’s vision on agile marketing and resource allocation" (conceptually relevant)
- "Quarterly Business Review - Performance and Future Planning" (broader context)
Hybrid search delivers: The AI twin now responds by giving the specific Q4 2024 Marketing Strategy and Budget Allocation document, which outlines the detailed spending plans. It also incorporates insights from the CEO’s broader vision on agile marketing, providing not just the numbers but also the strategic rationale and context for why the budget is structured that way. This gives the client both the precise information they asked for and a deeper understanding of the underlying strategy.
The results: From broken to best-in-class
Transformation after deploying hybrid search is undeniable:
- The hybrid approach significantly helped reduce hallucinations in RAG systems and accelerated enterprise AI adoption.
- Query Success Rate increased significantly, as employees consistently found the precise, relevant insights they needed from the AI twin.
- User Engagement increased, search abandonment dropped, indicating that employees were no longer giving up out of frustration.
- AI Twin Authenticity: The quality of the AI twin’s responses, as measured by internal scoring, has been consistently above 8 out of 10.
Implementation realities
"This sounds expensive and complicated."
Here is the truth: building a great search experience was never going to be cheap or simple. Our client Wizly expected state-of-the-art performance, and they knew what modern AI could deliver. The team had a narrow window to prove their system could not just tick the boxes, but exceed expectations.
Hybrid search wasn’t chosen because it was flashy. It was the only strategy that made sense. In a world where one critical miss can overshadow everything else, an approach that kept error rates low and relevance consistency high was an inevitable requirement.
And the payoff? Fewer hallucinations, more trustworthy answers, happier users, and a lighter load on support teams.
The bottom line
Building a great retrieval architecture is challenging. Building a great LLM is harder. But building a great RAG system on top of a mediocre retrieval architecture is impossible. The solution turned out to be simpler than expected: stop choosing between precision and understanding.
Ultimately, end-users don't care about the mathematical elegance of your embedding models or the technical sophistication of your ranking algorithms. They want to feel like they’re getting a reliable, intelligent, and trustworthy response to their questions. Give them both the precision of exact references and the intelligence of contextual understanding. Because in the end, the best RAG system is one that is reliable and trustworthy, regardless of how people choose to ask their questions.
