Qdrant, the leading high-performance open-source vector database, today announced the launch of BM42, a pure vector-based hybrid search approach that delivers more accurate and efficient retrieval for modern retrieval-augmented generation (RAG) applications. The BM42 search algorithm marks a significant step forward beyond traditional text-based search for RAG and AI applications.
BM42 provides enterprises another choice – not just traditional text search or traditional vector search. This pure vector-based hybrid search combines the best of both to achieve better results at lower costs in the realm of RAG. This will help users excel in the unfolding AI-centric world.
Shifting from Keyword to Vector-First Search
Traditional keyword-based search engines, using algorithms like BM25 that have been around for over 50 years, are not optimized for the precise retrieval needed in modern applications and so struggle with specific RAG demands, particularly with short text segments that require further context to inform successful search and retrieval.
“By moving away from keyword-based search to a fully vector-based approach, Qdrant sets a new industry standard,” said Andrey Vasnetsov, Qdrant CTO & Co-Founder. “BM42, for short texts which are more prominent in RAG scenarios, provides the efficiency of traditional text search approaches, plus the context of vectors, so is more flexible, precise and efficient. While Qdrant envisions a future centered on vector-based search, this release helps to make vector search more universally applicable and marks a significant step toward the inevitable shift toward a vector-first approach.”
Qdrant’s BM42 introduces a new way of classifying search results and is well suited for RAG applications. Unlike traditional keyword-based search suited for long-form content, Qdrant’s solution integrates sparse and dense vectors to accurately pinpoint relevant information within a document. A sparse vector handles exact term matching. Dense vectors handle semantic relevance and deep meaning.
Boosting Accuracy, Efficiency, and Scalability
Developers often face critical decisions about choosing between sparse or dense vectors or a hybrid approach. Many existing hybrid solutions struggle with scalability and accuracy or are prohibitively expensive. Qdrant’s new hybrid search system addresses these challenges, providing an efficient, and cost-effective solution for both new and existing users. Most importantly, BM42 will enable users to quickly jump from prototype to production quickly, and then scale the solution globally.
Source: Business Wire