Consensus upgrades academic research platform with advanced semantic search and AI tools from Elastic

Launched in 2022, Consensus is a pioneering search engine that uses advanced artificial intelligence (AI) and large language models (LLMs) to aggregate and distill insights from more than 200 million peer-reviewed papers from the Semantic Scholar database. All scientific domains are covered, making Consensus a preferred option among researchers in fields as diverse as medicine, anthropology, psychology, and climate science.

When the application first launched, it used an inference pipeline, based on a number of solutions including the default settings in Elastic Search. Christian Salem, CPO, Consensus, says, "At the time, we were one of a few companies in our space that used LLMs and AI summarization features on top of a search engine."

But, a year is a long time in artificial intelligence. With more companies entering the AI-search space, it became harder to stand out in an increasingly crowded marketplace. To stay ahead of the competition, Consensus hired engineers with search engine and AI expertise to improve search relevance and the end-user experience. Chris Varano, Consensus's lead search engineer with nearly a decade of experience at Amazon Search and google, helped lead the ELSER project from start to finish.

The team initially believed a vector search platform could be adapted to deliver better search performance, but it soon became clear that it lacked production-level support for millions of users and many useful lexical features. Varano says, "We could have built these features in-house, but for a small team in a fast-moving market, it was an expensive and time-consuming option."

The Consensus team turned to Elastic and its recently launched feature called ELSER (Elastic Learned Sparse Encoder). This new retrieval model, trained by Elastic, enables businesses to perform semantic searches based on contextual meaning and user intent, in addition to exact keyword matches. "We get all of the benefits associated with vector AI search while retaining the advantages of traditional keyword search functionality," says Varano.

With the launch of Consensus 2.0, powered by ELSER, end-users benefit from greater search accuracy and new generative AI features that provide a summary of the top results. The software takes the search query and runs a combination of keyword search and vector search over the abstracts and titles of all the papers. This gives Consensus an intelligent measure of the relevance of a document to the user's query.

This relevance score is then combined with many other pieces of metadata including citation count, velocity of citations, and publish date to re-rank the results and produce a top 20 possible results. The Consensus software then runs OpenAI's GPT-4 model over the top 10 results to produce a one sentence summary of the top studies.

As one of the earliest users of ELSER, the Consensus team collaborated closely with Elastic. "Once we got it working, it really exceeded our expectations. It blew away all the other vector search tests that we had done," says Varano.

The team also stresses the importance of Elastic's out-of-the-box functionality including lexical features, exact phrase matching, and keyword search in general. Filtering, term exclusion, and fuzzy matching also played a significant role.

ELSER and Elastic also provide underlying support for Consensus's retrieval augmented generation (RAG) workflow where Elastic search results form the basis for summaries generated by ChatGPT. "With Elastic and ELSER, we have high confidence in search quality and there's little or no risk of our AI layer hallucinating when generating summaries," says Salem.

Consensus also originally selected Elastic as a good fit for its underlying infrastructure and applications that are hosted on google Cloud. "One of the reasons that we went with Elastic was because it integrates so easily with google Cloud. Both are highly configurable, and we still feel like we have total control of the infrastructure," says Salem.

With the release of Consensus 2.0, powered by ELSER, end-users have seen a significant improvement in search relevance with question-specific information extracted from papers and AI-generated summaries of top results. Search is also faster, with an average semantic search time of almost four seconds reduced to less than a second with the new version. "Elastic is a huge differentiator in our space. We're not just a thin AI wrapper on top of someone else's data, we own the search engine and we're adding AI features on top of that," says Salem.

The numbers speak for themselves. Since launch, Consensus has seen a 30 percent increase in the number of queries that generate useful results for end-users. Salem has also received positive feedback from users who enjoy being able to use more flexible queries. "They don't have to use the exact same terms as the paper. Synonyms and colloquial terms also return relevant results," says Salem.

As an early-stage startup with about 8 people, Elastic's support was essential for Consensus. Salem says, "Our Elastic account executive rallied her engineers and experts around the project. Both parties saw it as an opportunity to build a new kind of search engine with the latest AI technology."

Although it is still in its early days, Salem and the rest of the Consensus team are looking forward to future Elastic AI releases, especially vector search features that can be combined with ELSER in the future. "Elastic is going full-speed ahead with AI- and LLM-powered search. They're a great partner to have because they keep us at the forefront of what is possible with the technology," says Salem.

Elastic also enables Consensus to maintain an advantage in the highly competitive AI-search space. Salem now wants to expand beyond academic research and include high quality data sets and expert knowledge outside of peer-reviewed journals. "We live in a world where there is a greater need than ever for accurate, precise information directly from the source. Elastic is a critical partner as we widen our search offering in hopes of filling this need," says Salem.