MarketResearch.com uses Elastic to search millions of documents on its subscription platform Profound and provide customers with critical information about markets, competitors, and consumers

The market research sector is undergoing massive disruption with artificial intelligence (AI) changing customer expectations while competition from new market entrants is exerting pressure on the $81 billion industry. To grow their share, Marketresearch.com is adopting new ways of leveraging search technologies to deliver the most timely and impactful insights.

MarketResearch.com, a leading provider of global market intelligence products and services, is a trailblazer in the industry and the first to use AI-powered search to help its clients get the insights they need. Its Profound subscription service is built atop a massive library of syndicated market research reports from top global publishers. Most user engagement comes via search on the platform, which not only aggregates more than 1 million reports but also slices them into chapters, sections, and tables so users can select just the content they need.

"We've built up the largest database of market research reports in the world," said Rob Granader, CEO and Founder of MarketResearch.com. "Now combined with this new search technology, we are delivering even more tangible benefits to our users." Eighty percent of the Fortune 1000 use Profound, including the world's biggest banks, consultancies, and well-known international brands.

When Profound's incumbent search platform reached its end of life, MarketResearch.com seized the opportunity to replace its current search with a future-proofed alternative that included machine learning and AI functionality. This led them to select Elastic Search, which was already implemented on a smaller scale to support MarketResearch.com's Academic platform, used by colleges and universities.

Their tech team was impressed by the performance of Elastic in these environments but was equally struck by the size of the wider Elastic community and the pace at which new features were released to users. "We were looking for a long-term search vendor that was agile and scalable enough to support the entire MarketResearch.com business. Elastic ticked all the boxes," says MarketResearch.com CTO Javier Zamudio.

The next step was to find an Elastic deployment partner who could support the small technology team at MarketResearch.com. Zamudio turned to Industrial Resolution, which specializes in helping Elastic customers take their implementations to the next level. "One of the biggest challenges we face is managing millions of documents that can be hundreds of pages long," Zamudio said. "Industrial Resolution understood the scale of the task and in just a couple of months we were exceeding the number of queries supported by the previous solution."

The team at MarketResearch.com was also keen to migrate from keyword to semantic search techniques as the volume of searchable content increased.

MarketResearch.com is currently deploying Elastic Reciprocal Rank Fusion (RRF), a method for scoring documents to combine multiple result sets with different relevance indicators into a single result set. RRF requires no normalization, and different relevance indicators do not have to be related to each other to achieve high-quality results.

Another important benefit of this Elastic deployment is that it opens the door to generative AI. "Relevance is paramount, especially given the continuous flow of new reports into Profound's index," states Paul Goehrke, AI Strategy Specialist at MarketResearch.com. "We knew we needed to consider how we would ground any future generative AI services on Profound. The quality of the output, in that regard, is largely influenced by the quality of the input, of the retrieval. Having this new search will lead to better user experiences with generative AI."

Retrieval Augmented Generation (RAG) is one example of how Elastic is helping MarketResearch.com remove as many barriers as possible to the information that its customers need.

"We're constantly uncovering features or deploying new releases that add massive value to our relationship with Elastic," Zamudio says. "Industrial Resolution also played a vital role in helping us identify new developments and how to apply them to our services."

Tim Schreyer, Technical Account Lead, Industrial Resolution, also calls out Elastic's own resources when it comes to understanding the importance of new features and releases. "It's great that there are so many new tools being added by Elastic, but it's equally important that decision makers understand their potential. Resources such as Elastic Search Labs help developers inside organizations stay up to date with the Elastic roadmap and prepare for the opportunities ahead."

It's important to remember the sheer volume of documents and data hosted by MarketResearch.com. The business has already used Elastic to keyword index eight million documents, while the move to semantic search involves an additional 56 million documents. To manage this and other spikes in capacity, MarketResearch.com has migrated the majority of its search infrastructure to the Cloud. "With Elastic Search running on Elastic Cloud and Microsoft Azure, we can scale up to 16 machine learning nodes, get all the processing done in a week, and return to a normal rate," Zamudio said.

Above all, Granader believes that Elastic will play a central role as the business cements its reputation as a source of trusted information and customer-centric innovation in the age of generative AI. "Recently we've seen increased activity from sources that don't apply the same rigorous diligence to market research as we do."