Beyond RAG basics: Advanced strategies for AI applications

Our recent virtual event with Cohere dove deep into the world of retrieval augmented generation (RAG), focusing on the critical considerations for building RAG applications beyond the proof-of-concept stage. Our speakers, Lily Adler, principal solutions architect at Elastic, and Maxime Voisin, senior product manager at Cohere, shared valuable insights on the challenges, solutions, and best practices in this evolving field of natural language processing (NLP).

Large language models (LLMs) are powerful but far from perfect. They often make absurd mistakes like suggesting putting glue on pizza or eating rocks — errors stemming from their training data without an inherent layer of logic. This is where RAG comes in, adding a crucial layer of control and context to help ground responses from the LLM. RAG is all about integrating relevant information retrieval systems with LLMs to enhance text generations. By grounding LLMs in contextually relevant data, RAG not only boosts response accuracy but also offers significant advantages in cost reduction and overall control. It helps in leveraging external knowledge sources, making the AI outputs more reliable and relevant.

A basic RAG architecture begins with user questions, using vector databases to retrieve relevant data, such as documents, images, and audio. This data then provides essential context for the LLM to generate a more accurate response.

However, an advanced RAG setup involves several layers with each playing a pivotal role:

• Data layer: Determines the type (structured or unstructured) and storage of information. Effective data management is crucial for high-quality information retrieval.

• Model layer: Incorporates foundational LLMs and embedding models. Fine-tuning these models is essential for handling specific tasks and improving performance in text generations.

• Application layer: Manages retrieval, prompts, and application logic, ensuring seamless integration of relevant documents into the workflow.

• Analysis and deployment layers: Ensure the solution is fit for purpose and efficiently deployed. Continuous analysis helps in refining model performance and adapting to new data.

Effective RAG solutions begin with a thorough understanding of the data landscape. When dealing with unstructured data like images or documents or structured data, such as databases, a robust chunking strategy is indispensable:

• Large vs. small chunks: Balancing context richness with precision. Large chunks provide more context but may reduce precision while small chunks are more precise but may lack complete information.

• Token overlapping: Ensuring continuous context across chunks, which helps in maintaining coherence in the retrieved information.

• Collapsing relevant chunks: Maintaining precision while always referencing the source for validation, ensuring the reliability of the information provided.

Security and legal considerations are also paramount. Access control mechanisms (LDAP, Active Directory) and privacy concerns, such as redacting sensitive information using named entity recognition, must be carefully managed to ensure compliance and user trust. These measures are essential to prevent data spills and unauthorized access to sensitive information.

• Parallel queries: Handling multipart questions with parallel search queries significantly improves response accuracy in RAG systems, making them adept at tackling complex user requests. This technique enables the system to break down and address different parts of a query simultaneously, ensuring a comprehensive and accurate response.

• RAG with tools: Extending RAG capabilities by integrating tools to handle complex data types (such as spreadsheets and SaaS apps) opens new possibilities for AI applications like workplace assistants. This incorporation allows RAG systems to interact with external knowledge sources, providing more comprehensive answers. For example, querying a database or a spreadsheet to provide data-driven responses can enhance the utility of the system in business and productivity applications.

• Agentic RAG: Equipping RAG systems with agentic capabilities allows for sequential reasoning and dynamic planning, making them robust against more complex queries. Agentic RAG systems can utilize multiple tools and adjust their plans based on the results they gather. This flexibility allows for more sophisticated problem-solving abilities and can handle intricate tasks that require multiple steps and logical reasoning.

Originally published on August 13, 2024; Updated December 17, 2024.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use. 

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.