From prompt engineering to fine‑tuning: Transforming document validation

Enterprise document validation, such as business partner deal submissions that include emails, meeting minutes, proposals, and quotes, presents a significant challenge for AI systems. Human reviewers traditionally evaluate these documents against established business engagement criteria and metadata requirements to determine deal approval, rejection, or discount eligibility. While large language models (LLMs) excel at general-purpose reasoning and natural language understanding, they often struggle with domain-specific evaluation tasks that require consistent, reproducible decision-making.

This article presents a hybrid architecture that addresses these limitations by combining a fine-tuned small language model (SLM) (such as Mistral 7B) with an LLM orchestrator (such as IBM Granite 3.3 8B Instruct). The fine-tuned Mistral 7B model, operating as a domain specialist, can perform this validation with greater consistency than prompt-engineered alternatives, while the IBM Granite 3.3 8B Instruct model handles orchestration and user-facing communication. Additionally, the article addresses a common architectural question: when to use retrieval augmented generation (RAG) versus fine-tuning and how these approaches serve complementary rather than competing purposes.

Limitations of prompt engineering for complex evaluation tasks

Before we adopted a fine-tuning approach, we experimented extensively with prompt engineering techniques, which revealed fundamental limitations. Standard prompting strategies, including few-shot examples, chain-of-thought reasoning, step-by-step templates, and structured frameworks such as CoSTAR, ReAct, and TAP, all provided incremental improvements but failed to achieve the consistency required for production deployment.

The core challenge lies in the nature of document evaluation itself. Determining whether a document demonstrates legitimate customer engagement requires nuanced judgment, such as distinguishing between template text and substantive evidence, recognizing when meeting invites contain meaningful agendas versus mere logistics, and understanding that keywords alone do not constitute proof. These distinctions require pattern recognition that emerges from repeated exposure to evaluated examples, which prompting alone cannot reliably produce.

Observed failure modes with prompt-only approaches included inconsistent application of criteria across similar documents, over-reliance on keyword matching instead of contextual evidence, non-deterministic reasoning paths for identical inputs across invocations, and incomplete adherence to the required evaluation schema. These limitations motivated the transition to a fine-tuning-based architecture.

RAG versus fine-tuning: Understanding the distinction

A common question in AI system design is whether to use RAG or fine-tuning. This framing is conceptually misleading. RAG and fine-tuning solve fundamentally different problems, and understanding this distinction is essential for effective AI system design.

What RAG provides: Dynamic knowledge access

RAG addresses the question of what information the model should reference. By retrieving relevant documents or data at inference time, RAG enables models to access knowledge that might not exist in their training data, including information that changes frequently, proprietary content, or domain-specific reference material. RAG excels when the challenge is knowledge currency or knowledge scope. RAG ensures the model has access to the right information at inference time to inform its response.

Typical RAG use cases include question-answering systems over proprietary document collections, providing up-to-date information beyond training cutoffs, grounding responses in authoritative sources to reduce hallucination, and accessing structured databases or knowledge graphs during inference.

What fine-tuning provides: Learned reasoning patterns

Fine-tuning addresses the question of how the model should reason. By training on examples that demonstrate the desired behavior, fine-tuning embeds judgment patterns, evaluation logic, and domain-specific reasoning directly into model weights. Fine-tuning excels when the challenge is reasoning consistency or domain adaptation. Fine-tuning teaches the model to think the way an expert thinks.

Typical fine-tuning use cases include consistent application of complex evaluation criteria, domain-specific classification or scoring tasks, maintaining strict output formats across invocations, and encoding tacit expert knowledge that resists explicit rule specification.

Comparative analysis of RAG and fine-tuning approaches

Why document validation requires fine-tuning

For our document validation use case, the primary challenge is not knowledge access, as the document content is already provided as input. The challenge is consistent evaluation by applying nuanced judgment criteria uniformly across thousands of diverse documents. RAG alone cannot solve this problem because retrieval augments available context without modifying the model’s underlying evaluation behavior or decision boundaries. A model that inconsistently applies evaluation criteria will remain inconsistent regardless of what documents are retrieved alongside the input.

This insight explains why prompt engineering reached a ceiling. Prompts can guide a model toward certain behaviors, but they cannot fundamentally alter its reasoning patterns. Fine-tuning, by contrast, modifies the model's internal representations, enabling it to develop the consistent judgment required for production-grade evaluation.

Use Case: Enterprise partner deal validation

Large enterprises with partner ecosystems often include thousands of business partners who resell products and services to end customers. When partners register deals for approval, which might qualify them for discounts, incentives, or preferred pricing, they must demonstrate legitimate customer engagement through supporting documentation. This documentation varies widely in format and content, ranging from email correspondence and meeting summaries to formal proposals and technical architecture documents.

The validation system for these registered deals evaluates documents against multiple business engagement criteria and metadata requirements. These typically include verifying genuine customer interaction, confirming alignment between documentation and registered deal details, and ensuring documents meet recency and relevance standards.

The evaluation logic enforces strict evidentiary standards. Keywords alone do not constitute proof; engagement requires explicit text evidence. Template text or checklists without supporting detail result in rejection. Meeting invites are accepted only when they include substantive agendas and explicit customer presence, not merely logistics. The system is constrained to cite only evidence that appears verbatim in the extracted document text, explicitly disallowing fabricated or inferred content. These principles require the model to internalize nuanced judgment patterns that resist encoding through prompts alone.

Hybrid SLM-LLM architecture (with optional RAG)

The hybrid architecture separates concerns between domain-specific evaluation, dynamic knowledge access, and general-purpose orchestration. This separation enables each component to excel at its designated function while maintaining clear interfaces between system layers.

This hybrid architecture includes these key components:

• A fine-tuned SLM
• An optional RAG system
• An LLM orchestrator

Fine-tuned SLM: The domain specialist

The fine-tuned Mistral 7B model serves as the evaluation specialist. Through supervised fine-tuning on hundreds of expert-annotated document reviews, this model learns the reasoning patterns that distinguish legitimate engagement evidence from insufficient documentation. Unlike prompt-based approaches, fine-tuning embeds evaluation logic directly into model weights, producing consistent outputs across invocations.

The SLM receives pre-processed document text along with contextual metadata (customer name, partner name, registered products, and date parameters) and produces structured output: a YES/NO determination for each evaluation criterion, accompanied by explanations citing specific evidence from the source text. This structured format enables downstream processing and audit trail generation.

Mistral 7B was selected as the fine-tuning base for several reasons. Its 7-billion parameter scale balances capability with computational efficiency, enabling local inference on Apple Silicon hardware (for example, M-series Macs) without requiring cloud GPU resources. The model's strong baseline performance on reasoning tasks provides a solid foundation for domain adaptation. Additionally, Mistral 7B's architecture supports efficient fine-tuning through parameter-efficient methods, reducing training time and resource requirements.

RAG system: Dynamic knowledge augmentation

While fine-tuning handles the core evaluation logic, RAG provides complementary value for knowledge that changes independently of reasoning patterns. The RAG system maintains a vector store that has three categories of reference material:

• First, the product catalog enables accurate metadata verification by providing current product family definitions, version mappings, and category hierarchies that may evolve as the organization's portfolio changes.
• Second, similar case retrieval allows the system to surface historical evaluations of comparable documents, providing contextual grounding for edge cases where confidence scores fall below predefined thresholds.
• Third, criteria definitions store the authoritative specification of each evaluation criterion, enabling updates to acceptance and rejection standards without model retraining.

This division illustrates the complementary nature of RAG and fine-tuning. The fine-tuned SLM knows how to evaluate documents consistently, while RAG provides what reference information might be relevant for specific verification tasks. Neither approach alone achieves what the combination provides.

LLM orchestrator: The workflow coordinator

The IBM Granite 3.3 8B Instruct LLM operates as the orchestration layer, handling responsibilities that benefit from general language understanding rather than domain-specific training. The LLM manages natural language summary generation for human reviewers, error recovery when document parsing fails or edge cases arise, clarification requests when submitted documentation is ambiguous, and coordination across multi-document submissions where a single deal may include multiple supporting files.

The orchestrator synthesizes outputs from both the fine-tuned SLM (structured evaluation results) and the RAG system (relevant reference material) to produce coherent, human-readable responses. This division reflects a practical insight: the LLM does not need to carry the burden of domain-specific reasoning or knowledge retrieval. By delegating evaluation to the fine-tuned specialist and knowledge lookup to RAG, the orchestrator can focus on communication and coordination tasks where its broad training provides value.

IBM Granite 3.3 8B Instruct was chosen for orchestration based on its instruction-following capabilities and alignment with enterprise deployment requirements. As an IBM-developed model, it integrates naturally with IBM's technology ecosystem while providing the conversational fluency needed for generating human-readable summaries and handling user interactions. The model's 8-billion parameter size maintains computational parity with the fine-tuned SLM, enabling balanced resource allocation across system components.

Fine-tuning methodology

Our fine-tuning methodology consists of:

• Training data preparation
• PEFT with LoRA and QLoRA
• Enterprise fine-tuning frameworks

Training data preparation

Effective fine-tuning requires high-quality training data that captures the reasoning patterns of expert human reviewers. The training dataset combines two sources: historical reviews where human experts evaluated real business partner submissions, providing ground truth for document-level decisions; and synthetic examples generated to cover edge cases and ensure balanced representation across all evaluation criteria, particularly for rejection scenarios that may be underrepresented in historical data.

Each training example follows the evaluation prompt structure, pairing document text and metadata inputs with the complete output schema. This format teaches the model not only what decisions to make but how to structure explanations and cite evidence appropriately.

Parameter-efficient fine-tuning with LoRA and QLoRA

Low-Rank Adaptation (LoRA) enables efficient fine-tuning by training small adapter layers rather than modifying all model parameters. This approach reduces memory requirements by an order of magnitude while preserving the base model's general capabilities. QLoRA extends this efficiency through 4-bit quantization of base model weights during training, further reducing VRAM requirements to enable fine-tuning on consumer hardware.

For this implementation, QLoRA fine-tuning targets the attention projection layers (q_proj, k_proj, v_proj, o_proj) with a rank of 64 and alpha of 128. These hyperparameters balance adaptation capacity with training efficiency, allowing the model to learn domain-specific evaluation patterns without overfitting to training examples. The parameter-efficient approach also minimizes catastrophic forgetting by freezing base model weights while training only low-rank adapter layers, preserving the model's general language capabilities.

Enterprise fine-tuning frameworks

Several frameworks support enterprise fine-tuning workflows. Unsloth provides optimized training kernels that accelerate fine-tuning by 2-5x while reducing memory consumption, making it particularly suitable for local development on constrained hardware. Hugging Face's PEFT (Parameter-Efficient Fine-Tuning) library offers production-grade implementations of LoRA, QLoRA, and related techniques with extensive documentation and community support. For teams preferring managed services, platforms like IBM watsonx.ai and AWS SageMaker provide integrated fine-tuning pipelines with enterprise security and compliance features.

RAG implementation for knowledge augmentation

Our RAG implementation consists of:

• Vector stores
• Knowledge categories and update patterns
• A retrieval strategy

Vector store architecture

The RAG component employs a vector database to store and retrieve reference material. For local deployment, ChromaDB or Milvus Lite provide lightweight vector storage with efficient similarity search. Production deployments may leverage managed services such as IBM watsonx.data, Pinecone, or Weaviate for scalability and enterprise features. The embedding model—typically a sentence transformer such as all-MiniLM-L6-v2 or IBM's Slate embedding models—converts both stored documents and queries into dense vector representations for semantic matching.

Knowledge categories and update patterns

Different knowledge categories require different update frequencies. The product catalog, which maps product names to registered categories for product metadata verification, updates quarterly as the organization releases new products or reorganizes portfolio categories. Similar case examples, drawn from evaluated documents that received expert review, accumulate continuously as new edge cases are resolved. Criteria definitions, which specify acceptance and rejection standards for each evaluation criterion, update infrequently but represent high-impact changes that would otherwise require model retraining.

This update pattern illustrates RAG's value proposition: knowledge changes that would require expensive model retraining can instead be handled through vector store updates, which complete in minutes rather than hours. The fine-tuned model's reasoning patterns remain stable while the knowledge it references evolves.

Retrieval strategy

The retrieval strategy balances precision with recall. For product catalog lookups, the system retrieves the top-3 most similar product definitions and applies a confidence threshold to filter weak matches. For similar case retrieval, the system identifies documents with comparable characteristics (document type, customer industry, product category) and surfaces historical evaluation outcomes with their reasoning. Hybrid retrieval combining dense vector search with sparse keyword matching (BM25) improves accuracy for queries containing specific product names or technical terminology that benefit from exact matching.

Evaluation framework

The evaluation framework compares the proposed architecture against multiple baselines to quantify performance gains and isolate the impact of fine-tuning. It tests four configurations: the baseline SLM (Mistral 7B without fine-tuning) using the complete evaluation prompt; the fine-tuned SLM applying learned evaluation patterns; the LLM with optimized prompting (Granite 3.3 8B) representing the best-case prompt engineering approach; and RAG-augmented LLM providing retrieved context alongside the evaluation prompt.

Evaluation metrics include accuracy (agreement with expert human judgments on held-out test documents), consistency (variance in outputs across multiple runs with identical inputs), schema compliance (adherence to the required output format without omissions or structural errors), and reasoning quality (human assessment of explanation coherence and evidence citation accuracy). These metrics capture both quantitative performance and qualitative factors essential for production deployment.

Notably, the RAG-augmented LLM configuration serves as a control to validate the architectural hypothesis: that for this evaluation task, fine-tuning provides benefits that RAG cannot replicate. The expectation is that RAG improves factual accuracy (particularly for product metadata matching) but does not substantially improve reasoning consistency—the fine-tuned SLM's primary advantage.

Implementation considerations

When implementing this architecture, consider these elements:

• Local inference configuration
• Pipeline orchestration and integration

Local inference configuration

The system runs entirely on local hardware, eliminating cloud API dependencies and associated latency and cost. On Apple Silicon, MLX provides optimized inference for both the fine-tuned Mistral 7B and Granite 3.3 8B models, leveraging unified memory architecture for efficient model loading. Alternative inference backends include llama.cpp for cross-platform deployment and Ollama for simplified model management.

Pipeline orchestration and integration

The evaluation workflow integrates into broader automation pipelines using frameworks such as LangChain or LlamaIndex for orchestration logic and RAG implementation, n8n or Apache Airflow for workflow automation and scheduling, and custom API layers exposing validation endpoints for integration with existing deal management systems. The fine-tuned SLM, RAG component, and LLM orchestrator communicate through structured JSON interfaces, enabling loose coupling and independent scaling of system components.

Benefits of the hybrid SLM-LLM-RAG architecture

The hybrid SLM-LLM-RAG architecture delivers advantages across multiple dimensions.

• Consistency improves dramatically because fine-tuned models, when configured with fixed decoding parameters (for example, temperature set to zero), produce highly consistent outputs for identical inputs, eliminating the run-to-run variance observed with prompt-based approaches.
• Adaptability comes from RAG's ability to update reference knowledge without model retraining, enabling rapid response to catalog changes or criteria updates.
• Cost efficiency follows from using smaller models optimized for specific tasks rather than routing all requests through larger, more expensive general-purpose models.
• Explainability benefits from the structured output format, which provides clear audit trails linking decisions to specific evidence in source documents.
• Scalability comes from encoding evaluation logic in model weights rather than extensive prompts, reducing per-request token consumption and enabling higher throughput.

Perhaps most significantly, the architecture transforms the developer experience. Rather than iterating endlessly on prompt refinements that never quite achieve production reliability, fine-tuning provides a direct path from expert knowledge to model capability. The model learns to evaluate documents through exposure to expert-annotated examples that encode tacit judgment patterns, rather than relying on brittle, explicitly defined rules. Meanwhile, RAG handles the dynamic knowledge requirements that would otherwise force frequent retraining cycles.

Summary

Prompt engineering has practical limits for complex evaluation tasks requiring consistent, domain-specific reasoning. RAG, while valuable for knowledge access, does not address these reasoning consistency challenges. The hybrid architecture presented in this article demonstrates an alternative approach: fine-tuning a small language model to serve as a domain specialist, augmenting it with RAG for dynamic knowledge access, and leveraging a larger model for orchestration and communication. This separation of concerns enables each component to excel at its designated function, producing a system that combines the consistency of specialized training with the flexibility of retrieval-augmented generation and general-purpose language models.

The distinction between RAG and fine-tuning is not a choice between competing approaches but an understanding of complementary capabilities. RAG answers the question of what knowledge to reference; fine-tuning answers the question of how to reason. Systems requiring both dynamic knowledge and consistent judgment (as most enterprise applications do) benefit from combining both approaches rather than choosing between them.

For practitioners facing similar challenges, fine-tuning offers a path beyond the ceiling imposed by prompt-only approaches. The tools and techniques described here, including LoRA, QLoRA, and frameworks like Unsloth, make this approach accessible on modest hardware, removing barriers that previously limited fine-tuning to organizations with substantial compute resources. Combined with RAG for knowledge management, this architecture provides a template for building enterprise AI systems that are both capable and reliable.