Lifting Chatbot Faithfulness with Semantic Chunking

Customer Story

A mid-sized financial services company used an internal knowledge assistant to help staff and account holders find information across a large library of documentation, including product handbooks, compliance notes, troubleshooting guides, and FAQs. These documents, stored as PDFs and HTML files, totaled several thousand pages. The assistant used a standard retrieval-augmented generation (RAG) setup: it processed and indexed documents, then retrieved relevant passages from the language model when users asked questions. Since support staff and external account holders depended on these answers, any mistakes could hurt the company’s reputation. As a result, the team focused on ensuring answers were faithful to the source material rather than just sounding polished. 

The company partnered with V4C to redesign the system’s chunking strategy and directly measure its impact on groundedness.

Problem Statement

During the pilot rollout of the assistant, they encountered critical challenges with response reliability. While the retrieval-augmented generation (RAG) architecture successfully surfaced relevant documents, the system frequently produced articulate, self-assured answers that failed factual review.

The root cause was the chunking layer. Fixed‑width tokenization and recursive separator‑based splitting ignored semantic boundaries: related sentences were split apart, and unrelated topics were blended. As a result, the model had to reason over fragmented, misaligned context that was unacceptable for high‑stakes compliance and troubleshooting content.

Solution

To isolate the impact of chunking on groundedness, V4C evaluated three strategies on a fixed evaluation set using a uniform retrieval-and-generation pipeline. By keeping prompts and retriever components the same, any performance differences could be traced directly to the chunking strategy.

The evaluated strategies included:

• Fixed-size Chunking: A standard token window with minor overlap. While computationally efficient and easy to deploy, it ignores where sentences or paragraphs naturally end.
• Recursive Splitting:  A structured hierarchy of separators (paragraphs → sentences → words). Although this produced cleaner structural breaks, it focused more on document layout than on keeping related topics together.

• Semantic Chunking (IQR): The final approach used an interquartile-range (IQR) threshold to identify significant shifts in sentence embeddings. By placing boundaries at the points of highest cosine distance divergence, the system effectively recognized topic transitions rather than just structural breaks.

Semantic Chunking - Technical Detail

The chunker works in five offline steps during document ingestion:

1. Sentence encoding: We split each document into a sequence of sentences s(1)…s(n) using a punctuation-aware tokenizer. To ensure the evaluation metric prioritizes domain-specific semantics over broad similarity, each sentence is embedded with an encoder trained on relevant in-domain text pairs.‍
2. Adjacency distance: For every sequential pair, the cosine distance d(i) = 1 - cos(e(i), e(i+1)) is computed. This generates a one-dimensional series in which significant spikes indicate potential topic transitions. ‍
3. Per-document threshold: We compute Q1 and Q3 from the distance series, and set the cutoff at T = Q3 + k · IQR. By recalculating these quartiles for each document, the algorithm automatically calibrates for varied content types, such as dense regulatory text or conversational FAQs, avoiding the single-cutoff brittleness of mean-plus-standard-deviation rules.‍
4. Boundary placement: A chunk boundary was set between sentences i and i+1 whenever d(i) surpasses the threshold T. This results in semantically cohesive sentence groups of varying lengths.‍
5. Guardrail post-processing: In the final stage, we applied soft size constraints: small chunks are merged with their nearest neighbour to retain enough context for retrieval, and oversized chunks are re-split at the highest sub-threshold distance, keeping each chunk within the retriever and generator context budgets.

We tuned the IQR multiplier k with a coarse grid search on a held‑out validation slice, balancing over‑segmentation (fragmented lookups) against under‑segmentation caused by multiple topics per chunk. Using an IQR-based approach instead of a rigid absolute cutoff ensures the system remains resilient against outlier distances that might otherwise skew a standard-deviation-driven threshold.

All three methods were evaluated on the same 81‑question set with an identical retriever, prompts, and model with varying chunking strategies.

Method Comparison

‍

Table 1: Three chunking strategies evaluated on identical retriever, prompt, and generator components (n = 81 user questions).

Semantic chunking improved grounded answers by 27.1 percentage points over fixed‑size chunking and 17.3 points over the recursive splitter.

Challenges and Mitigations

• Higher embedding load at ingestion. Embedding every sentence during ingestion rather than only the final chunks significantly increased the embedding workload. We mitigated this by embedding requests, a content-hash caching mechanism that prevents re-embedding unchanged sentences, and by running the ingestion process offline, ensuring there is no impact on user-facing latency.

• Variable chunk length. Semantic chunking could produce segments that were either too short to preserve sufficient context or too long to fit within the retriever’s token budget. We addressed this with soft minimum and maximum chunk-size thresholds in the final processing step, ensuring balanced chunk lengths while preserving semantic boundaries.

• Sensitivity of the IQR multiplier. The IQR multiplier (k) controls how aggressively topic transitions are identified, making segmentation behavior sensitive to its setting. We ran a validation sweep across document types to determine an optimal value that balanced accurate topic boundary detection with consistent segmentation.

• Domain vocabulary fit. Generic embedding models often fail to adequately capture the significance of specialized financial terminology, potentially reducing retrieval accuracy. To address this, the selected embedding model was evaluated by V4C on in-domain sentence pairs to verify its ability to effectively represent financial concepts before adoption for production use.

• Judge-model noise. Evaluation results can be influenced by variability in the judge model’s responses, making consistent comparison difficult. V4C minimized this by using a fixed evaluation prompt, deterministic decoding, and a stable set of 81 queries across all three evaluation runs, ensuring a reliable performance comparison.

Results

Using the same set of 81 questions and binary yes/no faithfulness judgments from a judge model, fixed-size chunking produced grounded responses for 45 of 81 queries (55.6%). At the same time, the recursive splitter improved this to 53 (65.4%). Our approach of using semantic chunking delivered the strongest performance, grounding 67 of 81 responses (82.7%). This represents an absolute improvement of 27.1 percentage points and an approximately 49% relative gain over the baseline.

By exceeding the client’s 80% faithfulness target, semantic chunking met the threshold for a staged production rollout. The remaining ungrounded responses were primarily associated with questions requiring reasoning across multiple, widely separated sections or documents. This suggests that future improvements are more likely to come from enhancements to multi-hop retrieval and query rewriting than from further refinements to the chunking strategy.

‍Key Takeaway

Chunking is a critical determinant of RAG performance rather than a mere implementation detail, as demonstrated by results showing that semantically aligned chunk boundaries outperform approaches based solely on document structure or fixed-size segmentation. Moreover, using an adaptive threshold based on the interquartile range (IQR) enables the chunking strategy to accommodate variations in writing style, content density, and document structure. Unlike fixed thresholds, which impose a one-size-fits-all compromise, the IQR-based approach by V4C dynamically adjusts to each document, delivering more consistent and effective segmentation across diverse content types.

How v4c.ai Can Help & CTA

V4C specializes in turning domain‑heavy, compliance‑sensitive documentation into reliable RAG systems. We combine semantic, IQR‑driven chunking, domain‑tuned embeddings, and rigorous evaluation to raise faithfulness well beyond generic baselines.

If your LLM assistant is articulate but untrustworthy, contact v4c.ai to schedule a technical architecture review and explore how we can lift your groundedness beyond the 80% production threshold.