HaroldConley/chunk-norris: Evaluate and compare chunking strategies for RAG pipelines

Evaluate and compare chunking strategies for RAG pipelines - HaroldConley/chunk-norris

Summary

chunk-norris is a Python library that empirically compares chunking strategies for Retrieval-Augmented Generation (RAG) pipelines. It chunks a document with multiple strategies, retrieves top-K chunks per question using hybrid search (BERT embeddings plus BM25 via Reciprocal Rank Fusion), and scores each strategy using deterministic metrics. The winning chunker is returned as an object that can be dropped directly into a RAG pipeline.

What it is useful for

It is useful when an engineer needs to pick a chunking strategy for a specific document rather than defaulting to a tutorial value like "512 tokens with 10% overlap." The library ships four chunkers: FixedChunker, ParagraphChunker, SentenceChunker, and RecursiveChunker. It is designed per-document, so a legal contract, technical manual, and FAQ can each get their own optimal strategy rather than a single compromise. Scoring uses two complementary metrics — answer span coverage (token recall) and semantic focus (BERT-based similarity) — which catch different failure modes; for example, a single full-document chunk would get perfect recall but very low semantic focus and be correctly penalised.

The evaluation has documented limitations. Token recall measures whether answer words are present in a chunk, not whether they appear in the right context, so a chunk containing the right words in a misleading sentence would score high. The library is optimised for English; BERT embeddings and NLTK sentence tokenisation may degrade on other languages, particularly for SentenceChunker. Multi-chunk questions (those requiring information from several sections) are not yet supported. Auto-generated questions are derived from the document's own vocabulary, so absolute scores run optimistic, even though the relative ranking between strategies remains reliable.

How engineers can use it

Installation is pip install git+https://github.com/HaroldConley/chunk-norris.git (Python 3.10+); a PyPI release is not yet available. The OpenAI key in .env is only needed for auto-generating questions, not for evaluation itself. Engineers can either call norris.generate_questions(text=..., llm=OpenAILLM(model="gpt-4o-mini-2024-07-18"), n=20) or supply a JSON list of {"question", "expected_answer"} pairs. norris.run(text=..., chunkers=[...], questions=...) produces a report; report.compare() prints a ranked table with Combined, Recall, Bert, and NRel columns, and report.best_chunker().chunk(text) returns the winning configuration. Results can be exported via report.to_excel("results.xlsx") or report.to_json("results.json"). Custom embedding models can be plugged in by implementing the BaseEmbedder interface.