sausi-7/rag-benchkit

Contribute to sausi-7/rag-benchkit development by creating an account on GitHub.

Summary

RAG BenchKit is a Python and Streamlit toolkit that benchmarks combinations of chunking strategies, embedding models, and retrieval methods against your own documents and queries, then visualizes the results in a dashboard. It ships with four chunkers, three embedding options that run locally, two hosted options, and three retrieval methods, evaluating every configured combination on six retrieval metrics (Precision@K, Recall@K, MRR, NDCG@K, MAP@K, Hit Rate@K). The goal, per the README, is to make it easy to revisit chunking and embedding choices that RAG teams often set once and never test.

What it is useful for

The toolkit targets engineers building Retrieval-Augmented Generation pipelines who want to compare how their corpus behaves across chunking/embedding/retrieval choices. Concretely useful for: selecting between FAISS dense retrieval, BM25 sparse retrieval, and Reciprocal Rank Fusion hybrid retrieval on your actual data; comparing semantic vs. recursive vs. fixed-size vs. markdown/code-aware chunking; and weighing free local embeddings (MiniLM, BGE Small) against OpenAI Small/Large and Cohere without writing custom eval code.

It includes a sample dataset (10 Python tutorial .txt files plus 15 labeled queries) so you can run a demo end-to-end before supplying your own data. Limitations worth flagging — it only accepts .txt and .md documents, custom embedding or chunker implementations require modifying the source (the README points to README_TECHNICAL.md for extension details), and OpenAI/Cohere results depend on API keys entered in the Streamlit sidebar.

How engineers can use it

Setup follows the README directly. Clone the repo, create a Python 3.9+ virtual environment, install with pip install . (or pip install -e ".[dev]" for the pytest/ruff test suite), then launch the UI:

git clone https://github.com/sausi-7/rag-benchkit.git
cd rag-benchkit
python3 -m venv .venv && source .venv/bin/activate
pip install .
streamlit run app.py

Once the dashboard opens, click Load Sample Data, pick strategies, and hit Run Benchmark to see the leaderboard. To use real data, upload .txt or .md files (each filename becomes its doc_id) and a JSON queries file shaped like [{"query_id": "q01", "query": "...", "relevant_doc_ids": ["doc_01_python_basics"]}], where relevant_doc_ids matches document filenames without extension — these are the ground-truth labels the metrics are computed against.

The dashboard tabs (Leaderboard, Heatmaps, Charts, Per-Query Details) show rankings, a chunker/embedder performance matrix, and per-query hit/miss breakdowns for any selected configuration. Results can be exported as CSV or JSON. For architecture internals and extension guidance, the README defers to README_TECHNICAL.md and CONTRIBUTING.md.