GitHub - ENDEVSOLS/LongParser: Privacy-first document intelligence engine — parse PDFs, DOCX, PPTX, XLSX & CSV into AI-ready chunks for RAG pipelines. Includes HITL review, 3-layer memory chat, and a production FastAPI server.

Privacy-first document intelligence engine — parse PDFs, DOCX, PPTX, XLSX & CSV into AI-ready chunks for RAG pipelines. Includes HITL review, 3-layer memory chat, and a production FastAPI serve...

Summary

LongParser is a Python document intelligence engine that converts PDFs, DOCX, PPTX, XLSX, and CSV files into validated chunks for retrieval-augmented generation (RAG) pipelines. It combines Docling- and Marker-based extraction, hybrid chunking, PII redaction, and a human-in-the-loop review stage, and ships with a FastAPI server plus LangChain and LlamaIndex adapters. The project is described as privacy-first, with all processing running locally.

What it is useful for

LongParser fits when an engineer needs to feed mixed office documents into a RAG system and wants a pipeline that goes beyond naive text splitting. Documented capabilities include token-aware, heading-hierarchy-aware, and table-aware chunking; semantic chunking using the all-MiniLM-L6-v2 model; PII redaction that combines Regex with spaCy NER; and deterministic cross-referencing of charts and figures. A LangGraph-based approve, edit, and reject workflow pauses for a reviewer before chunks are embedded, and the 3-layer memory chat combines short-term turns, a rolling summary, and long-term facts for question answering on private document sets. The README also documents multi-provider LLM support (OpenAI, Gemini, Groq, OpenRouter) and multiple vector backends (Chroma, FAISS, Qdrant), plus a drop-in LangChain BaseRetriever and LlamaIndex QueryEngine adapter so it can slot into existing RAG stacks without rewriting retrieval code.

How engineers can use it

For local experimentation, the README documents a minimal Python entry point that creates a DocumentPipeline with ProcessingConfig, calls process_file on a document, and returns the parsed chunks. Installation is pip install "longparser[gpu]" for the full bundle or pip install longparser for the core SDK, with smaller extras such as server, chroma, qdrant, faiss-cpu, langchain, llamaindex, and a cpu bundle that avoids the roughly 2 GB CUDA torch download. For a production-style setup, the README outlines copying .env.example to .env, starting MongoDB and Redis via docker-compose up -d mongo redis, running the API with uv run uvicorn longparser.server.app:app --reload --port 8000, then uploading files via POST /jobs, finalizing, embedding, and chatting through the /chat endpoints. The README does not list benchmark numbers for parsing accuracy or throughput, and the GitHub license field shows NOASSERTION while the README states MIT, so the LICENSE file should be checked before redistribution. Python 3.10 to 3.13 is required.