vunone/ennoia: Declarative Document Indexing (DDI) framework for Python. Define schemas, extract structured indices, search smarter.

Declarative Document Indexing (DDI) framework for Python. Define schemas, extract structured indices, search smarter. - vunone/ennoia

Summary

Ennoia is a Python framework for Declarative Document Indexing (DDI): you describe what to extract from each document as a typed Python schema, and the framework runs LLM extraction, persists the structured output, and serves hybrid filter + vector search. The same schema file drives an SDK, a CLI, a FastAPI REST server, and a FastMCP tool surface, so offline indexing and online serving cannot drift apart. Status is alpha (0.4.x) and the API may still shift.

What it is useful for

Ennoia targets retrieval-augmented generation pipelines where documents carry structure (dates, parties, categories, clauses) that naive chunking discards. It collapses three jobs into one framework: writing hand-rolled extraction prompts, choosing chunking heuristics, and gluing a filter store to a vector store. The README highlights hybrid filter + vector search on the same corpus as the main payoff, and the included ESCI benchmark reports Ennoia leading on vague queries that lack a lexical anchor for similarity search.

Three schema primitives cover the common cases:

• BaseStructure for typed scalars or enums used as filters (one row per document)
• BaseSemantic for free-form text embedded into the vector index
• BaseCollection for repeated entities (parties, citations) where each row is independently filterable and searchable

A Pipeline wires schemas to an LLM adapter, an embedding adapter, and a Store. At query time it does two-phase retrieval: structural filters narrow candidates, then the vector index ranks within. Filter-only and vector-only queries are also allowed.

How engineers can use it

Install with extras that match the stack. The zero-key local path is pip install "ennoia[ollama,sentence-transformers,cli]"; hosted stacks use pip install "ennoia[all]".

Scaffold a config and a first schema from a sample document:

ennoia init
ennoia craft ./contracts/sample.txt --output my_extractors.py \
  --llm openai:gpt-4o-mini \
  --task "filter by governing law, effective date, and contract value"

craft writes a three-class skeleton (Metadata, QuestionAnswer, Summary) and is explicitly a starting point for human editing, not production output. Iterate on it with ennoia try <doc> --schema my_extractors.py, which runs one extraction and prints per-field results without writing to any store.

Then run the index → search → serve loop:

ennoia index ./contracts --schema my_extractors.py --store ./contracts_index
ennoia search "payment obligations and late fees" \
  --filter "governing_law=Delaware" --filter "effective_date__gte=2025-01-01" --top-k 5
ennoia api --store ./contracts_index --schema my_extractors.py --api-key "$ENNOIA_API_KEY"
ennoia mcp --store ./contracts_index --schema my_extractors.py --transport sse

For SDK use, the README shows a 15-line Pipeline example combining BaseStructure and BaseSemantic with in-memory stores and a local Ollama model. Stores are pluggable (in-memory, filesystem Parquet + NumPy, Qdrant, pgvector), and ennoia.testing ships mocks and pytest fixtures. The full filter operator table, server flags, and a worked MCP agent loop live in docs/cli.md, docs/serve.md, and docs/cookbook/mcp-agent.md.