microsoft/thinkingbox: thinkingbox is a framework for defining tool as MCP servers, running LLM agents against them, and evaluating agent behavior — for offline training-data generation, reinforcement-learning training loops, and model evaluation.

thinkingbox is a framework for defining tool as MCP servers, running LLM agents against them, and evaluating agent behavior — for offline training-data generation, reinforcement-learning training l...

Summary

ThinkingBox is a Python framework from Microsoft for running LLM agents against mocked tools and evaluating their behavior. The core repo provides the tb CLI, an MCP Session Proxy that fronts a fleet of tool servers, and an agent/user/judge loop designed to produce conversations for offline training data, reinforcement-learning rollouts, and model evaluation. Real scenarios, datasets, and tool implementations live in the companion thinkingbox-data repo, while a separate thinkingbox-training repo adds RL post-training on top.

What it is useful for

Engineers working on agentic systems can use ThinkingBox to:

• Generate synthetic agent conversations offline. The framework spins up isolated MCP tool sessions, runs an agent against them, and captures the full transcript.
• Drive reinforcement-learning training loops. Because each session exposes deterministic tool state and side effects, the same setup feeds policy-optimization runs (group-relative policy optimization, LoRA adapters, multi-turn rollouts via thinkingbox-training).
• Benchmark agents on stateful business workflows. The MCP Session Proxy isolates state per session, so a test can assert against __reserved__geteffects after a tool call instead of scraping logs. Built-in example domains in thinkingbox-data include file storage, online banking, email, and a store FAQ.
• Simulate users and rubric judges. A separate "user" LLM and a "judge" LLM (including a RubricsJudge and a GeneratedAnswerEvaluator for RAG/QA) score multi-turn interactions without a human in the loop.

How engineers can use it

The README documents a Linux-only path (Python 3.12, uv). A typical first-time setup:

git clone https://github.com/microsoft/thinkingbox.git
git clone https://github.com/microsoft/thinkingbox-data.git
cd thinkingbox
uv venv --python 3.12
uv sync --group dev
uv run pre-commit install

To smoke-test the install without cloning real datasets, the repo ships one bundled scenario (cloud_drive). In one terminal:

uv run tb mcp-start

In another:

uv run tb infer -c config/config_o4mini.yaml --dataset ./dataset \
  --agent think --name cloud_drive.py:test_append_some_more_text \
  --output output.yaml
uv run tb pp output.yaml

For real workloads, point the proxy at the data repo's server config:

uv run tb mcp-start --servers ../thinkingbox-data/servers/servers.yaml

The tb CLI also exposes dump-tests, agg (aggregate metrics from JSONL), sbs (side-by-side candidate vs baseline), run-test, and an interactive tb tui for stepping through a single test case. Some tools (Typesense, embeddings) need extra services and a THINKINGBOX_DATA env var — see docs/tools_with_additional_setup.md. The TUI uses ESC then Enter to submit on Linux, not plain Enter.