Niko1221/Strata

Qwen3.8-Flash-Next (125B MoE) on a 8GB+ NVIDIA GPU: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input

Summary

Strata is a C++ inference engine that runs the 125-billion-parameter Qwen3.8-Flash-Next model on consumer gaming PCs (12 GB or more VRAM, NVIDIA or AMD, Windows or Linux). It ships a one-click installer that downloads a quantized model and starts a local server exposing OpenAI- and Anthropic-compatible HTTP APIs.

What it is useful for

Strata is useful when you want to run a very large open-weights model locally without sending prompts off your machine. Engineers can point coding agents, chat tools, and custom HTTP clients at a private endpoint and keep data on the host. The README documents use cases including code generation, long-document processing (a 32K-token prompt benchmarks at about 1,620-2,650 tokens/s on an RTX 5070), image reading, and experimenting with MoE inference behavior on real hardware. Working on both NVIDIA and AMD GPUs from one binary is unusual for llama.cpp-style stacks, which is helpful if your shop has mixed hardware.

How engineers can use it

Download the archive (or git clone) and run START-HERE.bat on Windows or ./setup.sh on Linux. The installer detects your GPU, asks which quantized model fits your RAM, how much context to keep, and whether to enable picture input, then downloads about 70 GB (resumable) and starts the server at http://127.0.0.1:8080. The first start loads 35-55 GB into RAM and may freeze the PC for 1-3 minutes, which the README flags as expected.

Once running, point OpenAI-compatible clients at http://127.0.0.1:8080/v1 (any API key, any model name), Anthropic-style apps at http://127.0.0.1:8080/v1/messages (Claude Code via ANTHROPIC_BASE_URL=http://127.0.0.1:8080), and Codex CLI at /v1/responses. Two or three cards can share the model per docs/MULTI_GPU.md, and an MCP server lets AI tools start and stop Strata.

Pick a model by RAM: Coder on 32 GB, IQ2_XS or Q2_0 on 48 GB, IQ3_S on 64 GB, and Unsloth's ~4-bit variants on 96 GB+. Documented limitations to plan around: requests are served one at a time unless you set it to 2 in docs/BATCHING.md, AMD cards cannot yet read images on Windows, the Coder model is weaker for non-code text including CJK, and the experimental ~4-bit model writes only 7-8.5 tokens/s on a 64 GB PC because it streams from SSD. For problems, the README points to docs/TROUBLESHOOTING.md and the GitHub issue tracker, where you should attach strata-<model>.log from the Strata folder.