pwilkin/trellis.cpp: TRELLIS.2 image-to-3D in C++/GGML (CUDA + Vulkan), with a resident HTTP server

TRELLIS.2 image-to-3D in C++/GGML (CUDA + Vulkan), with a resident HTTP server - pwilkin/trellis.cpp

Summary

trellis.cpp is a C++/GGML port of Microsoft's TRELLIS.2-4B image-to-3D pipeline. It runs background removal, image conditioning, three flow transformers, VAE decoders, mesh extraction, and UV-textured GLB export natively, with an optional long-running HTTP server and a Tauri-based desktop app called Trellis Studio. The project is licensed under MIT and offers a Python-free runtime that targets CUDA, ROCm/HIP, Vulkan, Metal, and Qualcomm HTP backends.

What it is useful for

The project turns a single 2D image into a textured 3D asset without Python at runtime, which is useful when you need to ship image-to-3D inside another product, embed it behind a service, or run it on hardware where Python and PyTorch are not practical. Concrete examples shown in the README include generating a textured GLB from assets/goblin.png end-to-end and serving generations through a /generate endpoint that returns model/gltf-binary. A trellis-server keeps the Vulkan context resident between requests, so it is well suited for batch or interactive use where startup cost matters.

It also exposes a text-to-3D path by chaining with stable-diffusion.cpp to produce the input image, plus a --model pixal3d switch for the Pixal3D fine-tune. Benchmarks against the reference Python implementation on the same GPU are included, with measured parity (for example, the Strix Halo iGPU runs a light goblin input in 6:09 versus 3:37 reference, and Apple M5 handles res-512 image-to-GLB in about 9 minutes at 5.6 GB peak RSS). Validation tolerances versus PyTorch are documented per stage, such as the sparse-structure sampler at relative 4.3e-3 and DINOv3 at 1.8e-2.

How engineers can use it

For a quick CLI run on a system with the prebuilt release binary, the README documents:

./build/trellis-cli assets/goblin.png out/goblin.glb

To run as a service, launch trellis-server and POST a multipart image file to /generate, with optional seed, resolution, and bg_removal fields; /health returns ok. The Trellis Studio installer auto-detects GPU runtime and downloads the server plus roughly 16.5 GB of weights, providing a drag-and-drop UI backed by the same binary.

To build from source, the README shows cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_VULKAN=ON (or CUDA, HIP, or no flag on macOS for Metal) followed by cmake --build build -j. Weights come from the ilintar/trellis2-gguf Hugging Face repo and are passed via --models DIR. For the optional all-quad retopology path, extra CMake flags (-DTRELLIS_RETOPO_MATCHING=ON -DTRELLIS_RETOPO_COLLISION=ON) and ample disk-backed scratch space under --retopo-workdir are required; this path is not included in the prebuilt binaries. Full installer options, troubleshooting, and the Pixal3D notes (including that MoGe-2 camera estimation is not ported) live in docs/getting-started.md and docs/pixal3d/README.md.