eugenioenko/textmate-go

Pure-Go TextMate grammar tokenization engine

Summary

textmate-go is a pure-Go port of Microsoft's vscode-textmate that performs line-oriented TextMate grammar tokenization with reusable state stacks, so editors can syntax-highlight code incrementally without CGO. It claims exact token-text and scope-stack parity with the upstream TypeScript implementation and ships an optional grammars package with 124 embedded language grammars covering Go, TypeScript, Python, Rust, and many more. The project is MIT-licensed, requires Go 1.25 or newer, and supports CGO_ENABLED=0 builds.

What it is useful for

The library is a fit when a Go program needs to colorize, analyze, or transform source code using the same grammar definitions that VS Code uses, but without dragging in Node, a JS runtime, or a C dependency. Concrete scenarios include embedding syntax highlighting in a Go HTTP service, building a code-search or linting tool, powering editor backends in tools like LSP servers, and processing untrusted or minified input where the README notes size and time limits bound tokenizer work. The interned scope stacks and immutable states are designed to make editor caches efficient across many open files.

The optional grammars subpackage embeds license-reviewed grammars for languages from ActionScript through Zig, plus a Markdown support grammar for inline HTML, which avoids the operational burden of shipping grammar JSON separately.

How engineers can use it

The README documents installation and a quick start. Install with:

go get github.com/eugenioenko/textmate-go@latest

The quick start shows creating a registry with textmate.NewRegistry, passing grammars.Load as the LoadGrammar option, calling registry.LoadGrammar(grammars.ScopeForFilename("main.go")), and then feeding lines into grammar.TokenizeLine(line, state). For the first line you pass nil or textmate.InitialState; for subsequent lines you carry the returned RuleStack so tokenization stays correct across line boundaries. Token offsets are rune offsets and scopes are ordered from the root grammar down to the most specific match.

Documented limitations and caveats: the project is under active development and the API may evolve before a stable release. Tokenization uses a maintained github.com/eugenioenko/regexp2/v2 fork rather than Go's RE2-based regexp because TextMate grammars require lookbehind, backreferences, and position-sensitive anchors; Oniguruma-only syntax is translated, but unsupported constructs surface through grammar.Diagnostics() rather than halting the tokenizer. For editable documents, theme integration, and tuning tokenization limits, the README points to docs/usage.md.