ClickHouse/walshadow

Contribute to ClickHouse/walshadow development by creating an account on GitHub.

Summary

walshadow replicates PostgreSQL rows into ClickHouse by reading the source database's physical write-ahead log. Written in Rust and licensed under AGPL-3.0, it covers initial load, continuous change streaming, schema evolution, restarts, and planned source switchover.

What it is useful for

The tool fits engineering teams that need to keep a ClickHouse analytics store in step with an operational PostgreSQL database. Rather than relying on logical decoding, triggers, or application-level change data capture, walshadow parses the physical WAL segments, which the project describes as enabling sub-second replication. Documented handling for schema evolution, restarts, and planned source switchover (see docs/failover.md) addresses common operational concerns. Three binaries ship from the workspace: walshadow-stream (the replication daemon), walshadow-filter (a segment-level filter for offline WAL files), and walshadow-classify (a record-level classifier for diagnostics). The filter and classifier let engineers inspect or trim WAL files offline without restarting the source database, which is useful for backfills and forensics.

How engineers can use it

The README points to two starting paths. A Quickstart uses Docker Compose to run PostgreSQL, ClickHouse, and walshadow together, while "Getting started" covers connecting existing databases. PostgreSQL 16 or newer is required, and the source plus the walshadow shadow must share the same PostgreSQL major version. Building from source clones submodules and uses Cargo:

git clone --recurse-submodules https://github.com/ClickHouse/walshadow.git
cd walshadow
cargo build --release
make -C pgext install

LZ4 compression is included by default; --no-default-features disables compression, and the zstd Cargo feature enables Zstandard. The walshadow PostgreSQL module is loaded via shared_preload_libraries rather than as a normal SQL extension, so the source must be restarted with that GUC set. Developers contributing changes run cargo nextest run --workspace --all-targets plus cargo clippy --all-targets -- -D warnings, which require initdb and pg_ctl on PATH. Operational details such as configuration, table selection, multi-database setups, and current limitations live under the docs/ tree. Engineers should review the AGPL-3.0 license terms before adopting it in commercial settings, and consult docs/limitations.md for the current scope of supported features.