debezium/debezium: Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.

Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues. - debezium/debezium

Summary

Debezium is an open source change data capture (CDC) platform built on Apache Kafka and Kafka Connect. It monitors databases and emits an event for every committed row-level change, exposing a uniform change-event model across different database management systems. Only committed changes are visible to consumers, and the event history is durable so applications can stop and resume without missing events.

What it is useful for

The README describes several scenarios where Debezium applies:

• Cache invalidation: refreshing entries in caches such as Redis, Memcache, or Infinispan when underlying records change or are removed.
• Simplifying monolithic applications: replacing "dual-writes" (updating a database and then separate systems like search indexes, caches, or notifications) by reacting to CDC events in separate processes.
• Sharing databases: allowing multiple applications to learn about each other's committed changes.
• Data integration: streaming changes into other systems for ETL-style synchronization.
• CQRS and Event Sourcing: capturing write-side changes into totally ordered streams consumed by services that update read-side views.

It supports two operating modes. Connectors can run inside a Kafka Connect cluster for durable, replicated, totally ordered event streams, or the embedded connector engine can be used inside an application to send events directly without Kafka.

How engineers can use it

To work with the codebase, the README lists Git 2.2.1+, JDK 21+, Docker Engine 1.9+ (used to run databases for integration tests), and Apache Maven 3.9.8+ as requirements. The documented setup commands are:

$ git clone https://github.com/debezium/debezium.git
$ cd debezium
$ ./mvnw clean verify

If Docker is not available, integration tests can be skipped with ./mvnw clean verify -DskipITs, and artifacts alone can be produced with ./mvnw clean verify -Dquick. The README also documents running Postgres connector tests with the wal2json or pgoutput logical decoding plug-ins and testing against an external database such as Amazon RDS. The README does not provide a single end-to-end "start streaming from my database" command, so for runtime deployment details engineers should consult the project's documentation.

Documented limitations and caveats include: a few Postgres tests do not pass with the wal2json plug-in (see io.debezium.connector.postgresql.DecoderDifferences); on Apple Silicon the Docker Maven Plugin must use the linux/amd64 platform; and the build expects Docker to be running for the full integration test suite, which automatically starts and stops containers for several database systems.