SyncFlow — Open-Source Data Sync Platform

Moving data between databases is a solved problem for point-to-point ETL tools, but most are either licensed, opaque, or single-source. I wanted an open, self-contained data sync platform where the capture → transport → apply pipeline was transparent end to end — tail a source's change events with CDC, backfill the initial copy with snapshots, transform rows through a declarative mapping, and land them in any connected target without locking users to one vendor.

The harder requirement was operational trust: every event had to be delivered at-least-once, failures had to be replayable rather than silent, and a platform hosting multiple tenants had to keep each tenant's data strictly isolated.

The platform had to be honest about failure and about isolation from day one.

• ›Delivery: no silent data loss — a failed row goes to a replayable dead-letter queue, never dropped
• ›Ordering: per-source change order preserved; downstream consumers can lag without blocking capture
• ›Multi-tenancy: every read and write must be scoped to the caller's tenant — no cross-tenant leakage at any tier
• ›Extensibility: new connectors and processing rules plug in without modifying the core
• ›Observability: capture/sync throughput, retries, errors, queue depth, and pipeline operations all surfaced in Grafana
• ›Composable: a pipeline is a declarative design (source → mappings → transformations → destination) that can be versioned, previewed, and validated before it ships

I split the platform into an 8-module Gradle build following a hexagonal (ports & adapters) shape. At the center, `syncflow-core` holds the domain — pipeline, CDC, snapshot, connection, governance, and repository domain models with zero framework coupling. Around it, `syncflow-api` is the inbound/adapter side: REST controllers, SSE, Kafka consumers, and orchestration services that translate application concerns into domain calls. On the outbound side, `syncflow-connectors` and the `syncflow-plugin-api` module define the ports — `CdcProvider`, `SnapshotProvider`, `DestinationWriterProvider`, `PluginConnector` — so a new source or target database is a new adapter behind an SPI, never a change to the core.

The hex boundary is what makes the platform extensible without forking it: the core compiles, tests, and ships independently of which connectors are wired in, and the plugin API is the contract a third party implements.

Capture is Debezium CDC against source binlogs, publishing Row-level change events to Kafka. Delivery is at-least-once: consumers commit only after the row is durably applied, and a replayable dead-letter queue captures anything that exhausts retries. Snapshot backfill handles the initial full-table copy that CDC can't — the two paths converge on the same target contract.

The pipeline designer treats a pipeline as a versioned design — source, destination, per-table mappings, and a transformation rule engine (rename, constant, concat, convert, trim, default, substring, expression) — with JSON-snapshot versioning, validation probes, and conflict detection feeding a live preview.

Multi-tenancy is enforced at the repository/store layer, so every read and write is scoped to the current tenant by construction, not by convention.

• ›Hexagonal layout: `syncflow-core` domain at the center; `syncflow-api` inbound adapters; `syncflow-connectors` + `syncflow-plugin-api` outbound SPI ports
• ›Plugin ports (`CdcProvider`, `SnapshotProvider`, `DestinationWriterProvider`) keep new connectors out of the core
• ›Debezium CDC → Kafka: at-least-once, ordered per source, with a replayable DLQ
• ›Snapshot executor for full-table backfill as the companion to live CDC
• ›Versioned pipeline designs with JSON snapshots, rollback, preview, and conflict detection
• ›Transformation rule engine for per-table column orchestration
• ›Distributed agent fleet + workflow scheduler for fan-out and retries
• ›Observability: Micrometer counters/gauges/timers → Prometheus → Grafana; SSE for live status
• ›Multi-tenant repo/store scoping guards against cross-tenant leakage

Engineering is the act of choosing what to give up. These are the deliberate tradeoffs I made and why.

• 01A readable data-sync contract is worth more than a marketing one: at-least-once + idempotent targets + a replayable DLQ is what makes an outage replayable instead of a fire drill.
• 02Multi-tenancy is a construction discipline, not a filter. If the repository layer can't leak, the controller never can — you stop trusting every endpoint to remember to filter.
• 03CDC and snapshots are complements, not rivals. Shipping both modes is what makes a new-target onboarding actually work.
• 04The versioned pipeline-as-document model pays off the first time you preview a change, roll it back, or answer 'what changed between last week and now' — the same reason we version code.
• 05A plugin API is the difference between a platform and a monolith. If the core can compile and ship without 'one more connector', the core will actually ship.