Real-Time Chat & Audit Pipeline
WebSocket-driven backend with Kafka-backed audit log
A production-shaped real-time chat backend with JWT/RBAC, S3-backed file sharing with antivirus scanning, and a Kafka audit pipeline designed to scale message ingestion independently from delivery.
role: Sole engineer — owned API design, persistence schema, Kafka topology, and observability.
Outcome
The Problem
Chat is a deceptively expensive product surface — message delivery has to be synchronous and ordered per-conversation, but audit, search indexing, and analytics want the same data streamed asynchronously without coupling them to the hot path.
I wanted to model a system where the sync delivery loop and the async fan-out could each fail and recover independently.
Constraints
Designed against requirements that mirror enterprise SaaS auditing workloads.
- ›Message order preserved per conversation, never globally
- ›Audit log durable before client receives delivery ack — no silent loss
- ›File uploads scanned by AV before any participant can download
- ›RBAC enforced at the resolver level, not just the controller
- ›All long-running ops (AV scan, S3 multipart) must be retriable without duplication
Approach
Two flows, one shared store. The sync flow accepts a message over a WebSocket, persists it inside a transaction, publishes to a Kafka audit topic in the same transactional outbox, then acks the client. Delivery to other participants happens off the same WebSocket session manager. The async flow consumes the audit topic to drive search indexing, retention, and downstream analytics — these can lag without affecting message delivery.
Authentication is JWT with role claims; authorization is enforced inside GraphQL resolvers using a guard that reads the claims and the conversation membership in one query.
- ›Transactional outbox for Kafka publish — no dual-write inconsistency
- ›Per-conversation partitioning key keeps order without serializing the whole topic
- ›S3 multipart upload with pre-signed URLs; AV scan triggered by Kafka event, not by a synchronous HTTP call
- ›Flyway migrations versioned with the deployment artifact — schema and code ship together
Tradeoffs
Engineering is the act of choosing what to give up. These are the deliberate tradeoffs I made and why.
Bidirectional and lower per-message overhead at the volumes I cared about.
Eliminates the dual-write problem — the database is the single source of truth, Kafka is eventually consistent off it.
GraphQL collapses N+1 client round-trips for chat lists and unread counts; WebSocket handles the stream cleanly.
Keeps upload latency bounded; user sees 'scanning…' state while the worker pool catches up.
What I'd take with me
- 01If you publish to Kafka inside a request handler, you've built a dual-write bug that hasn't fired yet. The outbox isn't optional.
- 02Sticky sessions for WebSockets are fine — until they aren't. Have a plan for connection draining before you have a deploy.
- 03Antivirus scanning is the single biggest source of upload-flow regret. Keep it async, gate downloads, and let the UI tell the truth.
- 04Per-conversation partitioning keys keep order cheap. Per-tenant keys do not — design the key for the consumer, not the tenant.
Happy to discuss tradeoffs, run the code live, or whiteboard the next iteration.