Event-Driven Architecture with Apache Kafka: Lessons from Production
· Samir Gautam
Event-driven architecture sounds simple in a diagram: a service publishes an event, other services react. In production, at real volume, the interesting problems all live in the gaps between those boxes.
The setup
On a recent NetSuite/HubSpot integration hub, we were buffering upwards of 100k transactions a day through Apache Kafka between an ERP sync daemon and a Postgres-backed CRM pipeline. That volume exposes issues that never show up in a demo.
What actually bit us
Consumer lag under bursty load. NetSuite inventory syncs don't arrive evenly — they spike after batch jobs. A consumer sized for average throughput falls behind during the spike and never catches up. We fixed this by partitioning by tenant ID and scaling consumer instances independently of the producer side, rather than assuming 1:1 capacity.
At-least-once delivery meets non-idempotent writes. Kafka's default delivery guarantee means a consumer can process the same message twice after a rebalance. If your downstream write is "insert a new deal in HubSpot," a duplicate delivery creates a duplicate deal. The fix isn't exactly-once semantics (which has its own costs) — it's making the write idempotent with a natural key:
hubSpotClient.upsertDeal(externalId, dealPayload);
// keyed on externalId, not a blind insert
Dead letters need a human-readable trail. Early on, poison messages just vanished into a DLQ topic with no context. Now every failed message is written with the original payload, the exception, and a retry count — because "why did lead #4821 never sync" needs an answer in under five minutes, not a Kafka archaeology session.
The pattern that held up
NetSuite RESTlets → Java Sync Daemon → Kafka → Postgres Buffer → HubSpot Webhook API
Kafka's job here isn't to be clever — it's to absorb bursts and give both sides of the integration room to fail independently without losing data. That's the whole value proposition of an event bus: not speed, but resilience.
If you're weighing whether your integration actually needs a message queue or if it's over-engineering for your volume, that's a conversation worth having before you build it, not after.