In my experience, Kinesis' performance makes it a nonstarter for low-latency systems and use cases with many consumers. Shards are limited to 5 reads/sec, so consumers are heavily throttled. Producers also have significant put latency. Kafka's latency and throughput are OOM better but of course come with the operational overhead. Depends on what your needs are, but Kafka is a better choice for "real-time" systems.
Interesting; I didn't know that Kafka had order-of-magnitude-better latency. Thanks for the data point.
I wonder if this is something inherent to Kinesis's design, or if Amazon will magically make it faster at some point in the future. Do you have a suspicion/indication either way?
Well, for one, there is no streaming socket API, just HTTP. I suspect the 5 reads/sec limit on shards is mainly due to multitenancy, but I could see this increasing in the future. Amazon has already improved the latency of Kinesis a fair amount, but it still has a ways to go before it matches Kafka.
We also have cases where we need many consumers reading a stream, so having that effectively limited to 5 concurrent consumers is a pretty tough limitation. Their solution to this is to just have consumers sleep on a failed poll, but that doesn't really help if you want to scale out your ingestion. You don't have that problem with Kafka.
Kinesis has gotten better about it's latencies, but it is inherently an HTTP protocol, which already gets you in pretty big trouble for serious real-time applications.
