This is the first in an ongoing ad-hoc series of posts on Apache Kafka performance. I have no general direction, I’ll just share interesting insights based on the performance testing I do on Apache Kafka. Recently I was curious to see if there was any general performance improvement since Kafka 3.X. So I ran a suite of benchmarks with Dimster (https://github.com/dimster-hq/dimster) against 3.7.2 and 4.3.0. I saw two common patterns: • *Pattern 1:**Low load benchmarks showed that end-to-end latency was higher with Kafka 4.3 compared to 3.7.2. The following is a 45 minute no-record-key workload of 5000 record/s, 20 topics (120 partitions), fan-out 2 (240 consumers), full TLS, on 3 brokers each allocated 8 SMT CPUs in k8s (on my Threadripper 9980X). Fig 1. Low load: end-to-end latency over time (p99 over 10 second intervals) • *Pattern 2:**On more stressful loads, 3.7.2 would show much more spiky end-to-end latency compared to 4.3. The following is for the same workload at 100K records/s (200K out). Fig 2. High load: end-to-end latency over time (p99 over 10 second intervals). Kafka 3.7.2 showed large latency spikes. Fig 3. High load: End-to-end latency distribution It seemed that somewhere between 3.7.2 and now, big performance gains had occurred. Then my subconscious kicked in and reminded me that at some point in that period, the default linger.ms had been changed from 0 to 5 ms. This would correlate with the low-load end-to-end latency result. Doing the math on linger.ms The linger.ms producer config controls how long the producer is willing to wait before sending a non-full batch (controlled by batch.size). If a batch reaches batch.size first, it can be sent earlier. The point of linger.ms is simple: give more records a chance to accumulate into the same batch, because larger batches are more efficient than many tiny batches. The important quantity is the rate “per producer, per partition” (rather than the aggregate rate). Kafka producers build batches per partition, so a producer sending 1,000 records/s to one partition has very different batching behavior from a producer sending 1,000 records/s evenly across 100 partitions. A rough way to reason about it is: expected records per batch ≈ 1 (first record) + linger_seconds × records_per_second_per_producer_per_partition For example, linger.ms=50 with a per-producer-per-partition rate of 100, we might expect 6 records per batch. This is only an approximation as it ignores arrival jitter, partition skew, batch.size config (default 16KB), compression, in-flight request limits, and broker backpressure. But it is good enough to build intuition. The math and the 5K workload In the 5K records/s workload, each producer was sending about 41 records/s: 5000 records/s / 120 producers = ~41 records/s per producer That is one record every: 1000 ms / 41 = ~24 ms This was also a no-record-key workload. With the default partitioning behavior, records from a producer tend to stick to one partition for a while before moving to another sticky partition. So, for batching purposes, the producer was usually sending roughly one record every 24 ms to its current sticky partition. That makes linger.ms=5 unlikely to help. A 5 ms linger is much shorter than the ~24 ms average gap between records, so most batches still contain a single record. To reliably get more than one record into a batch, the linger would need to be on the order of the inter-arrival time (tens of milliseconds), not 5 ms. So the low-load result made sense: Kafka 4.3’s default linger.ms=5 added a little extra waiting causing a higher end-to-end latency, but did not create meaningfully larger batches and its load was so low that larger batching wouldn't have helped anyway. The math and the 100K workload The 100K records/s workload was different. There, each producer was sending about 833 records/s: 100000 records/s / 120 producers = ~833 records/s per producer That is one record every: 1000 ms / 833 = ~1.2 ms At that rate, linger.ms=5 can make a real difference. A producer has time to collect several records before sending a batch. In this workload, I saw the average batch size reach about 5 KB, or roughly five 1 KB records per batch. That reduced the number of small produce batches the cluster had to process. It also improved downstream efficiency for the brokers and consumers. The result was a large reduction in tail latency: • the 3.7.2 run, with the old default linger.ms=0, had periodic p99.9 spikes around 700 ms, • while the 4.3.0 run, with the new default linger.ms=5, had a much lower and more stable p99.9 around 8 ms. So the benchmark was not necessarily showing a deep Kafka 3.7.2 versus 4.3.0 performance difference. A large part of the effect could be explained by one client-side default changing: linger.ms moved from 0 to 5 ms in Kafka 4.0. Running a linger.ms experiment on 3.7.2 and 4.3.0 I decided to run a similar benchmark again, explicitly setting linger rather than using defaults. This time I used half the producers (better for batching) but with record keys (much worse for batching). I ran Dimster on Kafka 3.7.2 (broker and clients) and 4.3.0 (broker and clients), with six test points across two scenarios: The 5K scenario If we look purely at the batching behavior, none of the linger values helped in the 5K records/s tests as the per-producer rate coupled with record keys meant that linger was ineffective at creating larger batches due to the low per-producer-per-partition rate. The chart below shows Kafka 4.3.0 over the three test points with linger of 0, 5 and 20. Only a linger of 20 slightly moved the needle. Fig 4. 5K workload. Batch sizes across lingers 0, 5 and 20 The exact same pattern occurred with 3.7.2. This workload did not need larger batches: the latency distribution for linger.ms=0 was already good. There was no difference in performance between 3.7.2 and 4.3.0. Fig 5. 5K workload, end-to-end latency distribution The 100K scenario The place where linger mattered was the 100K records/s keyed test. In that workload, linger.ms=20 showed a massive improvement over a linger of 0 and 5. Fig 6. 100K workload: end-to-end latency distributions for lingers of 0, 5 and 20 linger.ms=5 did not help much at all and we can understand why by doing the math: 100000 / 60 producers = 1666 records/s per producer. Due to record keys, 1666 / 6 partitions = 277 records/s per partition, per producer Send interval per partition = 1000 / 277 = 3.6 ms A simple estimate would predict about two records per batch at linger.ms=5 and about six at linger.ms=20, which lines up with the observed producer batch-size metrics below: Fig 7. 100K workload. Batch sizes across lingers 0, 5 and 20 The batching improvement with linger.ms=20 was reflected in the end-to-end latencies, with p99.9 of only 23 ms, compared to over 700 ms for a linger of linger.ms=5. Noteworthy is that the results for 3.7.2 and 4.3.0 with linger.ms=20 were essentially identical. 4.3.0 pulled ahead in the lower lingers, but there is often huge variance in the higher latencies, so from one run, this is inconclusive. Recommendations Don’t over-index on this one set of benchmarks. No benchmark is fully generalizable, and the right linger.ms value depends heavily on the workload. The main takeaway is simply this: pay attention to producer batch sizes. When producers are sending batches with only one record, Kafka can hit performance limits much sooner than you might expect. The broker has to process more produce requests, more record batches, more replication work, and more fetch-side batch metadata for the same logical throughput. A small amount of batching can make a large difference. The most important number to understand, with regard to likely batch sizes, is the per-producer-per-partition send rate. Total cluster throughput can be misleading. A workload doing 100K records/s may still produce tiny batches if each producer is spreading records across many partitions. Keyed workloads are especially prone to this, because the key determines the destination partition. If each producer writes to many keyed partitions, the effective rate into each producer-partition pair may be low. Under enough load, Kafka producers will often start batching more even with a low linger.ms, simply because the sender thread cannot drain records immediately. Broker latency, network saturation, throttling, or in-flight request limits can all cause records to accumulate in the producer. But relying on backpressure to create batching is not ideal. In some workloads, setting a higher linger.ms lets you get the batching benefit before the system is already under stress. The default linger.ms changed from 0 to 5 in Apache Kafka 4.0. That means some Kafka 4.x client upgrades may show performance improvements simply because the producer is now batching more by default. Conversely, if you are using Kafka 3.x clients, explicitly testing linger.ms=5 is a low-risk experiment. As for Kafka 3.7.2 versus 4.3.0, anecdotally, I’ve seen improvements in Kafka 4.x, and I may do more benchmarking to isolate those changes.