You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Today, as DLQ is currently implemented, the DLQ buffer is never "cleared" until consumer process is shut down. As a result, consumers will keep, in-memory, multiple partitions' worth of buffer data, if the consumers were assigned those partitions at any point in their lifetime. And, we know that during Kafka rebalancing, partitions are being shuffled around to various consumers.
This issue is for tracking the work to reset the buffers for specific partitions during partition revocation. The goal of this change is to ensure better memory management within the DLQ implementation of Arroyo.
The text was updated successfully, but these errors were encountered:
Today, as DLQ is currently implemented, the DLQ buffer is never "cleared" until consumer process is shut down. As a result, consumers will keep, in-memory, multiple partitions' worth of buffer data, if the consumers were assigned those partitions at any point in their lifetime. And, we know that during Kafka rebalancing, partitions are being shuffled around to various consumers.
This issue is for tracking the work to reset the buffers for specific partitions during partition revocation. The goal of this change is to ensure better memory management within the DLQ implementation of Arroyo.
The text was updated successfully, but these errors were encountered: