Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reset DLQ buffers during rebalances #405

Open
ayirr7 opened this issue Dec 12, 2024 · 0 comments
Open

Reset DLQ buffers during rebalances #405

ayirr7 opened this issue Dec 12, 2024 · 0 comments

Comments

@ayirr7
Copy link
Member

ayirr7 commented Dec 12, 2024

Today, as DLQ is currently implemented, the DLQ buffer is never "cleared" until consumer process is shut down. As a result, consumers will keep, in-memory, multiple partitions' worth of buffer data, if the consumers were assigned those partitions at any point in their lifetime. And, we know that during Kafka rebalancing, partitions are being shuffled around to various consumers.

This issue is for tracking the work to reset the buffers for specific partitions during partition revocation. The goal of this change is to ensure better memory management within the DLQ implementation of Arroyo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant