Fluentd Input S3 plugin Issue

[devinbfergy]

Hello I am running into issues with the fluentd s3 input plugin.

The plugin I am using is https://github.com/fluent/fluent-plugin-s3

I am running into this error:
2022-04-14 20:10:51 +0000 [warn]: #0 chunk bytes limit exceeds for an emitted event stream: 528456267bytes 2022-04-14 20:11:34 +0000 [warn]: #0 failed to write data into buffer by buffer overflow action=:throw_exception 2022-04-14 20:11:34 +0000 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.5/lib/fluent/plugin/buffer.rb:327:in write'" tag="input.s3"
2022-04-14 20:11:34 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.5/lib/fluent/plugin/buffer.rb:327:in write' 2022-04-14 20:11:34 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.5/lib/fluent/plugin/output.rb:1049:in block in handle_stream_simple'
2022-04-14 20:11:34 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.5/lib/fluent/plugin/output.rb:931:in write_guard' 2022-04-14 20:11:34 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.5/lib/fluent/plugin/output.rb:1048:in handle_stream_simple'
2022-04-14 20:11:34 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.5/lib/fluent/plugin/output.rb:921:in execute_chunking' 2022-04-14 20:11:34 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.5/lib/fluent/plugin/output.rb:851:in emit_buffered'
2022-04-14 20:11:34 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.5/lib/fluent/event_router.rb:115:in emit_stream' 2022-04-14 20:11:34 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-s3-1.6.1/lib/fluent/plugin/in_s3.rb:310:in process'
2022-04-14 20:11:34 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-s3-1.6.1/lib/fluent/plugin/in_s3.rb:192:in block in run' 2022-04-14 20:11:34 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/aws-sdk-sqs-1.51.0/lib/aws-sdk-sqs/queue_poller.rb:415:in block in yield_messages'
2022-04-14 20:11:34 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/aws-sdk-sqs-1.51.0/lib/aws-sdk-sqs/queue_poller.rb:414:in each' 2022-04-14 20:11:34 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/aws-sdk-sqs-1.51.0/lib/aws-sdk-sqs/queue_poller.rb:414:in yield_messages'
2022-04-14 20:11:34 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/aws-sdk-sqs-1.51.0/lib/aws-sdk-sqs/queue_poller.rb:407:in block in process_messages' 2022-04-14 20:11:34 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/aws-sdk-sqs-1.51.0/lib/aws-sdk-sqs/queue_poller.rb:406:in catch'
2022-04-14 20:11:34 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/aws-sdk-sqs-1.51.0/lib/aws-sdk-sqs/queue_poller.rb:406:in process_messages' 2022-04-14 20:11:34 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/aws-sdk-sqs-1.51.0/lib/aws-sdk-sqs/queue_poller.rb:338:in block (2 levels) in poll'
2022-04-14 20:11:34 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/aws-sdk-sqs-1.51.0/lib/aws-sdk-sqs/queue_poller.rb:333:in loop' 2022-04-14 20:11:34 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/aws-sdk-sqs-1.51.0/lib/aws-sdk-sqs/queue_poller.rb:333:in block in poll'
2022-04-14 20:11:34 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/aws-sdk-sqs-1.51.0/lib/aws-sdk-sqs/queue_poller.rb:332:in catch' 2022-04-14 20:11:34 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/aws-sdk-sqs-1.51.0/lib/aws-sdk-sqs/queue_poller.rb:332:in poll'
2022-04-14 20:11:34 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-s3-1.6.1/lib/fluent/plugin/in_s3.rb:186:in run' 2022-04-14 20:11:34 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.5/lib/fluent/plugin_helper/thread.rb:78:in block in thread_create'
2022-04-14 20:11:34 +0000 [warn]: #0 error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data"`

The buffer settings I have tried are:

@type memory
flush_thread_count 2
flush_interval 5s
chunk_limit_size 2M
queue_limit_length 60

I've mess around with increase the chunk limit size to be quite large 48M, more flush threads, smaller flush intervals. I haven't tried any file buffer.

Your help will be greatly appreciated.