Fluentd doesnt send logs

[Venture200]

What is a problem?

I am getting this error - enqueueing all chunks in buffer for specific index

Describe the configuration of Fluentd

I am using 1.16.2 and other node 1.16.6 as test rocky linux9

Describe the logs of Fluentd

enqueueing all chunks in buffer

Environment

- Fluentd version: 1.16.2
- TD Agent version:
- Fluent Package version:
- Docker image (tag):
- Operating system:
- Kernel version:

[daipom]

• from Fluentd keeps on enqueueing all chunks in buffer #2379

[daipom]

enqueueing all chunks in buffer is not an error log.
No problem with that log.
The problem is that some logs do not be sent, right?

Could you please share the steps to reproduce the phenomenon?
Does this occur if changing the output to a simple plugin, such as out_file?
First, we need to be sure if the problem is specific to a particular Output plugin.

[Venture200]

Hey, @daipom
Problem happens with specific log source, when I add tracing it doesnt show exact logs instead enqueueing but when I enable tracing for others, can clearly see the logs

also in debug mode can see these

2025-01-21 14:50:33 +0200 [info]: #0 Oj is not installed, and failing back to Yajl for json parser
2025-01-21 14:50:33 +0200 [info]: #0 [turbo] 'flush_interval' is configured at out side of <buffer>. 'flush_mode' is set to 'interval' to keep existing behaviour
2025-01-21 14:50:33 +0200 [debug]: #0 [turbo] 'host localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: localhost' doesn't have tag placeholder
2025-01-21 14:50:33 +0200 [debug]: #0 [turbo] 'index_name aks-write-alias' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: aks-write-alias' doesn't have tag placeholder
2025-01-21 14:50:33 +0200 [debug]: #0 [turbo] 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2025-01-21 14:50:33 +0200 [debug]: #0 [turbo] 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2025-01-21 14:50:33 +0200 [debug]: #0 [turbo] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2025-01-21 14:50:33 +0200 [debug]: #0 [turbo] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2025-01-21 14:50:33 +0200 [debug]: #0 [turbo] 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2025-01-21 14:50:33 +0200 [debug]: #0 [turbo] 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2025-01-21 14:50:33 +0200 [debug]: #0 [turbo] 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2025-01-21 14:50:33 +0200 [debug]: #0 [turbo] Need substitution: false
2025-01-21 14:50:33 +0200 [debug]: #0 [turbo] 'host_placeholder localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: localhost' doesn't have tag placeholder

[Venture200]

Hello, yes

but specific match setting i only get enqueueing all chunks in buffer
other match setting in separate config when i enable tracing at least i can see logs from fluent-bit. So the problem is that atm I have no clue what logs are arrriving to fluentd because logs dont tell anything

[daipom]

Hmm, could you share that match setting?

[Venture200]

<filter myapp.**>
 @type record_transformer
   remove_keys stream
</filter>

<match myapp.**>
  @type copy

  <store>
    @type elasticsearch
    @log_level trace
    @id myapp
    @include /etc/td-agent/config.d/defaults/elasticsearch.params
    index_name myapp-logs-write-alias
    log_es_400_reason true
  </store>

  @include /etc/td-agent/config.d/defaults/prometheus.store

</match>

[daipom]

Thanks! Could you share the settings for source that work and those that don't as well?

[Venture200]

working version

<filter secondapp.**>
   @type parser
   key_name log
   @log_level debug
   reserve_data true
   <parse>
      @type json
   </parse>
</filter>

<filter secondapp.**>
  @type record_transformer
  <record>
   namespace ${tag_parts[1]}
   application ${tag_parts[2]}
  </record>
</filter>

<filter secondapp.**>
 @type record_transformer
   remove_keys loggerFqcn,$.instant.epochSecond,$.instant.nanoOfSecond,endOfBatch,threadId,thread,threadPriority
</filter>


<match secondapp.**>
  @type copy

  <store>
    @type elasticsearch
    @id secondapp
    @include /etc/td-agent/config.d/defaults/elasticsearch.params
    index_name secondapp-app-write-alias
  </store>

  @include /etc/td-agent/config.d/defaults/prometheus.store

</match>

[jchalk222]

Any answer to this?
Thanks

[daipom]

Please give us more information.