text:
temporarily failed to flush the buffer Encoding::UndefinedConversionError" error=""\xC2" from ASCII-8BIT to UTF-8

Description
fluentd-kubernetes-daemonset does not send anything to the Cloudwatch group, it fails when trying to send with this error in the log:
2017-04-21 09:14:23 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:14:23 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:133:in `encode'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:133:in `to_json'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:133:in `block (2 levels) in write'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:126:in `each'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:126:in `block in write'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:104:in `each'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:104:in `write'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/buffer.rb:354:in `write_chunk'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/buffer.rb:333:in `pop'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/output.rb:342:in `try_flush'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/output.rb:149:in `run'

This is the complete log
$ kubectl logs dev-fluentd-cloudwatch-035t9 -n logging
2017-04-21 09:13:16 +0000 [info]: reading config file path="/fluentd/etc/fluent.conf"
2017-04-21 09:13:16 +0000 [info]: starting fluentd-0.12.33
2017-04-21 09:13:17 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2017-04-21 09:13:17 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.3.3'
2017-04-21 09:13:17 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '0.26.3'
2017-04-21 09:13:17 +0000 [info]: gem 'fluent-plugin-record-reformer' version '0.9.0'
2017-04-21 09:13:17 +0000 [info]: gem 'fluent-plugin-secure-forward' version '0.4.3'
2017-04-21 09:13:17 +0000 [info]: gem 'fluentd' version '0.12.33'
2017-04-21 09:13:17 +0000 [info]: adding match pattern="fluent.**" type="null"
2017-04-21 09:13:17 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2017-04-21 09:13:21 +0000 [info]: adding match pattern="**" type="cloudwatch_logs"
2017-04-21 09:13:21 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: adding source type="tail"
2017-04-21 09:13:22 +0000 [info]: using configuration file: <ROOT>
  <match fluent.**>
    type null
  </match>
  <source>
    type tail
    path /var/log/containers/*.log
    pos_file /var/log/fluentd-containers.log.pos
    time_format %Y-%m-%dT%H:%M:%S.%NZ
    tag kubernetes.*
    format json
    read_from_head true
  </source>
  <source>
    type tail
    format /^(?<time>[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?<severity>[^ \]]*) *\] (?<message>.*)$/
    time_format %Y-%m-%d %H:%M:%S
    path /var/log/salt/minion
    pos_file /var/log/fluentd-salt.pos
    tag salt
  </source>
  <source>
    type tail
    format syslog
    path /var/log/startupscript.log
    pos_file /var/log/fluentd-startupscript.log.pos
    tag startupscript
  </source>
  <source>
    type tail
    format /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
    path /var/log/docker.log
    pos_file /var/log/fluentd-docker.log.pos
    tag docker
  </source>
  <source>
    type tail
    format none
    path /var/log/etcd.log
    pos_file /var/log/fluentd-etcd.log.pos
    tag etcd
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/kubelet.log
    pos_file /var/log/fluentd-kubelet.log.pos
    tag kubelet
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/kube-proxy.log
    pos_file /var/log/fluentd-kube-proxy.log.pos
    tag kube-proxy
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/kube-apiserver.log
    pos_file /var/log/fluentd-kube-apiserver.log.pos
    tag kube-apiserver
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/kube-controller-manager.log
    pos_file /var/log/fluentd-kube-controller-manager.log.pos
    tag kube-controller-manager
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/kube-scheduler.log
    pos_file /var/log/fluentd-kube-scheduler.log.pos
    tag kube-scheduler
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/rescheduler.log
    pos_file /var/log/fluentd-rescheduler.log.pos
    tag rescheduler
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/glbc.log
    pos_file /var/log/fluentd-glbc.log.pos
    tag glbc
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/cluster-autoscaler.log
    pos_file /var/log/fluentd-cluster-autoscaler.log.pos
    tag cluster-autoscaler
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <filter kubernetes.**>
    type kubernetes_metadata
  </filter>
  <match **>
    type cloudwatch_logs
    log_group_name k8s-testq
    auto_create_stream true
    use_tag_as_stream true
  </match>
</ROOT>
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/kube-apiserver-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_POD-8c84359f13adea39ed4a55a5125d1d0c0b7cd3a8d2cafa3bd28f693bafcd896a.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/etcd-server-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_etcd-container-53daba4bbd8bf05809af2b38de842804849a344f909da17a8b65592812c98f44.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/prometheus-prometheus-node-exporter-b3nkb_monitoring_prometheus-node-exporter-cd080c1c0153db79442f30f8203c650ece2cf461e903b63f75e5fb8549d6ba69.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/kube-scheduler-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_kube-scheduler-fedcf148381b1376b40d02baab445a0cf6a9646fa864347b6d22662be6df585f.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/etcd-server-events-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_etcd-container-2b476b585a1fd76bd9aeb794ab768e7b517e8f19ef60683d4af55dcd17f9b678.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/dns-controller-3905064033-7dpzl_kube-system_POD-52d24c9afa094d70aff656bd86ab80520f3c55d461598749d10606c4cbc58769.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/kube-proxy-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_POD-aab68b3db2ba54044fd14cca911c3a0c404816518d85897f8811b87dbe130a5b.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/dev-fluentd-cloudwatch-035t9_logging_dev-fluentd-cloudwatch-295565aca55dfe7fe9dfacb6c55e693de6eedfd045a8c276593730ccf437e704.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/etcd-server-events-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_POD-f4e74260b2e6060169bbd10a04cbae1d4a6537511033cd30ceb8c08a7b469517.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/kube-controller-manager-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_kube-controller-manager-9b2ce2a7b180b03ddb6980dada16a75fd93e63791fddb710b902f51d8279310e.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/dev-fluentd-cloudwatch-0cf0x_logging_dev-fluentd-cloudwatch-5f7aeeffb3e6837a17d9fc44cdcbbd71025dc9e09b1be54d4b8cbc72e487f643.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/dns-controller-3905064033-7dpzl_kube-system_dns-controller-df93b047d0936b1cef30f3c4c7bb9004a0f51873279aeb25c7f31419c31f7f1f.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/kube-controller-manager-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_kube-controller-manager-e33d89c54cec783c7b1c93a1e1c71851373a4331c25496351747a59369cbbe24.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/kube-apiserver-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_kube-apiserver-c9d15e2dbbd44265d369cf26ecef754eb6fdf9ba26f7aa3648eacc8b0036735c.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/dev-fluentd-cloudwatch-0cf0x_logging_POD-6788ad180875f93ced6ea831b2c32e3159219f6807de0b42184b66130554cd23.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/calico-node-8s1nt_kube-system_install-cni-4ef4fd7464bcd03261eb8a92089c773ba12bfed7fb4b8e616ad4aa58890f9046.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/etcd-server-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_POD-fcb6afeb1431e326f918e8d007a3804f92030721fe0270487e86ea79899e2825.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/kube-scheduler-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_POD-912beb2045e51c78644cb32508f2da0f6186f82575b3f5457afbb8f7fcb0b046.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/prometheus-prometheus-node-exporter-b3nkb_monitoring_POD-ca353f808c94b2151b6ea49ddc77544612cf7eabbd185676eba3b41bf20f3261.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/kube-proxy-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_kube-proxy-984aa615033c397dfe53a7d946f4aa340d3e9befc921be0165a6500ac611cf62.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/calico-node-8s1nt_kube-system_calico-node-1858f09a9872eb301f48df6a6854ed119adee7b26f042c3fd4f53f0c2a27da21.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/calico-node-8s1nt_kube-system_POD-595063d9b99efb2b6bb24289467e73350985f0e3c64e561a24cab3aac45fae51.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/dev-fluentd-cloudwatch-035t9_logging_POD-58a8d71f2511784ed483503cbaea39502de9257cefe9807d95f763ddb295750c.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/containers/kube-controller-manager-ip-10-0-104-242.eu-west-1.compute.internal_kube-system_POD-dbe4ddf33c9c294d1497bb5ce5859d66a9903f87d5ca110b2d34768e00862f73.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/etcd.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/kube-proxy.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/kube-apiserver.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/kube-controller-manager.log
2017-04-21 09:13:23 +0000 [info]: following tail of /var/log/kube-scheduler.log
2017-04-21 09:13:52 +0000 [info]: detected rotation of /var/log/containers/dev-fluentd-cloudwatch-0cf0x_logging_dev-fluentd-cloudwatch-5f7aeeffb3e6837a17d9fc44cdcbbd71025dc9e09b1be54d4b8cbc72e487f643.log; waiting 5 seconds
2017-04-21 09:13:52 +0000 [info]: detected rotation of /var/log/containers/dev-fluentd-cloudwatch-0cf0x_logging_POD-6788ad180875f93ced6ea831b2c32e3159219f6807de0b42184b66130554cd23.log; waiting 5 seconds
2017-04-21 09:14:23 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:14:23 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:133:in `encode'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:133:in `to_json'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:133:in `block (2 levels) in write'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:126:in `each'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:126:in `block in write'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:104:in `each'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.3.3/lib/fluent/plugin/out_cloudwatch_logs.rb:104:in `write'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/buffer.rb:354:in `write_chunk'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/buffer.rb:333:in `pop'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/output.rb:342:in `try_flush'
  2017-04-21 09:14:23 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/output.rb:149:in `run'
2017-04-21 09:14:24 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:14:26 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:14:24 +0000 [warn]: suppressed same stacktrace
2017-04-21 09:14:26 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:14:30 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:14:26 +0000 [warn]: suppressed same stacktrace
2017-04-21 09:14:30 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:14:37 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:14:30 +0000 [warn]: suppressed same stacktrace
2017-04-21 09:14:38 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:14:53 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:14:38 +0000 [warn]: suppressed same stacktrace
2017-04-21 09:14:54 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:15:21 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:14:54 +0000 [warn]: suppressed same stacktrace
2017-04-21 09:15:22 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:16:19 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:15:22 +0000 [warn]: suppressed same stacktrace
2017-04-21 09:16:19 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:18:37 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:16:19 +0000 [warn]: suppressed same stacktrace
2017-04-21 09:18:38 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:23:09 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:18:38 +0000 [warn]: suppressed same stacktrace
2017-04-21 09:23:09 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:32:17 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:23:09 +0000 [warn]: suppressed same stacktrace
2017-04-21 09:32:17 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 09:49:24 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:32:17 +0000 [warn]: suppressed same stacktrace
2017-04-21 09:49:24 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 10:20:37 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 09:49:24 +0000 [warn]: suppressed same stacktrace
2017-04-21 10:20:37 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 11:29:40 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 10:20:37 +0000 [warn]: suppressed same stacktrace
2017-04-21 11:29:41 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-21 13:57:55 +0000 error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="object:2aeda49bb36c"
  2017-04-21 11:29:41 +0000 [warn]: suppressed same stacktrace

Repro steps:

Create a namespace so you can clean up easilly afterwards:
kubectl create namespace encodingbug

Put the following into fluentd-cloudwatch.yaml

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    app: fluentd-cloudwatch
    release: dev
  name: dev-fluentd-cloudwatch
spec:
  selector:
    matchLabels:
      app: fluentd-cloudwatch
      release: dev
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: fluentd-cloudwatch
        release: dev
    spec:
      containers:
      - env:
        - name: AWS_REGION
          value: eu-west-1
        - name: LOG_GROUP_NAME
          value: k8s-testq
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              key: aws_access_key_id
              name: dev-fluentd-cloudwatch
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              key: aws_secret_access_key
              name: dev-fluentd-cloudwatch
        image: fluent/fluentd-kubernetes-daemonset:v0.12.33-cloudwatch
        imagePullPolicy: IfNotPresent
        name: dev-fluentd-cloudwatch
        resources:
          limits:
            cpu: 100m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        terminationMessagePath: /dev/termination-log
        volumeMounts:
        - mountPath: /var/log
          name: varlog
        - mountPath: /var/lib/docker/containers
          name: varlibdockercontainers
          readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /var/log
        name: varlog
      - hostPath:
          path: /var/lib/docker/containers
        name: varlibdockercontainers
---
apiVersion: v1
data:
  aws_access_key_id: REDACTED_AWS_ACCESS_KEY_ID
  aws_secret_access_key: REDACTED_AWS_SECRET_ACCESS_KEY
kind: Secret
metadata:
  labels:
    app: fluentd-cloudwatch
    release: dev
  name: dev-fluentd-cloudwatch
type: Opaque

Replace REDACTED_AWS_ACCESS_KEY_ID and REDACTED_AWS_SECRET_ACCESS_KEY by a valid AWS user with proper permissions in file fluentd-cloudwatch.yaml

Run kubectl create fluentd-cloudwatch.yaml --namespace encodingbug

Check the pods got created:
kubectl get pods --namespace encodingbug -l "app=fluentd-cloudwatch,release=dev"

Check the logs:
This checks all pods:
kubectl  logs --namespace encodingbug -l "app=fluentd-cloudwatch,release=dev"
You can (should) specify a specific pod:
kubectl  logs dev-fluentd-cloudwatch-035t9 --namespace encodingbug

Clean up step:
kubectl delete namespace encodingbug

Suggestion:
Check this Stackoverflow entry
Versions:
Docker image: fluent/fluentd-kubernetes-daemonset:v0.12.33-cloudwatch
Kubernetes: 1.5.2

Thanks for detailed report @edsiper - seems linked to https://github.com/ryotarai/fluent-plugin-cloudwatch-logs/issues/65

I'm having the same issue, looks like https://github.com/ryotarai/fluent-plugin-cloudwatch-logs/pull/58 could fix it, tho. Any ideas of why the PR haven't been merged?

Hey @kenden , I have a question:
I've been trying to deploy this daemon to integrate fluentd + CloudWatch with little success.
Even though I've followed these instructions I keep getting this error:
https://github.com/fluent/fluentd-kubernetes-daemonset/issues/42
Have you ever seen it before? I'm running out of options here, maybe you can help

After a lot of tests it looks like the issue is related to the version of the fluent-plugin-cloudwatch-logs gem. If you update it to 0.4.0 the encoding error goes away.
I build a docker image using
FROM fluent/fluentd:v0.12.33-debian

USER root
WORKDIR /home/fluent
ENV PATH /home/fluent/.gem/ruby/2.3.0/bin:$PATH

RUN buildDeps="sudo make gcc g++ libc-dev ruby-dev libffi-dev" \
     && apt-get update \
     && apt-get install \
     -y --no-install-recommends \
     $buildDeps \
    && echo 'gem: --no-document' >> /etc/gemrc \
    && gem install fluent-plugin-secure-forward \
    && gem install fluent-plugin-record-reformer \
    && gem install fluent-plugin-cloudwatch-logs -v 0.4.0 \
    && gem install fluent-plugin-kubernetes_metadata_filter \
    && gem install ffi \
    && gem install fluent-plugin-systemd -v 0.0.8 \
    && SUDO_FORCE_REMOVE=yes \
    apt-get purge -y --auto-remove \
                  -o APT::AutoRemove::RecommendsImportant=false \
                  $buildDeps \
 && rm -rf /var/lib/apt/lists/* \
    && gem sources --clear-all \
    && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

# Copy configuration files
COPY ./conf/fluent.conf /fluentd/etc/
COPY ./conf/systemd.conf /fluentd/etc/
COPY ./conf/kubernetes.conf /fluentd/etc/

# Copy plugins
COPY plugins /fluentd/plugins/

# Environment variables
ENV FLUENTD_OPT=""
ENV FLUENTD_CONF="fluent.conf"

# jemalloc is memory optimization only available for td-agent
# td-agent is provided and QA'ed by treasuredata as rpm/deb/.. package
# -> td-agent (stable) vs fluentd (edge)
#ENV LD_PRELOAD="/usr/lib/libjemalloc.so.2"

# Run Fluentd
CMD exec fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT

Uploaded it to the Docker Hub (https://hub.docker.com/r/alterx/fluentd-cloudwatch/) and it works like a charm. As you can see in the Dockerfile above, the only thing I changed was the the version of the fluent-plugin-cloudwatch-logs gem

@alterx Thanks for the feedback. there's a PR to update the version of the plugin:
https://github.com/fluent/fluentd-kubernetes-daemonset/pull/44

Perfect, will keep an eye on that PR :D

@alterx The PR was merged, the version of the ruby package fluent-plugin-cloudwatch-logs used is now 0.4.0.

Fixed by #44

authors: alterx edsiper kenden so0k

repo: fluent/fluentd-kubernetes-daemonset


text:
container logs are not getting update continuously

Hi ,
i have configured kubernetes cluster in AWS environment using KOPS.
i  have used  the below git hub (fluentd-kubernetes-daemonset/docker-image/v0.12/debian-cloudwatch/).
i am trying to send application & kubernetes logs to cloud watch through fluentd. using fluent-plugin-cloudwatch-logs -v 0.4.0 plugin. however the application & cluster level logs stopped updating in cloudwatch within 10-15 mins.
it doesn't work even recreating the fluentd pod.
help me to fix this issue.

Possibly related issue and fix: https://github.com/kubernetes/charts/issues/5514
Someone else said v1.1 also fixes a similar issue.
I tried v1.2 but it just crashes (#150).

source: gharchive/issue

authors: kvranjith whereisaaron

license_type: permissive

license_source: github-api


text:
_key is listed as fluentd.warn

We are facing an issue where in if the fluentd docker container comes up before the elk, all the K8 logs have the key as fluentd.warn. This issue is seen even after i restart the docker container of fluentd.
However in case if the fluentd docker container is brought up after the elk the key is set to the log file name. The following is the tdagent.conf
<source>
  type tail
  format multiline
  format_firstline /^\w\d{4}/
  format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
  time_format %m%d %H:%M:%S.%N
  path /varlog/*controller*.log
  pos_file /var/log/es-kube-controller-manager.log.pos
  tag kube-controller-manager
</source>

<source>
  type tail
  format multiline
  format_firstline /^\w\d{4}/
  format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
  time_format %m%d %H:%M:%S.%N
  path /varlog/*apiserver*.log
  pos_file /var/log/es-kube-apiserver.log.pos
  tag kube-apiserver
</source>

<source>
  type tail
  format multiline
  format_firstline /^\w\d{4}/
  format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
  time_format %m%d %H:%M:%S.%N
  path /varlog/*scheduler*.log
  pos_file /var/log/es-kube-scheduler.log.pos
  tag kube-scheduler
</source>
<filter kubernetes.**>
  type kubernetes_metadata
</filter>

<match **>
   type elasticsearch
   log_level info
   include_tag_key true
   tag_key _key
   host elk
   port 9200
   logstash_format true
   # Set the chunk limit the same as for fluentd-gcp.
   buffer_chunk_limit 2M
   # Cap buffer memory usage to 2MiB/chunk * 32 chunks = 64 MiB
   buffer_queue_limit 32
   flush_interval 5s
   # Never wait longer than 5 minutes between retries.
   max_retry_wait 30
   # Disable the limit on the number of retries (retry forever).
   disable_retry_limit
</match>

Appreciate any help/workaround.

Could you show me the log message?

fluentd.txt
Attaching the logs

Where is _key related warn?

It was seen in Kibana. I dismantled my setup, shall bring it up and update.

It was seen in Kibana.

Ah, I see.
You use ** in match and it is why your es has fluent.warn.
Check this article: http://docs.fluentd.org/articles/logging#capture-fluentd-logs

It's better to use @label ... in all <source> sections, and <label> section to handle events from input plugins excluding events from Fluentd internal logs.
http://docs.fluentd.org/articles/config-file#5-group-filter-and-output-the-ldquolabelrdquo-directive

authors: maverick-racheal repeatedly tagomoris

repo: fluent/fluentd


text:
warn message should include right status

Check CONTRIBUTING guideline first and here is the list to help us investigate the problem.
Describe the bug

I received a log message like below
2020-07-14 01:24:15 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted
 event stream: 574180bytes

But I set to  chunk_limit_size is 8M and chunk_full_threshold is 0.8.
8M*0.8 = 6.4M
chunk has free space more than 1.6M but log message say 574K.
I checked the chunk and realized that there are many records.
The message received was a bit unfriendly.
To Reproduce

set large chunk_limit_size
set small chunk_limit_records
start fluentd
push large number of minimum records
show warn message

Expected behavior

2020-07-14 01:24:15 +0000 [warn]: #0 [rawlog2elasticsearch] chunk size limit exceeds for an emitted
 event stream: 1400records

Your Environment

Fluentd or td-agent version: fluentd --version or td-agent --version

$ td-agent --version
td-agent 1.11.1

Operating system: cat /etc/os-release

$ cat /etc/os-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

Kernel version: uname -r

$ uname -r
4.14.177-139.254.amzn2.x86_64

If you hit the problem with older fluentd version, try latest version first.
Your Configuration
  <source>
    @type forward
    @id input_forward
    port 24224
    bind "0.0.0.0"
  </source>
  <filter **.es>
    @type prometheus
    <metric>
      name fluentd_input_status_num_records_total
      type counter
      desc The total number of incoming records
      <labels>
        tag ${tag}
        hostname ${hostname}
      </labels>
    </metric>
  </filter>
  <match *.*.es>
    @type elasticsearch
    @id rawlog2elasticsearch
    hosts "xxxx"
    log_es_400_reason true
    reconnect_on_error true
    reload_on_failure true
    reload_connections false
    logstash_format true
    logstash_prefix "raw-${tag[1]}-${tag[0]}"
    <buffer tag>
      @type "file"
      path "/data/buffers/raw"
      chunk_limit_size 8M
      total_limit_size 1G
      chunk_limit_records 1000
      chunk_full_threshold 0.8
      flush_mode interval
      flush_interval 60s
      flush_thread_count 1
      retry_forever true
      retry_type periodic
      retry_wait 30
    </buffer>
  </match>

Your Error Log
2020-07-13 04:59:59 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 474709bytes
2020-07-13 05:00:24 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 553495bytes
2020-07-13 05:04:19 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 603165bytes
2020-07-13 05:04:24 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 551115bytes
2020-07-13 05:06:24 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 581636bytes
2020-07-13 05:07:19 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 562004bytes
2020-07-13 05:07:24 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 572290bytes
2020-07-13 05:11:10 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 627635bytes
2020-07-13 05:12:15 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 460620bytes
2020-07-13 05:13:20 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 485934bytes
2020-07-13 05:14:15 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 464083bytes
2020-07-13 05:16:20 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 466951bytes

Additional context

Thanks for the report. Will check soon.

Patch: https://github.com/fluent/fluentd/pull/3079

authors: paihu repeatedly


text:
in_tail plugin crashes on multiple gracefulReloads of fluentd

Describe the bug
We were performing tests that were issuing multiple /api/config.gracefulReload calls to fluentd. After issuing 2+ gracefulReloads one right after another (without letting fluentd fully go through the reload) we can see the below error in the logs:
2021-04-16 01:34:19 +0000 [error]: #0 [in_tail_container_logs] closed stream
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail/position_file.rb:212:in `pos='
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail/position_file.rb:212:in `block in update_pos'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail/position_file.rb:211:in `synchronize'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail/position_file.rb:211:in `update_pos'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:933:in `block in handle_notify'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:964:in `with_io'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:912:in `handle_notify'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:895:in `block in on_notify'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:895:in `synchronize'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:895:in `on_notify'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:724:in `on_notify'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:376:in `setup_watcher'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:410:in `block in start_watchers'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:396:in `each_value'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:396:in `start_watchers'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:359:in `refresh_watchers'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:234:in `start'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:200:in `block in start'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:189:in `block (2 levels) in lifecycle'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:188:in `each'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:188:in `block in lifecycle'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:175:in `each'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:175:in `lifecycle'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:199:in `start'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/engine.rb:244:in `start_phase'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/engine.rb:201:in `reload_config'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/supervisor.rb:915:in `block (2 levels) in reload_config'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/variable_store.rb:32:in `try_to_reset'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/supervisor.rb:914:in `block in reload_config'
2021-04-16 01:34:19 +0000 [error]: #0 fluent/log.rb:371:error: failed to reload config: undefined method `synchronize' for nil:NilClass

To Reproduce
Reload fluentd config 2+ times via /api/config.gracefulReload call one right after another (without letting fluentd fully go through the reload).
Expected behavior
in_tail plugin shouldn't crash on multiple fluentd reloads.
Your Environment

Fluentd or td-agent version: fluentd 1.12.2 we also tested with 1.9.1 and 1.11.2 and saw same issue in those as well.
Operating system: Ubuntu 20.04.1 LTS
Kernel version: 5.4.0-42-generic

Your Configuration
<ROOT>
  <system>
    log_level info
    rpc_endpoint "127.0.0.1:24444"
  </system>
  <source>
    @type systemd
    @id in_systemd_docker
    path "/var/log/journal"
    tag "systemd.unit"
    read_from_head false
    <storage>
      @type "local"
      persistent true
      path "/var/log/-kube-fluentd-operator--fluentd-journald-cursor.json"
    </storage>
    <entry>
      field_map {"_SYSTEMD_UNIT":"unit","MESSAGE":"log","_PID":["pid"],"_PRIORITY":"priority","_COMM":"cmd","_HOSTNAME":"hostname"}
      field_map_strict true
      fields_lowercase true
    </entry>
  </source>
  <match systemd.unit>
    @type rewrite_tag_filter
    <rule>
      key "unit"
      pattern ^(.+)$
      tag "systemd.$1"
    </rule>
  </match>
  <filter systemd.kubelet.service>
    @type parser
    reserve_data true
    key_name "log"
    emit_invalid_record_to_error false
    <parse>
      @type "kubernetes"
      time_format "%m%d %H:%M:%S.%N"
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
    </parse>
  </filter>
  <source>
    @type tail
    @id in_tail_container_logs
    path "/var/log/containers/*.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-containers.log.pos"
    tag "kubernetes.*"
    read_from_head true
    <parse>
      @type "multiline"
      format1 /^(?<partials>([^\n]+ (stdout|stderr) P [^\n]+\n)*)/
      format2 /(?<time>[^\n]+) (?<stream>stdout|stderr) F (?<log>[^\n]*)/
      format3 /|(?<json>{.*})/
      time_format "%Y-%m-%dT%H:%M:%S.%N%:z"
      unmatched_lines
    </parse>
  </source>
  <filter kubernetes.**>
    @type record_transformer
    @id filter_crio_container_logs
    enable_ruby true
    remove_keys partials
    <record>
      log ${record["partials"]&.gsub(/.+ (stdout|stderr) P (.+)\n/, '\\2')}${record["log"]}
    </record>
  </filter>
  <filter kubernetes.**>
    @type parser
    @id filter_docker_container_logs
    key_name "json"
    remove_key_name_field true
    reserve_data true
    emit_invalid_record_to_error false
    <parse>
      @type "json"
      time_format "%Y-%m-%dT%H:%M:%S.%NZ"
      time_type string
    </parse>
  </filter>
  <source>
    @type tail
    @id in_tail_startupscript
    path "/var/log/startupscript.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-startupscript.log.pos"
    tag "startupscript"
    <parse>
      @type "syslog"
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_docker
    path "/var/log/docker.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-docker.log.pos"
    tag "docker"
    <parse>
      @type "regexp"
      expression /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_etcd
    path "/var/log/etcd.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-etcd.log.pos"
    tag "k8s.etcd"
    <parse>
      @type "none"
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kubelet
    multiline_flush_interval 5s
    path "/var/log/kubelet.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kubelet.log.pos"
    tag "k8s.kubelet"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_proxy
    multiline_flush_interval 5s
    path "/var/log/kube-proxy.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-proxy.log.pos"
    tag "k8s.kube-proxy"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_apiserver
    multiline_flush_interval 5s
    path "/var/log/kube-apiserver.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-apiserver.log.pos"
    tag "k8s.kube-apiserver"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_controller_manager
    multiline_flush_interval 5s
    path "/var/log/kube-controller-manager.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-controller-manager.log.pos"
    tag "k8s.kube-controller-manager"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_scheduler
    multiline_flush_interval 5s
    path "/var/log/kube-scheduler.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-scheduler.log.pos"
    tag "k8s.kube-scheduler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_rescheduler
    multiline_flush_interval 5s
    path "/var/log/rescheduler.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-rescheduler.log.pos"
    tag "k8s.rescheduler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_glbc
    multiline_flush_interval 5s
    path "/var/log/glbc.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-glbc.log.pos"
    tag "k8s.glbc"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_cluster_autoscaler
    multiline_flush_interval 5s
    path "/var/log/cluster-autoscaler.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-cluster-autoscaler.log.pos"
    tag "k8s.cluster-autoscaler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_apiserver_audit
    multiline_flush_interval 5s
    path "/var/log/kubernetes/kube-apiserver-audit.log"
    pos_file "/var/log/-kube-fluentd-operator--kube-apiserver-audit.log.pos"
    tag "k8s.kube-apiserver-audit"
    <parse>
      @type "multiline"
      format_firstline "/^\\S+\\s+AUDIT:/"
      format1 /^(?<time>\S+) AUDIT:(?: (?:id="(?<id>(?:[^"\\]|\\.)*)"|ip="(?<ip>(?:[^"\\]|\\.)*)"|method="(?<method>(?:[^"\\]|\\.)*)"|user="(?<user>(?:[^"\\]|\\.)*)"|groups="(?<groups>(?:[^"\\]|\\.)*)"|as="(?<as>(?:[^"\\]|\\.)*)"|asgroups="(?<asgroups>(?:[^"\\]|\\.)*)"|namespace="(?<namespace>(?:[^"\\]|\\.)*)"|uri="(?<uri>(?:[^"\\]|\\.)*)"|response="(?<response>(?:[^"\\]|\\.)*)"|\w+="(?:[^"\\]|\\.)*"))*/
      time_format "%Y-%m-%dT%T.%L%Z"
      unmatched_lines
    </parse>
  </source>
  <filter kubernetes.**>
    @type kubernetes_metadata
    @id filter_kube_metadata
  </filter>
  <filter kubernetes.**>
    @type kubernetes_metadata
  </filter>
  <filter kubernetes.**>
    @type record_transformer
    enable_ruby true
    <record>
      kubernetes_namespace_container_name ${record["kubernetes"]["namespace_name"]}.${record["kubernetes"]["pod_name"]}.${record["kubernetes"]["container_name"]}
      container_info ${record["docker"]["container_id"]}-${record["stream"]}
    </record>
  </filter>
  <match kubernetes.**>
    @type rewrite_tag_filter
    <rule>
      key "kubernetes_namespace_container_name"
      pattern ^(.+)$
      tag "kube.$1"
    </rule>
  </match>
  <filter kube.*.*.*>
    @type record_modifier
    remove_keys "dummy_"
    <record>
      dummy_ ${record["kubernetes"]&.delete("master_url"); record["kubernetes"]&.delete("namespace_id"); if record["kubernetes"]&.has_key?("labels"); record["kubernetes"]["labels"].delete("pod-template-generation"); record["kubernetes"]["labels"].delete("controller-revision-hash");  record["kubernetes"]["labels"].delete("pod-template-hash"); end; nil}
    </record>
  </filter>
  <filter kube.*.*.*>
    @type record_transformer
    remove_keys kubernetes_namespace_container_name
  </filter>
  <filter kube.kube-system.**>
    @type parser
    reserve_data true
    key_name "log"
    emit_invalid_record_to_error false
    <parse>
      @type "kubernetes"
      time_format "%m%d %H:%M:%S.%N"
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
    </parse>
  </filter>
  <source>
    @type prometheus
  </source>
  <source>
    @type prometheus_monitor
  </source>
  <source>
    @type prometheus_output_monitor
  </source>
  <match systemd.** kube.kube-system.**>
    @type copy
    <store>
      @id X
      @type "vmware_loginsight"
      host "X.X.X.X"
      include_tag_key true
      log_text_keys ["log","msg","message"]
      port 9000
      scheme "http"
      serializer "json"
      ssl_verify false
      tag_key "tag"
    </store>
  </match>
  <match **>
    @type null
  </match>
</ROOT>

Your Error Log
2021-04-16 01:34:19 +0000 [error]: #0 [in_tail_container_logs] closed stream
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail/position_file.rb:212:in `pos='
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail/position_file.rb:212:in `block in update_pos'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail/position_file.rb:211:in `synchronize'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail/position_file.rb:211:in `update_pos'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:933:in `block in handle_notify'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:964:in `with_io'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:912:in `handle_notify'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:895:in `block in on_notify'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:895:in `synchronize'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:895:in `on_notify'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:724:in `on_notify'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:376:in `setup_watcher'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:410:in `block in start_watchers'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:396:in `each_value'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:396:in `start_watchers'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:359:in `refresh_watchers'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:234:in `start'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:200:in `block in start'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:189:in `block (2 levels) in lifecycle'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:188:in `each'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:188:in `block in lifecycle'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:175:in `each'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:175:in `lifecycle'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:199:in `start'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/engine.rb:244:in `start_phase'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/engine.rb:201:in `reload_config'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/supervisor.rb:915:in `block (2 levels) in reload_config'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/variable_store.rb:32:in `try_to_reset'
  2021-04-16 01:34:19 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/supervisor.rb:914:in `block in reload_config'
2021-04-16 01:34:19 +0000 [error]: #0 fluent/log.rb:371:error: failed to reload config: undefined method `synchronize' for nil:NilClass

Additional context

We are also seeing similar, but still different stack trace under the same conditions:
2021-04-16 03:54:12 +0000 [error]: #0 [in_tail_container_logs] closed stream
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail/position_file.rb:212:in `pos='
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail/position_file.rb:212:in `block in update_pos'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail/position_file.rb:211:in `synchronize'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail/position_file.rb:211:in `update_pos'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:933:in `block in handle_notify'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:964:in `with_io'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:912:in `handle_notify'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:895:in `block in on_notify'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:895:in `synchronize'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:895:in `on_notify'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:724:in `on_notify'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:376:in `setup_watcher'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:410:in `block in start_watchers'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:396:in `each_value'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:396:in `start_watchers'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:359:in `refresh_watchers'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:234:in `start'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:200:in `block in start'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:189:in `block (2 levels) in lifecycle'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:188:in `each'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:188:in `block in lifecycle'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:175:in `each'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:175:in `lifecycle'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:199:in `start'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/engine.rb:248:in `start'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/engine.rb:147:in `run'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/supervisor.rb:700:in `block in run_worker'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/supervisor.rb:951:in `main_process'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/supervisor.rb:691:in `run_worker'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/command/fluentd.rb:365:in `<top (required)>'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/bin/fluentd:8:in `<top (required)>'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/bin/fluentd:23:in `load'
  2021-04-16 03:54:12 +0000 [error]: #0 /usr/local/bin/fluentd:23:in `<main>'
2021-04-16 03:54:12 +0000 [error]: #0 fluent/log.rb:371:error: unexpected error error_class=NoMethodError error="undefined method `synchronize' for nil:NilClass"
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin_helper/event_loop.rb:50:in `event_loop_detach'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:386:in `block in setup_watcher'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:385:in `each'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:385:in `rescue in setup_watcher'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:362:in `setup_watcher'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:410:in `block in start_watchers'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:396:in `each_value'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:396:in `start_watchers'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:359:in `refresh_watchers'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/plugin/in_tail.rb:234:in `start'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:200:in `block in start'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:189:in `block (2 levels) in lifecycle'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:188:in `each'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:188:in `block in lifecycle'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:175:in `each'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:175:in `lifecycle'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/root_agent.rb:199:in `start'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/engine.rb:248:in `start'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/engine.rb:147:in `run'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/supervisor.rb:700:in `block in run_worker'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/supervisor.rb:951:in `main_process'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/supervisor.rb:691:in `run_worker'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/lib/fluent/command/fluentd.rb:365:in `<top (required)>'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.12.2/bin/fluentd:8:in `<top (required)>'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/bin/fluentd:23:in `load'
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:951:main_process: /usr/local/bin/fluentd:23:in `<main>'
2021-04-16 03:54:12 +0000 [error]: #0 fluent/log.rb:371:error: unexpected error error_class=NoMethodError error="undefined method `synchronize' for nil:NilClass"
  2021-04-16 03:54:12 +0000 [error]: #0 fluent/supervisor.rb:950:main_process: suppressed same stacktrace
2021-04-16 03:54:13 +0000 [info]: fluent/log.rb:329:info: Worker 0 finished unexpectedly with status 1

The current code doesn't seem have guards to prevent multiple gracefulReload so that it will spawn multiple reload thread, it might be the cause.

I saw a similar error at https://github.com/fluent/fluentd/issues/3323#issuecomment-830634918 and it's fixed by #3380.
Now I think this issue should be addressed by #3380 too.

#3380 has been merged.
v1.13.0 will ship it (will be released at the end of this month).

authors: alex-vmw ashie


text:
Fix CloseHandle definition

The defining place for CloseHandle is wrong.
I fixed to move it from WinSock to RbWinSock.

Looks good :-)
As we talked offline, welcome to the ServerEngine committers group :+1:
Please merge this commit, and release a new version. https://github.com/fluent/serverengine/commit/a7df7d8613be9cb51ab018e2362ca7672a010e24 will be helpful for the release process.

Thank you very much!
I'll merge this.

source: gharchive/pull-request

authors: frsyuki naritta

repo: fluent/serverengine

license_source: bigquery


text:
modify pkg/utils/ufs_path_builder_test.go from 'Author' to 'Authors'

Ⅰ. Describe what this PR does
Ⅱ. Does this pull request fix one issue?

fixes #XXXX
Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.
Ⅳ. Describe how to verify it
Ⅴ. Special notes for reviews

@JiulinJun please fix the DCO by following the guide:https://github.com/fluid-cloudnative/fluid/pull/3596/checks?check_run_id=19249189923

authors: JiulinJun cheyang

repo: fluid-cloudnative/fluid


text:
Packages repo should pin its dependencies

We are experiencing a tree breakage today because Mockito deprecated an API and published it an hour or so ago.
google_maps_flutter_web's integration tests were using ^5.0.0 for mockito.
This caused analysis to fail after mockito was published, and is requiring some manual work arounds to avoid.
There is a fix forward at https://github.com/flutter/plugins/pull/6153, but we can make this more robust by pinning all deps for tests in the plugins repo. This still won't help for user facing deps though.
/cc @stuartmorgan @christopherfujino @Piinks
/cc @keyonghan @godofredoc @Hixie - this is a case where manual intervention that is not a revert ends up being needed to get the tree green again. I don't see how we can avoid this kind of situation given the cycles we've introduced in our infra (testing plugins on framework in this case).

I don't see how we can avoid this kind of situation given the cycles we've introduced in our infra (testing plugins on framework in this case).

The cycle isn't really the problem here; this was just as bad for flutter/plugins. The cycle just extended the pain to flutter/flutter.
The combination of:

dependencies we don't control
that are unpinned
with deprecations as errors

Leaves us open to out-of-band breakage, because deprecation doesn't require a major version change in semver. We either need to pin everything from outside the tree, or turn off deprecations as errors. The latter would be annoying for long-term code health, so we should definitely try to do the former.

I need to verify that dev_dependencies can be pinned without pub publish complaining. If it doesn't we can add tooling to have a list of dependencies that must be pinned (or more safely, a list of dependencies that are allowed to be unpinned).
If it does, we can do something more complex like make the analyze step locally edit everything to convert ranges to pins before starting the analysis.

We either need to pin everything from outside the tree, or turn off deprecations as errors. The latter would be annoying for long-term code health, so we should definitely try to do the former.

Agreed, this sounds best. I built a pub autoroller for the framework repo, would it be useful to extend this to flutter/plugins? https://github.com/flutter/flutter/issues/106371

Potentially, although we might be able to just use the new Dart Dependabot instead. I'd need to look into whether we can easily configure it to specific packages (since we don't want to constantly roll our inter-package deps).

We would still be exposed to non-dev dependencies breaking us OOB like this though.
And deprecation breakages are not the only way to get broken.

We would still be exposed to non-dev dependencies breaking us OOB like this though.

Yes, but we have very few of those that aren't repo-internal.
If we want complete coverage there we'll have to do CI-time munging, because we absolutely don't want to pin exact versions for all plugin clients.

Potentially, although we might be able to just use the new Dart Dependabot instead. I'd need to look into whether we can easily configure it to specific packages (since we don't want to constantly roll our inter-package deps).

Oh yeah, good idea.

combination of:

dependencies we don't control
that are unpinned
with deprecations as errors

Leaves us open to out-of-band breakage, because deprecation doesn't require a major version change in semver.

Semver doesn't always help either, even if you ignore the deprecation issue. When we weren't pinning flutter/flutter's dependencies we often found that breaking changes leaked into "non-breaking" updates, even after the relevant package author had carefully thought about whether what they were doing was breaking or not. The reality is that all changes are potentially breaking. Semver only applies to the contract that you intend to export, but dependencies actually depend on everything, even the unintended contracts. Make an algorithm internally slightly faster with no change to anything else and suddenly a race condition manifests differently and a downstream package's tests flaky tests start manifesting failures where they previously were (by sheer chance) not failing. And of course sometimes people make breaking changes by mistake or forget to update the version appropriately.

Right; I meant that we're subject to breakage even within the context of the official contract. We're currently explicitly allowing picking up deprecations, and also explicitly making deprecations errors.

If we want complete coverage there we'll have to do CI-time munging, because we absolutely don't want to pin exact versions for all plugin clients.

It occurred to me that the better way of getting complete pinning would be to just check in a lock file for every example app in the repo. We would need a bot that updated them all very frequently though, since otherwise we wouldn't be testing our federated plugins the way (new or ugradeing) clients would experience them.

engine and flutter/flutter already create lock files but we don't have a way to updating them automatically. I tried to enable dependabot for java ecosystem but it doesn't generate lock files. Let me reach out to OSSF and GOSST on this topic, they may have a few ideas on how to handle java lock files.
The bonus points of generating lock files is that we will get vulnerability scanning for free.

Thanks for the clarification. Dependabot for the pub ecosystem is setup in a per directory. E.g. https://github.com/flutter/codelabs/blob/main/.github/dependabot.yaml.
This is an example of how we use it in cocoon:

Dependabot configuration for app_dart folder in cocoon https://github.com/flutter/cocoon/blob/main/.github/dependabot.yml#L55
Example PR https://github.com/flutter/cocoon/pull/2633/files

I believe there is a dependabot cli tool that may help run updates from a luci build with more control over the parameters and freq.

I'm going to downgrade this to P2; we've pinned various dev dependencies, and also switch flutter/flutter's version of analyze of flutter/packages to downgrade first which insulates it from OOB breakage. So while this isn't solved yet, we've reduced the impact in practice.
I was re-reading the discussion above about PRs and dependabot and such, and was thinking that we could probably make this much simpler since we don't need to isolate specific deps; we want everything to stay as current as possible, we just want to control it so we don't get OOB breakage. Given that, we could probably have a task that runs, say, once a day that just deletes and recreates all the lock files.

This just came up again: web 0.4.1 deprecated things we were using, which caused OOB breakage in both flutter/packages and flutter/flutter, and the ultimate result was that the deprecation was rolled back in 0.4.2 just for us. So that was pretty much the worst of all worlds, where we both disrupted the Flutter team and the maintainers of the package we depend on.
I'll put together a doc to more thoroughly explore options for addressing this.

I'll put together a doc to more thoroughly explore options for addressing this.

I've deprioritized this due to the change in Flutter repo policy that means we no longer include deprecation in analysis, which was the source of almost all of the remaining problems around not pinning exact versions.
There's still value in things being fully repeatable, but we aren't actively experiencing issues around this at the moment.

authors: Hixie christopherfujino dnfield godofredoc stuartmorgan

repo: flutter/flutter


text:
CanvasKit CPU rendering is incorrectly cropped

Summary
CanvasKit renderer can use WebGl or CPU, while using it it generates a Canvas to paint the surface on it, usually this canvas is larger than the current surface, so it draws the surface:

WebGl: On the lower-left corner
Cpu: On the upper-left corner

Demo:
Screencast from 04-08-22 12:58:35.webm
Provided demo:
CanvasKit Application with using webgl:  https://splendid-expansion.surge.sh/
CanvasKit Application with using cpu rendering:  https://decorous-fowl.surge.sh/#/
Make the window bigger and smaller to force the app to use a bigger than surface canvas.
Click anywhere to download a png of the canvas.
Analysis
The CanvasKit surface renderer, expects the surface to be drawn in the lower-left corner of the canvas, so it crops the view to that sector.
https://github.com/flutter/engine/blob/ec1be5b9c3c4b51c4d81cb4170563f8078fd01a7/lib/web_ui/lib/src/engine/canvaskit/surface.dart#L216-L229
  /// Translate the canvas so the surface covers the visible portion of the
  /// screen.
  ///
  /// The <canvas> may be larger than the visible screen, but the SkSurface is
  /// exactly the size of the visible screen. Unfortunately, the SkSurface is
  /// drawn in the lower left corner of the <canvas>, and without translation,
  /// only the top left of the <canvas> is visible. So we shift the canvas up so
  /// the bottom left corner is visible.
  void _translateCanvas() {
    final int surfaceHeight = _currentSurfaceSize!.height.ceil();
    final double offset =
        (_pixelHeight - surfaceHeight) / window.devicePixelRatio;
    htmlCanvas!.style.transform = 'translate(0, -${offset}px)';
  }

expected: It should crop the correct place when using CPU, or make both to render on the same portion of canvas.
Related to #77084

Flutter Doctor
[✓] Flutter (Channel master, 3.1.0-0.0.pre.2059, on Ubuntu 22.04 LTS 5.15.0-41-generic, locale en_US.UTF-8)
    • Flutter version 3.1.0-0.0.pre.2059 on channel master at /home/bri/sdk/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 5d52d3103a (25 hours ago), 2022-08-03 11:51:10 -0400
    • Engine revision 51296a62d9
    • Dart version 2.19.0 (build 2.19.0-58.0.dev)
    • DevTools version 2.16.0

[✓] Android toolchain - develop for Android devices (Android SDK version 32.1.0-rc1)
    • Android SDK at /home/bri/Android/Sdk
    • Platform android-32, build-tools 32.1.0-rc1
    • Java binary at: /home/bri/app/android-studio/jre/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)
    • All Android licenses accepted.

[✓] Chrome - develop for the web
    • Chrome at google-chrome

[✓] Linux toolchain - develop for Linux desktop
    • Ubuntu clang version 14.0.0-1ubuntu1
    • cmake version 3.22.1
    • ninja version 1.10.1
    • pkg-config version 0.29.2

[✓] Android Studio (version 2021.2)
    • Android Studio at /home/bri/app/android-studio
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)

[✓] VS Code (version 1.68.1)
    • VS Code at /usr/share/code
    • Flutter extension version 3.46.0

[✓] Connected device (2 available)
    • Linux (desktop) • linux  • linux-x64      • Ubuntu 22.04 LTS 5.15.0-41-generic
    • Chrome (web)    • chrome • web-javascript • Google Chrome 103.0.5060.53

[✓] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!

Hi @BrianCraig, could you please provide a completed and minimal reproducible code sample from your demo so that we may verify this.
This issue seems similar to #77084 that you mentioned. I have seen there is a PR, let's wait for it to be merged and try again to see if the issue still reproduces.
Thank you!

@huycozy Hi!
Code
Add this to web/index.html in any project
<script>
  window.flutterConfiguration = {
    canvasKitForceCpuOnly: true
  };
</script>

Steps

Run the application on web with canvaskit engine.
Make the window bigger
Make the window smaller.

I'm currently using a machine that is quite old, and chrome seems to have some problems, so it fallbacks to cpu rendering often  in various flutter apps. This doesn't happen on Firefox, so this is why i found this problem, i'll add a record of this
Screencast from 05-08-22 11:21:20.webm
I advert you that #33695 will not fix this issue becasue that calls _translateCanvas in a new situation, the main problem with cpu rendering is the function _translateCanvas because it expects the surfafe in another section of the canvas. If you remove the usage of this function, cpu rendering crops fine, but webgl not. I add you an example compiled with #33695 and cpu forced: https://acoustic-room.surge.sh/, this doesn't depend on browser viewport, just rendering engine used.
Thanks, Brian

Hi @BrianCraig, thanks for the detailed description.
This issue is reproducible on the latest stable and master channels (already pulled the merged PR https://github.com/flutter/engine/pull/33695 recently).
But I also see there's an open issue addressing the case you described #96919. And see https://github.com/flutter/flutter/issues/96919#issuecomment-1058290430 that explains the issue comes from using snapd to install Flutter. Do you find that issue similar to yours?

@huycozy thanks for the reply.
I use the non-snap version, and it's not related to that.
Let me explain, that issue references a dev installing flutter and having problems with the chrome instance that comes within. I'm having client problems, if I open any actual demo from the internet compiled fine, or the dartpad examples, they have the problem I mentioned in this issue. I don't think that I have issues regarding my flutter setup.
Brian

Hi @BrianCraig, I can confirm this issue happens with manual installation Flutter as below.
This issue is reproducible on the latest stable and master channels and happens when setting canvasKitForceCpuOnly: true as your comment in https://github.com/flutter/flutter/issues/108983#issuecomment-1206545316
This log comes from the console:
WARNING: Falling back to CPU-only rendering. CPU rendering forced by application.

Canvaskit: ❌
HTML: ✅

✅: No Issue ❌: Issue reproduced

Demo

Canvaskit
HTML

flutter doctor -v
[✓] Flutter (Channel stable, 3.0.5, on Ubuntu 20.04.4 LTS 5.13.0-44-generic, locale en_US.UTF-8)
    • Flutter version 3.0.5 at /home/huynq/Documents/Working/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision f1875d570e (2 days ago), 2022-07-13 11:24:16 -0700
    • Engine revision e85ea0e79c
    • Dart version 2.17.6
    • DevTools version 2.12.2

[✓] Android toolchain - develop for Android devices (Android SDK version 32.1.0-rc1)
    • Android SDK at /home/huynq/Android/Sdk
    • Platform android-32, build-tools 32.1.0-rc1
    • Java binary at: /home/huynq/Documents/android-studio/jre/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.11+0-b60-7590822)
    • All Android licenses accepted.

[✓] Chrome - develop for the web
    • Chrome at google-chrome

[✓] Linux toolchain - develop for Linux desktop
    • clang version 10.0.0-4ubuntu1
    • cmake version 3.16.3
    • ninja version 1.10.0
    • pkg-config version 0.29.1

[✓] Android Studio (version 2021.1)
    • Android Studio at /home/huynq/Documents/android-studio
    • Flutter plugin version 67.0.1
    • Dart plugin version 211.7817
    • Java version OpenJDK Runtime Environment (build 11.0.11+0-b60-7590822)

[✓] VS Code (version 1.64.2)
    • VS Code at /usr/share/code
    • Flutter extension can be installed from:
      🔨 https://marketplace.visualstudio.com/items?itemName=Dart-Code.flutter

[✓] Connected device (2 available)
    • Linux (desktop) • linux  • linux-x64      • Ubuntu 20.04.4 LTS 5.13.0-44-generic
    • Chrome (web)    • chrome • web-javascript • Google Chrome 101.0.4951.54

[✓] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!

[✓] Flutter (Channel master, 3.1.0-0.0.pre.2160, on Ubuntu 20.04.4 LTS 5.15.0-43-generic, locale en_US.UTF-8)
    • Flutter version 3.1.0-0.0.pre.2160 on channel master at /home/huynq/Documents/Working/flutter_master
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 612e536282 (51 minutes ago), 2022-08-08 22:54:07 -0400
    • Engine revision 6c14fb339f
    • Dart version 2.19.0 (build 2.19.0-75.0.dev)
    • DevTools version 2.16.0

[✓] Android toolchain - develop for Android devices (Android SDK version 32.1.0-rc1)
    • Android SDK at /home/huynq/Android/Sdk
    • Platform android-32, build-tools 32.1.0-rc1
    • Java binary at: /home/huynq/Documents/android-studio/jre/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.11+0-b60-7590822)
    • All Android licenses accepted.

[✓] Chrome - develop for the web
    • Chrome at google-chrome

[✓] Linux toolchain - develop for Linux desktop
    • clang version 10.0.0-4ubuntu1
    • cmake version 3.16.3
    • ninja version 1.10.0
    • pkg-config version 0.29.1

[✓] Android Studio (version 2021.1)
    • Android Studio at /home/huynq/Documents/android-studio
    • Flutter plugin version 67.0.1
    • Dart plugin version 211.7817
    • Java version OpenJDK Runtime Environment (build 11.0.11+0-b60-7590822)

[✓] VS Code (version 1.69.2)
    • VS Code at /usr/share/code
    • Flutter extension version 3.46.0

[✓] Connected device (2 available)
    • Linux (desktop) • linux  • linux-x64      • Ubuntu 20.04.4 LTS 5.15.0-43-generic
    • Chrome (web)    • chrome • web-javascript • Google Chrome 103.0.5060.134

[✓] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!

Labeling this issue for further investigation from the team.

Is there any update to this issue?
I am facing the same issue.

Are there any updates on this issue?
Or any known work arounds besides forcing html renderer?
I am having this same issue on web for iOS
Using Flutter 3.22.2

On Flutter 3.24 , the flutter team is deprecating the html renderer https://docs.flutter.dev/platform-integration/web/renderers#choosing-a-renderer-at-runtime
What is the status of this issue then?

cc @kevmoo

@mit-mit We have corporative clients who block WebGL by security reasons. CanvasKit should work correct without it even if performance is poor.

authors: BrianCraig LuisReyes98 OlehSv huycozy mit-mit yugeshiii