Losing connection to database, not recovering

[velorix92]

Hi

Having fluentd running as DaemonSet in kubernetes based on
fluentd:v1.2.6-debian using the current 1.0.0 fluent-plugin-pgjson plugin version.
After some time we lose connection to the database and then miss some logs in the db.

2018-10-26 06:59:21 +0000 [warn]: #0 fluent/log.rb:342:warn: failed to flush the buffer. retry_time=16 next_retry_seconds=2018-10-26 06:59:21 +0000 chunk="5790cd573c914755aed4e344e7b110b9" error_class=PG::UnableToSend error="no connection to the server\n"
2018-10-26 23:09:19 +0000 [warn]: #0 fluent/log.rb:342:warn: failed to flush the buffer. retry_time=17 next_retry_seconds=2018-10-26 23:09:19 +0000 chunk="5790cd573c914755aed4e344e7b110b9" error_class=PG::UnableToSend error="no connection to the server\n"
2018-10-28 08:00:18 +0000 [warn]: #0 fluent/log.rb:342:warn: failed to flush the buffer. retry_time=18 next_retry_seconds=2018-10-28 08:00:18 +0000 chunk="5790cd573c914755aed4e344e7b110b9" error_class=PG::UnableToSend error="no connection to the server\n"
2018-10-28 12:36:12 +0000 [error]: #0 fluent/log.rb:362:error: failed to flush the buffer, and hit limit for retries. dropping all chunks in the buffer queue. retry_times=19 records=46 error_class=PG::UnableToSend error="no connection to the server\n"

Not being a ruby programmer myself, I'm wondering how to recover from that, creating a new connection to the db somehow, add some retry logic somewhere ?!

Additional Info:

• We split the log stream ("@type copy") and one part goes to graylog, that part is working fine. We don't lose any loglines. It all ends up in Graylog.
• Before, we were using fluentd 0.12 with pg gem 0.21.0. With this setup we reproducibly lost connection to the db after some hours. Then I upgraded fluentd to 1.2.6 to be able to use the latest version of the plugin and a more current version of the pg gem.

Any hints or help appreciated.

Regards

[hellower]

I have same problem.

[jdevalk2]

I submitted a pull request for this which should fix it. Just need a Ruby expert to take a look at it because I am not a Ruby developer by trade.

[zlymeda]

is this repo dead?

[cosmo0920]

#27 is merged. Closing.

[pigi-spm]

@cosmo0920 - I have modified the function "write", where the problem was being generated, you can reopen the ticket and add reviewers to check the PR --> #35
Thanks a lot :)

[hcderaad]

@cosmo0920 apologies for commenting on this closed ticket, I'm suffering from the same symptom, yet my failure mode seems to be addressed by #36 by @pigi-spm is there any chance to merge that as well?