ActiveRecord::ConnectionTimeoutError

[j-mcnally]

[2014-01-23 00:43:04] [ERROR] ActiveRecord::ConnectionTimeoutError, could not obtain a database connection within 5.000 seconds (waited 5.000 seconds)
2014-01-23T00:43:04.558462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:190:in `block in wait_poll'
2014-01-23T00:43:04.558462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:181:in `loop'
2014-01-23T00:43:04.558462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:181:in `wait_poll'
2014-01-23T00:43:04.558462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:136:in `block in poll'
2014-01-23T00:43:04.558462+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
2014-01-23T00:43:04.558462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:423:in `acquire_connection'
2014-01-23T00:43:04.558462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:356:in `block in checkout'
2014-01-23T00:43:04.558841+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
2014-01-23T00:43:04.558841+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:355:in `checkout'
2014-01-23T00:43:04.558841+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:265:in `block in connection'
2014-01-23T00:43:04.558462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:134:in `poll'
2014-01-23T00:43:04.558462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:146:in `synchronize'
2014-01-23T00:43:04.558841+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/store/active_record/reconnectable.rb:17:in `with_database_reconnect_and_retry'
2014-01-23T00:43:04.558841+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/store/active_record.rb:12:in `deliverable_notifications'
2014-01-23T00:43:04.558841+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/feeder.rb:42:in `enqueue_notifications'
2014-01-23T00:43:04.559166+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/feeder.rb:27:in `loop'
2014-01-23T00:43:04.559166+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/feeder.rb:27:in `feed_forever'
2014-01-23T00:43:04.558841+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:294:in `with_connection'
2014-01-23T00:43:04.558841+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:264:in `connection'
2014-01-23T00:43:04.558841+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
2014-01-23T00:43:04.559166+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/feeder.rb:11:in `block in start'
2014-01-23T00:43:04.558841+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rapns-3.4.1/lib/rapns/daemon/feeder.rb:28:in `block in feed_forever'

I am running on heroku and have been getting quite a bit of these, i think its a heroku issue but wanted confirm i dont need to be catching and reconnecting anywhere?

[j-mcnally]

The solution here was to add

ActiveRecord::ConnectionTimeoutError

to the ADAPTER_ERRORS in reconnectable.

Does this make anysense, without it the db wouldnt try the connection again

[ileitch]

Yes, checking for ConnectionTimeoutError does make sense. Usually this error is a symptom of not having a large enough connection pool. What is the sum of all connections in your rapns_apps table?

[j-mcnally]

i was afraid this was an issue, i have to set small DB_POOL size because we are wroking on heroku postgres_hobby which limits pool size to 20. Probably why I ran into this, while probably a bit slower, checking ConnectionTimeoutError seems to have resolved the symptoms.