916886 - [stage][traceback] 500 Error: ConnectionError: HTTPConnectionPool

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Matt Brandt [:mbrandt]]

I'm seeing spurious but fairly consistent 500 errors on stage with this signature - ConnectionError: HTTPConnectionPool. This is occurring when I visit pages such as https://crash-stats.allizom.org/topcrasher/products/Firefox/versions/23.0.1?days=7

Here's the errormill goodness; https://errormill.mozilla.org/webtools/socorro-stage/group/97549/

Comment 1

[Peter Bengtsson [:peterbe]]

All of these `ConnectionError` exceptions are basically due to annoying temporary network I/O outtages when the webhead is unable to connect to the middleware server. 

Since the problem almost always goes away a short while later, why don't we wrap our requests in a try:except with a manual sleep time. 
We should make it back off if it attempts too much.

Comment 2

[Peter Bengtsson [:peterbe]]

So I did some research, ConnectionError basically happens when *you* don't have an internet connection. If you can connect the server but that server refuses to connect you get a HTTPError. 
It's easy to reproduce this. Just switch off your Wifi and try to load Socorro with using the dev server for the middleware. 

urllib3 (part of requests as a module), by default, uses 3 retries when it tries to connect. After the third attempt it raises a MaxRetryError which when bubbled up into requests becomes a ConnectionError.

In other words, it's a good idea to add a sleeper that retries.

Comment 3

[Peter Bengtsson [:peterbe]]

Pull request: https://github.com/mozilla/socorro/pull/1517

Comment 4

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/ea632ccedce341bbd0fbca07c536682e1aa86c2b
fixes bug 916886 - retry on ConnectionError on middleware, r=rhelmer