868811 - l10n dashboard needs a way to recover from urllib2.HTTPError Error 500

Status: RESOLVED FIXED

(Webtools Graveyard :: Elmo, defect, P3)

Description

[Francesco Lodolo [:flod]]

As commented in bug 865678 comment 2, periodically the dashboard get stuck with this "urllib2.HTTPError: HTTP Error 500: Internal Server Error." and must be manually kicked.

A possible solution could be to automatically wait and retry when the error appears in the log.

Comment 1

[Axel Hecht [:Pike]]

This should be fixed by bug 882110, we'll see how it goes once that's deployed.

Comment 2

[Axel Hecht [:Pike]]

Another infra-related error surfaced today:

2013-07-08 16:57:02-0500 [Uninitialized] failed to load json for releases/l10n-miramar/hi-IN, adding back
	Traceback (most recent call last):
	Failure: twisted.internet.error.TCPTimedOutError: TCP connection timed out: 110: Connection timed out.

Comment 3

[Axel Hecht [:Pike]]

2013-07-08 18:09:46-0500 [-] failed to load json for l10n-central/eo, adding back
	Traceback (most recent call last):
	Failure: twisted.internet.error.DNSLookupError: DNS lookup failed: address 'hg.mozilla.org' not found: [Errno -5] No address associated with hostname.

Comment 4

[Francesco Lodolo [:flod]]

I haven't seen the dashboard stuck in a while, also situation on hg.mozilla.org seems to be improved lately (I didn't get many 500s in the last weeks).

Should we just close this as WORKSFORME or similar?

Comment 5

[Axel Hecht [:Pike]]

Let's keep this open for a bit more, I think there are still a bunch of scenarios in which we hit unexpected hg errors. Though now with the new poller, we should recover from more of them.

Comment 6

[Axel Hecht [:Pike]]

We're out of the woods on this one, marking FIXED.