1514431 - Very frequent failed to fetch ... from https://tooltool.mozilla-releng.net/

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Cosmin Sabou [:CosminS]]

There was a window in which several test, mainly raptor tests failed with 
HTTPError: HTTP Error 503: Service Unavailable, for eg:

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=217123340&repo=mozilla-beta&lineNumber=594
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=217118101&repo=autoland&lineNumber=1335
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=217125465&repo=autoland&lineNumber=970
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=217128440&repo=autoland&lineNumber=968

This is how it looked on autoland: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&failure_classification_id=5&fromchange=90a1eb1a99de210c7148860d187baf2d13c2c455&tochange=d8a944c012a7a814ee5a5fea4c9075969451a48b

Beta: https://treeherder.mozilla.org/?#/jobs?repo=mozilla-beta&resultStatus=testfailed%2Cbusted%2Cexception&searchStr=raptor&tochange=9eff0feefcf92ed0cd1232c484686108965f4474&fromchange=6f8e2bdfd8b61347c2ccd7a29841ef277d055d04&selectedJob=217116195

00:02:37     INFO - Calling ['c:\\mozilla-build\\python\\python.exe', '-u', 'Z:\\task_1544830079\\mozharness\\external_tools\\tooltool.py', '--url', 'https://tooltool.mozilla-releng.net/', '--authentication-file', 'c:\\builds\\relengapi.tok', 'fetch', '-m', 'Z:\\task_1544830079\\build\\tests\\config/tooltool-manifests/win32/releng.manifest', '-o'] with output_timeout 600
00:02:37     INFO -  INFO - Attempting to fetch from 'https://tooltool.mozilla-releng.net/'...
00:03:11     INFO -  INFO - ...failed to fetch 'win32-minidump_stackwalk.exe' from https://tooltool.mozilla-releng.net/
00:03:11    ERROR -  Traceback (most recent call last):
00:03:11     INFO -    File "Z:\task_1544830079\mozharness\external_tools\tooltool.py", line 480, in fetch_file
00:03:11     INFO -      f = urllib2.urlopen(req)
00:03:11     INFO -    File "c:\mozilla-build\python\lib\urllib2.py", line 154, in urlopen
00:03:11     INFO -      return opener.open(url, data, timeout)
00:03:11     INFO -    File "c:\mozilla-build\python\lib\urllib2.py", line 435, in open
00:03:11     INFO -      response = meth(req, response)
00:03:11     INFO -    File "c:\mozilla-build\python\lib\urllib2.py", line 548, in http_response
00:03:11     INFO -      'http', request, response, code, msg, hdrs)
00:03:11     INFO -    File "c:\mozilla-build\python\lib\urllib2.py", line 473, in error
00:03:11     INFO -      return self._call_chain(*args)
00:03:11     INFO -    File "c:\mozilla-build\python\lib\urllib2.py", line 407, in _call_chain
00:03:11     INFO -      result = func(*args)
00:03:11     INFO -    File "c:\mozilla-build\python\lib\urllib2.py", line 556, in http_error_default
00:03:11     INFO -      raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
00:03:11     INFO -  HTTPError: HTTP Error 503: Service Unavailable
00:03:11    ERROR -  ERROR - The following files failed: 'win32-minidump_stackwalk.exe'
00:03:11    ERROR - Return code: 1
00:03:11     INFO - retry: Failed, sleeping 60 seconds before retrying

16:00:53     INFO -  raptor-mitmproxy INFO - ...failed to fetch 'mitmproxy-2.0.2-osx.tar.gz' from https://tooltool.mozilla-releng.net/
16:00:53    ERROR -  raptor-mitmproxy Traceback (most recent call last):
16:00:53     INFO -  raptor-mitmproxy   File "/Users/cltbld/tasks/task_1544831922/mozharness/external_tools/tooltool.py", line 480, in fetch_file
16:00:53     INFO -  raptor-mitmproxy     f = urllib2.urlopen(req)
16:00:53     INFO -  raptor-mitmproxy   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen
16:00:53     INFO -  raptor-mitmproxy     return opener.open(url, data, timeout)
16:00:53     INFO -  raptor-mitmproxy   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 437, in open
16:00:53     INFO -  raptor-mitmproxy     response = meth(req, response)
16:00:53     INFO -  raptor-mitmproxy   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 550, in http_response
16:00:53     INFO -  raptor-mitmproxy     'http', request, response, code, msg, hdrs)
16:00:53     INFO -  raptor-mitmproxy   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 475, in error
16:00:53     INFO -  raptor-mitmproxy     return self._call_chain(*args)
16:00:53     INFO -  raptor-mitmproxy   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
16:00:53     INFO -  raptor-mitmproxy     result = func(*args)
16:00:53     INFO -  raptor-mitmproxy   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 558, in http_error_default
16:00:53     INFO -  raptor-mitmproxy     raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
16:00:53     INFO -  raptor-mitmproxy HTTPError: HTTP Error 503: Service Unavailable
16:00:53     INFO -  raptor-mitmproxy ERROR - The following files failed: 'mitmproxy-2.0.2-osx.tar.gz'
16:00:53     INFO -  raptor-mitmproxy downloading mitmproxy pageset
16:00:53     INFO -  raptor-mitmproxy INFO - Attempting to fetch from 'https://tooltool.mozilla-releng.net/'...
16:01:01     INFO -  raptor-mitmproxy INFO - File raptor-tp6-3.zip fetched from https://tooltool.mozilla-releng.net/ as /Users/cltbld/tasks/task_1544831922/testing/raptor/tmpGkggJP
16:01:01     INFO -  raptor-mitmproxy INFO - File integrity verified, renaming tmpGkggJP to raptor-tp6-3.zip
16:01:01     INFO -  raptor-mitmproxy INFO - unzipping "raptor-tp6-3.zip"
16:01:02     INFO -  raptor-mitmproxy mitmdump path: /Users/cltbld/tasks/task_1544831922/testing/raptor/mitmdump
16:01:02     INFO -  raptor-mitmproxy recording path: /Users/cltbld/tasks/task_1544831922/testing/raptor
16:01:02     INFO -  raptor-mitmproxy recordings list: ['imdb.mp']
16:01:02     INFO -  raptor-mitmproxy browser path: /Users/cltbld/tasks/task_1544831922/build/application/Firefox Nightly.app/Contents/MacOS/firefox
16:01:02     INFO -  raptor-mitmproxy Starting mitmproxy playback using env path: /Users/cltbld/tasks/task_1544831922/build/application/Firefox Nightly.app/Contents/MacOS;/Users/cltbld/tasks/task_1544831922/build/venv/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
16:01:02     INFO -  raptor-mitmproxy Starting mitmproxy playback using command: /Users/cltbld/tasks/task_1544831922/testing/raptor/mitmdump -k -q -s /Users/cltbld/tasks/task_1544831922/build/tests/raptor/raptor/playback/alternate-server-replay.py /Users/cltbld/tasks/task_1544831922/testing/raptor/imdb.mp
16:01:02    ERROR -  Traceback (most recent call last):

Comment 1

[Cosmin Sabou [:CosminS]]

Chris, could you take a look over this? It's still failing pretty often.

Comment 2

[Intermittent Failures Robot]

14 failures in 2485 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 5
* mozilla-inbound: 4
* mozilla-central: 4
* mozilla-beta: 1

Platform breakdown:
* windows10-64: 1
* windows7-32: 5
* windows10-64-ccov: 2
* windows10-64-devedition: 1
* linux64: 2
* windows10-64-qr: 1
* osx-10-10: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1514431&startday=2018-12-10&endday=2018-12-16&tree=all

Comment 3

[Intermittent Failures Robot]

3 failures in 2627 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 3

Platform breakdown:

• windows7-32-nightly: 1
• windows7-32-devedition: 1
• windows10-64-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1514431&startday=2019-01-07&endday=2019-01-13&tree=all

Comment 4

[Intermittent Failures Robot]

2 failures in 2378 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform breakdown:

• windows10-64-ccov: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1514431&startday=2019-01-14&endday=2019-01-20&tree=all

Comment 5

[Roland Mutter Michael (:rmutter)]

Hey :garbas . I wonder if there is something wrong with tooltool

Comment 6

[Rok Garbas [:garbas]]

:rmutter it looks like something was causing really long responses which resulted sometime in timeouts. after all tooltool instances automatically restarted around 8:00AM CET everything went back to normal.

Comment 7

[Intermittent Failures Robot]

4 failures in 2628 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 1
• mozilla-central: 3

Platform breakdown:

• osx-10-10: 1
• windows7-32: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1514431&startday=2019-01-28&endday=2019-02-03&tree=all

Comment 8

[Intermittent Failures Robot]

1 failures in 4710 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 1

Platform breakdown:

• android-hw-g5-7-0-arm7-api-16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1514431&startday=2019-07-08&endday=2019-07-14&tree=all