1532504 - Intermittent Uncaught exception: Traceback (most recent call last): after Finished get-secrets step (failed)

Status: RESOLVED INCOMPLETE

(Release Engineering :: Applications: MozharnessCore, defect, P5)

Description

[Treeherder Bug Filer]

#[markdown(off)]
Filed by: rmaries [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=231794794&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/UeXIzGN_S9uIp0NlA4p_Cw/runs/0/artifacts/public/logs/live_backing.log

02:02:04 INFO - [mozharness: 2019-03-05 02:02:04.715000Z] Finished get-secrets step (failed)
02:02:04 FATAL - Uncaught exception: Traceback (most recent call last):
02:02:04 FATAL - File "z:\build\build\src\testing\mozharness\mozharness\base\script.py", line 2089, in run
02:02:04 FATAL - self.run_action(action)
02:02:04 FATAL - File "z:\build\build\src\testing\mozharness\mozharness\base\script.py", line 2028, in run_action
02:02:04 FATAL - self._possibly_run_method(method_name, error_if_missing=True)
02:02:04 FATAL - File "z:\build\build\src\testing\mozharness\mozharness\base\script.py", line 1983, in _possibly_run_method
02:02:04 FATAL - return getattr(self, method_name)()
02:02:04 FATAL - File "z:\build\build\src\testing\mozharness\mozharness\mozilla\secrets.py", line 74, in get_secrets
02:02:04 FATAL - secret = self._fetch_secret(secret_name)
02:02:04 FATAL - File "z:\build\build\src\testing\mozharness\mozharness\mozilla\secrets.py", line 23, in _fetch_secret
02:02:04 FATAL - res = urllib2.urlopen(url)
02:02:04 FATAL - File "c:\mozilla-build\python\lib\urllib2.py", line 154, in urlopen
02:02:04 FATAL - return opener.open(url, data, timeout)
02:02:04 FATAL - File "c:\mozilla-build\python\lib\urllib2.py", line 429, in open
02:02:04 FATAL - response = self._open(req, data)
02:02:04 FATAL - File "c:\mozilla-build\python\lib\urllib2.py", line 447, in _open
02:02:04 FATAL - '_open', req)
02:02:04 FATAL - File "c:\mozilla-build\python\lib\urllib2.py", line 407, in _call_chain
02:02:04 FATAL - result = func(*args)
02:02:04 FATAL - File "c:\mozilla-build\python\lib\urllib2.py", line 1228, in http_open
02:02:04 FATAL - return self.do_open(httplib.HTTPConnection, req)
02:02:04 FATAL - File "c:\mozilla-build\python\lib\urllib2.py", line 1198, in do_open
02:02:04 FATAL - raise URLError(err)
02:02:04 FATAL - URLError: <urlopen error [Errno 11001] getaddrinfo failed>
02:02:04 FATAL - Running post_fatal callback...
02:02:04 ERROR - setting return code to 2 because fatal was called
02:02:04 WARNING - setting return code to 2
02:02:04 FATAL - Exiting -1

Comment 1

[Intermittent Failures Robot]

4 failures in 2818 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 1
• autoland: 3

Platform breakdown:

• windows2012-64: 2
• windows2012-32: 1
• windows2012-aarch64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1532504&startday=2019-03-04&endday=2019-03-10&tree=all

Comment 2

[Intermittent Failures Robot]

1 failures in 2561 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1532504&startday=2019-03-18&endday=2019-03-24&tree=all

Comment 3

[Intermittent Failures Robot]

1 failures in 3910 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• android-hw-p2-8-0-android-aarch64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1532504&startday=2019-03-25&endday=2019-03-31&tree=all

Comment 4

[Firefox Bug Husbandry Bot]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup

Comment 5

[Andreea Pavel [:apavel]]

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=253291928&repo=autoland&lineNumber=499

[task 2019-06-25T10:41:20.753Z] 10:41:20 INFO - Running main action method: get_secrets
[task 2019-06-25T10:41:20.753Z] 10:41:20 INFO - fetching secret project/releng/gecko/build/level-3/gls-gapi.data from API
[task 2019-06-25T10:41:20.920Z] 10:41:20 INFO - fetching secret project/releng/gecko/build/level-3/sb-gapi.data from API
[task 2019-06-25T10:41:21.090Z] 10:41:21 INFO - fetching secret project/releng/gecko/build/level-3/mozilla-fennec-geoloc-api.key from API
[task 2019-06-25T10:41:21.258Z] 10:41:21 INFO - fetching secret project/releng/gecko/build/level-3/adjust-sdk.token from API
[task 2019-06-25T10:41:21.428Z] 10:41:21 INFO - fetching secret project/releng/gecko/build/level-3/adjust-sdk-beta.token from API
[task 2019-06-25T10:41:21.592Z] 10:41:21 INFO - fetching secret project/releng/gecko/build/level-3/leanplum-sdk-release.token from API
[task 2019-06-25T10:41:21.764Z] 10:41:21 INFO - fetching secret project/releng/gecko/build/level-3/leanplum-sdk-beta.token from API
[task 2019-06-25T10:41:21.930Z] 10:41:21 INFO - fetching secret project/releng/gecko/build/level-3/leanplum-sdk-nightly.token from API
[task 2019-06-25T10:41:22.099Z] 10:41:22 INFO - fetching secret project/releng/gecko/build/level-3/pocket-api-release.token from API
[task 2019-06-25T10:41:22.269Z] 10:41:22 INFO - fetching secret project/releng/gecko/build/level-3/pocket-api-beta.token from API
[task 2019-06-25T10:41:22.322Z] 10:41:22 INFO - [mozharness: 2019-06-25 10:41:22.322646Z] Finished get-secrets step (failed)
[task 2019-06-25T10:41:22.323Z] 10:41:22 FATAL - Uncaught exception: Traceback (most recent call last):
[task 2019-06-25T10:41:22.323Z] 10:41:22 FATAL - File "/builds/worker/workspace/build/src/testing/mozharness/mozharness/base/script.py", line 2097, in run
[task 2019-06-25T10:41:22.323Z] 10:41:22 FATAL - self.run_action(action)
[task 2019-06-25T10:41:22.323Z] 10:41:22 FATAL - File "/builds/worker/workspace/build/src/testing/mozharness/mozharness/base/script.py", line 2036, in run_action
[task 2019-06-25T10:41:22.324Z] 10:41:22 FATAL - self._possibly_run_method(method_name, error_if_missing=True)
[task 2019-06-25T10:41:22.324Z] 10:41:22 FATAL - File "/builds/worker/workspace/build/src/testing/mozharness/mozharness/base/script.py", line 1991, in _possibly_run_method
[task 2019-06-25T10:41:22.324Z] 10:41:22 FATAL - return getattr(self, method_name)()
[task 2019-06-25T10:41:22.324Z] 10:41:22 FATAL - File "/builds/worker/workspace/build/src/testing/mozharness/mozharness/mozilla/secrets.py", line 74, in get_secrets
[task 2019-06-25T10:41:22.324Z] 10:41:22 FATAL - secret = self._fetch_secret(secret_name)
[task 2019-06-25T10:41:22.324Z] 10:41:22 FATAL - File "/builds/worker/workspace/build/src/testing/mozharness/mozharness/mozilla/secrets.py", line 23, in _fetch_secret
[task 2019-06-25T10:41:22.324Z] 10:41:22 FATAL - res = urllib2.urlopen(url)
[task 2019-06-25T10:41:22.324Z] 10:41:22 FATAL - File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
[task 2019-06-25T10:41:22.324Z] 10:41:22 FATAL - return opener.open(url, data, timeout)
[task 2019-06-25T10:41:22.324Z] 10:41:22 FATAL - File "/usr/lib/python2.7/urllib2.py", line 429, in open
[task 2019-06-25T10:41:22.324Z] 10:41:22 FATAL - response = self._open(req, data)
[task 2019-06-25T10:41:22.324Z] 10:41:22 FATAL - File "/usr/lib/python2.7/urllib2.py", line 447, in _open
[task 2019-06-25T10:41:22.324Z] 10:41:22 FATAL - '_open', req)
[task 2019-06-25T10:41:22.324Z] 10:41:22 FATAL - File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
[task 2019-06-25T10:41:22.324Z] 10:41:22 FATAL - result = func(*args)
[task 2019-06-25T10:41:22.324Z] 10:41:22 FATAL - File "/usr/lib/python2.7/urllib2.py", line 1228, in http_open
[task 2019-06-25T10:41:22.324Z] 10:41:22 FATAL - return self.do_open(httplib.HTTPConnection, req)
[task 2019-06-25T10:41:22.324Z] 10:41:22 FATAL - File "/usr/lib/python2.7/urllib2.py", line 1201, in do_open
[task 2019-06-25T10:41:22.324Z] 10:41:22 FATAL - r = h.getresponse(buffering=True)
[task 2019-06-25T10:41:22.324Z] 10:41:22 FATAL - File "/usr/lib/python2.7/httplib.py", line 1121, in getresponse
[task 2019-06-25T10:41:22.324Z] 10:41:22 FATAL - response.begin()
[task 2019-06-25T10:41:22.324Z] 10:41:22 FATAL - File "/usr/lib/python2.7/httplib.py", line 438, in begin
[task 2019-06-25T10:41:22.324Z] 10:41:22 FATAL - version, status, reason = self._read_status()
[task 2019-06-25T10:41:22.325Z] 10:41:22 FATAL - File "/usr/lib/python2.7/httplib.py", line 402, in _read_status
[task 2019-06-25T10:41:22.325Z] 10:41:22 FATAL - raise BadStatusLine(line)
[task 2019-06-25T10:41:22.325Z] 10:41:22 FATAL - BadStatusLine: ''
[task 2019-06-25T10:41:22.325Z] 10:41:22 FATAL - Running post_fatal callback...
[task 2019-06-25T10:41:22.325Z] 10:41:22 ERROR - setting return code to 2 because fatal was called
[task 2019-06-25T10:41:22.325Z] 10:41:22 WARNING - setting return code to 2

Comment 6

[Intermittent Failures Robot]

1 failures in 4409 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• android-4-0-armv7-api16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1532504&startday=2019-06-24&endday=2019-06-30&tree=all

Comment 8

[Firefox Bug Husbandry Bot]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup

Comment 9

[Cristian Brindusan [:cbrindusan]]

New occurrence:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=299450105&repo=autoland&lineNumber=677

Comment 10

[Intermittent Failures Robot]

1 failures in 5582 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• android-5-0-x86_64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1532504&startday=2020-04-20&endday=2020-04-26&tree=all

Comment 11

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Comment 12

[Razvan Maries]

New occurrence: https://treeherder.mozilla.org/logviewer?job_id=322193465&repo=autoland&lineNumber=530

Comment 13

[Bob Clary [:bc] (inactive)]

I'm not sure if this new occurrence is this same bug. This one was due to urllib.error.HTTPError: HTTP Error 401: Unauthorized

[task 2020-11-18T17:07:05.294Z] 17:07:05    FATAL - Uncaught exception: Traceback (most recent call last):
[task 2020-11-18T17:07:05.294Z] 17:07:05    FATAL -   File "/builds/worker/checkouts/gecko/testing/mozharness/mozharness/base/script.py", line 2358, in run
[task 2020-11-18T17:07:05.294Z] 17:07:05    FATAL -     self.run_action(action)
[task 2020-11-18T17:07:05.294Z] 17:07:05    FATAL -   File "/builds/worker/checkouts/gecko/testing/mozharness/mozharness/base/script.py", line 2292, in run_action
[task 2020-11-18T17:07:05.294Z] 17:07:05    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
[task 2020-11-18T17:07:05.294Z] 17:07:05    FATAL -   File "/builds/worker/checkouts/gecko/testing/mozharness/mozharness/base/script.py", line 2244, in _possibly_run_method
[task 2020-11-18T17:07:05.294Z] 17:07:05    FATAL -     return getattr(self, method_name)()
[task 2020-11-18T17:07:05.294Z] 17:07:05    FATAL -   File "/builds/worker/checkouts/gecko/testing/mozharness/mozharness/mozilla/secrets.py", line 74, in get_secrets
[task 2020-11-18T17:07:05.294Z] 17:07:05    FATAL -     secret = self._fetch_secret(secret_name)
[task 2020-11-18T17:07:05.294Z] 17:07:05    FATAL -   File "/builds/worker/checkouts/gecko/testing/mozharness/mozharness/mozilla/secrets.py", line 23, in _fetch_secret
[task 2020-11-18T17:07:05.294Z] 17:07:05    FATAL -     res = urllib.request.urlopen(url)
[task 2020-11-18T17:07:05.294Z] 17:07:05    FATAL -   File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen
[task 2020-11-18T17:07:05.294Z] 17:07:05    FATAL -     return opener.open(url, data, timeout)
[task 2020-11-18T17:07:05.295Z] 17:07:05    FATAL -   File "/usr/lib/python3.6/urllib/request.py", line 532, in open
[task 2020-11-18T17:07:05.295Z] 17:07:05    FATAL -     response = meth(req, response)
[task 2020-11-18T17:07:05.295Z] 17:07:05    FATAL -   File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
[task 2020-11-18T17:07:05.295Z] 17:07:05    FATAL -     'http', request, response, code, msg, hdrs)
[task 2020-11-18T17:07:05.295Z] 17:07:05    FATAL -   File "/usr/lib/python3.6/urllib/request.py", line 570, in error
[task 2020-11-18T17:07:05.295Z] 17:07:05    FATAL -     return self._call_chain(*args)
[task 2020-11-18T17:07:05.295Z] 17:07:05    FATAL -   File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
[task 2020-11-18T17:07:05.295Z] 17:07:05    FATAL -     result = func(*args)
[task 2020-11-18T17:07:05.295Z] 17:07:05    FATAL -   File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default
[task 2020-11-18T17:07:05.295Z] 17:07:05    FATAL -     raise HTTPError(req.full_url, code, msg, hdrs, fp)
[task 2020-11-18T17:07:05.295Z] 17:07:05    FATAL - urllib.error.HTTPError: HTTP Error 401: Unauthorized
[task 2020-11-18T17:07:05.295Z] 17:07:05    FATAL - Running post_fatal callback...
[task 2020-11-18T17:07:05.295Z] 17:07:05    ERROR - setting return code to 2 because fatal was called

Comment 14

[Intermittent Failures Robot]

1 failures in 4810 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• linux64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1532504&startday=2020-11-16&endday=2020-11-22&tree=all

Comment 15

[Cosmin Sabou [:CosminS]]

[Errno 11001] getaddrinfo failed> failures are investigated in Bug 1720121.