1473071 - Jobs timeouts because of external_tools/tooltool.py

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Cosmin Sabou [:CosminS]]

Trees are closed because of this issue.

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&selectedJob=186217147

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=186217144&repo=mozilla-inbound&lineNumber=811

[task 2018-07-03T15:56:47.431Z]     INFO - Copy/paste: /usr/bin/python2.7 -u /builds/worker/workspace/mozharness/external_tools/tooltool.py --url http://relengapi/tooltool/ fetch -m /builds/worker/workspace/build/.android/releng.manifest -o -c /builds/worker/tooltool_cache
[task 2018-07-03T15:56:47.431Z]     INFO - Calling ['/usr/bin/python2.7', '-u', '/builds/worker/workspace/mozharness/external_tools/tooltool.py', '--url', 'http://relengapi/tooltool/', 'fetch', '-m', '/builds/worker/workspace/build/.android/releng.manifest', '-o', '-c', '/builds/worker/tooltool_cache'] with output_timeout 600
[task 2018-07-03T15:56:47.477Z]     INFO -  INFO - File AVDs-x86-android-4.2_r1-build-2016-11-18.tar.gz not present in local cache folder /builds/worker/tooltool_cache
[task 2018-07-03T15:56:47.477Z]     INFO -  INFO - Attempting to fetch from 'http://relengapi/tooltool/'...
[task 2018-07-03T15:56:49.435Z] compiz (core) - Warn: Attempted to restack relative to 0x1200006 which is not a child of the root window or a window compiz owns
[task 2018-07-03T16:06:47.505Z]     INFO - Automation Error: mozprocess timed out after 600 seconds running ['/usr/bin/python2.7', '-u', '/builds/worker/workspace/mozharness/external_tools/tooltool.py', '--url', 'http://relengapi/tooltool/', 'fetch', '-m', '/builds/worker/workspace/build/.android/releng.manifest', '-o', '-c', '/builds/worker/tooltool_cache']
[task 2018-07-03T16:06:47.538Z]    ERROR - timed out after 600 seconds of no output
[task 2018-07-03T16:06:47.538Z]    ERROR - Return code: -15
[task 2018-07-03T16:06:47.539Z]     INFO - retry: Failed, sleeping 60 seconds before retrying
[task 2018-07-03T16:07:47.586Z]     INFO - retry: Calling run_command with args: (['/usr/bin/python2.7', '-u', '/builds/worker/workspace/mozharness/external_tools/tooltool.py', '--url', 'http://relengapi/tooltool/', 'fetch', '-m', '/builds/worker/workspace/build/.android/releng.manifest', '-o', '-c', '/builds/worker/tooltool_cache'],), kwargs: {'output_timeout': 600, 'error_list': [{'substr': 'command not found', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x7f696be742c0>, 'level': 'warning'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x7f696c74dc48>, 'level': 'critical'}, {'regex': <_sre.SRE_Pattern object at 0x7f696c6e1648>, 'level': 'critical'}, {'substr': 'ERROR - ', 'level': 'error'}], 'cwd': '/builds/worker/workspace/build/.android', 'privileged': False}, attempt #2
[task 2018-07-03T16:07:47.586Z]     INFO - Running command: ['/usr/bin/python2.7', '-u', '/builds/worker/workspace/mozharness/external_tools/tooltool.py', '--url', 'http://relengapi/tooltool/', 'fetch', '-m', '/builds/worker/workspace/build/.android/releng.manifest', '-o', '-c', '/builds/worker/tooltool_cache'] in /builds/worker/workspace/build/.android
[task 2018-07-03T16:07:47.587Z]     INFO - Copy/paste: /usr/bin/python2.7 -u /builds/worker/workspace/mozharness/external_tools/tooltool.py --url http://relengapi/tooltool/ fetch -m /builds/worker/workspace/build/.android/releng.manifest -o -c /builds/worker/tooltool_cache
[task 2018-07-03T16:07:47.587Z]     INFO - Calling ['/usr/bin/python2.7', '-u', '/builds/worker/workspace/mozharness/external_tools/tooltool.py', '--url', 'http://relengapi/tooltool/', 'fetch', '-m', '/builds/worker/workspace/build/.android/releng.manifest', '-o', '-c', '/builds/worker/tooltool_cache'] with output_timeout 600
[task 2018-07-03T16:07:47.630Z]     INFO -  INFO - File AVDs-x86-android-4.2_r1-build-2016-11-18.tar.gz not present in local cache folder /builds/worker/tooltool_cache
[task 2018-07-03T16:07:47.631Z]     INFO -  INFO - Attempting to fetch from 'http://relengapi/tooltool/'...
[task 2018-07-03T16:07:53.261Z]     INFO -  INFO - File AVDs-x86-android-4.2_r1-build-2016-11-18.tar.gz fetched from http://relengapi/tooltool/ as /builds/worker/workspace/build/.android/tmpD1Er2v
[task 2018-07-03T16:07:53.971Z]     INFO -  INFO - File integrity verified, renaming tmpD1Er2v to AVDs-x86-android-4.2_r1-build-2016-11-18.tar.gz
[task 2018-07-03T16:07:53.971Z]     INFO -  INFO - Updating local cache /builds/worker/tooltool_cache...
[task 2018-07-03T16:07:54.120Z]     INFO -  INFO - Local cache /builds/worker/tooltool_cache updated with AVDs-x86-android-4.2_r1-build-2016-11-18.tar.gz
[task 2018-07-03T16:07:54.121Z]     INFO -  INFO - untarring "AVDs-x86-android-4.2_r1-build-2016-11-18.tar.gz"
[task 2018-07-03T16:07:59.257Z]     INFO - Return code: 0

Comment 1

[Aki Sasaki (not active)]

[task 2018-07-03T16:23:22.755Z]     INFO -  INFO - ...failed to fetch 'host-utils-61.0a1.en-US.linux-x86_64.tar.gz' from http://relengapi/tooltool/
[task 2018-07-03T16:23:22.755Z]    ERROR -  ERROR - The following files failed: 'host-utils-61.0a1.en-US.linux-x86_64.tar.gz'
[task 2018-07-03T16:23:22.777Z]    ERROR - Return code: 1
[task 2018-07-03T16:23:22.778Z]    FATAL - Tooltool /builds/worker/workspace/build/hostutils/releng.manifest fetch failed!
[task 2018-07-03T16:23:22.778Z]    FATAL - Running post_fatal callback...
[task 2018-07-03T16:23:22.779Z]    FATAL - Exiting -1

Comment 2

[Aki Sasaki (not active)]

https://hg.mozilla.org/integration/mozilla-inbound/log/default/testing/config/tooltool-manifests/linux64/hostutils.manifest hasn't changed since April.

:garbas, do you know of any tooltool issues with host-utils ?

Comment 3

[Geoff Brown [:gbrown]]

Any chance this is related to recent changes in bug 1471541?

Comment 4

[Rok Garbas [:garbas]]

relengapi-proxy was released pointing to new tooltool location (tooltool.mozilla-releng.net) which was under provisioned. I increased the dynos from 4 to 8 and metrics for the heroku app looks good again.