1599032 - Intermittent Can't download from https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BIS36236QZqv-i4RngxDZw/artifacts/public/build/target.common.tests.tar.gz | Caught exception: <urlopen error timed out>

Status: RESOLVED INACTIVE

(Taskcluster :: General, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: malexandru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=277958654&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Z8-HmkcYSTi14_eYhVlt2g/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Z8-HmkcYSTi14_eYhVlt2g/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

---

[task 2019-11-25T07:12:10.201Z] 07:12:10 INFO - accel:
[task 2019-11-25T07:12:10.202Z] 07:12:10 INFO - 0
[task 2019-11-25T07:12:10.202Z] 07:12:10 INFO - KVM (version 12) is installed and usable.
[task 2019-11-25T07:12:10.202Z] 07:12:10 INFO - accel
[task 2019-11-25T07:12:10.202Z] 07:12:10 INFO - Return code: 0
[task 2019-11-25T07:12:10.202Z] 07:12:10 INFO - Launching the emulator with: emulator -avd test-1 -gpu on -skip-adb-auth -verbose -show-kernel -ranchu -selinux permissive -memory 3072 -cores 4
[task 2019-11-25T07:12:10.203Z] 07:12:10 INFO - Writing log to /builds/worker/workspace/build/blobber_upload_dir/emulator-NW1DDu.log
[task 2019-11-25T07:12:10.204Z] 07:12:10 INFO - Running post-action listener: _resource_record_post_action
[task 2019-11-25T07:12:10.204Z] 07:12:10 INFO - [mozharness: 2019-11-25 07:12:10.204560Z] Finished start-emulator step (success)
[task 2019-11-25T07:12:10.204Z] 07:12:10 INFO - [mozharness: 2019-11-25 07:12:10.204619Z] Running download-and-extract step.
[task 2019-11-25T07:12:10.204Z] 07:12:10 INFO - Running pre-action listener: _resource_record_pre_action
[task 2019-11-25T07:12:10.204Z] 07:12:10 INFO - Running main action method: download_and_extract
[task 2019-11-25T07:12:10.204Z] 07:12:10 INFO - mkdir: /builds/worker/workspace/build/tests
[task 2019-11-25T07:12:10.205Z] 07:12:10 INFO - Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BIS36236QZqv-i4RngxDZw/artifacts/public/build/target.test_packages.json to /builds/worker/workspace/build/target.test_packages.json
[task 2019-11-25T07:12:10.205Z] 07:12:10 INFO - retry: Calling _download_file with args: (), kwargs: {'url': u'https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BIS36236QZqv-i4RngxDZw/artifacts/public/build/target.test_packages.json', 'file_name': u'/builds/worker/workspace/build/target.test_packages.json'}, attempt #1
[task 2019-11-25T07:14:20.358Z] 07:14:20 INFO - Downloaded 1357 bytes.
[task 2019-11-25T07:14:20.358Z] 07:14:20 INFO - Reading from file /builds/worker/workspace/build/target.test_packages.json
[task 2019-11-25T07:14:20.358Z] 07:14:20 INFO - Downloading packages: [u'target.common.tests.tar.gz', u'target.reftest.tests.tar.gz'] for test suite categories: ['reftest']
[task 2019-11-25T07:14:20.358Z] 07:14:20 INFO - Downloading and extracting to /builds/worker/workspace/build/tests these dirs * from https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BIS36236QZqv-i4RngxDZw/artifacts/public/build/target.common.tests.tar.gz
[task 2019-11-25T07:14:20.358Z] 07:14:20 INFO - retry: Calling fetch_url_into_memory with args: (), kwargs: {'url': u'https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BIS36236QZqv-i4RngxDZw/artifacts/public/build/target.common.tests.tar.gz'}, attempt #1
[task 2019-11-25T07:14:20.358Z] 07:14:20 INFO - Fetch https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BIS36236QZqv-i4RngxDZw/artifacts/public/build/target.common.tests.tar.gz into memory
[task 2019-11-25T07:14:55.364Z] 07:14:55 INFO - retry: attempt #1 caught URLError exception: <urlopen error timed out>
[task 2019-11-25T07:14:55.364Z] 07:14:55 INFO - retry: Failed, sleeping 30 seconds before retrying
[task 2019-11-25T07:15:25.365Z] 07:15:25 INFO - retry: Calling fetch_url_into_memory with args: (), kwargs: {'url': u'https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BIS36236QZqv-i4RngxDZw/artifacts/public/build/target.common.tests.tar.gz'}, attempt #2
[task 2019-11-25T07:15:25.365Z] 07:15:25 INFO - Fetch https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BIS36236QZqv-i4RngxDZw/artifacts/public/build/target.common.tests.tar.gz into memory
[task 2019-11-25T07:15:55.395Z] 07:15:55 INFO - retry: attempt #2 caught URLError exception: <urlopen error timed out>
[task 2019-11-25T07:15:55.395Z] 07:15:55 INFO - retry: Failed, sleeping 60 seconds before retrying
[task 2019-11-25T07:16:55.409Z] 07:16:55 INFO - retry: Calling fetch_url_into_memory with args: (), kwargs: {'url': u'https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BIS36236QZqv-i4RngxDZw/artifacts/public/build/target.common.tests.tar.gz'}, attempt #3
[task 2019-11-25T07:16:55.409Z] 07:16:55 INFO - Fetch https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BIS36236QZqv-i4RngxDZw/artifacts/public/build/target.common.tests.tar.gz into memory
[task 2019-11-25T07:17:30.434Z] 07:17:30 INFO - retry: attempt #3 caught URLError exception: <urlopen error timed out>
[task 2019-11-25T07:17:30.434Z] 07:17:30 INFO - retry: Failed, sleeping 120 seconds before retrying
[task 2019-11-25T07:19:30.496Z] 07:19:30 INFO - retry: Calling fetch_url_into_memory with args: (), kwargs: {'url': u'https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BIS36236QZqv-i4RngxDZw/artifacts/public/build/target.common.tests.tar.gz'}, attempt #4
[task 2019-11-25T07:19:30.496Z] 07:19:30 INFO - Fetch https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BIS36236QZqv-i4RngxDZw/artifacts/public/build/target.common.tests.tar.gz into memory
[task 2019-11-25T07:20:00.508Z] 07:20:00 INFO - retry: attempt #4 caught URLError exception: <urlopen error timed out>
[task 2019-11-25T07:20:00.508Z] 07:20:00 INFO - retry: Failed, sleeping 240 seconds before retrying
[task 2019-11-25T07:24:00.509Z] 07:24:00 INFO - retry: Calling fetch_url_into_memory with args: (), kwargs: {'url': u'https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BIS36236QZqv-i4RngxDZw/artifacts/public/build/target.common.tests.tar.gz'}, attempt #5
[task 2019-11-25T07:24:00.509Z] 07:24:00 INFO - Fetch https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BIS36236QZqv-i4RngxDZw/artifacts/public/build/target.common.tests.tar.gz into memory
[task 2019-11-25T07:24:30.539Z] 07:24:30 INFO - retry: attempt #5 caught URLError exception: <urlopen error timed out>
[task 2019-11-25T07:24:30.539Z] 07:24:30 FATAL - Can't download from https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BIS36236QZqv-i4RngxDZw/artifacts/public/build/target.common.tests.tar.gz
[task 2019-11-25T07:24:30.539Z] 07:24:30 FATAL - Caught exception: <urlopen error timed out>
[task 2019-11-25T07:24:30.539Z] 07:24:30 FATAL - Caught exception: <urlopen error timed out>
[task 2019-11-25T07:24:30.539Z] 07:24:30 FATAL - Caught exception: <urlopen error timed out>
[task 2019-11-25T07:24:30.540Z] 07:24:30 FATAL - Caught exception: <urlopen error timed out>
[task 2019-11-25T07:24:30.540Z] 07:24:30 FATAL - Caught exception: <urlopen error timed out>
[task 2019-11-25T07:24:30.540Z] 07:24:30 FATAL - Running post_fatal callback...
[task 2019-11-25T07:24:30.540Z] 07:24:30 FATAL - Exiting -1
[task 2019-11-25T07:24:30.540Z] 07:24:30 INFO - Running post-action listener: _resource_record_post_action
[task 2019-11-25T07:24:30.540Z] 07:24:30 INFO - Running post-action listener: find_tests_for_coverage
[task 2019-11-25T07:24:30.540Z] 07:24:30 INFO - Running post-action listener: find_tests_for_verification
[task 2019-11-25T07:24:30.540Z] 07:24:30 INFO - Running post-action listener: set_extra_try_arguments
[task 2019-11-25T07:24:30.540Z] 07:24:30 INFO - Running post-action listener: setup_coverage_tools
[task 2019-11-25T07:24:30.540Z] 07:24:30 INFO - [mozharness: 2019-11-25 07:24:30.540363Z] Finished download-and-extract step (failed)
[task 2019-11-25T07:24:30.540Z] 07:24:30 INFO - Running post-run listener: _resource_record_post_run
[task 2019-11-25T07:24:30.546Z] cleanup
[task 2019-11-25T07:24:30.546Z] + cleanup
[task 2019-11-25T07:24:30.546Z] + local rv=255
[task 2019-11-25T07:24:30.546Z] + [[ -s /builds/worker/.xsession-errors ]]
[task 2019-11-25T07:24:30.546Z] + cp /builds/worker/.xsession-errors /builds/worker/artifacts/public/xsession-errors.log
[task 2019-11-25T07:24:30.549Z] + true
[task 2019-11-25T07:24:30.549Z] + cleanup_xvfb
[task 2019-11-25T07:24:30.549Z] pidof Xvfb
[task 2019-11-25T07:24:30.549Z] ++ pidof Xvfb
[task 2019-11-25T07:24:30.552Z] + local xvfb_pid=25
[task 2019-11-25T07:24:30.552Z] + local vnc=false
[task 2019-11-25T07:24:30.552Z] + local interactive=false
[task 2019-11-25T07:24:30.552Z] + '[' -n 25 ']'
[task 2019-11-25T07:24:30.555Z] + [[ false == false ]]
[task 2019-11-25T07:24:30.555Z] + [[ false == false ]]
[task 2019-11-25T07:24:30.555Z] + kill 25
[task 2019-11-25T07:24:30.555Z] + screen -XS xvfb quit
[task 2019-11-25T07:24:30.902Z] No screen session found.
[task 2019-11-25T07:24:30.902Z] + true
[task 2019-11-25T07:24:30.902Z] + exit 255
[taskcluster 2019-11-25 07:24:34.936Z] === Task Finished ===
[taskcluster 2019-11-25 07:26:46.993Z] Unsuccessful task run with exit code: 255 completed in 909.243 seconds

Comment 1

[Alexandru Michis [:malexandru]]

Dustin, can you please take a look at this?

We've also had these issues appear around the same time:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=277959771&repo=autoland&lineNumber=14
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=277959793&repo=autoland&lineNumber=14

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

I think that last time this happened it had to do with network errors at packet.net? That these all occurred in packet as well suggests it's the same thing here.

Comment 3

[Intermittent Failures Robot]

4 failures in 4331 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-central: 1

Platform breakdown:

• android-em-7-0-x86_64-qr: 1
• android-em-7-0-x86_64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1599032&startday=2019-11-25&endday=2019-12-01&tree=all

Comment 4

[Intermittent Failures Robot]

15 failures in 4955 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 11
• mozilla-beta: 3
• try: 1

Platform breakdown:

• android-em-7-0-x86_64-qr: 1
• android-em-7-0-x86_64: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1599032&startday=2020-03-16&endday=2020-03-22&tree=all

Comment 5

[Intermittent Failures Robot]

1 failures in 5582 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• android-em-7-0-x86_64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1599032&startday=2020-04-20&endday=2020-04-26&tree=all

Comment 6

[Intermittent Failures Robot]

1 failures in 5488 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1599032&startday=2020-05-11&endday=2020-05-17&tree=all

Comment 7

[Intermittent Failures Robot]

1 failures in 4949 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• android-em-7-0-x86_64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1599032&startday=2020-06-01&endday=2020-06-07&tree=all

Comment 8

[Intermittent Failures Robot]

1 failures in 4195 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform breakdown:

• windows7-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1599032&startday=2020-09-28&endday=2020-10-04&tree=all

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

(closed as part of mass closure of old intermittent bugs)