1838979 - Intermittent Uncaught exception: Traceback (most recent call last): after Finished download-and-extract step (failed)

Status: REOPENED

(Release Engineering :: Applications: MozharnessCore, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: nfay [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=419671413&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BhG8PjRmT0ePdxbK-fol6g/runs/0/artifacts/public/logs/live_backing.log

---

[task 2023-06-16T22:19:51.542Z] 22:19:22     INFO - Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Z9eQUNyeSjGDI7ABaphYyw/artifacts/public/build/geckoview-test_runner.apk to /builds/task_168695393418223/workspace/build/geckoview-test_runner.apk
[task 2023-06-16T22:19:51.542Z] 22:19:22     INFO - retry: Calling _download_file with args: (), kwargs: {'url': 'https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Z9eQUNyeSjGDI7ABaphYyw/artifacts/public/build/geckoview-test_runner.apk', 'file_name': '/builds/task_168695393418223/workspace/build/geckoview-test_runner.apk'}, attempt #1
[task 2023-06-16T22:19:51.542Z] 22:19:27     INFO - Downloaded 100909566 bytes.
[task 2023-06-16T22:19:51.542Z] 22:19:27     INFO - retry: Calling query_symbols_url with args: (), kwargs: {'raise_on_failure': True}, attempt #1
[task 2023-06-16T22:19:51.542Z] 22:19:27     INFO - Downloading and extracting to /builds/task_168695393418223/workspace/build/symbols these dirs * from https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Z9eQUNyeSjGDI7ABaphYyw/artifacts/public/build/target.crashreporter-symbols.zip
[task 2023-06-16T22:19:51.542Z] 22:19:27     INFO - retry: Calling fetch_url_into_memory with args: (), kwargs: {'url': 'https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Z9eQUNyeSjGDI7ABaphYyw/artifacts/public/build/target.crashreporter-symbols.zip'}, attempt #1
[task 2023-06-16T22:19:51.542Z] 22:19:27     INFO - Fetch https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Z9eQUNyeSjGDI7ABaphYyw/artifacts/public/build/target.crashreporter-symbols.zip into memory
[task 2023-06-16T22:19:51.542Z] 22:19:37     INFO - Content-Length response header: 236904684
[task 2023-06-16T22:19:51.542Z] 22:19:37     INFO - Bytes received: 236904684
[task 2023-06-16T22:19:51.542Z] 22:19:44     INFO - Running post-action listener: _resource_record_post_action
[task 2023-06-16T22:19:51.542Z] 22:19:44     INFO - Running post-action listener: find_tests_for_coverage
[task 2023-06-16T22:19:51.542Z] 22:19:44     INFO - Running post-action listener: find_tests_for_verification
[task 2023-06-16T22:19:51.542Z] 22:19:44     INFO - Running post-action listener: set_extra_try_arguments
[task 2023-06-16T22:19:51.542Z] 22:19:44     INFO - Running post-action listener: setup_coverage_tools
[task 2023-06-16T22:19:51.542Z] 22:19:44     INFO - [mozharness: 2023-06-16 22:19:44.568370Z] Finished download-and-extract step (failed)
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL - Uncaught exception: Traceback (most recent call last):
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL -   File "/builds/task_168695393418223/workspace/mozharness/mozharness/base/script.py", line 2393, in run
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL -     self.run_action(action)
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL -   File "/builds/task_168695393418223/workspace/mozharness/mozharness/base/script.py", line 2327, in run_action
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL -   File "/builds/task_168695393418223/workspace/mozharness/mozharness/base/script.py", line 2279, in _possibly_run_method
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL -     return getattr(self, method_name)()
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL -   File "/builds/task_168695393418223/workspace/mozharness/scripts/android_hardware_unittest.py", line 371, in download_and_extract
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL -     super(AndroidHardwareTest, self).download_and_extract(
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL -   File "/builds/task_168695393418223/workspace/mozharness/mozharness/mozilla/testing/testbase.py", line 589, in download_and_extract
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL -     self._download_and_extract_symbols()
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL -   File "/builds/task_168695393418223/workspace/mozharness/mozharness/mozilla/testing/testbase.py", line 546, in _download_and_extract_symbols
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL -     self.download_unpack(self.symbols_url, self.symbols_path)
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL -   File "/builds/task_168695393418223/workspace/mozharness/mozharness/base/script.py", line 824, in download_unpack
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL -     function(**kwargs)
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL -   File "/builds/task_168695393418223/workspace/mozharness/mozharness/base/script.py", line 704, in unzip
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL -     bundle.extract(entry, path=extract_to)
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL -   File "/usr/lib/python3.9/zipfile.py", line 1625, in extract
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL -     return self._extract_member(member, path, pwd)
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL -   File "/usr/lib/python3.9/zipfile.py", line 1697, in _extract_member
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL -     shutil.copyfileobj(source, target)
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL -   File "/usr/lib/python3.9/shutil.py", line 208, in copyfileobj
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL -     fdst_write(buf)
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL - OSError: [Errno 12] Cannot allocate memory
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL - Running post_fatal callback...
[task 2023-06-16T22:19:51.542Z] 22:19:46    FATAL - Exiting -1
[task 2023-06-16T22:19:51.542Z] 22:19:46     INFO - Running post-run listener: _resource_record_post_run
[task 2023-06-16T22:19:51.542Z] cleanup

Comment 1

[Intermittent Failures Robot]

2 failures in 4079 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform and build breakdown:

• android-hw-p5-13-0-android-aarch64-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2023-06-12&endday=2023-06-18&tree=all

Comment 2

[Intermittent Failures Robot]

1 failures in 3880 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux1804-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2023-06-19&endday=2023-06-25&tree=all

Comment 3

[Intermittent Failures Robot]

1 failures in 3594 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• android-hw-p5-13-0-android-aarch64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2023-06-26&endday=2023-07-02&tree=all

Comment 4

[Intermittent Failures Robot]

4 failures in 3520 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr102: 3
• mozilla-central: 1

Platform and build breakdown:

• windows7-32-shippable-qr: 1
  • opt: 1
• windows10-32-2004-qr: 1
  • debug: 1
• windows10-64-2004-qr: 1
  • debug: 1
• linux1804-64-tsan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2023-07-17&endday=2023-07-23&tree=all

Comment 5

[Intermittent Failures Robot]

8 failures in 3403 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr102: 2
• try: 6

Platform and build breakdown:

• windows10-32-2004-qr: 1
  • debug: 1
• windows10-64-2004-qr: 1
  • debug: 1
• linux1804-64-qr: 6
  • debug: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2023-07-24&endday=2023-07-30&tree=all

Comment 6

[Intermittent Failures Robot]

1 failures in 3276 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr102: 1

Platform and build breakdown:

• windows10-32-2004-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2023-07-31&endday=2023-08-06&tree=all

Comment 7

[Intermittent Failures Robot]

5 failures in 2218 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5

Platform and build breakdown:

• windows11-32-2009-qr: 4
  • opt: 3
  • debug: 1
• windows11-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2023-08-21&endday=2023-08-27&tree=all

Comment 8

[Intermittent Failures Robot]

15 failures in 682 pushes (0.022 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 1
• mozilla-beta: 1
• autoland: 13

Platform and build breakdown:

• linux1804-64-qr: 15
  • debug: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2023-09-08&endday=2023-09-08&tree=all

Comment 9

[Intermittent Failures Robot]

16 failures in 3267 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 2
• mozilla-central: 1
• autoland: 13

Platform and build breakdown:

• linux1804-64-tsan-qr: 1
  • opt: 1
• linux1804-64-qr: 15
  • debug: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2023-09-04&endday=2023-09-10&tree=all

Comment 10

[Intermittent Failures Robot]

1 failures in 3850 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux1804-64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2023-09-18&endday=2023-09-24&tree=all

Comment 11

[Intermittent Failures Robot]

2 failures in 3726 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform and build breakdown:

• android-hw-p5-13-0-android-aarch64-shippable-qr: 1
  • opt: 1
• windows11-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2023-09-25&endday=2023-10-01&tree=all

Comment 12

[Intermittent Failures Robot]

2 failures in 3764 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform and build breakdown:

• linux1804-64-ccov-qr: 1
  • opt: 1
• windows11-64-2009-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2023-10-09&endday=2023-10-15&tree=all

Comment 13

[Intermittent Failures Robot]

1 failures in 3484 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform and build breakdown:

• linux1804-64-tsan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2023-10-16&endday=2023-10-22&tree=all

Comment 14

[Intermittent Failures Robot]

1 failures in 3657 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform and build breakdown:

• linux1804-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2023-10-23&endday=2023-10-29&tree=all

Comment 15

[Intermittent Failures Robot]

1 failures in 3288 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• android-em-7-0-x86_64-lite-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2023-11-06&endday=2023-11-12&tree=all

Comment 16

[Intermittent Failures Robot]

21 failures in 4108 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-beta: 11
• mozilla-central: 2
• try: 6

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 1
  • debug: 1
• linux1804-64-tsan-qr: 6
  • opt: 6
• linux1804-64-qr: 13
  • debug: 13
• linux2204-64-wayland: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2023-11-27&endday=2023-12-03&tree=all

Comment 17

[Intermittent Failures Robot]

21 failures in 3671 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 15
• mozilla-beta: 5

Platform and build breakdown:

• linux1804-64-qr: 20
  • debug: 20
• linux2204-64-wayland: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2023-12-04&endday=2023-12-10&tree=all

Comment 18

[Intermittent Failures Robot]

2 failures in 3349 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform and build breakdown:

• linux1804-64-qr: 1
  • debug: 1
• linux1804-64-tsan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2023-12-18&endday=2023-12-24&tree=all

Comment 19

[Intermittent Failures Robot]

2 failures in 3217 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 1

Platform and build breakdown:

• android-hw-a51-11-0-aarch64-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2024-01-08&endday=2024-01-14&tree=all

Comment 20

[Intermittent Failures Robot]

1 failures in 3599 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• android-hw-a51-11-0-aarch64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2024-01-15&endday=2024-01-21&tree=all

Comment 21

[Intermittent Failures Robot]

1 failures in 4034 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows10-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2024-01-22&endday=2024-01-28&tree=all

Comment 22

[Intermittent Failures Robot]

3 failures in 3952 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-central: 1

Platform and build breakdown:

• android-5-0-aarch64: 2
  • pgo: 2
• android-em-7-0-x86_64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2024-01-29&endday=2024-02-04&tree=all

Comment 23

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 24

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=449622972&repo=autoland

Comment 25

[Intermittent Failures Robot]

1 failures in 4198 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• android-em-7-0-x86_64-lite-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2024-03-04&endday=2024-03-10&tree=all

Comment 26

[Intermittent Failures Robot]

1 failures in 3791 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2024-03-18&endday=2024-03-24&tree=all

Comment 27

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 28

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=466425060&repo=autoland

Comment 29

[Intermittent Failures Robot]

1 failures in 3799 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2024-07-15&endday=2024-07-21&tree=all

Comment 30

[Intermittent Failures Robot]

1 failures in 3459 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• android-hw-p5-13-0-android-aarch64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2024-08-05&endday=2024-08-11&tree=all

Comment 31

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 32

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=479632215&repo=mozilla-central

Comment 33

[Intermittent Failures Robot]

1 failures in 4456 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2024-10-21&endday=2024-10-27&tree=all

Comment 34

[Intermittent Failures Robot]

1 failures in 4515 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr128: 1

Platform and build breakdown:

• linux1804-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838979&startday=2024-11-04&endday=2024-11-10&tree=all

Comment 35

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 36

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=491479786&repo=autoland