Intermittent Exception: ['tar', 'xf', '-'] exited 2 after raise Exception('%r exited %d' % (args, p.returncode))
Categories
(Firefox Build System :: Android Studio and Gradle Integration, defect)
Tracking
(firefox-esr91 unaffected, firefox99 wontfix, firefox100 wontfix, firefox101 wontfix)
Tracking | Status | |
---|---|---|
firefox-esr91 | --- | unaffected |
firefox99 | --- | wontfix |
firefox100 | --- | wontfix |
firefox101 | --- | wontfix |
People
(Reporter: intermittent-bug-filer, Unassigned)
References
(Regression)
Details
(Keywords: intermittent-failure, regression, Whiteboard: [stockwell unknown])
Attachments
(3 obsolete files)
Filed by: ccoroiu [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=304307358&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BggZ3iRsROqu_W1jHz9RBA/runs/0/artifacts/public/logs/live_backing.log
fetches 2020-05-29T20:10:57.232Z] /builds/worker/fetches/rustc.tar.zst extracted in 12.668s
[fetches 2020-05-29T20:10:57.232Z] Removing /builds/worker/fetches/rustc.tar.zst
[fetches 2020-05-29T20:10:57.831Z] /builds/worker/fetches/clang.tar.zst extracted in 13.066s
[fetches 2020-05-29T20:10:57.831Z] Removing /builds/worker/fetches/clang.tar.zst
[fetches 2020-05-29T20:17:21.148Z] http://taskcluster/api/queue/v1/task/X4VFC7FvQJCHgH12wheNSg/artifacts/project/gecko/android-ndk/android-ndk.tar.zst resolved to 0 bytes with sha256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 in 398.939s
[fetches 2020-05-29T20:17:21.148Z] http://taskcluster/api/queue/v1/task/X6NGz-iRRKSKYWva45tGtg/artifacts/project/gecko/android-sdk/android-sdk-linux.tar.zst resolved to 0 bytes with sha256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 in 398.937s
[fetches 2020-05-29T20:17:21.148Z] Extracting /builds/worker/fetches/android-sdk-linux.tar.zst to /builds/worker/fetches
[fetches 2020-05-29T20:17:21.148Z] Extracting /builds/worker/fetches/android-ndk.tar.zst to /builds/worker/fetches
[fetches 2020-05-29T20:17:21.155Z] tar: This does not look like a tar archive
[fetches 2020-05-29T20:17:21.155Z] tar: Exiting with failure status due to previous errors
[fetches 2020-05-29T20:17:21.156Z] tar: This does not look like a tar archive
[fetches 2020-05-29T20:17:21.156Z] tar: Exiting with failure status due to previous errors
[fetches 2020-05-29T20:17:21.157Z] Traceback (most recent call last):
[fetches 2020-05-29T20:17:21.157Z] File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 681, in <module>
[fetches 2020-05-29T20:17:21.157Z] sys.exit(main())
[fetches 2020-05-29T20:17:21.157Z] File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 677, in main
[fetches 2020-05-29T20:17:21.157Z] return args.func(args)
[fetches 2020-05-29T20:17:21.157Z] File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 611, in command_task_artifacts
[fetches 2020-05-29T20:17:21.157Z] fetch_urls(downloads)
[fetches 2020-05-29T20:17:21.157Z] File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 482, in fetch_urls
[fetches 2020-05-29T20:17:21.157Z] f.result()
[fetches 2020-05-29T20:17:21.157Z] File "/usr/lib/python3.5/concurrent/futures/_base.py", line 405, in result
[fetches 2020-05-29T20:17:21.157Z] return self.__get_result()
[fetches 2020-05-29T20:17:21.157Z] File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
[fetches 2020-05-29T20:17:21.158Z] raise self._exception
[fetches 2020-05-29T20:17:21.158Z] File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
[fetches 2020-05-29T20:17:21.158Z] result = self.fn(*self.args, **self.kwargs)
[fetches 2020-05-29T20:17:21.158Z] File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 468, in fetch_and_extract
[fetches 2020-05-29T20:17:21.158Z] extract_archive(dest_path, dest_dir, typ)
[fetches 2020-05-29T20:17:21.158Z] File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 356, in extract_archive
[fetches 2020-05-29T20:17:21.158Z] raise Exception('%r exited %d' % (args, p.returncode))
[fetches 2020-05-29T20:17:21.158Z] Exception: ['tar', 'xf', '-'] exited 2
[fetches 2020-05-29T20:17:21.171Z] removing /builds/worker/fetches
[fetches 2020-05-29T20:17:22.102Z] finished
[taskcluster 2020-05-29 20:17:22.685Z] === Task Finished ===
Comment 1•5 years ago
|
||
Bugbug thinks this bug should belong to this component, but please revert this change in case of error.
Comment hidden (Intermittent Failures Robot) |
Comment 3•4 years ago
|
||
Updated•4 years ago
|
Comment 4•4 years ago
|
||
Updated•4 years ago
|
Comment 5•4 years ago
|
||
There are various binaries that are included in test archives that are
necessary for artifact builds to be repackaged. Don't generate artifact build
variants of tasks without them.
Comment 6•4 years ago
|
||
Comment on attachment 9156429 [details]
Bug 1642070: Don't try run multi-l10n in artifact builds; r?nalexander
Revision D79565 was moved to bug 1643842. Setting attachment 9156429 [details] to obsolete.
Comment 7•4 years ago
|
||
Comment on attachment 9156450 [details]
Bug 1642070: Don't try generate artifact builds of builds without tests; r?nalexander
Revision D79575 was moved to bug 1643842. Setting attachment 9156450 [details] to obsolete.
Comment 8•4 years ago
|
||
https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.
Comment hidden (Intermittent Failures Robot) |
Comment 10•4 years ago
|
||
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment 16•4 years ago
|
||
https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.
Comment 17•4 years ago
|
||
New occurrence: https://treeherder.mozilla.org/logviewer?job_id=325825939&repo=autoland&lineNumber=620
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment 22•4 years ago
|
||
https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.
Comment 23•4 years ago
|
||
Recent log: https://treeherder.mozilla.org/logviewer?job_id=339982696&repo=autoland&lineNumber=264
[fetches 2021-05-18T06:12:21.099Z] Extracting /builds/worker/fetches/clang.tar.zst to /builds/worker/fetches
[fetches 2021-05-18T06:12:21.303Z] /builds/worker/fetches/sysroot.tar.zst extracted in 45.119s
[fetches 2021-05-18T06:12:21.303Z] Removing /builds/worker/fetches/sysroot.tar.zst
[fetches 2021-05-18T06:12:24.550Z] /builds/worker/fetches/binutils.tar.xz extracted in 49.154s
[fetches 2021-05-18T06:12:24.550Z] Removing /builds/worker/fetches/binutils.tar.xz
[fetches 2021-05-18T06:12:47.806Z] tar: Unexpected EOF in archive
[fetches 2021-05-18T06:12:47.806Z] tar: Unexpected EOF in archive
[fetches 2021-05-18T06:12:47.806Z] tar: Error is not recoverable: exiting now
[fetches 2021-05-18T06:13:36.094Z] /builds/worker/fetches/clang.tar.zst extracted in 74.995s
[fetches 2021-05-18T06:13:36.094Z] Removing /builds/worker/fetches/clang.tar.zst
[fetches 2021-05-18T06:13:36.123Z] Traceback (most recent call last):
[fetches 2021-05-18T06:13:36.123Z] File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 736, in <module>
[fetches 2021-05-18T06:13:36.123Z] sys.exit(main())
[fetches 2021-05-18T06:13:36.123Z] File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 732, in main
[fetches 2021-05-18T06:13:36.123Z] return args.func(args)
[fetches 2021-05-18T06:13:36.123Z] File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 664, in command_task_artifacts
[fetches 2021-05-18T06:13:36.123Z] fetch_urls(downloads)
[fetches 2021-05-18T06:13:36.123Z] File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 506, in fetch_urls
[fetches 2021-05-18T06:13:36.123Z] f.result()
[fetches 2021-05-18T06:13:36.123Z] File "/usr/lib/python3.7/concurrent/futures/_base.py", line 425, in result
[fetches 2021-05-18T06:13:36.123Z] return self.__get_result()
[fetches 2021-05-18T06:13:36.123Z] File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
[fetches 2021-05-18T06:13:36.123Z] raise self._exception
[fetches 2021-05-18T06:13:36.123Z] File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
[fetches 2021-05-18T06:13:36.124Z] result = self.fn(*self.args, **self.kwargs)
[fetches 2021-05-18T06:13:36.124Z] File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 492, in fetch_and_extract
[fetches 2021-05-18T06:13:36.124Z] extract_archive(dest_path, dest_dir, typ)
[fetches 2021-05-18T06:13:36.124Z] File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 377, in extract_archive
[fetches 2021-05-18T06:13:36.124Z] raise Exception('%r exited %d' % (args, p.returncode))
[fetches 2021-05-18T06:13:36.124Z] Exception: ['tar', 'xf', '-'] exited 2
[taskcluster 2021-05-18 06:13:40.670Z] === Task Finished ===
[taskcluster 2021-05-18 06:13:40.826Z] Artifact "public/logs" not found at "/builds/worker/logs/"
[taskcluster 2021-05-18 06:13:40.918Z] Artifact "public/build" not found at "/builds/worker/artifacts/"
[taskcluster 2021-05-18 06:13:41.002Z] Unsuccessful task run with exit code: 1 completed in 367.408 seconds
Comment hidden (Intermittent Failures Robot) |
Comment 25•3 years ago
|
||
https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.
Comment 26•3 years ago
|
||
Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=345901842&repo=autoland&lineNumber=88
Comment hidden (Intermittent Failures Robot) |
Comment 28•3 years ago
|
||
Agi: Are the recent failures are related to bug 1718341 - the downloaded files are 0 bytes.
Comment 29•3 years ago
|
||
Comment 30•3 years ago
•
|
||
The artifact in the build job linked above is from the taskcluster task dSM-5RnoS9ynNepcjTgAPg
, and it's 278MB (even though in the logs it says it's 0 bytes). This is a taskcluster failure, doesn't look like it's related to my change (just happens to be that artifact for some reason).
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment 40•3 years ago
|
||
There are 50 total failures in the last 7 days on
- android-em-7-0-x86_64-lite-qr debug and opt
- android-em-7-0-x86_64-qr debug and opt
Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=351491651&repo=autoland&lineNumber=81
[fetches 2021-09-14T14:33:25.810Z] Extracting /builds/worker/fetches/android-system-images-linux.tar.zst to /builds/worker/fetches
[fetches 2021-09-14T14:33:25.815Z] tar: This does not look like a tar archive
[fetches 2021-09-14T14:33:25.815Z] tar: Exiting with failure status due to previous errors
[fetches 2021-09-14T14:33:25.826Z] Traceback (most recent call last):
[fetches 2021-09-14T14:33:25.826Z] File "/builds/worker/bin/fetch-content", line 799, in <module>
[fetches 2021-09-14T14:33:25.826Z] sys.exit(main())
[fetches 2021-09-14T14:33:25.826Z] File "/builds/worker/bin/fetch-content", line 795, in main
[fetches 2021-09-14T14:33:25.827Z] return args.func(args)
[fetches 2021-09-14T14:33:25.827Z] File "/builds/worker/bin/fetch-content", line 712, in command_task_artifacts
[fetches 2021-09-14T14:33:25.827Z] fetch_urls(downloads)
[fetches 2021-09-14T14:33:25.827Z] File "/builds/worker/bin/fetch-content", line 527, in fetch_urls
[fetches 2021-09-14T14:33:25.827Z] f.result()
[fetches 2021-09-14T14:33:25.827Z] File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
[fetches 2021-09-14T14:33:25.827Z] return self.__get_result()
[fetches 2021-09-14T14:33:25.827Z] File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
[fetches 2021-09-14T14:33:25.828Z] raise self._exception
[fetches 2021-09-14T14:33:25.828Z] File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
[fetches 2021-09-14T14:33:25.828Z] result = self.fn(*self.args, **self.kwargs)
[fetches 2021-09-14T14:33:25.828Z] File "/builds/worker/bin/fetch-content", line 513, in fetch_and_extract
[fetches 2021-09-14T14:33:25.828Z] extract_archive(dest_path, dest_dir, typ)
[fetches 2021-09-14T14:33:25.828Z] File "/builds/worker/bin/fetch-content", line 387, in extract_archive
[fetches 2021-09-14T14:33:25.828Z] raise Exception("%r exited %d" % (args, p.returncode))
[fetches 2021-09-14T14:33:25.828Z] Exception: ['tar', 'xf', '-'] exited 2
[taskcluster 2021-09-14 14:33:26.252Z] === Task Finished ===
[taskcluster 2021-09-14 14:33:26.274Z] Artifact "public/logs" not found at "/builds/worker/workspace/logs/"
[taskcluster 2021-09-14 14:33:26.286Z] Artifact "public/test" not found at "/builds/worker/artifacts/"
[taskcluster 2021-09-14 14:33:26.301Z] Artifact "public/test_info" not found at "/builds/worker/workspace/build/blobber_upload_dir/"
[taskcluster 2021-09-14 14:33:26.397Z] Unsuccessful task run with exit code: 1 completed in 48.117 seconds
Brian, can you take a look?
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Updated•3 years ago
|
Comment 61•3 years ago
|
||
There are 34 total failures in the last 7 days on
- android-em-7-0-x86_64-lite-qr debug
- android-em-7-0-x86_64-qr debug
- android-em-7-0-x86_64-shippable-lite-qr opt
Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=363931067&repo=mozilla-central&lineNumber=81
[fetches 2022-01-12T23:40:13.168Z] http://taskcluster/api/queue/v1/task/blXt76b9T3ubFF4tGJlsLw/artifacts/project/gecko/android-emulator/android-emulator-linux.tar.zst resolved to 0 bytes with sha256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 in 15.639s
[fetches 2022-01-12T23:40:13.170Z] Extracting /builds/worker/fetches/android-emulator-linux.tar.zst to /builds/worker/fetches
[fetches 2022-01-12T23:40:13.176Z] tar: This does not look like a tar archive
[fetches 2022-01-12T23:40:13.176Z] tar: Exiting with failure status due to previous errors
[fetches 2022-01-12T23:40:13.382Z] http://taskcluster/api/queue/v1/task/KIeRg040SLi9HDdTrdEE9Q/artifacts/project/gecko/android-system-images/android-system-images-linux.tar.zst resolved to 0 bytes with sha256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 in 15.853s
[fetches 2022-01-12T23:40:13.382Z] Extracting /builds/worker/fetches/android-system-images-linux.tar.zst to /builds/worker/fetches
[fetches 2022-01-12T23:40:13.387Z] tar: This does not look like a tar archive
[fetches 2022-01-12T23:40:13.387Z] tar: Exiting with failure status due to previous errors
[fetches 2022-01-12T23:40:13.402Z] Traceback (most recent call last):
[fetches 2022-01-12T23:40:13.402Z] File "/builds/worker/bin/fetch-content", line 841, in <module>
[fetches 2022-01-12T23:40:13.403Z] sys.exit(main())
[fetches 2022-01-12T23:40:13.403Z] File "/builds/worker/bin/fetch-content", line 837, in main
[fetches 2022-01-12T23:40:13.403Z] return args.func(args)
[fetches 2022-01-12T23:40:13.403Z] File "/builds/worker/bin/fetch-content", line 751, in command_task_artifacts
[fetches 2022-01-12T23:40:13.403Z] fetch_urls(downloads)
[fetches 2022-01-12T23:40:13.403Z] File "/builds/worker/bin/fetch-content", line 527, in fetch_urls
[fetches 2022-01-12T23:40:13.403Z] f.result()
[fetches 2022-01-12T23:40:13.403Z] File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
[fetches 2022-01-12T23:40:13.404Z] return self.__get_result()
[fetches 2022-01-12T23:40:13.404Z] File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
[fetches 2022-01-12T23:40:13.404Z] raise self._exception
[fetches 2022-01-12T23:40:13.404Z] File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
[fetches 2022-01-12T23:40:13.404Z] result = self.fn(*self.args, **self.kwargs)
[fetches 2022-01-12T23:40:13.404Z] File "/builds/worker/bin/fetch-content", line 513, in fetch_and_extract
[fetches 2022-01-12T23:40:13.404Z] extract_archive(dest_path, dest_dir, typ)
[fetches 2022-01-12T23:40:13.404Z] File "/builds/worker/bin/fetch-content", line 387, in extract_archive
[fetches 2022-01-12T23:40:13.404Z] raise Exception("%r exited %d" % (args, p.returncode))
[fetches 2022-01-12T23:40:13.404Z] Exception: ['tar', 'xf', '-'] exited 2
[taskcluster 2022-01-12 23:40:13.867Z] === Task Finished ===
[taskcluster 2022-01-12 23:40:13.886Z] Artifact "public/logs" not found at "/builds/worker/workspace/logs/"
[taskcluster 2022-01-12 23:40:13.902Z] Artifact "public/test" not found at "/builds/worker/artifacts/"
[taskcluster 2022-01-12 23:40:13.915Z] Artifact "public/test_info" not found at "/builds/worker/workspace/build/blobber_upload_dir/"
[taskcluster 2022-01-12 23:40:14.006Z] Unsuccessful task run with exit code: 1 completed in 23.633 seconds
Brian, can you please take a look?
Comment 62•3 years ago
|
||
Redirecting to Pete if anything can be done about this.
Comment hidden (Intermittent Failures Robot) |
Comment 64•3 years ago
•
|
||
(In reply to Andreea Pavel [:apavel] from comment #61)
There are 34 total failures in the last 7 days on
- android-em-7-0-x86_64-lite-qr debug
- android-em-7-0-x86_64-qr debug
- android-em-7-0-x86_64-shippable-lite-qr opt
Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=363931067&repo=mozilla-central&lineNumber=81[fetches 2022-01-12T23:40:13.168Z] http://taskcluster/api/queue/v1/task/blXt76b9T3ubFF4tGJlsLw/artifacts/project/gecko/android-emulator/android-emulator-linux.tar.zst resolved to 0 bytes with sha256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 in 15.639s [fetches 2022-01-12T23:40:13.170Z] Extracting /builds/worker/fetches/android-emulator-linux.tar.zst to /builds/worker/fetches [fetches 2022-01-12T23:40:13.176Z] tar: This does not look like a tar archive [fetches 2022-01-12T23:40:13.176Z] tar: Exiting with failure status due to previous errors [fetches 2022-01-12T23:40:13.382Z] http://taskcluster/api/queue/v1/task/KIeRg040SLi9HDdTrdEE9Q/artifacts/project/gecko/android-system-images/android-system-images-linux.tar.zst resolved to 0 bytes with sha256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 in 15.853s [fetches 2022-01-12T23:40:13.382Z] Extracting /builds/worker/fetches/android-system-images-linux.tar.zst to /builds/worker/fetches [fetches 2022-01-12T23:40:13.387Z] tar: This does not look like a tar archive [fetches 2022-01-12T23:40:13.387Z] tar: Exiting with failure status due to previous errors [fetches 2022-01-12T23:40:13.402Z] Traceback (most recent call last): [fetches 2022-01-12T23:40:13.402Z] File "/builds/worker/bin/fetch-content", line 841, in <module> [fetches 2022-01-12T23:40:13.403Z] sys.exit(main()) [fetches 2022-01-12T23:40:13.403Z] File "/builds/worker/bin/fetch-content", line 837, in main [fetches 2022-01-12T23:40:13.403Z] return args.func(args) [fetches 2022-01-12T23:40:13.403Z] File "/builds/worker/bin/fetch-content", line 751, in command_task_artifacts [fetches 2022-01-12T23:40:13.403Z] fetch_urls(downloads) [fetches 2022-01-12T23:40:13.403Z] File "/builds/worker/bin/fetch-content", line 527, in fetch_urls [fetches 2022-01-12T23:40:13.403Z] f.result() [fetches 2022-01-12T23:40:13.403Z] File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result [fetches 2022-01-12T23:40:13.404Z] return self.__get_result() [fetches 2022-01-12T23:40:13.404Z] File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result [fetches 2022-01-12T23:40:13.404Z] raise self._exception [fetches 2022-01-12T23:40:13.404Z] File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run [fetches 2022-01-12T23:40:13.404Z] result = self.fn(*self.args, **self.kwargs) [fetches 2022-01-12T23:40:13.404Z] File "/builds/worker/bin/fetch-content", line 513, in fetch_and_extract [fetches 2022-01-12T23:40:13.404Z] extract_archive(dest_path, dest_dir, typ) [fetches 2022-01-12T23:40:13.404Z] File "/builds/worker/bin/fetch-content", line 387, in extract_archive [fetches 2022-01-12T23:40:13.404Z] raise Exception("%r exited %d" % (args, p.returncode)) [fetches 2022-01-12T23:40:13.404Z] Exception: ['tar', 'xf', '-'] exited 2 [taskcluster 2022-01-12 23:40:13.867Z] === Task Finished === [taskcluster 2022-01-12 23:40:13.886Z] Artifact "public/logs" not found at "/builds/worker/workspace/logs/" [taskcluster 2022-01-12 23:40:13.902Z] Artifact "public/test" not found at "/builds/worker/artifacts/" [taskcluster 2022-01-12 23:40:13.915Z] Artifact "public/test_info" not found at "/builds/worker/workspace/build/blobber_upload_dir/" [taskcluster 2022-01-12 23:40:14.006Z] Unsuccessful task run with exit code: 1 completed in 23.633 seconds
Brian, can you please take a look?
Looking at https://firefox-ci-tc.services.mozilla.com/tasks/blXt76b9T3ubFF4tGJlsLw#artifacts the artifact project/gecko/android-emulator/android-emulator-linux.tar.zst
does exist, and is 307MB, so the following line is suspicious:
[fetches 2022-01-12T23:40:13.168Z] http://taskcluster/api/queue/v1/task/blXt76b9T3ubFF4tGJlsLw/artifacts/project/gecko/android-emulator/android-emulator-linux.tar.zst resolved to 0 bytes with sha256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 in 15.639s
From the output it looks like /builds/worker/bin/fetch-content believed it had fetched the content, but retrieved 0 bytes, and then tried to extract the archive. Presumably the fetch failed, but the code didn't retry, and didn't notice the failure. I suspect this is lack of error handling in /builds/worker/bin/fetch-content in method/function fetch_and_extract.
From https://searchfox.org/mozilla-central/rev/72c7cef167829b6f1e24cae216fa261934c455fc/taskcluster/scripts/misc/fetch-content#495 it looks like that happens in download_to_path
. That seems to call stream_download
so maybe there is something up there?
Since the log line above seemed to show that the download completed in 15.639 seconds, which is less than the sleep time for retries (60s) I suspect one of the following things is happening:
- It downloaded completely (i.e. the data was fetched in its entirety from the download url), but due to some bug, it resulted in a 0 byte file
- It didn't download at all, but due to some bug, no error was caught, so the code believed the file had been downloaded
- The download started but control returned to the caller before the download had completed and flushed data to the file
These are just guesses, I haven't stepped through the code in detail, but hopefully these could be pointers to someone who wants to look more closely. It looks like this is all code in-tree, not inside taskcluster - i.e. this is appears to be a task failure, rather than a taskcluster failure. Tasks that fetch data from urls need to provide adequate retry support. It looks like this code attempts to do this, but probably there is a bug in there somewhere, since it looks like the retry method didn't engage.
Oh, last option:
- At the time the task ran to download the file, the URL genuinely served a 0 byte file. Note, the sha256 (e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855) is actually correct for a zero byte file. In contrast to the other three options, that would actually be a taskcluster bug. Looking at the Queue Service logs might show whether zero bytes were actually transferred. Another option is to add logging to /builds/worker/bin/fetch-content to show the exact Headers and Body of the HTTP request and response, in the case that the artifact is zero bytes, and write that to a private log file (e.g. artifact project/gecko/android-emulator/android-emulator-linux.tar.zst.fetch.log).
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Updated•3 years ago
|
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment 73•3 years ago
|
||
Update:
There have been 35 failures within the last 7 days:
- 14 failures on Android 7.0 x86-64 WebRender opt/debug
- 21 failures on Android 7.0 x86-64 Lite WebRender opt/debug
Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=369261972&repo=autoland&lineNumber=81
[fetches 2022-02-25T20:37:53.034Z] tar: This does not look like a tar archive
[fetches 2022-02-25T20:37:53.034Z] tar: Exiting with failure status due to previous errors
[fetches 2022-02-25T20:37:53.037Z] tar: This does not look like a tar archive
[fetches 2022-02-25T20:37:53.037Z] tar: Exiting with failure status due to previous errors
[fetches 2022-02-25T20:37:53.054Z] Traceback (most recent call last):
[fetches 2022-02-25T20:37:53.054Z] File "/builds/worker/bin/fetch-content", line 841, in <module>
[fetches 2022-02-25T20:37:53.055Z] sys.exit(main())
[fetches 2022-02-25T20:37:53.055Z] File "/builds/worker/bin/fetch-content", line 837, in main
[fetches 2022-02-25T20:37:53.055Z] return args.func(args)
[fetches 2022-02-25T20:37:53.055Z] File "/builds/worker/bin/fetch-content", line 751, in command_task_artifacts
[fetches 2022-02-25T20:37:53.055Z] fetch_urls(downloads)
[fetches 2022-02-25T20:37:53.055Z] File "/builds/worker/bin/fetch-content", line 527, in fetch_urls
[fetches 2022-02-25T20:37:53.055Z] f.result()
[fetches 2022-02-25T20:37:53.055Z] File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
[fetches 2022-02-25T20:37:53.056Z] return self.__get_result()
[fetches 2022-02-25T20:37:53.056Z] File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
[fetches 2022-02-25T20:37:53.056Z] raise self._exception
[fetches 2022-02-25T20:37:53.056Z] File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
[fetches 2022-02-25T20:37:53.056Z] result = self.fn(*self.args, **self.kwargs)
[fetches 2022-02-25T20:37:53.056Z] File "/builds/worker/bin/fetch-content", line 513, in fetch_and_extract
[fetches 2022-02-25T20:37:53.056Z] extract_archive(dest_path, dest_dir, typ)
[fetches 2022-02-25T20:37:53.056Z] File "/builds/worker/bin/fetch-content", line 387, in extract_archive
[fetches 2022-02-25T20:37:53.056Z] raise Exception("%r exited %d" % (args, p.returncode))
[fetches 2022-02-25T20:37:53.056Z] Exception: ['tar', 'xf', '-'] exited 2
Pete, is there any chance you could take a look at this again?
Maybe help us assign it to someone?
Thank you.
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment 77•3 years ago
|
||
There have been 42 total failures in the last 7 days, recent failure log.
Affected platforms are:
- android-5-0-x86_64
- android-em-7-0-x86_64-lite-qr
- android-em-7-0-x86_64-qr
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment 82•3 years ago
|
||
(In reply to Natalia Csoregi [:nataliaCs] from comment #73)
Pete, is there any chance you could take a look at this again?
Maybe help us assign it to someone?
Thank you.
Hi Mike,
It looks like may have some experience with the code here (https://searchfox.org/mozilla-central/rev/72c7cef167829b6f1e24cae216fa261934c455fc/taskcluster/scripts/misc/fetch-content#165-222) - see comment 64. It would be great to get some extra debugging to see what is going on. Is this something you could help with, or point the sheriffs to someone who might be able to help?
Thanks!
Comment hidden (Intermittent Failures Robot) |
Comment 84•3 years ago
|
||
There is a race condition between the unpacking of *android-system-image*
and *android-sdk*
, whereby the former assumes the directory from the latter exists, but there is no guarantee that the latter is unpacked first.
The general rule is that toolchain artifacts shouldn't unpack in directories of other toolchain artifacts. The even stricter general rule is that toolchain artifacts should unpack to a directory corresponding to the artifact name without extensions.
Updated•3 years ago
|
Comment 85•3 years ago
|
||
Set release status flags based on info from the regressing bug 1718341
Comment hidden (Intermittent Failures Robot) |
Updated•3 years ago
|
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Updated•3 years ago
|
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Updated•2 years ago
|
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment 128•2 years ago
|
||
There have been 34 total failures in the last 7 days.
There are:
- 1 failure on android-5-0-aarch64 debug
- 2 failures on android-5-0-aarch64-shippable-lite opt
- 1 failure on android-5-0-x86_64 asan
- 1 failure on android-5-0-x86_64-shippable-lite opt
- 1 failure on android-em-7-0-x86_64-lite-qr opt
- 4 failures on android-em-7-0-x86_64-qr debug
- 1 failure on android-em-7-0-x86_64-shippable-lite-qr opt
- 1 failure on android-em-7-0-x86_64-shippable-qr opt
- 1 failure on linux1804-64-asan-qr opt
- 2 failures on linux1804-64-shippable-qr opt
- 6 failures on linux64 debug, opt and asan
- 1 failure on linux64-asan-reporter opt
- 2 failures on linux64-shippable opt
- 2 failures on osx-cross opt and debug
- 2 failures on osx-shippable opt
- 1 failure on windows2012-32 opt
- 2 failures on windows2012-32-shippable opt
- 2 failures on windows2012-64 debug and opt
- 1 failure on windows2012-64-shippable opt
Nick, as you are the Triage Owner, can you please take a look? Thank you
Recent failure log.
Updated•2 years ago
|
Comment 129•2 years ago
|
||
(In reply to Cristina Horotan [:chorotan] from comment #128)
There have been 34 total failures in the last 7 days.
There are:
- 1 failure on android-5-0-aarch64 debug
- 2 failures on android-5-0-aarch64-shippable-lite opt
- 1 failure on android-5-0-x86_64 asan
- 1 failure on android-5-0-x86_64-shippable-lite opt
- 1 failure on android-em-7-0-x86_64-lite-qr opt
- 4 failures on android-em-7-0-x86_64-qr debug
- 1 failure on android-em-7-0-x86_64-shippable-lite-qr opt
- 1 failure on android-em-7-0-x86_64-shippable-qr opt
- 1 failure on linux1804-64-asan-qr opt
- 2 failures on linux1804-64-shippable-qr opt
- 6 failures on linux64 debug, opt and asan
- 1 failure on linux64-asan-reporter opt
- 2 failures on linux64-shippable opt
- 2 failures on osx-cross opt and debug
- 2 failures on osx-shippable opt
- 1 failure on windows2012-32 opt
- 2 failures on windows2012-32-shippable opt
- 2 failures on windows2012-64 debug and opt
- 1 failure on windows2012-64-shippable opt
Nick, as you are the Triage Owner, can you please take a look? Thank you
Recent failure log.
This is just a general build system/CI issue. I think it's just flaky disk IO in automation.
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment 181•10 months ago
|
||
This is just a general build system/CI issue. I think it's just flaky disk IO in automation.
See comment 84
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Description
•