Open Bug 1642070 Opened 5 years ago Updated 3 days ago

Intermittent Exception: ['tar', 'xf', '-'] exited 2 after raise Exception('%r exited %d' % (args, p.returncode))

Categories

(Firefox Build System :: Android Studio and Gradle Integration, defect)

defect

Tracking

(firefox-esr91 unaffected, firefox99 wontfix, firefox100 wontfix, firefox101 wontfix)

Tracking Status
firefox-esr91 --- unaffected
firefox99 --- wontfix
firefox100 --- wontfix
firefox101 --- wontfix

People

(Reporter: intermittent-bug-filer, Unassigned)

References

(Regression)

Details

(Keywords: intermittent-failure, regression, Whiteboard: [stockwell unknown])

Attachments

(3 obsolete files)

Filed by: ccoroiu [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=304307358&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BggZ3iRsROqu_W1jHz9RBA/runs/0/artifacts/public/logs/live_backing.log


fetches 2020-05-29T20:10:57.232Z] /builds/worker/fetches/rustc.tar.zst extracted in 12.668s
[fetches 2020-05-29T20:10:57.232Z] Removing /builds/worker/fetches/rustc.tar.zst
[fetches 2020-05-29T20:10:57.831Z] /builds/worker/fetches/clang.tar.zst extracted in 13.066s
[fetches 2020-05-29T20:10:57.831Z] Removing /builds/worker/fetches/clang.tar.zst
[fetches 2020-05-29T20:17:21.148Z] http://taskcluster/api/queue/v1/task/X4VFC7FvQJCHgH12wheNSg/artifacts/project/gecko/android-ndk/android-ndk.tar.zst resolved to 0 bytes with sha256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 in 398.939s
[fetches 2020-05-29T20:17:21.148Z] http://taskcluster/api/queue/v1/task/X6NGz-iRRKSKYWva45tGtg/artifacts/project/gecko/android-sdk/android-sdk-linux.tar.zst resolved to 0 bytes with sha256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 in 398.937s
[fetches 2020-05-29T20:17:21.148Z] Extracting /builds/worker/fetches/android-sdk-linux.tar.zst to /builds/worker/fetches
[fetches 2020-05-29T20:17:21.148Z] Extracting /builds/worker/fetches/android-ndk.tar.zst to /builds/worker/fetches
[fetches 2020-05-29T20:17:21.155Z] tar: This does not look like a tar archive
[fetches 2020-05-29T20:17:21.155Z] tar: Exiting with failure status due to previous errors
[fetches 2020-05-29T20:17:21.156Z] tar: This does not look like a tar archive
[fetches 2020-05-29T20:17:21.156Z] tar: Exiting with failure status due to previous errors
[fetches 2020-05-29T20:17:21.157Z] Traceback (most recent call last):
[fetches 2020-05-29T20:17:21.157Z] File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 681, in <module>
[fetches 2020-05-29T20:17:21.157Z] sys.exit(main())
[fetches 2020-05-29T20:17:21.157Z] File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 677, in main
[fetches 2020-05-29T20:17:21.157Z] return args.func(args)
[fetches 2020-05-29T20:17:21.157Z] File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 611, in command_task_artifacts
[fetches 2020-05-29T20:17:21.157Z] fetch_urls(downloads)
[fetches 2020-05-29T20:17:21.157Z] File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 482, in fetch_urls
[fetches 2020-05-29T20:17:21.157Z] f.result()
[fetches 2020-05-29T20:17:21.157Z] File "/usr/lib/python3.5/concurrent/futures/_base.py", line 405, in result
[fetches 2020-05-29T20:17:21.157Z] return self.__get_result()
[fetches 2020-05-29T20:17:21.157Z] File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
[fetches 2020-05-29T20:17:21.158Z] raise self._exception
[fetches 2020-05-29T20:17:21.158Z] File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
[fetches 2020-05-29T20:17:21.158Z] result = self.fn(*self.args, **self.kwargs)
[fetches 2020-05-29T20:17:21.158Z] File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 468, in fetch_and_extract
[fetches 2020-05-29T20:17:21.158Z] extract_archive(dest_path, dest_dir, typ)
[fetches 2020-05-29T20:17:21.158Z] File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 356, in extract_archive
[fetches 2020-05-29T20:17:21.158Z] raise Exception('%r exited %d' % (args, p.returncode))
[fetches 2020-05-29T20:17:21.158Z] Exception: ['tar', 'xf', '-'] exited 2
[fetches 2020-05-29T20:17:21.171Z] removing /builds/worker/fetches
[fetches 2020-05-29T20:17:22.102Z] finished
[taskcluster 2020-05-29 20:17:22.685Z] === Task Finished ===

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Component: General → Task Configuration
Assignee: nobody → mozilla
Status: NEW → ASSIGNED
Attachment #9156407 - Attachment is obsolete: true

There are various binaries that are included in test archives that are
necessary for artifact builds to be repackaged. Don't generate artifact build
variants of tasks without them.

See Also: → 1641293

Comment on attachment 9156429 [details]
Bug 1642070: Don't try run multi-l10n in artifact builds; r?nalexander

Revision D79565 was moved to bug 1643842. Setting attachment 9156429 [details] to obsolete.

Attachment #9156429 - Attachment is obsolete: true

Comment on attachment 9156450 [details]
Bug 1642070: Don't try generate artifact builds of builds without tests; r?nalexander

Revision D79575 was moved to bug 1643842. Setting attachment 9156450 [details] to obsolete.

Attachment #9156450 - Attachment is obsolete: true
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → INCOMPLETE
Status: REOPENED → RESOLVED
Closed: 4 years ago4 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 4 years ago4 years ago
Resolution: --- → INCOMPLETE

Recent log: https://treeherder.mozilla.org/logviewer?job_id=339982696&repo=autoland&lineNumber=264

[fetches 2021-05-18T06:12:21.099Z] Extracting /builds/worker/fetches/clang.tar.zst to /builds/worker/fetches
[fetches 2021-05-18T06:12:21.303Z] /builds/worker/fetches/sysroot.tar.zst extracted in 45.119s
[fetches 2021-05-18T06:12:21.303Z] Removing /builds/worker/fetches/sysroot.tar.zst
[fetches 2021-05-18T06:12:24.550Z] /builds/worker/fetches/binutils.tar.xz extracted in 49.154s
[fetches 2021-05-18T06:12:24.550Z] Removing /builds/worker/fetches/binutils.tar.xz
[fetches 2021-05-18T06:12:47.806Z] tar: Unexpected EOF in archive
[fetches 2021-05-18T06:12:47.806Z] tar: Unexpected EOF in archive
[fetches 2021-05-18T06:12:47.806Z] tar: Error is not recoverable: exiting now
[fetches 2021-05-18T06:13:36.094Z] /builds/worker/fetches/clang.tar.zst extracted in 74.995s
[fetches 2021-05-18T06:13:36.094Z] Removing /builds/worker/fetches/clang.tar.zst
[fetches 2021-05-18T06:13:36.123Z] Traceback (most recent call last):
[fetches 2021-05-18T06:13:36.123Z]   File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 736, in <module>
[fetches 2021-05-18T06:13:36.123Z]     sys.exit(main())
[fetches 2021-05-18T06:13:36.123Z]   File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 732, in main
[fetches 2021-05-18T06:13:36.123Z]     return args.func(args)
[fetches 2021-05-18T06:13:36.123Z]   File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 664, in command_task_artifacts
[fetches 2021-05-18T06:13:36.123Z]     fetch_urls(downloads)
[fetches 2021-05-18T06:13:36.123Z]   File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 506, in fetch_urls
[fetches 2021-05-18T06:13:36.123Z]     f.result()
[fetches 2021-05-18T06:13:36.123Z]   File "/usr/lib/python3.7/concurrent/futures/_base.py", line 425, in result
[fetches 2021-05-18T06:13:36.123Z]     return self.__get_result()
[fetches 2021-05-18T06:13:36.123Z]   File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
[fetches 2021-05-18T06:13:36.123Z]     raise self._exception
[fetches 2021-05-18T06:13:36.123Z]   File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
[fetches 2021-05-18T06:13:36.124Z]     result = self.fn(*self.args, **self.kwargs)
[fetches 2021-05-18T06:13:36.124Z]   File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 492, in fetch_and_extract
[fetches 2021-05-18T06:13:36.124Z]     extract_archive(dest_path, dest_dir, typ)
[fetches 2021-05-18T06:13:36.124Z]   File "/builds/worker/checkouts/gecko/taskcluster/scripts/misc/fetch-content", line 377, in extract_archive
[fetches 2021-05-18T06:13:36.124Z]     raise Exception('%r exited %d' % (args, p.returncode))
[fetches 2021-05-18T06:13:36.124Z] Exception: ['tar', 'xf', '-'] exited 2
[taskcluster 2021-05-18 06:13:40.670Z] === Task Finished ===
[taskcluster 2021-05-18 06:13:40.826Z] Artifact "public/logs" not found at "/builds/worker/logs/"
[taskcluster 2021-05-18 06:13:40.918Z] Artifact "public/build" not found at "/builds/worker/artifacts/"
[taskcluster 2021-05-18 06:13:41.002Z] Unsuccessful task run with exit code: 1 completed in 367.408 seconds
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 4 years ago3 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

Agi: Are the recent failures are related to bug 1718341 - the downloaded files are 0 bytes.

Assignee: mozilla → nobody
Status: REOPENED → NEW
Flags: needinfo?(agi)

The artifact in the build job linked above is from the taskcluster task dSM-5RnoS9ynNepcjTgAPg, and it's 278MB (even though in the logs it says it's 0 bytes). This is a taskcluster failure, doesn't look like it's related to my change (just happens to be that artifact for some reason).

Component: Task Configuration → General
Flags: needinfo?(agi)
Priority: P5 → --
Product: Firefox Build System → Taskcluster

There are 50 total failures in the last 7 days on

  • android-em-7-0-x86_64-lite-qr debug and opt
  • android-em-7-0-x86_64-qr debug and opt

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=351491651&repo=autoland&lineNumber=81

[fetches 2021-09-14T14:33:25.810Z] Extracting /builds/worker/fetches/android-system-images-linux.tar.zst to /builds/worker/fetches
[fetches 2021-09-14T14:33:25.815Z] tar: This does not look like a tar archive
[fetches 2021-09-14T14:33:25.815Z] tar: Exiting with failure status due to previous errors
[fetches 2021-09-14T14:33:25.826Z] Traceback (most recent call last):
[fetches 2021-09-14T14:33:25.826Z]   File "/builds/worker/bin/fetch-content", line 799, in <module>
[fetches 2021-09-14T14:33:25.826Z]     sys.exit(main())
[fetches 2021-09-14T14:33:25.826Z]   File "/builds/worker/bin/fetch-content", line 795, in main
[fetches 2021-09-14T14:33:25.827Z]     return args.func(args)
[fetches 2021-09-14T14:33:25.827Z]   File "/builds/worker/bin/fetch-content", line 712, in command_task_artifacts
[fetches 2021-09-14T14:33:25.827Z]     fetch_urls(downloads)
[fetches 2021-09-14T14:33:25.827Z]   File "/builds/worker/bin/fetch-content", line 527, in fetch_urls
[fetches 2021-09-14T14:33:25.827Z]     f.result()
[fetches 2021-09-14T14:33:25.827Z]   File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
[fetches 2021-09-14T14:33:25.827Z]     return self.__get_result()
[fetches 2021-09-14T14:33:25.827Z]   File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
[fetches 2021-09-14T14:33:25.828Z]     raise self._exception
[fetches 2021-09-14T14:33:25.828Z]   File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
[fetches 2021-09-14T14:33:25.828Z]     result = self.fn(*self.args, **self.kwargs)
[fetches 2021-09-14T14:33:25.828Z]   File "/builds/worker/bin/fetch-content", line 513, in fetch_and_extract
[fetches 2021-09-14T14:33:25.828Z]     extract_archive(dest_path, dest_dir, typ)
[fetches 2021-09-14T14:33:25.828Z]   File "/builds/worker/bin/fetch-content", line 387, in extract_archive
[fetches 2021-09-14T14:33:25.828Z]     raise Exception("%r exited %d" % (args, p.returncode))
[fetches 2021-09-14T14:33:25.828Z] Exception: ['tar', 'xf', '-'] exited 2
[taskcluster 2021-09-14 14:33:26.252Z] === Task Finished ===
[taskcluster 2021-09-14 14:33:26.274Z] Artifact "public/logs" not found at "/builds/worker/workspace/logs/"
[taskcluster 2021-09-14 14:33:26.286Z] Artifact "public/test" not found at "/builds/worker/artifacts/"
[taskcluster 2021-09-14 14:33:26.301Z] Artifact "public/test_info" not found at "/builds/worker/workspace/build/blobber_upload_dir/"
[taskcluster 2021-09-14 14:33:26.397Z] Unsuccessful task run with exit code: 1 completed in 48.117 seconds

Brian, can you take a look?

Flags: needinfo?(bstack01)
Whiteboard: [stockwell needswork:owner]
Flags: needinfo?(bstack01)

There are 34 total failures in the last 7 days on

[fetches 2022-01-12T23:40:13.168Z] http://taskcluster/api/queue/v1/task/blXt76b9T3ubFF4tGJlsLw/artifacts/project/gecko/android-emulator/android-emulator-linux.tar.zst resolved to 0 bytes with sha256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 in 15.639s
[fetches 2022-01-12T23:40:13.170Z] Extracting /builds/worker/fetches/android-emulator-linux.tar.zst to /builds/worker/fetches
[fetches 2022-01-12T23:40:13.176Z] tar: This does not look like a tar archive
[fetches 2022-01-12T23:40:13.176Z] tar: Exiting with failure status due to previous errors
[fetches 2022-01-12T23:40:13.382Z] http://taskcluster/api/queue/v1/task/KIeRg040SLi9HDdTrdEE9Q/artifacts/project/gecko/android-system-images/android-system-images-linux.tar.zst resolved to 0 bytes with sha256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 in 15.853s
[fetches 2022-01-12T23:40:13.382Z] Extracting /builds/worker/fetches/android-system-images-linux.tar.zst to /builds/worker/fetches
[fetches 2022-01-12T23:40:13.387Z] tar: This does not look like a tar archive
[fetches 2022-01-12T23:40:13.387Z] tar: Exiting with failure status due to previous errors
[fetches 2022-01-12T23:40:13.402Z] Traceback (most recent call last):
[fetches 2022-01-12T23:40:13.402Z]   File "/builds/worker/bin/fetch-content", line 841, in <module>
[fetches 2022-01-12T23:40:13.403Z]     sys.exit(main())
[fetches 2022-01-12T23:40:13.403Z]   File "/builds/worker/bin/fetch-content", line 837, in main
[fetches 2022-01-12T23:40:13.403Z]     return args.func(args)
[fetches 2022-01-12T23:40:13.403Z]   File "/builds/worker/bin/fetch-content", line 751, in command_task_artifacts
[fetches 2022-01-12T23:40:13.403Z]     fetch_urls(downloads)
[fetches 2022-01-12T23:40:13.403Z]   File "/builds/worker/bin/fetch-content", line 527, in fetch_urls
[fetches 2022-01-12T23:40:13.403Z]     f.result()
[fetches 2022-01-12T23:40:13.403Z]   File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
[fetches 2022-01-12T23:40:13.404Z]     return self.__get_result()
[fetches 2022-01-12T23:40:13.404Z]   File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
[fetches 2022-01-12T23:40:13.404Z]     raise self._exception
[fetches 2022-01-12T23:40:13.404Z]   File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
[fetches 2022-01-12T23:40:13.404Z]     result = self.fn(*self.args, **self.kwargs)
[fetches 2022-01-12T23:40:13.404Z]   File "/builds/worker/bin/fetch-content", line 513, in fetch_and_extract
[fetches 2022-01-12T23:40:13.404Z]     extract_archive(dest_path, dest_dir, typ)
[fetches 2022-01-12T23:40:13.404Z]   File "/builds/worker/bin/fetch-content", line 387, in extract_archive
[fetches 2022-01-12T23:40:13.404Z]     raise Exception("%r exited %d" % (args, p.returncode))
[fetches 2022-01-12T23:40:13.404Z] Exception: ['tar', 'xf', '-'] exited 2
[taskcluster 2022-01-12 23:40:13.867Z] === Task Finished ===
[taskcluster 2022-01-12 23:40:13.886Z] Artifact "public/logs" not found at "/builds/worker/workspace/logs/"
[taskcluster 2022-01-12 23:40:13.902Z] Artifact "public/test" not found at "/builds/worker/artifacts/"
[taskcluster 2022-01-12 23:40:13.915Z] Artifact "public/test_info" not found at "/builds/worker/workspace/build/blobber_upload_dir/"
[taskcluster 2022-01-12 23:40:14.006Z] Unsuccessful task run with exit code: 1 completed in 23.633 seconds

Brian, can you please take a look?

Flags: needinfo?(bstack01)
Whiteboard: [stockwell unknown] → [stockwell needswork:owner]

Redirecting to Pete if anything can be done about this.

Flags: needinfo?(bstack01) → needinfo?(pmoore)

(In reply to Andreea Pavel [:apavel] from comment #61)

There are 34 total failures in the last 7 days on

[fetches 2022-01-12T23:40:13.168Z] http://taskcluster/api/queue/v1/task/blXt76b9T3ubFF4tGJlsLw/artifacts/project/gecko/android-emulator/android-emulator-linux.tar.zst resolved to 0 bytes with sha256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 in 15.639s
[fetches 2022-01-12T23:40:13.170Z] Extracting /builds/worker/fetches/android-emulator-linux.tar.zst to /builds/worker/fetches
[fetches 2022-01-12T23:40:13.176Z] tar: This does not look like a tar archive
[fetches 2022-01-12T23:40:13.176Z] tar: Exiting with failure status due to previous errors
[fetches 2022-01-12T23:40:13.382Z] http://taskcluster/api/queue/v1/task/KIeRg040SLi9HDdTrdEE9Q/artifacts/project/gecko/android-system-images/android-system-images-linux.tar.zst resolved to 0 bytes with sha256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 in 15.853s
[fetches 2022-01-12T23:40:13.382Z] Extracting /builds/worker/fetches/android-system-images-linux.tar.zst to /builds/worker/fetches
[fetches 2022-01-12T23:40:13.387Z] tar: This does not look like a tar archive
[fetches 2022-01-12T23:40:13.387Z] tar: Exiting with failure status due to previous errors
[fetches 2022-01-12T23:40:13.402Z] Traceback (most recent call last):
[fetches 2022-01-12T23:40:13.402Z]   File "/builds/worker/bin/fetch-content", line 841, in <module>
[fetches 2022-01-12T23:40:13.403Z]     sys.exit(main())
[fetches 2022-01-12T23:40:13.403Z]   File "/builds/worker/bin/fetch-content", line 837, in main
[fetches 2022-01-12T23:40:13.403Z]     return args.func(args)
[fetches 2022-01-12T23:40:13.403Z]   File "/builds/worker/bin/fetch-content", line 751, in command_task_artifacts
[fetches 2022-01-12T23:40:13.403Z]     fetch_urls(downloads)
[fetches 2022-01-12T23:40:13.403Z]   File "/builds/worker/bin/fetch-content", line 527, in fetch_urls
[fetches 2022-01-12T23:40:13.403Z]     f.result()
[fetches 2022-01-12T23:40:13.403Z]   File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
[fetches 2022-01-12T23:40:13.404Z]     return self.__get_result()
[fetches 2022-01-12T23:40:13.404Z]   File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
[fetches 2022-01-12T23:40:13.404Z]     raise self._exception
[fetches 2022-01-12T23:40:13.404Z]   File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
[fetches 2022-01-12T23:40:13.404Z]     result = self.fn(*self.args, **self.kwargs)
[fetches 2022-01-12T23:40:13.404Z]   File "/builds/worker/bin/fetch-content", line 513, in fetch_and_extract
[fetches 2022-01-12T23:40:13.404Z]     extract_archive(dest_path, dest_dir, typ)
[fetches 2022-01-12T23:40:13.404Z]   File "/builds/worker/bin/fetch-content", line 387, in extract_archive
[fetches 2022-01-12T23:40:13.404Z]     raise Exception("%r exited %d" % (args, p.returncode))
[fetches 2022-01-12T23:40:13.404Z] Exception: ['tar', 'xf', '-'] exited 2
[taskcluster 2022-01-12 23:40:13.867Z] === Task Finished ===
[taskcluster 2022-01-12 23:40:13.886Z] Artifact "public/logs" not found at "/builds/worker/workspace/logs/"
[taskcluster 2022-01-12 23:40:13.902Z] Artifact "public/test" not found at "/builds/worker/artifacts/"
[taskcluster 2022-01-12 23:40:13.915Z] Artifact "public/test_info" not found at "/builds/worker/workspace/build/blobber_upload_dir/"
[taskcluster 2022-01-12 23:40:14.006Z] Unsuccessful task run with exit code: 1 completed in 23.633 seconds

Brian, can you please take a look?

Looking at https://firefox-ci-tc.services.mozilla.com/tasks/blXt76b9T3ubFF4tGJlsLw#artifacts the artifact project/gecko/android-emulator/android-emulator-linux.tar.zst does exist, and is 307MB, so the following line is suspicious:

[fetches 2022-01-12T23:40:13.168Z] http://taskcluster/api/queue/v1/task/blXt76b9T3ubFF4tGJlsLw/artifacts/project/gecko/android-emulator/android-emulator-linux.tar.zst resolved to 0 bytes with sha256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 in 15.639s

From the output it looks like /builds/worker/bin/fetch-content believed it had fetched the content, but retrieved 0 bytes, and then tried to extract the archive. Presumably the fetch failed, but the code didn't retry, and didn't notice the failure. I suspect this is lack of error handling in /builds/worker/bin/fetch-content in method/function fetch_and_extract.

From https://searchfox.org/mozilla-central/rev/72c7cef167829b6f1e24cae216fa261934c455fc/taskcluster/scripts/misc/fetch-content#495 it looks like that happens in download_to_path. That seems to call stream_download so maybe there is something up there?

Since the log line above seemed to show that the download completed in 15.639 seconds, which is less than the sleep time for retries (60s) I suspect one of the following things is happening:

  1. It downloaded completely (i.e. the data was fetched in its entirety from the download url), but due to some bug, it resulted in a 0 byte file
  2. It didn't download at all, but due to some bug, no error was caught, so the code believed the file had been downloaded
  3. The download started but control returned to the caller before the download had completed and flushed data to the file

These are just guesses, I haven't stepped through the code in detail, but hopefully these could be pointers to someone who wants to look more closely. It looks like this is all code in-tree, not inside taskcluster - i.e. this is appears to be a task failure, rather than a taskcluster failure. Tasks that fetch data from urls need to provide adequate retry support. It looks like this code attempts to do this, but probably there is a bug in there somewhere, since it looks like the retry method didn't engage.

Oh, last option:

  1. At the time the task ran to download the file, the URL genuinely served a 0 byte file. Note, the sha256 (e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855) is actually correct for a zero byte file. In contrast to the other three options, that would actually be a taskcluster bug. Looking at the Queue Service logs might show whether zero bytes were actually transferred. Another option is to add logging to /builds/worker/bin/fetch-content to show the exact Headers and Body of the HTTP request and response, in the case that the artifact is zero bytes, and write that to a private log file (e.g. artifact project/gecko/android-emulator/android-emulator-linux.tar.zst.fetch.log).
Flags: needinfo?(pmoore)
Summary: Intermittent Exception: ['tar', 'xf', '-'] exited 2 after aise Exception('%r exited %d' % (args, p.returncode)) → Intermittent Exception: ['tar', 'xf', '-'] exited 2 after raise Exception('%r exited %d' % (args, p.returncode))

Update:

There have been 35 failures within the last 7 days:

  • 14 failures on Android 7.0 x86-64 WebRender opt/debug
  • 21 failures on Android 7.0 x86-64 Lite WebRender opt/debug

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=369261972&repo=autoland&lineNumber=81

[fetches 2022-02-25T20:37:53.034Z] tar: This does not look like a tar archive
[fetches 2022-02-25T20:37:53.034Z] tar: Exiting with failure status due to previous errors
[fetches 2022-02-25T20:37:53.037Z] tar: This does not look like a tar archive
[fetches 2022-02-25T20:37:53.037Z] tar: Exiting with failure status due to previous errors
[fetches 2022-02-25T20:37:53.054Z] Traceback (most recent call last):
[fetches 2022-02-25T20:37:53.054Z]   File "/builds/worker/bin/fetch-content", line 841, in <module>
[fetches 2022-02-25T20:37:53.055Z]     sys.exit(main())
[fetches 2022-02-25T20:37:53.055Z]   File "/builds/worker/bin/fetch-content", line 837, in main
[fetches 2022-02-25T20:37:53.055Z]     return args.func(args)
[fetches 2022-02-25T20:37:53.055Z]   File "/builds/worker/bin/fetch-content", line 751, in command_task_artifacts
[fetches 2022-02-25T20:37:53.055Z]     fetch_urls(downloads)
[fetches 2022-02-25T20:37:53.055Z]   File "/builds/worker/bin/fetch-content", line 527, in fetch_urls
[fetches 2022-02-25T20:37:53.055Z]     f.result()
[fetches 2022-02-25T20:37:53.055Z]   File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
[fetches 2022-02-25T20:37:53.056Z]     return self.__get_result()
[fetches 2022-02-25T20:37:53.056Z]   File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
[fetches 2022-02-25T20:37:53.056Z]     raise self._exception
[fetches 2022-02-25T20:37:53.056Z]   File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
[fetches 2022-02-25T20:37:53.056Z]     result = self.fn(*self.args, **self.kwargs)
[fetches 2022-02-25T20:37:53.056Z]   File "/builds/worker/bin/fetch-content", line 513, in fetch_and_extract
[fetches 2022-02-25T20:37:53.056Z]     extract_archive(dest_path, dest_dir, typ)
[fetches 2022-02-25T20:37:53.056Z]   File "/builds/worker/bin/fetch-content", line 387, in extract_archive
[fetches 2022-02-25T20:37:53.056Z]     raise Exception("%r exited %d" % (args, p.returncode))
[fetches 2022-02-25T20:37:53.056Z] Exception: ['tar', 'xf', '-'] exited 2

Pete, is there any chance you could take a look at this again?
Maybe help us assign it to someone?
Thank you.

Flags: needinfo?(pmoore)
Whiteboard: [stockwell unknown] → [stockwell needswork]

There have been 42 total failures in the last 7 days, recent failure log.
Affected platforms are:

  • android-5-0-x86_64
  • android-em-7-0-x86_64-lite-qr
  • android-em-7-0-x86_64-qr

(In reply to Natalia Csoregi [:nataliaCs] from comment #73)

Pete, is there any chance you could take a look at this again?
Maybe help us assign it to someone?
Thank you.

Hi Mike,

It looks like may have some experience with the code here (https://searchfox.org/mozilla-central/rev/72c7cef167829b6f1e24cae216fa261934c455fc/taskcluster/scripts/misc/fetch-content#165-222) - see comment 64. It would be great to get some extra debugging to see what is going on. Is this something you could help with, or point the sheriffs to someone who might be able to help?

Thanks!

Flags: needinfo?(pmoore) → needinfo?(mh+mozilla)

There is a race condition between the unpacking of *android-system-image* and *android-sdk*, whereby the former assumes the directory from the latter exists, but there is no guarantee that the latter is unpacked first.
The general rule is that toolchain artifacts shouldn't unpack in directories of other toolchain artifacts. The even stricter general rule is that toolchain artifacts should unpack to a directory corresponding to the artifact name without extensions.

Component: General → Android Studio and Gradle Integration
Flags: needinfo?(mh+mozilla)
Product: Taskcluster → Firefox Build System
Regressed by: 1718341

Set release status flags based on info from the regressing bug 1718341

Severity: normal → S3

There have been 34 total failures in the last 7 days.
There are:

  • 1 failure on android-5-0-aarch64 debug
  • 2 failures on android-5-0-aarch64-shippable-lite opt
  • 1 failure on android-5-0-x86_64 asan
  • 1 failure on android-5-0-x86_64-shippable-lite opt
  • 1 failure on android-em-7-0-x86_64-lite-qr opt
  • 4 failures on android-em-7-0-x86_64-qr debug
  • 1 failure on android-em-7-0-x86_64-shippable-lite-qr opt
  • 1 failure on android-em-7-0-x86_64-shippable-qr opt
  • 1 failure on linux1804-64-asan-qr opt
  • 2 failures on linux1804-64-shippable-qr opt
  • 6 failures on linux64 debug, opt and asan
  • 1 failure on linux64-asan-reporter opt
  • 2 failures on linux64-shippable opt
  • 2 failures on osx-cross opt and debug
  • 2 failures on osx-shippable opt
  • 1 failure on windows2012-32 opt
  • 2 failures on windows2012-32-shippable opt
  • 2 failures on windows2012-64 debug and opt
  • 1 failure on windows2012-64-shippable opt

Nick, as you are the Triage Owner, can you please take a look? Thank you

Recent failure log.

Flags: needinfo?(nalexander)
Whiteboard: [stockwell unknown] → [stockwell needswork:owner]

(In reply to Cristina Horotan [:chorotan] from comment #128)

There have been 34 total failures in the last 7 days.
There are:

  • 1 failure on android-5-0-aarch64 debug
  • 2 failures on android-5-0-aarch64-shippable-lite opt
  • 1 failure on android-5-0-x86_64 asan
  • 1 failure on android-5-0-x86_64-shippable-lite opt
  • 1 failure on android-em-7-0-x86_64-lite-qr opt
  • 4 failures on android-em-7-0-x86_64-qr debug
  • 1 failure on android-em-7-0-x86_64-shippable-lite-qr opt
  • 1 failure on android-em-7-0-x86_64-shippable-qr opt
  • 1 failure on linux1804-64-asan-qr opt
  • 2 failures on linux1804-64-shippable-qr opt
  • 6 failures on linux64 debug, opt and asan
  • 1 failure on linux64-asan-reporter opt
  • 2 failures on linux64-shippable opt
  • 2 failures on osx-cross opt and debug
  • 2 failures on osx-shippable opt
  • 1 failure on windows2012-32 opt
  • 2 failures on windows2012-32-shippable opt
  • 2 failures on windows2012-64 debug and opt
  • 1 failure on windows2012-64-shippable opt

Nick, as you are the Triage Owner, can you please take a look? Thank you

Recent failure log.

This is just a general build system/CI issue. I think it's just flaky disk IO in automation.

Component: Android Studio and Gradle Integration → General
Flags: needinfo?(nalexander)
See Also: → 1839395

This is just a general build system/CI issue. I think it's just flaky disk IO in automation.

See comment 84

Component: General → Android Studio and Gradle Integration
Depends on: 1884685
See Also: → 1885155
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: