Closed Bug 1802331 Opened 2 years ago Closed 2 years ago

Jobs on NSS fail when migrating to GCP with invalid header on tar file

Categories

(NSS :: Build, defect)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: masterwayz, Assigned: jcristau)

References

(Regression)

Details

(Keywords: regression)

Attachments

(1 file)

No description provided.

Fails on GCP, passes on AWS.
https://treeherder.mozilla.org/jobs?repo=nss&revision=a3669ed2c60675dd79c446aa98795c05f79cb591&selectedTaskRun=7p9bUxdXTbCK64QHFR3z5Q.0

[taskcluster 2022-11-24 13:09:48.824Z] Task ID: 7p9bUxdXTbCK64QHFR3z5Q
[taskcluster 2022-11-24 13:09:48.824Z] Worker ID: 2650636084908839504
[taskcluster 2022-11-24 13:09:48.824Z] Worker Group: us-west1
[taskcluster 2022-11-24 13:09:48.824Z] Worker Node Type: projects/970387039909/machineTypes/n2-standard-4
[taskcluster 2022-11-24 13:09:48.824Z] Worker Pool: nss-3/linux-gcp
[taskcluster 2022-11-24 13:09:48.824Z] Worker Version: 38.0.4
[taskcluster 2022-11-24 13:09:48.824Z] Public IP: 34.82.59.78
[taskcluster 2022-11-24 13:09:48.824Z] Hostname: nss-3-linux-gcp-d9bhbytusvkwffeanbdrwq

[taskcluster 2022-11-24 13:10:00.351Z] Downloading artifact "public/image.tar" from task ID: m1bD4ii1QzCLupCMv7zx_g.
[taskcluster 2022-11-24 13:10:04.158Z] Downloaded artifact successfully.
[taskcluster 2022-11-24 13:10:04.158Z] Downloaded 264.909 mb
[taskcluster 2022-11-24 13:10:04.158Z] Loading docker image from downloaded archive.
[taskcluster 2022-11-24 13:10:04.220Z] Downloading artifact "public/image.tar" from task ID: m1bD4ii1QzCLupCMv7zx_g.
[taskcluster 2022-11-24 13:10:08.344Z] Downloaded artifact successfully.
[taskcluster 2022-11-24 13:10:08.344Z] Downloaded 264.909 mb
[taskcluster 2022-11-24 13:10:08.344Z] Loading docker image from downloaded archive.
[taskcluster 2022-11-24 13:10:08.396Z] Downloading artifact "public/image.tar" from task ID: m1bD4ii1QzCLupCMv7zx_g.
[taskcluster 2022-11-24 13:10:12.235Z] Downloaded artifact successfully.
[taskcluster 2022-11-24 13:10:12.235Z] Downloaded 264.909 mb
[taskcluster 2022-11-24 13:10:12.235Z] Loading docker image from downloaded archive.
[taskcluster 2022-11-24 13:10:12.337Z] Downloading artifact "public/image.tar" from task ID: m1bD4ii1QzCLupCMv7zx_g.
[taskcluster 2022-11-24 13:10:15.053Z] Downloaded artifact successfully.
[taskcluster 2022-11-24 13:10:15.053Z] Downloaded 264.909 mb
[taskcluster 2022-11-24 13:10:15.054Z] Loading docker image from downloaded archive.

[taskcluster:error] Pulling docker image has failed.

[taskcluster:error] Error: Error loading docker image. Invalid tar header. Maybe the tar is corrupted or it needs to be gunzipped?
[taskcluster 2022-11-24 13:10:15.102Z] Unsuccessful task run with exit code: -1 completed in 26.279 seconds

The Bugbug bot thinks this bug is a defect, but please change it back in case of error.

Type: task → defect

When we moved tasks to run on GCP from AWS in bug 1799315, we started
using a newer version of docker-worker including the changes from bug
1637302; as a result, artifacts are compressed with gzip before upload
to s3, and downloads now come with a "content-encoding: gzip" header and
compressed content, regardless of the client's "accept-encoding".
Unfortunately docker-worker doesn't handle that encoding and expects an
artifact called image.tar to be uncompressed. To work around that
issue, we now compress docker images in image_builder with zstd before
upload.

[Ideally we'd install the zstd package in the nssdev/image_builder
docker image itself instead of doing it in every task, however I'm not
sure who owns that or how it's built so this might be good enough for
right now.]

Assignee: nobody → jcristau
Status: NEW → ASSIGNED
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: