Closed Bug 1304333 Opened 3 years ago Closed 2 years ago

50.0b1 source generation task failed because it's using the old gecko hierarchy

Categories

(Release Engineering :: Release Automation: Other, defect, P3)

defect

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: mtabara, Unassigned)

References

Details

Attachments

(1 file, 1 obsolete file)

During 50.0b1 we've got the source generation task[1] failing with:

+ . /home/worker/workspace/build/src/testing/taskcluster/scripts/builder/build-linux.sh
/home/worker/bin/build.sh: line 27: /home/worker/workspace/build/src/testing/taskcluster/scripts/builder/build-linux.sh: No such file or directory 

Turns out we're using the old gecko hierarchy as the builder scripts are now under /taskcluster/scripts in gecko, no longer under /testing/taskcluster .

Rail suggested we should start using images generated by TC[2] instead of the hardcoded "taskcluster/desktop-build:0.1.11".

[1]: https://tools.taskcluster.net/task-inspector/#58xr7SxhSXWi3U4METZs3w/
[2]: https://tools.taskcluster.net/task-inspector/#IGxW7rJdQpOZhTF1oIXMEQ/
This can be fixed at least in 2 ways:

1) we can create a CI task to generate source tarballs and bundles (maybe we should stop generating bundles?). Release runner would need to find this task and set the dependencies (so we don't wait for it).

We just sign the artifacts (the checksums?) and beetmove it properly.

The only downside would be a waist of S3 storage - we'd be generating tarballs and bundles for every push. We can set the expiration to something like a month to reduce the usage.


2) Change the image from obsoleted "desktop-builder" to run-time generated ones. The task itself should refer to the image as

"image": {
    "path": "public/image.tar",
    "type": "task-image",
    "taskId": "IGxW7rJdQpOZhTF1oIXMEQ"
  },

Release runner would need to find the corresponding image task, pass the taskId to the graph and set the dependencies. This won't help with "OMG beta1" issues though.


As a temporary solution I'm going to take one of the images and push it to my docker hub:

$ curl -L -o image.tar https://queue.taskcluster.net/v1/task/IGxW7rJdQpOZhTF1oIXMEQ/artifacts/public/image.tar
$ image=$(tar xf image.tar -O repositories | jq -r 'keys[0]')
$ image_tag=$(tar xf image.tar -O repositories | jq -r '.[keys[0]] | keys[0]')
$ image_name=$image:$image_tag
$ echo $image_name
mozilla-beta:b454ff70f9ca649d74ee3319faee0c0a9d2d9cbea3b026814d7fcf788dc073f5
$ docker tag mozilla-beta:b454ff70f9ca649d74ee3319faee0c0a9d2d9cbea3b026814d7fcf788dc073f5 rail/source-builder
$ docker push rail/source-builder
...
latest: digest: sha256:20ea2d9aff06740d6a35ff2f42af98c8d8b65efec9d0dea48692fbb3025d5d8c size: 6956

IGxW7rJdQpOZhTF1oIXMEQ
Attached file PR: temp image (obsolete) —
Attachment #8793273 - Flags: review?
Attachment #8793273 - Flags: review? → review?(mtabara)
Attachment #8793273 - Flags: review?(mtabara) → review+
Comment on attachment 8793273 [details] [review]
PR: temp image

merged, deployed
Attachment #8793273 - Flags: checked-in+
Status update: temporary fix worked like a charm in follow-up 50.0b1-build2 \o/
Status update after talking about this over the post-mortem:
* CI builds solution falls off the board as it adds kind off redundant logic and resource-consuming stuff

We'll stick to the temporary fix for a while now, pending another decision soon.
Unfortunately the 49.0.1 source builder hit
chmod: cannot access `/home/worker/workspace/build/src/taskcluster/scripts/builder/build-linux.sh': No such file or directory

(https://public-artifacts.taskcluster.net/j9gMug_SRMykjHfTA9WEtQ/0/public/logs/live_backing.log)

So at rail's suggestion I've backed out the temp fix
 https://github.com/mozilla/releasetasks/commit/7661e2e8d44eb0577365659806efc753aa0a237b
which I expect will fix release but break beta again.
build2 had the same problem, but that was my fault because I didn't remember to update bm85:/home/cltbld/releasetasks/ before starting it.
20:10 <rail> I have a band-aid idea... :/
20:10 <rail> test -e path/to/linux-build.sh || ln -s ...
20:10 <rail> in the task definition
Attachment #8793273 - Attachment is obsolete: true
Attachment #8794865 - Flags: review?(mtabara)
Attachment #8794865 - Flags: review?(mtabara) → review+
Priority: -- → P3
We're tackling this differently in in-tree scheduling relpro so we can close this.
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.