Source builder: chmod: cannot access `/home/worker/workspace/build/src/testing/taskcluster/scripts/builder/build-linux.sh': No such file or directory

RESOLVED FIXED

Status

enhancement
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: jlorenzo, Assigned: jlorenzo)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments, 1 obsolete attachment)

Example: https://tools.taskcluster.net/groups/Zi-rCIvSRUm7t-CFbAaUYw/tasks/Pxzm3a3kTfS5pBRW5wmCbA/runs/5/logs/public%2Flogs%2Flive_backing.log

> + script=/home/worker/workspace/build/src/testing/taskcluster/scripts/builder/build-linux.sh
> chmod +x $script
> + chmod +x /home/worker/workspace/build/src/testing/taskcluster/scripts/builder/build-linux.sh
> chmod: cannot access `/home/worker/workspace/build/src/testing/taskcluster/scripts/builder/build-linux.sh': No such file or directory
Posted file releasetasks PR
Attachment #8908492 - Flags: review?(nthomas)
Duplicate of this bug: 1400119
A new issue came up, on the task I created manually[1] (with the path for build-linux):

> /home/worker/workspace/build/src/taskcluster/scripts/builder/build-linux.sh: line 7: /builds/worker/scripts/xvfb.sh: No such file or directory

I published a new version of [2]: sha256:5cfe1601e804cf561f31ec14df8abd1a825749aaf26581a93a659711a3d30b5d. However, I doubt this is the right long-term solution. I see at [3], that docker image was just temporary. Moreover, I haven't found a Dockerfile for it. Then, the new version of the image was made by manually changing the docker image and commit it. 

I'd be in favor to use an in-tree docker image again. What do you think, Rail? 

In the meantime "truncate()" is troublesome on releasetasks. I'll go with Nick's suggestion to turn off source on 57.

There was another hack Nick suggested: force complete the source checksum tc task. It's not feasible, because that particular task is unscheduled due to the source failure. Then we'd have to force complete every source task. I didn't managed to do so because a task needs to be pending if we want to complete it. If it runs, then TC is too smart and refuses the request, because another worker is working on the task. I tried to quickly cancel/rerun/complete a task, but our signing worker are always too fast to pick up new tasks.

Then, I think build 2 is our best chance.

[1] https://tools.taskcluster.net/groups/Zi-rCIvSRUm7t-CFbAaUYw/tasks/HBCqP5gJQa-OD7tT6GJUjw/runs/0/logs/public%2Flogs%2Flive_backing.log
[2] https://hub.docker.com/r/mozillareleases/source-builder/
[3] https://github.com/mozilla-releng/releasetasks/pull/202/commits
Comment on attachment 8908534 [details]
Bug 1400157 - Temporarily disable source on the 57 branch

https://reviewboard.mozilla.org/r/180194/#review185354
Attachment #8908534 - Flags: review?(sfraser) → review+
I forgot to set the NI in comment 4.
Flags: needinfo?(rail)
Depends on: 1304333
Comment on attachment 8908492 [details] [review]
releasetasks PR

If we had some way to figure out the current image we use for building linux, why not use that to build the source too ? Not sure why we maintain a separate image which is vulnerable to getting out of sync. No doubt everything is easier with in-tree scheduling.
Attachment #8908492 - Flags: review?(nthomas)
IIRC when we started using releasetasks, CI was using pre-built linux build images. We picked one of them and used them. Then CI became smarter and started using periodically built images (like in https://tools.taskcluster.net/groups/IFTjJi6QRoewnFziWW14Rw/tasks/Qk7VIJxjQEyELnyj1vI-AA/details). Maybe we should add the source task to the nightly graph and just fetch the generated tarball.
Flags: needinfo?(rail)
Thanks for the idea. I'll give it a try.
Assignee: nobody → jlorenzo
I tested the in-tree task by manually submitting it at https://tools.taskcluster.net/groups/D9O7hsF8S5OSL_qLwms-2Q/tasks/D9O7hsF8S5OSL_qLwms-2Q.
Comment on attachment 8909323 [details]
Bug 1400157 - Generate source in-tree

https://reviewboard.mozilla.org/r/180884/#review186020

In overal this looks good, but we need to solve 2 other things before we can land this:

1) We need to sign `public/build/*.source.checksums`. This will probably require another task added.
2) We need to integrate this into releasetasks. Release runner should resolve the task ID by its route, pass it to releasetasks, and releasetasks should beetmove the files.

I'm not sure if it's worth to invest more time into this integration with releasetasks, since they are going away once we switch in in-tree scheduling.
Attachment #8909323 - Flags: review?(rail)
Comment on attachment 8908492 [details] [review]
releasetasks PR

Agreed. Let's keep the fix in releasetasks alone while we get tasks in-tree.

I found a way to not use "truncate" anymore. The tests are now passing. What do you think?

I'll back out the tool patch, if we're good.
Attachment #8908492 - Flags: review?(rail)
Attachment #8909323 - Attachment is obsolete: true
Comment on attachment 8908492 [details] [review]
releasetasks PR

More tweaks are needed, actually. The image has some discrepancies like: https://public-artifacts.taskcluster.net/fh7wmVqpRQK-6bsAbTpNPw/0/public/logs/live_backing.log
Attachment #8908492 - Flags: review?(rail)
(In reply to Johan Lorenzo [:jlorenzo] from comment #14)
> Comment on attachment 8908492 [details] [review]
> releasetasks PR
> 
> Agreed. Let's keep the fix in releasetasks alone while we get tasks in-tree.
> 
> I found a way to not use "truncate" anymore. The tests are now passing. What
> do you think?

I think this is an acceptable solution.
Comment on attachment 8908492 [details] [review]
releasetasks PR

All required changes in both the task def and the docker image were manually tested in https://tools.taskcluster.net/groups/QLqur6CgTsWIaybylIDU4w/tasks/QLqur6CgTsWIaybylIDU4w/runs/0/logs/public%2Flogs%2Flive.log
Attachment #8908492 - Flags: review?(rail)
Attachment #8908492 - Flags: review?(rail) → review+
Tasks templates landed at [1]. I backed out the temporary workaround (attachment 8908534 [details]) at [2]. Both have been deployed to bm83 and bm85. 

[1] https://github.com/mozilla-releng/releasetasks/commit/89ab91371a9a25956d04ec341909b39969f2b2ff
[2] https://hg.mozilla.org/build/tools/rev/d069f89487226dbc22631edfb5b2f8127a36f83b
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.