1400157 - Source builder: chmod: cannot access `/home/worker/workspace/build/src/testing/taskcluster/scripts/builder/build-linux.sh': No such file or directory

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, enhancement)

Description

[Johan Lorenzo [:jlorenzo]]

Example: https://tools.taskcluster.net/groups/Zi-rCIvSRUm7t-CFbAaUYw/tasks/Pxzm3a3kTfS5pBRW5wmCbA/runs/5/logs/public%2Flogs%2Flive_backing.log

> + script=/home/worker/workspace/build/src/testing/taskcluster/scripts/builder/build-linux.sh
> chmod +x $script
> + chmod +x /home/worker/workspace/build/src/testing/taskcluster/scripts/builder/build-linux.sh
> chmod: cannot access `/home/worker/workspace/build/src/testing/taskcluster/scripts/builder/build-linux.sh': No such file or directory

Comment 1

[Johan Lorenzo [:jlorenzo]]

Attachment: releasetasks PR

Comment 3

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1400157 - Temporarily disable source on the 57 branch

Review commit: https://reviewboard.mozilla.org/r/180194/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/180194/

Comment 4

[Johan Lorenzo [:jlorenzo]]

A new issue came up, on the task I created manually[1] (with the path for build-linux):

> /home/worker/workspace/build/src/taskcluster/scripts/builder/build-linux.sh: line 7: /builds/worker/scripts/xvfb.sh: No such file or directory

I published a new version of [2]: sha256:5cfe1601e804cf561f31ec14df8abd1a825749aaf26581a93a659711a3d30b5d. However, I doubt this is the right long-term solution. I see at [3], that docker image was just temporary. Moreover, I haven't found a Dockerfile for it. Then, the new version of the image was made by manually changing the docker image and commit it. 

I'd be in favor to use an in-tree docker image again. What do you think, Rail? 

In the meantime "truncate()" is troublesome on releasetasks. I'll go with Nick's suggestion to turn off source on 57.

There was another hack Nick suggested: force complete the source checksum tc task. It's not feasible, because that particular task is unscheduled due to the source failure. Then we'd have to force complete every source task. I didn't managed to do so because a task needs to be pending if we want to complete it. If it runs, then TC is too smart and refuses the request, because another worker is working on the task. I tried to quickly cancel/rerun/complete a task, but our signing worker are always too fast to pick up new tasks.

Then, I think build 2 is our best chance.

[1] https://tools.taskcluster.net/groups/Zi-rCIvSRUm7t-CFbAaUYw/tasks/HBCqP5gJQa-OD7tT6GJUjw/runs/0/logs/public%2Flogs%2Flive_backing.log
[2] https://hub.docker.com/r/mozillareleases/source-builder/
[3] https://github.com/mozilla-releng/releasetasks/pull/202/commits

Comment 5

[Simon Fraser [:sfraser] ⌚️GMT]

Comment on attachment 8908534 [details]
Bug 1400157 - Temporarily disable source on the 57 branch

https://reviewboard.mozilla.org/r/180194/#review185354

Comment 6

[Johan Lorenzo [:jlorenzo]]

I forgot to set the NI in comment 4.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8908492 [details] [review]
releasetasks PR

If we had some way to figure out the current image we use for building linux, why not use that to build the source too ? Not sure why we maintain a separate image which is vulnerable to getting out of sync. No doubt everything is easier with in-tree scheduling.

Comment 8

[Rail Aliiev [:rail]]

IIRC when we started using releasetasks, CI was using pre-built linux build images. We picked one of them and used them. Then CI became smarter and started using periodically built images (like in https://tools.taskcluster.net/groups/IFTjJi6QRoewnFziWW14Rw/tasks/Qk7VIJxjQEyELnyj1vI-AA/details). Maybe we should add the source task to the nightly graph and just fetch the generated tarball.

Comment 9

[Johan Lorenzo [:jlorenzo]]

Thanks for the idea. I'll give it a try.

Comment 10

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1400157 - Generate source in-tree

Review commit: https://reviewboard.mozilla.org/r/180884/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/180884/

Comment 11

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8909323 [details]
Bug 1400157 - Generate source in-tree

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/180884/diff/1-2/

Comment 12

[Johan Lorenzo [:jlorenzo]]

I tested the in-tree task by manually submitting it at https://tools.taskcluster.net/groups/D9O7hsF8S5OSL_qLwms-2Q/tasks/D9O7hsF8S5OSL_qLwms-2Q.

Comment 13

[Rail Aliiev [:rail]]

Comment on attachment 8909323 [details]
Bug 1400157 - Generate source in-tree

https://reviewboard.mozilla.org/r/180884/#review186020

In overal this looks good, but we need to solve 2 other things before we can land this:

1) We need to sign `public/build/*.source.checksums`. This will probably require another task added.
2) We need to integrate this into releasetasks. Release runner should resolve the task ID by its route, pass it to releasetasks, and releasetasks should beetmove the files.

I'm not sure if it's worth to invest more time into this integration with releasetasks, since they are going away once we switch in in-tree scheduling.

Comment 14

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8908492 [details] [review]
releasetasks PR

Agreed. Let's keep the fix in releasetasks alone while we get tasks in-tree.

I found a way to not use "truncate" anymore. The tests are now passing. What do you think?

I'll back out the tool patch, if we're good.

Comment 15

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8908492 [details] [review]
releasetasks PR

More tweaks are needed, actually. The image has some discrepancies like: https://public-artifacts.taskcluster.net/fh7wmVqpRQK-6bsAbTpNPw/0/public/logs/live_backing.log

Comment 16

[Rail Aliiev [:rail]]

(In reply to Johan Lorenzo [:jlorenzo] from comment #14)
> Comment on attachment 8908492 [details] [review]
> releasetasks PR
> 
> Agreed. Let's keep the fix in releasetasks alone while we get tasks in-tree.
> 
> I found a way to not use "truncate" anymore. The tests are now passing. What
> do you think?

I think this is an acceptable solution.

Comment 17

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8908492 [details] [review]
releasetasks PR

All required changes in both the task def and the docker image were manually tested in https://tools.taskcluster.net/groups/QLqur6CgTsWIaybylIDU4w/tasks/QLqur6CgTsWIaybylIDU4w/runs/0/logs/public%2Flogs%2Flive.log

Comment 18

[Johan Lorenzo [:jlorenzo]]

Tasks templates landed at [1]. I backed out the temporary workaround (attachment 8908534 [details]) at [2]. Both have been deployed to bm83 and bm85. 

[1] https://github.com/mozilla-releng/releasetasks/commit/89ab91371a9a25956d04ec341909b39969f2b2ff
[2] https://hg.mozilla.org/build/tools/rev/d069f89487226dbc22631edfb5b2f8127a36f83b