Closed Bug 1543932 Opened 8 months ago Closed 6 months ago

partner repacks bustage in 68.0


(Release Engineering :: Release Automation: Other, defect)

Not set


(geckoview66 unaffected, firefox-esr60 unaffected, firefox67 unaffected, firefox67.0.1 unaffected, firefox68 fixed, firefox69 fixed)

Tracking Status
geckoview66 --- unaffected
firefox-esr60 --- unaffected
firefox67 --- unaffected
firefox67.0.1 --- unaffected
firefox68 --- fixed
firefox69 --- fixed


(Reporter: nthomas, Assigned: nthomas)




(1 file) (bug 1532747) modified the python used to call mozharness, to the one in the virtualenv in the gecko source. The clobber action of then deletes all that source. Mozharness doesn't care for itself but we get a failure when it tries to use the same python to call

eg log:

[task 2019-04-09T06:14:39.299Z] 06:14:39     INFO - [mozharness: 2019-04-09 06:14:39.299903Z] Running clobber step.
[task 2019-04-09T06:14:39.299Z] 06:14:39     INFO - Running main action method: clobber
[task 2019-04-09T06:14:39.300Z] 06:14:39     INFO - rmtree: /builds/worker/workspace/build
[task 2019-04-09T06:14:39.300Z] 06:14:39     INFO - retry: Calling rmtree with args: ('/builds/worker/workspace/build',), kwargs: {}, attempt #1
[task 2019-04-09T06:14:43.179Z] 06:14:43     INFO - [mozharness: 2019-04-09 06:14:43.179221Z] Finished clobber step (success)
[task 2019-04-09T06:14:51.120Z] 06:14:51     INFO - [mozharness: 2019-04-09 06:14:51.120472Z] Running repack step.
[task 2019-04-09T06:14:51.120Z] 06:14:51     INFO - Running main action method: repack
[task 2019-04-09T06:14:51.120Z] 06:14:51     INFO - Running command: ['/builds/worker/workspace/build/src/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python', '', '-v', '68.0b8', '-n', '1', '--platform', 'macosx64', '--taskid', 'PcyZJLIUQ4ups3kqzyC2WA', '--taskid', 'HdatpxDYQvCknJm6s-O-ew', '--limit-locale', 'ja-JP-mac', '--limit-locale', 'en-US', '--limit-locale', 'ja'] in /builds/worker/workspace/build/scripts
[task 2019-04-09T06:14:51.120Z] 06:14:51     INFO - Copy/paste: /builds/worker/workspace/build/src/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python -v 68.0b8 -n 1 --platform macosx64 --taskid PcyZJLIUQ4ups3kqzyC2WA --taskid HdatpxDYQvCknJm6s-O-ew --limit-locale ja-JP-mac --limit-locale en-US --limit-locale ja
[task 2019-04-09T06:14:51.122Z] 06:14:51    FATAL - caught OS error 2: No such file or directory while running ['/builds/worker/workspace/build/src/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python', '', '-v', '68.0b8', '-n', '1', '--platform', 'macosx64', '--taskid', 'PcyZJLIUQ4ups3kqzyC2WA', '--taskid', 'HdatpxDYQvCknJm6s-O-ew', '--limit-locale', 'ja-JP-mac', '--limit-locale', 'en-US', '--limit-locale', 'ja']

I've verified that it's the python that is missing rather than

We're using taskcluster/scripts/builder/ as a tc-friendly way to run mozharness, but it leaves us with this problem as well as re-using the workspace cache for repackage jobs.

Is this as simple as not clobbering? Taskcluster docker-worker doesn't have leftover cruft in-tree, so we can probably stop clobbering.

I'm not sure yet. On a beta EME task (YkBO2xynRZO6uTEw8mBnAw) I see a scope
providing a cache at /builds/worker/workspace, which makes me think we might be sharing with genuine repackage jobs. If so, clobbering is unhelpful to those, and a cleanup action at the end of the partner repacks would be better.

Do you know where I can read about docker-worker and how working directories are treated ?

It does look like that is using the same cache as cross builds. However, run-task always clobbers the checkout.

Wow, that's from calling robustcheckout with --purge ? Doesn't that mean that every compile is a clobber regardless of the other mechanisms we use for that ?

Yes, though sccache mitigates that to a reasonable degree, I think.

[Tracking Requested - why for this release]:
We'll have partner repack bustage in 68.0b8 without this, due to infra changes since 67.

Pushed by
partner repacks bustage in 68.0, r=tomprince
Closed: 6 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.