Closed Bug 1543932 Opened 8 months ago Closed 6 months ago

partner repacks bustage in 68.0

Categories

(Release Engineering :: Release Automation: Other, defect)

defect
Not set

Tracking

(geckoview66 unaffected, firefox-esr60 unaffected, firefox67 unaffected, firefox67.0.1 unaffected, firefox68 fixed, firefox69 fixed)

RESOLVED FIXED
Tracking Status
geckoview66 --- unaffected
firefox-esr60 --- unaffected
firefox67 --- unaffected
firefox67.0.1 --- unaffected
firefox68 --- fixed
firefox69 --- fixed

People

(Reporter: nthomas, Assigned: nthomas)

References

Details

Attachments

(1 file)

https://hg.mozilla.org/mozilla-central/rev/39e6ec535eb4 (bug 1532747) modified the python used to call mozharness, to the one in the virtualenv in the gecko source. The clobber action of desktop_partner_repacks.py then deletes all that source. Mozharness doesn't care for itself but we get a failure when it tries to use the same python to call tc-parter-repacks.py.

eg log:

[task 2019-04-09T06:14:39.299Z] 06:14:39     INFO - [mozharness: 2019-04-09 06:14:39.299903Z] Running clobber step.
[task 2019-04-09T06:14:39.299Z] 06:14:39     INFO - Running main action method: clobber
[task 2019-04-09T06:14:39.300Z] 06:14:39     INFO - rmtree: /builds/worker/workspace/build
[task 2019-04-09T06:14:39.300Z] 06:14:39     INFO - retry: Calling rmtree with args: ('/builds/worker/workspace/build',), kwargs: {}, attempt #1
[task 2019-04-09T06:14:43.179Z] 06:14:43     INFO - [mozharness: 2019-04-09 06:14:43.179221Z] Finished clobber step (success)
...
[task 2019-04-09T06:14:51.120Z] 06:14:51     INFO - [mozharness: 2019-04-09 06:14:51.120472Z] Running repack step.
[task 2019-04-09T06:14:51.120Z] 06:14:51     INFO - Running main action method: repack
[task 2019-04-09T06:14:51.120Z] 06:14:51     INFO - Running command: ['/builds/worker/workspace/build/src/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python', 'tc-partner-repacks.py', '-v', '68.0b8', '-n', '1', '--platform', 'macosx64', '--taskid', 'PcyZJLIUQ4ups3kqzyC2WA', '--taskid', 'HdatpxDYQvCknJm6s-O-ew', '--limit-locale', 'ja-JP-mac', '--limit-locale', 'en-US', '--limit-locale', 'ja'] in /builds/worker/workspace/build/scripts
[task 2019-04-09T06:14:51.120Z] 06:14:51     INFO - Copy/paste: /builds/worker/workspace/build/src/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python tc-partner-repacks.py -v 68.0b8 -n 1 --platform macosx64 --taskid PcyZJLIUQ4ups3kqzyC2WA --taskid HdatpxDYQvCknJm6s-O-ew --limit-locale ja-JP-mac --limit-locale en-US --limit-locale ja
[task 2019-04-09T06:14:51.122Z] 06:14:51    FATAL - caught OS error 2: No such file or directory while running ['/builds/worker/workspace/build/src/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python', 'tc-partner-repacks.py', '-v', '68.0b8', '-n', '1', '--platform', 'macosx64', '--taskid', 'PcyZJLIUQ4ups3kqzyC2WA', '--taskid', 'HdatpxDYQvCknJm6s-O-ew', '--limit-locale', 'ja-JP-mac', '--limit-locale', 'en-US', '--limit-locale', 'ja']

I've verified that it's the python that is missing rather than tc-partner-repacks.py.

We're using taskcluster/scripts/builder/repackage.sh as a tc-friendly way to run mozharness, but it leaves us with this problem as well as re-using the workspace cache for repackage jobs.

Is this as simple as not clobbering? Taskcluster docker-worker doesn't have leftover cruft in-tree, so we can probably stop clobbering.

I'm not sure yet. On a beta EME task (YkBO2xynRZO6uTEw8mBnAw) I see a scope
docker-worker:cache:gecko-level-3-mozilla-beta-build-macosx64-nightly-opt-workspace-v3-33ea6ead87f10b63cd64
providing a cache at /builds/worker/workspace, which makes me think we might be sharing with genuine repackage jobs. If so, clobbering is unhelpful to those, and a cleanup action at the end of the partner repacks would be better.

Do you know where I can read about docker-worker and how working directories are treated ?

It does look like that is using the same cache as cross builds. However, run-task always clobbers the checkout.

Wow, that's from calling robustcheckout with --purge ? Doesn't that mean that every compile is a clobber regardless of the other mechanisms we use for that ?

Yes, though sccache mitigates that to a reasonable degree, I think.

[Tracking Requested - why for this release]:
We'll have partner repack bustage in 68.0b8 without this, due to infra changes since 67.

Pushed by nthomas@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c4f64fa957d9
partner repacks bustage in 68.0, r=tomprince
Status: NEW → RESOLVED
Closed: 6 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.