If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

update clobberer to take a custom workdir

RESOLVED INCOMPLETE

Status

Release Engineering
General
P3
normal
RESOLVED INCOMPLETE
9 years ago
4 years ago

People

(Reporter: aki, Assigned: aki)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments, 3 obsolete attachments)

(Assignee)

Description

9 years ago
For scratchbox we build outside of the standard build tree... I set self.baseWorkDir for that.  It would be cool if the clobberer were able to work with that so the Maemo builds+repacks can take advantage of it.
(Assignee)

Updated

9 years ago
Assignee: nobody → aki
(Assignee)

Updated

8 years ago
Assignee: aki → jford
I have looked at this and it looks like what needs to happen is to set the workdir of the MozillaClobberer step in the MozillaBuild addInitialSteps method.
Status: NEW → ASSIGNED
Priority: -- → P3
(Assignee)

Updated

8 years ago
Duplicate of this bug: 509652
(Assignee)

Comment 3

8 years ago
jhford and I talked about this on the whiteboard.

There seem to be two approaches to fix scratchbox clobbers:

1) create a self.clobberDir.  This would be set to builddir/objdir for srcdir builds; builddir/build for standard builds; /scratchbox/users/cltbld/home/cltbld/build/branch for scratchbox builds.  The clobberer would then clobber self.clobberDir.

This is fairly straightforward to implement, and the complexities involving scratchbox stay pretty much the same.

2) bind mount /builds/slave to /scratchbox/users/cltbld/home/cltbld/slave (or elsewhere within scratchbox).  Then we can do all host-based operations (mercurial etc) in the builddir as a normal build; the build, repack, and package operations would happen in /scratchbox/users/cltbld/path-to-slave/builddir/...

The downside of this is the complexity of setting up and verifying that this works properly.

The upside is we end up with scratchbox builds that behave mostly like the other builds, differing only in build/repack/package steps.
Created attachment 402466 [details] [diff] [review]
[WIP] patch

This is a partial implementation of this patch.  It does not work on the clobberer steps as it changes the workdir of the Clobber step but does not have access to the tools repository clone.  

The easiest way to implement this is still Option 1 by teaching clobberer.py how to clobber a directory that is not the pwd of the script's invocation.
Created attachment 402530 [details]
proof of concept -- bind mount

This is a simple, non-buildbot, implementation of option 2, which is what I am leaning towards doing.  This is what the bind mount would let us do.  In the appropriate factories, we would only need to override build, repack and packaging steps instead of overriding almost every step to set a custom workdir.
Created attachment 402539 [details]
proof of concept 2 -- bind mount

I don't know where my mind is tonight, but the above comment shouldn't say implementation, rather it is a proof of concept of a build using the bind mount.  This is an example of the way we would divide scratchbox and non-scratchbox paths.  The new script is because I forgot to check out mobile-browser in the first one.  Also of note is that the mount can only be done as root and scratchbox can not be done as root.  If you run just the first line as root, it should work fine if you run the whole thing as a user
Attachment #402530 - Attachment is obsolete: true
Created attachment 406056 [details] [diff] [review]
clobberer-workdir

This patch adds a -d/--dir option to the clobberer script that will allow us to specify arbitrary locations on the filesystem to act as the top level directory for the clobber.  I have another version of the patch that allows for multiple -d/--dir options, each being clobbered but I don't know if we want this feature.
Attachment #402466 - Attachment is obsolete: true
Attachment #402539 - Attachment is obsolete: true
Attachment #406056 - Flags: review?(catlee)
Created attachment 406057 [details] [diff] [review]
specify mutliple clobber directories on command line

This is an alternate version of my last patch which allows you to specify more than one directory to clobber.
Assignee: jford → nobody
Component: Release Engineering: Future → Release Engineering
Assignee: nobody → jford
Comment on attachment 406056 [details] [diff] [review]
clobberer-workdir

This looks good on its own...How are you thinking of using it?
Attachment #406056 - Flags: review?(catlee) → review+
For the maemo builders we currently clobber the slave directory instead of the place where the actual build and checkout occur.  My plan is to have the maemo builders override the default clobber step and use the -d parameter to specify the build directory for the maemo builds.
Comment on attachment 406056 [details] [diff] [review]
clobberer-workdir

Is this ok to check in?
Attachment #406056 - Flags: checked-in?
Verified to work in staging (http://staging-master.build:8011/builders/Linux%20electrolysis%20nightly/builds/23)

Using this version of the patch, we can specify in the maemo build factories that the clobber will be deleting directories that are under the -d parameter if there is not enough free space.
(Assignee)

Updated

8 years ago
Attachment #406056 - Flags: checked-in? → checked-in+
(Assignee)

Comment 13

8 years ago
Comment on attachment 406056 [details] [diff] [review]
clobberer-workdir

http://hg.mozilla.org/build/tools/rev/871b27c215b1
(Assignee)

Updated

8 years ago
Blocks: 522719
(Assignee)

Comment 14

8 years ago
Biting us in the butt =(
Currently clobbering all pm02 linux slaves' /scratchboxes manually.
(Assignee)

Updated

8 years ago
Blocks: 528431
(Assignee)

Updated

8 years ago
Blocks: 537426
(Assignee)

Updated

8 years ago
Assignee: jhford → aki
No longer blocks: 528431
(Assignee)

Comment 15

8 years ago
We're not going to need this w/ jhford's new bind mount.
Status: ASSIGNED → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → INCOMPLETE
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.