For scratchbox we build outside of the standard build tree... I set self.baseWorkDir for that. It would be cool if the clobberer were able to work with that so the Maemo builds+repacks can take advantage of it.
I have looked at this and it looks like what needs to happen is to set the workdir of the MozillaClobberer step in the MozillaBuild addInitialSteps method.
jhford and I talked about this on the whiteboard. There seem to be two approaches to fix scratchbox clobbers: 1) create a self.clobberDir. This would be set to builddir/objdir for srcdir builds; builddir/build for standard builds; /scratchbox/users/cltbld/home/cltbld/build/branch for scratchbox builds. The clobberer would then clobber self.clobberDir. This is fairly straightforward to implement, and the complexities involving scratchbox stay pretty much the same. 2) bind mount /builds/slave to /scratchbox/users/cltbld/home/cltbld/slave (or elsewhere within scratchbox). Then we can do all host-based operations (mercurial etc) in the builddir as a normal build; the build, repack, and package operations would happen in /scratchbox/users/cltbld/path-to-slave/builddir/... The downside of this is the complexity of setting up and verifying that this works properly. The upside is we end up with scratchbox builds that behave mostly like the other builds, differing only in build/repack/package steps.
Created attachment 402466 [details] [diff] [review] [WIP] patch This is a partial implementation of this patch. It does not work on the clobberer steps as it changes the workdir of the Clobber step but does not have access to the tools repository clone. The easiest way to implement this is still Option 1 by teaching clobberer.py how to clobber a directory that is not the pwd of the script's invocation.
Created attachment 402530 [details] proof of concept -- bind mount This is a simple, non-buildbot, implementation of option 2, which is what I am leaning towards doing. This is what the bind mount would let us do. In the appropriate factories, we would only need to override build, repack and packaging steps instead of overriding almost every step to set a custom workdir.
Created attachment 402539 [details] proof of concept 2 -- bind mount I don't know where my mind is tonight, but the above comment shouldn't say implementation, rather it is a proof of concept of a build using the bind mount. This is an example of the way we would divide scratchbox and non-scratchbox paths. The new script is because I forgot to check out mobile-browser in the first one. Also of note is that the mount can only be done as root and scratchbox can not be done as root. If you run just the first line as root, it should work fine if you run the whole thing as a user
Created attachment 406056 [details] [diff] [review] clobberer-workdir This patch adds a -d/--dir option to the clobberer script that will allow us to specify arbitrary locations on the filesystem to act as the top level directory for the clobber. I have another version of the patch that allows for multiple -d/--dir options, each being clobbered but I don't know if we want this feature.
Created attachment 406057 [details] [diff] [review] specify mutliple clobber directories on command line This is an alternate version of my last patch which allows you to specify more than one directory to clobber.
Comment on attachment 406056 [details] [diff] [review] clobberer-workdir This looks good on its own...How are you thinking of using it?
For the maemo builders we currently clobber the slave directory instead of the place where the actual build and checkout occur. My plan is to have the maemo builders override the default clobber step and use the -d parameter to specify the build directory for the maemo builds.
Comment on attachment 406056 [details] [diff] [review] clobberer-workdir Is this ok to check in?
Verified to work in staging (http://staging-master.build:8011/builders/Linux%20electrolysis%20nightly/builds/23) Using this version of the patch, we can specify in the maemo build factories that the clobber will be deleting directories that are under the -d parameter if there is not enough free space.
Comment on attachment 406056 [details] [diff] [review] clobberer-workdir http://hg.mozilla.org/build/tools/rev/871b27c215b1
Biting us in the butt =( Currently clobbering all pm02 linux slaves' /scratchboxes manually.
We're not going to need this w/ jhford's new bind mount.