Closed Bug 554343 Opened 10 years ago Closed 8 years ago

Release builders should always clobber

Categories

(Release Engineering :: Release Automation: Other, defect, P3)

x86
Linux
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: catlee, Assigned: aki)

References

Details

(Whiteboard: [release-process-improvement][automation][clobberer])

Attachments

(2 files, 1 obsolete file)

If the build directory of a release builder isn't empty on a slave, it should do some initial sanity checking that the build directory contains the proper version.

For everything except l10n repacks, perhaps a simple assertion that the build directory is empty is sufficient.
Blocks: 478420
Priority: -- → P5
Priority: P5 → --
Priority: -- → P3
Whiteboard: [automation]
Assignee: nobody → aki
Now that we chunk up our release l10n jobs, does "clobber all release builders" work?

Or maybe "clobber all non-l10n-repack-release-builders, and check version in create-release-repacks.py" ?
Bug 554752 covers repacks.
Since I haven't heard back re: comment 1, I'll morph this into always clobbering release builders.
Summary: Release builders should check prior build version on slave before proceeding → Release builders should always clobber
Whiteboard: [automation] → [automation][triagefollowup]
Well we don't want to clobber repacks EVERY time, (or really anything we chunk) if we get the same builder with same version.

The setup/teardown cost is real, and is why we don't clobber by default for these.

That said, we can certainly make non-chunked release builders auto-clobber, but chunked probably need to check version.

For SeaMonkey I also always hit the clobberer clobber button for release builders when I do a release, which we can possibly require as part of the process, or teach release_sanity.py to do it for the master its executed on. :-)
I think you've got it backwards. You don't want non-chunked l10n builders to clobber every time, because they need to inherit the checkouts from the previous one, or else we'll have to redo them for every locale. In the chunked world, you have one N chunks, each of which corresponds to 1 build, and builds X locales, so even if you clobber at the start of every single l10n repack chunk, you still build many locales without needing to reclone repositories.
Triage: Marked as triagefollowup since I won't have time in Q4 for this.
(In reply to Aki Sasaki [:aki] from comment #5)
> Triage: Marked as triagefollowup since I won't have time in Q4 for this.

Don't think that's a problem. We have an established process for clobbering by hand before each release, but this will be nice-to-have eventually.

Sounds like we still have some discussion to do about under which circumstances we automatically clobber anyway.
Whiteboard: [automation][triagefollowup] → [release-process-improvement][automation][clobberer]
What if we only auto-clobbered release builders when the tag and/or build number changed since the last clobber?
Mass move of bugs to Release Automation component.
Component: Release Engineering → Release Engineering: Automation (Release Automation)
No longer blocks: hg-automation
There's lots of approaches discussed here. We need to solve it one way or another for rapid betas.
Blocks: daily-beta
This is my current approach:

1) add always_clobber.php from bug 639566 to both production and staging clobberers
2) look in releaseConfig for base_clobber_url first; fall back to branchConfig if needed
3) set base_clobber_url to the appropriate always_clobber.php for desktop betas

We can then set that per release config file as desired, while leaving other branches/products alone.

I'm going to test this in staging.
Attachment #618840 - Attachment is obsolete: true
Attachment #618846 - Flags: review?(rail)
We probably want to wait to land this until we actually want to start always clobbering.

From a staging beta linux build:

Checking clobber URL: http://build.mozilla.org/stage-clobberer/always_clobber.php?master=http%3A%2F%2Fdev-master01.build.scl1.mozilla.com%3A8052%2F&slave=linux-ix-slave05&builddir=rel-m-beta-lnx-bld&branch=mozilla-beta&buildername=release-mozilla-beta-linux_build
rel-m-beta-lnx-bld:Our last clobber date:  2012-04-26 16:25:55
rel-m-beta-lnx-bld:Server clobber date:    2012-04-26 17:30:01
rel-m-beta-lnx-bld:Server is forcing a clobber, initiated by clobber_always
rel-m-beta-lnx-bld:Clobbering...
Skipping tools
Skipping last-clobber
Removing build/
program finished with exit code 0
Attachment #618847 - Flags: review?(rail)
Attachment #618846 - Flags: review?(rail) → review+
Comment on attachment 618847 [details] [diff] [review]
[configs] set staging+release firefox betas to always clobber

Please keep in mind that this clobberer behaves a little bit differently: it has no idea about explicitly set clobberers and removes the current builder's directory only.
Attachment #618847 - Flags: review?(rail) → review+
(In reply to Rail Aliiev [:rail] from comment #13)
> Comment on attachment 618847 [details] [diff] [review]
> [configs] set staging+release firefox betas to always clobber
> 
> Please keep in mind that this clobberer behaves a little bit differently: it
> has no idea about explicitly set clobberers and removes the current
> builder's directory only.

Purge builds will take care of disk space issues, and any subsequent non-release job will clobber those explicitly set clobbers, right?
Why do we want an external dependency on anything if we always want to clobber?  Why not have step one of all release jobs be to rm -rf "*" in the slave build dir?
(In reply to John Ford [:jhford] from comment #15)
> Why do we want an external dependency on anything if we always want to
> clobber?  Why not have step one of all release jobs be to rm -rf "*" in the
> slave build dir?

I'd be happy to do that; it sounds like other products don't want that.
Maybe call a method to add the step, and child classes can override...
Comment on attachment 618847 [details] [diff] [review]
[configs] set staging+release firefox betas to always clobber

http://hg.mozilla.org/build/buildbot-configs/rev/639661929d34

Now all Firefox desktop betas will clobber.
purge_builds should handle the disk space cleanup.

I'm open to either closing this bug as fixed, or keeping it open until we add the above line to the desktop release and mobile {beta,release} config files as well.
Attachment #618847 - Flags: checked-in+
Landed to production today
Looks like this worked in 13.0b2.
I'm going to mark as fixed; it's easy to add this to any release as wanted/needed.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.