439778 - Use a Nightly scheduler to generate firefox 1.9 nightlies

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P3)

Description

[Armen [:armenzg]]

We generate nightly builds that use different source code.
Using a Nightly scheduler and using this https://bugzilla.mozilla.org/attachment.cgi?id=325321 with Buildproperties should allow us to get nightly builds with the same source stamp.

For instance, last night in production1.8
linux = 10:01 +0000
mac   = 10:25 +0000
win32 = 11:03 +0000

A quick solution might be to have another bootstrap config for dependent builds (this way we don't have them uploading to ftp the "nightly builds" they generate) and for the nightly have a bootstrap file that will put its builds to ftp.

This way we get nightly synchronized builds right away generated from the common pool of slaves and we won't have to worry about what dep_scheduler does with its 24/7 "dedicated slave" even if it generates "nightly builds" that will go to nowhere BUT we won't mind that dependent builds start once a day from a clean build (which won't go to ftp as I mentioned before)

Having a Nightly scheduler will also help me generate "real" l10n repackages with the same checkout time as the en-US nightly build


##################

http://production-1.8-master.build.mozilla.org:8810/builders/linux_dep_build/builds/11755/steps/shell_10/logs/stdio
|Most recent nightly build: Sun Jun 15 03:04:16 2008
|Most recent build hour:    Mon Jun 16 03:00:00 2008
|Starting nightly release build
|MOZ_CO_DATE=06/16/2008 10:01 +0000

http://production-1.8-master.build.mozilla.org:8810/builders/win32_dep_build/builds/5033/steps/shell_10/logs/stdio
|Most recent nightly build: Sun Jun 15 03:18:51 2008
|Most recent build hour:    Mon Jun 16 03:00:00 2008
|Starting nightly release build
|MOZ_CO_DATE=06/16/2008 11:03 +0000

http://production-1.8-master.build.mozilla.org:8810/builders/macosx_dep_build/builds/4197/steps/shell_10/logs/stdio
|Most recent nightly build: Sun Jun 15 03:00:34 2008
|Most recent build hour:    Mon Jun 16 03:00:00 2008
|Starting nightly release build
|MOZ_CO_DATE=06/16/2008 10:25 +0000

Comment 1

[[e-mail] (:bhearsum)]

Note that we don't have appropriate Buildbot code in our tree to do this. It requires scheduler properties, which were landed post-0.7.7.

Comment 2

[Axel Hecht [:Pike]]

I think it'd be easier with build properties, but I guess that if we encoded "do the 4 am nightly" in the reason, we could make a buildstep set the MOZ_CO_DATE to 4am, and then the builder would use that to pull the 4 am source, independent of when the actual build is run, even. Which should be this way, as we can schedule the build at 4am, but that doesn't guarantee that it's immediately run.

Thinking on how to port this forward to moz2, do we need a pushlog enhancement to give us the mercurial changeset for a timestamp?

Comment 3

[[e-mail] (:bhearsum)]

I'm not saying we shouldn't do it with scheduler properties - in fact, I think that *is* the proper way. I'm just pointing out that we need to either a) pre-land the schedule properties patches (which I was looking at doing, but had trouble figuring out exactly which changesets were necessary), b) merge all of the post-0.7.7 changes (which I really don't want to do because I'm not entirely confident of stability), or c) wait until 0.7.8 is released & merged to our tree.

Comment 4

[Armen [:armenzg]]

(In reply to comment #1)
> Note that we don't have appropriate Buildbot code in our tree to do this. It
> requires scheduler properties, which were landed post-0.7.7.
> 
I have been able to use build properties on my local machine and I am using buildbot from our repository (I believe)
I have use this script to set up my machine https://bugzilla.mozilla.org/attachment.cgi?id=323439&action=edit
which buildbot shows this: /tools/buildbot/bin/buildbot
which python shows this: /tools/python/bin/python
echo $PYTHONPATH shows: /tools/buildbotcustom:/tools/buildbot/lib/python2.5/site-packages:/tools/twisted/lib/python2.5/site-packages:/tools/twisted-core/lib/python2.5/site-packages:/tools/zope-interface/lib/python2.5/site-packages/
I will try to delete my buildbot and reinstall it manually and see how it goes with a fresh install

(In reply to comment #2)
> we could make a buildstep set the
> MOZ_CO_DATE to 4am, and then the builder would use that to pull the 4 am
> source, independent of when the actual build is run, even. Which should be this
> way, as we can schedule the build at 4am, but that doesn't guarantee that it's
> immediately run.
>
Yeah, this would work to have synchronized en-US builds and the same to have l10n repacks with the same en-US source stamp

Comment 5

[[e-mail] (:bhearsum)]

(In reply to comment #4)
> (In reply to comment #1)
> > Note that we don't have appropriate Buildbot code in our tree to do this. It
> > requires scheduler properties, which were landed post-0.7.7.
> > 
> I have been able to use build properties on my local machine and I am using
> buildbot from our repository (I believe)

Please double check that. I do not see any properties stuff in http://mxr.mozilla.org/mozilla/source/tools/buildbot/buildbot/scheduler.py#22

Comment 6

[Armen [:armenzg]]

(In reply to comment #5)
> (In reply to comment #4)
> > (In reply to comment #1)
> > > Note that we don't have appropriate Buildbot code in our tree to do this. It
> > > requires scheduler properties, which were landed post-0.7.7.
> > > 
> > I have been able to use build properties on my local machine and I am using
> > buildbot from our repository (I believe)
> 
> Please double check that. I do not see any properties stuff in
> http://mxr.mozilla.org/mozilla/source/tools/buildbot/buildbot/scheduler.py#22
> 
Isn't it this one? http://mxr.mozilla.org/mozilla/source/tools/buildbot/buildbot/status/builder.py#1077

Comment 7

[[e-mail] (:bhearsum)]

From IRC:
I misunderstood what Armen was doing. Turns out he doesn't need Scheduler Properties.

Comment 8

[Armen [:armenzg]]

Attachment: staging/master.cfg - it adds Nightly scheduler that runs e

We would not be able to set the MOZ_CO_DATE variable until we have this r-plused patch landed https://bugzilla.mozilla.org/attachment.cgi?id=325329&action=edit, but I would like to see it working in staging

I still have to add and modify the file configs/fx-moz18-dependent-staging-bootstrap.cfg

Comment 9

[Armen [:armenzg]]

Attachment: release/configs/fx-moz18-dependent-staging-bootstrap.cfg - for dependent builds

Needed for the previous patch - It is a copy of nightly-staging and changing some folder to build on.

Any way on changing where the nightly builds of dependent builder puts its nightly builds?? It wouldn't be nice that it would overwrite the nightly builds by the nightly builders

Comment 10

[Armen [:armenzg]]

Folders to create in slaves:

Linux & Mac
- /builds/tinderbox/Fx-Mozilla1.8-Dependent
- /builds/tinderbox/Fx-Mozilla1.8-l10n-Dependent

Windows
- /cygdrive/c/builds/tinderbox/Fx-Mozilla1.8-Dependent
- /cygdrive/c/builds/tinderbox/Fx-Mozilla1.8-l10n-Dependent

All 3 platforms
- /builds/logs.dependent

NOTE: We might never need anymore the l10n folders since at some point my work on Bug 434878 might probably leads us on not having to set them up

Comment 11

[[e-mail] (:bhearsum)]

To be clear, the goal of this bug is to find a way to distinguish between dep builds/regular clobbers, and our official "nightlies".

Comment 12

[[e-mail] (:bhearsum)]

To expand on comment#11...the reason we want to be able to distinguish is so we can eventually trigger l10n nightlies w/ an en-US one.
Here are the action items Armen and I came up with:
* find a way to have the "dep" builders do only non-nightlies (but not necessarily only dependent builds)
* find a way to have the "nightly" builder do only nightlies

Updating the summary.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

I'm sure if tinderbox supports doing point 1. IIRC you need to make sure that cachebuild is not 1, maybe setting $OfficialBuildMachinery=0 in the tinder-config.pl lets you do that. We don't use that normally so it'd need some investigation.

The second point is easy I think, just call schedule a builder every 24 hours and trigger it after the $build-hour in the tinder-config.pl.

What's going to happen in the hourly case ? The same alignment of timestamps for en-US and l10n ?

Comment 14

[Axel Hecht [:Pike]]

Nick, I would expect that l10n boxens always try to build l10n tip, against the right en-US source, i.e., non-aligned timestamps.

If we do real l10n nightlies, that's different, then I'd align timestamps. With l10n-merge, I think I'll get something hackerish up for that next week.

Comment 15

[Armen [:armenzg]]

Attachment: misc.py - It allows to set the same checkout time for a build in 3 different ways

This patch was initially started in bug 398954 but I believe it makes more sense to have it here

This attachment allows to use a step in which you can set a given hour and all slave will checkout that timestamp no matter when they start their step

1) You can set different timestamp format's
2) you can pass no parameters and it will give the Build's start time 
3) You can give a whole datetime
4) or just set a time (which is not dangerous if we are going to use it for a Nightly scheduler)

When I say dangerous, I refer that if this class is used in other usages in which builds could be queued and therefore a Build object assigned to a slave before midnight would be "TODAY at specified time" and the next build if assigned after midnight it would be "TOMORROW at specified time"

We might even have the problem, that we schedule a Nightly scheduler to start at 4am but we give a wrong parameter, let's say 5am; the slave would be checking out something in the future (which in CVS might default to latest), but this more of a human error rather than a machine error (I think)

Comment 16

[Armen [:armenzg]]

(In reply to comment #13)
> I'm sure if tinderbox supports doing point 1. IIRC you need to make sure that
> cachebuild is not 1, maybe setting $OfficialBuildMachinery=0 in the
> tinder-config.pl lets you do that. We don't use that normally so it'd need some
> investigation.
> 
To investigate this I would need to make changes on mozilla/tools/tinderbox-configs/firefox/linux mac and windows but which branch do we use for staging? I have not seen a branch for staging which I believe it makes sense since we don't want different behaviors between staging and production

Could I use MOZILLA_1_8_BRANCH_test to test it in staging before moving to MOZILLA_1_8_BRANCH??
|http://bonsai.mozilla.org/cvslog.cgi?file=/mozilla/tools/tinderbox-configs|/firefox/linux/tinder-config.pl&rev=MOZILLA_1_8_BRANCH_test

Comment 17

[Armen [:armenzg]]

(In reply to comment #13)
> I'm sure if tinderbox supports doing point 1. IIRC you need to make sure that
> cachebuild is not 1, maybe setting $OfficialBuildMachinery=0 in the
> tinder-config.pl lets you do that. 
>
As Nick recalls, it should work:
http://mxr.mozilla.org/mozilla/source/tools/tinderbox/post-mozilla-rel.pl#1240 (only match)

Comment 18

[Armen [:armenzg]]

Attachment: tinder-config.pl - MOZILLA_1_8_BRANCH - Disable $OfficialBuildMachinery to stop official nigthly clobbers

This should affect any LINUX machines that are doing dependent builds like in staging-1.8 and production-1.8, which would stop them from generating 1.8 Nightly builds
We could tell the effect of this patch immediately by looking at their logs which would show "Configured to release hourly builds only\n" instead of "Starting nightly release build\n" or "Starting non-release build\n"

I would like to test one night a Nightly release with the Nightly scheduler and turning off the Depedent Scheduler before trying to apply this patch.

Comment 19

[Armen [:armenzg]]

Attachment: master.cfg - Staging and production to have Nightly Scheduler

This patch only makes the "linux" slaves to take part of the Nightly scheduler since the previous patch affects "linux" machines only and it would affect Staging-1.8 and Production-1.8.
Once we see that it works fine for linux, we can then add the patches for windows and mac.

Who would be affected if these two patches goes in? 
Firefox 2 users who are still using nightly builds since if not mistaken they still should be getting a prompt saying that the next "nightly" is available

Comment 20

[Armen [:armenzg]]

Attachment: tinder-config.pl - 1.8 BRANCH, osx & w32 configuration

Once we see that the other patch is working fine for the linux machine, we will then add this patch as well

Comment 21

[Armen [:armenzg]]

>Index: tools/tinderbox-configs/firefox/linux/tinder-config.pl
>===================================================================
>RCS file: /cvsroot/mozilla/tools/tinderbox-configs/firefox/linux/tinder-config.pl,v
>retrieving revision 1.1.12.27
>diff -u -8 -p -r1.1.12.27 tinder-config.pl
>--- tools/tinderbox-configs/firefox/linux/tinder-config.pl	13 May 2008 19:04:40 -0000	1.1.12.27
>+++ tools/tinderbox-configs/firefox/linux/tinder-config.pl	1 Jul 2008 17:48:47 -0000
> $DependToDated = 1; # Push the hourly to <host>-<milestone>/<build_start_time>?
>+$OfficialBuildMachinery = 0; # Allow official clobber nightlies.  When false, $cachebuild in post-mozilla-rel.pl can never be true.
> $build_hour    = "3";

This will stop the depend_scheduler from generating nightly builds (which is what we want) but it will stop the nightly scheduler as well since they share the same bootstrap.cfg which specifies the version to nightly
http://mxr.mozilla.org/mozilla/source/tools/release/configs/fx-moz18-nightly-bootstrap.cfg#1
Having the version to nightly will reach this code: http://mxr.mozilla.org/mozilla/source/tools/release/Bootstrap/Step/TinderConfig.pm#197
which will pull MOZILLA_1_8 and then for release it pulls MOZILLA_1_8_release

There should be another version of tinder-config.pl called MOZILLA_1_8_nightly, which should have $OfficialBuildMachinery = 1; to allow nightly builds

Sounds reasonable?

I will attach patch to add bootstrap configuration for dependent builds and I don't really know how to branch (if that is how you call the process) tinder-config.pl as MOZILLA_1_8_nightly

Comment 22

[Armen [:armenzg]]

From discussion with joduinn, bhearsum and coop I extracted that let's make this work for 1.8 by branching tinder-config.pl to MOZILLA_1_8_nightly

joduinn, could we go this way for 1.8 since maintaining synchronized two configurations files (one for nightly and the other one for dependent) would only be until the decease of 1.8 which should be by December 2008?

For 1.9.0, we will have to generate dependent/nightly builds under the umbrella of our 1.9-master instead of using fx-linux-tbox, bm-xserve08 and fx-win32-tbox and then have the dependent-nightly split for them as well

From fx-linux-tbox's log I can tell that the version of tinder-config.pl is 1.29
which is HEAD
What version do we use for 1.9 releases?

Comment 23

[Armen [:armenzg]]

(In reply to comment #22)
> From fx-linux-tbox's log I can tell that the version of tinder-config.pl is
> 1.29
> which is HEAD
> What version do we use for 1.9 releases?
> 
It seems that we use the branch "release"
cvs -d:pserver:[e-mail]:/cvsroot co -r release mozilla/tools/tinderbox-configs/firefox/linux

This means we could have a branch "nightly"?? "MOZILLA_1_9_0_nightly"?? and keep "HEAD" or a new branch "MOZILLA_1_9_0" for dependent builds?

Comment 24

[Armen [:armenzg]]

Comment on attachment 327634 [details] [diff] [review]
master.cfg - Staging and production to have Nightly Scheduler

Canceling some of the patches. The only reason I have been doing these changes of separating the dependent process from the nightly process is to have a Nightly scheduler from which we can trigger L10n repackages. From the discussion with joduinn, I am going to be give the work done by the tbox slaves in 1.9 dedicated to the slaves under our master1.9 and bring the build upon commit into use on 1.9 staging first and this I can have the Nightly scheduler that we need for l10n

Comment 25

[Armen [:armenzg]]

Comment on attachment 327631 [details] [diff] [review]
tinder-config.pl - MOZILLA_1_8_BRANCH - Disable $OfficialBuildMachinery to stop official nigthly clobbers

Obsoleting so nobody decides to check this in.

coop, no branching needed for this file

Comment 26

[Axel Hecht [:Pike]]

I don't understand comment 24.

Comment 27

[[e-mail] (:bhearsum)]

Comment on attachment 327486 [details] [diff] [review]
misc.py - It allows to set the same checkout time for a build in 3 different ways

>Index: tools/buildbotcustom/steps/misc.py
>===================================================================
>+class SetCvsCoDate(BuildStep):
>+    """
>+    but you can specify an specific
>+    date and time, just the time or even give another format to the build start
>+    time. Check "man date" for formatting information
>+    

How are we going to make use of this? I don't see any situation where we would be passing in a hardcoded date/time from the master.cfg.

Comment 28

[Armen [:armenzg]]

(In reply to comment #26)
> I don't understand comment 24.
> 
On 1.9.0, we build continually dependent/nightly builds, therefore, there is no way for me to know when a nightly build has happened and when to trigger the L10n repackages.
Having the Nightly scheduler as we have in 1.9.1 will give us room to set the same Source Stamp in en-US for all 3 platforms and we can have proper L10n repackages afterward
One way of having Nightly schedulers is to separate the dependent/nightly process into dependent builds in one side and nightly builds in another.
I will be working on migrating firefox 1.9 nightlies to release automation (bug 421411) and have build upon checkin which give us Nightly scheduler. There is some work that was started by rhelmer which will accelerate this move

Comment 29

[Axel Hecht [:Pike]]

'k

Comment 30

[[e-mail] (:bhearsum)]

Comment on attachment 327486 [details] [diff] [review]
misc.py - It allows to set the same checkout time for a build in 3 different ways

OK. I chatted with Armen on the phone about this, here's the summary:

* SetCvsCoDate should _not_ get hardcoded with a certain checkout time in the master.cfg. This would break respinning with a CLOBBER file, ie, we would have to manually update the master.cfg && reconfig to respin. We do not want to lose this.
* We want to avoid splitting the tinder-config/mozconfig into a dep and nightly branch if at all possible. This would eliminate the need for an extra tinderbox tree on the build machines, and again, prevent breaking the CLOBBER files.

Armen is going to investigate using a new bootstrap.cfg parameter to distinguish between a dep and nightly. If, eg, 'generateNightly' is set in the bootstrap.cfg maybe we can pass an option to build-seamonkey.pl to force it to do a nightly rather than a dep build (this would only apply if version=nightly).

The above would *not* let us generate nightlies on different platforms with the same checkout time, but not breaking CLOBBER files outweighs the benefits of that IMHO.

Comment 31

[Armen [:armenzg]]

If bug 421411 won't be fixed, then this bug will not fix either
RESOLVING WONDTFIX

Comment 32

[Chris Cooper [:coop] (he/him)]

Moving closed Future bugs into Release Engineering in preparation for removing the Future component.