444957 - Separate dependent/clobber build from nightly build in tinderbox code to have a Dependent and a Nightly schedulers in automation 1.9

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P3)

Description

[Armen [:armenzg]]

Attachment: Bootstrap/Build.pm, build-seamonkey-util.pl - We can call build-seamonkey with extraArguments to generate different behaviours

Currently, we use the tinderbox code to do 3 things at the same time:
 a) Dependent builds
 b) Scheduled nightly builds
 c) Nightly builds if a developer touches the CLOBBER file associated to a specific platform

##########

The separation I am requesting will allow us to run scheduled nightly builds completely separated from dependent/clobber builds

dependent_scheduler will generate:
 a) Dependent builds which will go to tinderbox builds in ftp
 c) Clobber builds which are exactly as nightly builds but they happen only if a developer touches the CLOBBER file of specified platform

nightly_scheduler will generate:
 b) Scheduled nightly builds

##########

Why I am requesting this? Because this will allow us to have a Nightly scheduler in buildbot and therefore be able to trigger L10n repackages afterward

Why not use another tinder-config.pl? Because we would have to branch it again per platform, while in this way we only add *ONE* new bootstrap configuration file under release/ per branch

##########

I have to figure out how to report with the different column header if the machine has generated a dependent/clobber build or a nightly build. If not every night we would have a dependent and a nightly happening at the same time in a tinderbox page

You can see some of the builds that I have run in the fx-linux-slave1 in here:
http://tinderbox.mozilla.org/showbuilds.cgi?tree=Firefox-Staging
with some comments

Comment 1

[Armen [:armenzg]]

I have found how the naming of the header is created:

build-seamonkey-util.pl:

sub ParseArgs
185         $args->{BuildDepend} = 0, next if $arg eq '--clobber';
186         $args->{BuildDepend} = 1, next if $arg eq '--depend';

sub GetSystemInfo
312     my $build_type = $Settings::BuildDepend ? 'Depend' : 'Clobber';
362     $Settings::BuildName = "$Settings::OS ${os_ver} $host $build_type";
416     $Settings::BuildName .= " $Settings::BuildNameExtra";

#########
BuildNameExtra comes from the tinder-config.pl (http://mxr.mozilla.org/mozilla/source/tools/tinderbox-configs/firefox/linux/tinder-config.pl#161)
BuildDepend comes from the way we call build-seamonkey.pl and we always call it with --depend from bootstrap

I wonder what is the best way to go about this fix...

Comment 2

[[e-mail] (:bhearsum)]

Comment on attachment 329261 [details] [diff] [review]
Bootstrap/Build.pm, build-seamonkey-util.pl - We can call build-seamonkey with extraArguments to generate different behaviours

>Index: tools/release/Bootstrap/Step/Build.pm
>===================================================================
>RCS file: /cvsroot/mozilla/tools/release/Bootstrap/Step/Build.pm,v
>retrieving revision 1.29
>diff -u -8 -p -r1.29 Build.pm
>--- tools/release/Bootstrap/Step/Build.pm	28 Apr 2008 13:48:22 -0000	1.29
>+++ tools/release/Bootstrap/Step/Build.pm	13 Jul 2008 01:07:13 -0000
>@@ -43,19 +43,25 @@ sub Execute {
>     if ($sysname =~ /cygwin/i) {
>         $this->Shell(
>           cmd => 'mount',
>           cmdArgs => ['-b', '-sc', '/cygdrive'],
>           dir => $buildDir,
>         );
>     }
>   
>+    my $extraBuildSeamonkeyArguments = '';
>+    if ($config->Exists('var' => 'extraBuildSeamonkeyArguments')) {
>+       $extraBuildSeamonkeyArguments = $config->Get('var' => 'extraBuildSeamonkeyArguments');
>+    } 
>+ 

I'm not sure how I feel about this. On one hand I like the idea of not having to patch Bootstrap if we need to pass different arguments to Tinderbox. On the other hand, I don't think we need another variable. We already special case 'version' for 'nightly' builds. We could do the same for dep builds when we separate nightly/dep. Something like..

my $extraArg = '';
if ($version eq 'nightly') {
  $extraArg = '--only-nightly';
}
else {
...
}

I don't have _really_ strong feelings about this, so if others think a new argument would be better I'm OK with that, as long as we rename it to something more sensible.

>     $this->Shell(
>       cmd => './build-seamonkey.pl',
>       cmdArgs => ['--once', '--mozconfig', 'mozconfig', '--depend', 
>+                  $extraBuildSeamonkeyArguments,
>                   '--config-cvsup-dir', 
>                   catfile($buildDir, 'tinderbox-configs')],
>       dir => $buildDir,
>       logFile => $buildLog,
>       timeout => 36000
>     );
> 
>     if ($version eq 'nightly') {
>Index: tools/tinderbox/build-seamonkey-util.pl
>===================================================================
>RCS file: /cvsroot/mozilla/tools/tinderbox/build-seamonkey-util.pl,v
>retrieving revision 1.393
>diff -u -8 -p -r1.393 build-seamonkey-util.pl
>--- tools/tinderbox/build-seamonkey-util.pl	3 Jul 2008 12:19:01 -0000	1.393
>+++ tools/tinderbox/build-seamonkey-util.pl	13 Jul 2008 01:07:13 -0000
>@@ -179,16 +179,18 @@ END_USAGE
> sub ParseArgs {
>     PrintUsage() if $#ARGV == -1;
> 
>     my $args = {};
>     my $arg;
>     while ($arg = shift @ARGV) {
>         $args->{BuildDepend} = 0, next if $arg eq '--clobber';
>         $args->{BuildDepend} = 1, next if $arg eq '--depend';
>+        $args->{BuildOnly} = 'nightly', next if $arg eq '--onlyNightly';
>+        $args->{BuildOnly} = 'dependent', next if $arg eq '--onlyDependent';

What's the difference between --depend and --onlyDependent? If --onlyDependent *never* clobbers, that's bad, and we should just use --depend. Same thing for --clobber and --onlyNightly.

If we *need* to have these new arguments please follow the existing format. Eg, --only-nightly instead of --onlyNightly.

>Index: tools/tinderbox/post-mozilla-rel.pl
>===================================================================
>+  elsif ($Settings::BuildOnly eq 'nightly') {
>+    TinderUtils::print_log("Configured to release nightly builds only\n");
>+    $cachebuild = 1;
>+  }
>+  elsif ($Settings::BuildOnly eq 'dependent' && $Setting::ForceRebuild) {
>+    TinderUtils::print_log("Configured to release dependent builds".
>+            "but clobber requested. Therefore a nightly will be generated\n");
>+    $cachebuild = 1;
>+  }
>+  elsif ($Settings::BuildOnly eq 'dependent') {
>+    TinderUtils::print_log("Configured to release dependent builds only\n");
>+    $cachebuild = 0;
>+  }

It looks like this code won't change any existing Tinderbox behaviour, that's good. Like we talked about on the phone though, it's *important* for dependent builds to clobber every night, even if they don't upload their builds. Does this code break that?


r-'ing specifically because of the Tinderbox code, and maybe the Bootstrap stuff, depending what Nick & Coop think.

Comment 3

[Armen [:armenzg]]

We can't even use this until we migrate en-US under buildbot automation (bug 421411)

Will resume a little later

Comment 4

[Chris Cooper [:coop] (he/him)]

(In reply to comment #2)
> I'm not sure how I feel about this. On one hand I like the idea of not having
> to patch Bootstrap if we need to pass different arguments to Tinderbox. On the
> other hand, I don't think we need another variable. We already special case
> 'version' for 'nightly' builds. We could do the same for dep builds when we
> separate nightly/dep. Something like..

How often are we going to change how we call/trigger these builds through tinderbox or Bootstrap? I'd rather have specific variables for specific functionality to make the code easier to understand and to make it easier to reproduce problems should they occur. One could pass anything through the extraArguments and I'd hate for it to be opaque. 

Building the extraArgs inline as Ben demonstrated would be acceptable though.

> What's the difference between --depend and --onlyDependent? If --onlyDependent
> *never* clobbers, that's bad, and we should just use --depend. Same thing for
> --clobber and --onlyNightly.
> [deletia]
> It looks like this code won't change any existing Tinderbox behaviour, that's
> good. Like we talked about on the phone though, it's *important* for dependent
> builds to clobber every night, even if they don't upload their builds. Does
> this code break that?

It's still unclear to me how this will work in practice. Do we maintain two separate source trees on each box, one for dependent builds and one for nightlies? That seems like a waste to me, unless we will also have >1 slave on each box.

Comment 5

[Armen [:armenzg]]

It has been a while since I worked on this, but I believe that instead of the whole big changed, we could keep on doing dependent builds the same way we have been doing but turning off the upload of the nightly builds (we should clean the objdir and source once a day)
In the other side we could do nightly en-US builds without Bootstrap but I believe this idea might not work since will nightly builds would be being generated differently from releases (even though we would be calling the same things that build-seamonkey would call)

Comment 6

[Armen [:armenzg]]

If bug 421411 won't be fixed, then this bug will not fix either
RESOLVING WONDTFIX

Comment 7

[Chris Cooper [:coop] (he/him)]

Moving closed Future bugs into Release Engineering in preparation for removing the Future component.