404768 - make nightly builds create partial .mar

Status: RESOLVED WONTFIX

(Webtools Graveyard :: Tinderbox, enhancement)

Description

[ause]

Attachment: my idea of teaching tinderbox to create partial .mar

after moving from aus2.mozilla.org to aus2-community.mozilla.org this previous mechanism to create partial .mar for nightly build is no longer available.

this job can be done by the tinderbox clients. the attached patch implements this.

Comment 1

[ause]

found off-by-one error in

my $old_build_id = $cachelist[$#cachelist - 1];

needs to be

my $old_build_id = $cachelist[$#cachelist];

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Rob, do you want to take a look at this patch too ?

Comment 3

[ause]

resulting partials are unusable as the used make_incremental_update.sh is meant to compare dirs with unpacked products not complete .mar files. would have been too easy...

back to workplace. sorry for the noise...

Comment 4

[ause]

Attachment: next iteration of that patch

Comment 5

[ause]

Attachment: updated to trunk; minor fixes for edge cases

Comment 6

[ause]

got hit by the second part of this: https://bugzilla.mozilla.org/show_bug.cgi?id=375415#c11
makes partial mar files on trunk quite pointless...

unfortunately, the python script doesn't even exist on MOZILLA_1_8_BRANCH

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 292060 [details] [diff] [review]
updated to trunk; minor fixes for edge cases

Sorry for the delay in getting to this. There are quite a few comments inline, and before you consider addressing them, please talk to Rob Helmer about his plans for generating partials on tinderboxes.

>Index: post-mozilla-rel.pl
...
> sub packit {
>-  my ($packaging_dir, $package_location, $url, $stagedir, $builddir, $cachebuild) = @_;
>+  my ($packaging_dir, $package_location, $url, $stagedir, $builddir, $cachebuild, $tb_builddir) = @_;

tb_builddir is short for Tinderbox or Thunderbird ? Seems this is really the root dir for the current tinderbox build. Please make the name less ambiguous.

>@@ -572,20 +573,21 @@
> sub update_create_package {
...
>     if ($^O eq "cygwin") {
>       # need to convert paths for use by Cygwin utilities which treat
>       # "drive:/path" interpret drive as a host.
>       $objdir = `cygpath -u $objdir`;

Have you tried this code on a win32 Moz1.8 box ? Probably need to do this cygpath trick on tb_dir too.

>     TinderUtils::run_shell_command("make -C $objdir/tools/update-packaging full-update STAGE_DIR=$up_temp_stagedir DIST=$up_distdir AB_CD=$locale");
> 
>+    my $buildid = get_buildid( dist => $distdir );
>+    TinderUtils::print_log("Got build ID $buildid.\n");
>+
>+    if ( $update_partials == 1 ) {
>+      my $current_build_id = $buildid;
>+      my $current_build_dir = $up_temp_stagedir;
>+      opendir(DIR, $tb_dir) || die "can’t opendir $tb_dir: $!";

You have a UTF char here for ' - please replace with plain ASCII char.

>+      my @cachelist = grep { /^[0-9]{10}$/ } readdir(DIR);
>+      closedir DIR;
>+      if ( scalar(@cachelist) == -1 ) {
>+        $update_partials = 0;
>+      } else {
>+        my @sorted_cachelist = sort @cachelist;
>+        my $old_build_id = $sorted_cachelist[$#sorted_cachelist];
>+        if ( !defined($old_build_id) || "$old_build_id" eq "$current_build_id" ) {
>+          # something nasty has happened...
>+          $update_partials = 0;
>+        } else {
>+          my $old_build_dir = "$tb_dir/$old_build_id/packages";
>+
>+          # build diff tool first
>+          TinderUtils::run_shell_command("make -C $objdir/other-licenses/bsdiff ");

I think it's sufficient to checkout that code with MOZ_CO_MODULE in the mozconfig, and --enable-update-packaging does the rest.

>+          # now unpack the complete .mar
>+          mkdir("$up_temp_stagedir/previous") || return 1;
>+          mkdir("$up_temp_stagedir/current") || return 1;

use run_shell_command here, like other calls to mkdir in this file

>+          $ENV{MAR}="$objdir/dist/host/bin/mar";

What is this for ? The makefile in update-packaging seems to override the environment, and the complete mar is happily made without it.

>+          my $currentdir = getcwd();
>+
>+          chdir("$up_temp_stagedir/previous"); # better cd inside run_shell_command?
>+          TinderUtils::run_shell_command("$Settings::TopsrcdirFull/tools/update-packaging/unwrap_full_update.pl $old_build_dir/*complete*.mar");

Might as well do the cd in run_shell_command too, so that you get log output in case of errors.

Why not *complete.mar ? This is hard-coded in update-packaging/Makefile.in

>+          chdir("$up_temp_stagedir/current"); # better cd inside run_shell_command?
>+          TinderUtils::run_shell_command("$Settings::TopsrcdirFull/tools/update-packaging/unwrap_full_update.pl $current_build_dir/*complete*.mar");

Same again.

>+          chdir("$currentdir");
>+          TinderUtils::run_shell_command("make -C $objdir/tools/update-packaging partial-patch STAGE_DIR=$up_temp_stagedir DIST=$up_distdir AB_CD=$locale SRC_BUILD=$up_temp_stagedir/previous SRC_BUILD_ID=$old_build_id DST_BUILD=$up_temp_stagedir/current DST_BUILD_ID=$current_build_id ");
>+        }
>+      }
>+    }
>+
>+    # all this heavily relies on knowing the naming conventions...
>     my $update_file = "update.mar";
>     my @updatemar;
>-    my $update_glob = "${temp_stagedir}/*.mar";
>+    my $update_glob = "${temp_stagedir}/*complete*.mar";

*complete.mar will be fine.

>+    my $update_part_file = "update.mar";
>+    if ( $update_partials == 1 ) {
>+      my $update_part_glob = "${temp_stagedir}/*partial*.mar";
>+      @updatemar = grep { -f $_ } glob($update_part_glob);
>+      if ( scalar(@updatemar) ge 1 ) {
>+        $update_part_file = $updatemar[0];
>+        $update_part_file =~ s:^$temp_stagedir/(.*)$:$1:g;
>+      } else {
>+        TinderUtils::print_log("No MAR file found matching '$update_part_glob', update generation failed.\n");
>+        $status = 1;
>+        return $status;
>+      }
>+      TinderUtils::run_shell_command("rsync -av $temp_stagedir/$update_part_file $stagedir/");
>+    }

Arrgh, this is truly terrible code to replicate. If you end up doing another patch, then feel free to rewrite this. 

>       # Gather stats for update file.
>       update_create_stats( update => $update_path,
>                            type => "complete",
>                            output_file => "$objdir/dist/update/update.snippet",
>                            url => $update_fullurl,
>                            buildid => $buildid,
>                            appversion => $update_appv,
>                            extversion => $update_extv,
>                          );
> 
>+      if ( $update_partials == 1 ) {
>+        update_create_stats( update => $update_part_path,
>+                             type => "partial",
>+                             output_file => "$objdir/dist/update/update.snippet",
>+                             url => "$update_part_fullurl",
>+                             buildid => $buildid,
>+                             appversion => $update_appv,
>+                             extversion => $update_extv,
>+                           );
>+      }

This is definitely going to go wrong. The only difference between update.snippet.0 and .1 is appversion and extversion are at the end of the .1 file. You need to give them unique names, eg complete.snippet and partial.snippet, and then only push up the .1 files (see below).

>       # Push update information to update-staging/auslite.
> 
>       # Push the build schema 2 data.
>       if ( $Settings::update_pushinfo ) {
>           TinderUtils::print_log("\nPushing third-gen update info...\n");
>           my $path = "/opt/aus2/build/0";
>           $path = "$path/$update_product/$update_version/$update_platform/$buildid/$locale";

If you're wanting to push snippets into their final location, then $path will need to depend on $update_partials. When this code was originally written, tinderbox didn't know the previous buildid, so it does
  * push up to .../build/0/.../<new_buildid>/...
  * other code comes along and grabs all the snippets, figures out the update graph, builds partials, and makes new snippets which go to 
               .../incoming/2/.../<previous_buildid>/

AUS then offers the latest complete to all requests on the nightly channel, and the partial if you are one build behind. It needs the snippets in the right directory to offer the partial correctly - ie to know that <appname>...partial.$old_buildid-$buildid.mar updates $old_build_id to $buildid, the snippet needs to live in .../$old_build_id/...

So you'll need something like

  my $path = "/opt/aus2/";
  if ($update_partials == 1 ) {
     $path = "$path/incoming/2/$update_product/$update_version/$update_platform/$old_buildid/$locale";
  } else {
     $path = "$path/build/0/$update_product/$update_version/$update_platform/$buildid/$locale";
  }

Naturally, you need to be very careful _not_ to push anything into the release area in .../incoming/3/

...
>           TinderUtils::run_shell_command("ssh $ssh_opts $Settings::ssh_user\@$update_aus_host mkdir -p $path");
>           TinderUtils::run_shell_command("scp $scp_opts $objdir/dist/update/update.snippet.1 $Settings::ssh_user\@$update_aus_host:$path/complete.txt");
>+          if ( $update_partials == 1 ) {
>+            TinderUtils::run_shell_command("scp $scp_opts $objdir/dist/update/update.snippet.0 $Settings::ssh_user\@$update_aus_host:$path/partial.txt");
>+          }
>+

See comments above - always push complete.snippet.1, conditionally push partial.snippet.1

Comment 8

[Stefan Sitter [:ssitter]]

The Sunbird 0.8 release is coming closer and closer. Will partial updates work again until then?

Comment 9

[ause]

(In reply to comment #8)
> The Sunbird 0.8 release is coming closer and closer. Will partial updates work
> again until then?
> 

this bug is only about partial updates from/for nightly builds. creating partial updates for release versions is a completely different process (see http://wiki.mozilla.org/Software_Update:GeneratingSnippetsAndPartialsForReleases).

Comment 10

[Serge Gautherie (:sgautherie)]

Any update ?
(Does this have to wait for "comm-central & all" ?)

Comment 11

[ause]

afaik, comm-central won't use tinderbox scripting at all. there is also some python scripting on trunk to support partial .mar. i will probably revisit this bug once the switch is done.

Comment 12

[Serge Gautherie (:sgautherie)]

(In reply to comment #11)
> i will probably revisit this bug once the switch is done.

The Calendar switch to comm-central !?

NB: I can't find a/the bug about that :-/

Comment 13

[[e-mail] (:bhearsum)]

I'm doing some work on this related to our Mercurial Release Automation over in bug 454261. I'll likely be importing a script specifically designed to do nightly partial updates. You guys may be able to call out to it to do what you need to do.

Comment 14

[ause]

obsoleted by several other efforts, e.g. Bug 410806, and the switch to buildbots.

Comment 15

[mikedoak]

Hi Bug Catchers,

I'm a novice, at this, and "on the fly", meaning I am learning as I go, and confused at times, but working hardily to address my computer and restoring it to a better state than it is currently at, at present....

My phone # is 760/729~8111, in Carlsbad, CA, and do not hesitate to phone me as necessary, or email me as appropriate....

Thanx,

Mike Doak