375852 - update the scripts on the nightly build machine to use bsmedberg's fix for bug #375415 (UPDATE_PACKAGING_R1 tag)

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[(not reading, please use [e-mail] instead)]

update the scripts on the nightly build machine to use bsmedberg's fix for bug #375415 

I believe the scripts that generate the partial and complete mars for the nightly builds may not be updated even though bsmedberg has checked in a fix for bug #375415.

So, we may have to update things by hand.  Or, perhaps soon we will fix the nightly build system to use the same scripts / process as the releases.

Preed asked me to remind him about this issue, so I've logged this bug.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

There are two parts to this

1) complete mar generation - nightly tinderboxen pull mozilla/tools/update-packaging from their respective branches, which are now out of sync with the trunk. We should look at some diffs and decide if the changes need to be pushed to the branches.

2) partial mar generation - this done on prometheus-vm, and may not be autoupdating. I'll check into this.

Seth, could you remind me if you are seeing the update hang with partial updates on the mac-trunk ?

Comment 2

[Robert Helmer [:rhelmer]]

(In reply to comment #1)
> There are two parts to this
> 
> 1) complete mar generation - nightly tinderboxen pull
> mozilla/tools/update-packaging from their respective branches, which are now
> out of sync with the trunk. We should look at some diffs and decide if the
> changes need to be pushed to the branches.


Right, these are automatically pulled but pull from whatever branch the product is coming from.

 
> 2) partial mar generation - this done on prometheus-vm, and may not be
> autoupdating. I'll check into this.


I know that it is not; ping me and I'll show you where it's set up.

Comment 3

[(not reading, please use [e-mail] instead)]

> Seth, could you remind me if you are seeing the update hang with partial
> updates on the mac-trunk ?

yes, I am seeing it with both.

Also, to answer an earlier question, the 1.8.0.x and 1.8.x mars are not
affected, possibly because the scripts that generate the mars for those
branches do not contain the change from bsmedberg that caused the 0 byte
problem.

I downloaded:

http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.8/firefox-2.0.0.4pre.en-US.mac.complete.mar
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.8.0/firefox-1.5.0.12pre.en-US.mac.complete.mar

and neither of those contained any 0 byte files.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #3)
> yes, I am seeing it with both.

Ok, that'll be because the busted complete mars are used to make the partials.
 
> Also, to answer an earlier question, the 1.8.0.x and 1.8.x mars are not
> affected, possibly because the scripts that generate the mars for those
> branches do not contain the change from bsmedberg that caused the 0 byte
> problem.

That sounds right.

On prometheus-vm, cvs stat reveals that 
  common.sh is v1.8 (latest is v1.9) - just bsmedberg's change from bug 375435
  make_full_update.sh is 1.7 (1.9) - the recent problem (373121) and 375435
  make_incremental_update.sh is 1.9 (1.11) - ditto
  unwrap_full_update.sh is 1.1 (1.2) - just 373121

The other files in mozilla/tools/update-packaging have not changed. So updating this dir would be a relatively safe thing to do.

Comment 5

[Robert Helmer [:rhelmer]]

(In reply to comment #4)
> (In reply to comment #3)
> > yes, I am seeing it with both.
> 
> Ok, that'll be because the busted complete mars are used to make the partials.
> 
> > Also, to answer an earlier question, the 1.8.0.x and 1.8.x mars are not
> > affected, possibly because the scripts that generate the mars for those
> > branches do not contain the change from bsmedberg that caused the 0 byte
> > problem.
> 
> That sounds right.
> 
> On prometheus-vm, cvs stat reveals that 
>   common.sh is v1.8 (latest is v1.9) - just bsmedberg's change from bug 375435
>   make_full_update.sh is 1.7 (1.9) - the recent problem (373121) and 375435
>   make_incremental_update.sh is 1.9 (1.11) - ditto
>   unwrap_full_update.sh is 1.1 (1.2) - just 373121
> 
> The other files in mozilla/tools/update-packaging have not changed. So updating
> this dir would be a relatively safe thing to do.

Was this done? 

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #5)
> Was this done? 
 
No, the file revisions are unchanged. In the meantime, this file also changed:
  unwrap_full_update.pl is 1.1 (latest is 1.2) - Bug 380859 (fix perl warning)

Comment 7

[Robert Helmer [:rhelmer]]

(In reply to comment #6)
> (In reply to comment #5)
> > Was this done? 
> 
> No, the file revisions are unchanged. In the meantime, this file also changed:
>   unwrap_full_update.pl is 1.1 (latest is 1.2) - Bug 380859 (fix perl warning)

How about updating to UPDATE_PACKAGING_R1?

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #7)
> How about updating to UPDATE_PACKAGING_R1?

I'm testing this in a side-by-side setup on prometheus-vm. 

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #8)
> I'm testing this in a side-by-side setup on prometheus-vm. 

Results are in and look good. There are very slight differences in the mar files generated by the existing ("old") and UPDATE_PACKAGING_R1 ("new") utils, limited to the ordering of the update manifest. Eg:

/builds/nightly-partial-generation/ftp/firefox/nightly/2008/02/2008-02-13-04-mozilla1.8/firefox-2.0.0.13pre.en-US.win32.partial.2008021304-2008021403.mar
diff -ru old/update.manifest new/update.manifest
--- old/update.manifest 2008-02-15 10:01:21.000000000 -0800
+++ new/update.manifest 2008-02-15 10:01:24.000000000 -0800
@@ -23,9 +23,9 @@
 patch "ssl3.dll.patch" "ssl3.dll"
 patch "uninstall/helper.exe.patch" "uninstall/helper.exe"
 patch "updater.exe.patch" "updater.exe"
-patch "xpcom.dll.patch" "xpcom.dll"
 patch "xpcom_compat.dll.patch" "xpcom_compat.dll"
 patch "xpcom_core.dll.patch" "xpcom_core.dll"
+patch "xpcom.dll.patch" "xpcom.dll"
 patch "xpicleanup.exe.patch" "xpicleanup.exe"
 patch "xpistub.dll.patch" "xpistub.dll"
 remove "chrome/US.jar"

This sort of thing happens for win32 and Mac manifests but not for linux, and I think is down to different orderings in the sort on each platform (in particular for . and _). If you sort the unpacked manifest then there are no differences; the patch instructions also stay before any remove ones.

UPDATE_PACKAGING_R1 is setup in /builds/new-mozilla-checkout/ and can be enabled easily by switching mozilla-checkout symlink in /builds/nightly-update-generation/. I'm going to hold off doing that for the moment, since I'm going to have other changes pending in bug 399848.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Forgot to mention that I tested all the mar files from the 13th and 14th, so that covers both Firefox & Thunderbird, the 3 branches, and 3 platforms.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Done as part of teaching the nightly update generator to use the role keys instead of cltbld (bug 399848).