Closed Bug 340879 Opened 18 years ago Closed 18 years ago

Repackaged builds cannot update without forcing updates of certain files

Categories

(Firefox Build System :: General, defect)

1.5.0.x Branch
x86
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jbecerra, Assigned: coop)

References

Details

Partial updates (1.5.0.3-1.5.0.4) for Google CJK builds failed to apply, and it probably has to do with the repackaging process. The last-update.log file after getting the partial update and restarting says:

PREPARE PATCH AccessibleMarshal.dll
PREPARE PATCH chrome/browser.jar
CRC check failed
LoadSourceFile failed
failed: 4
calling QuitProgressUI

Quoting Paul:

"We need to investigate this problem, because right now, we're force-updating all the locale file, and now we're force-updating browser.jar. So basically, we're force-updating every file we change in the repackaging step... so there's either an issue with the repackaging step that is violating an invariant we're assuming, or a problem with the way we're calculating diffs, or... *something*."
Let me take a first swing at this. I'll try altering the process so that all the files get unpacked and then repacked without actually modifying any files in the middle. That should help narrow down what is causing the delta.
Assignee: nobody → ccooper
This bug is currently marked as OS X 10.3, but I know preed has said that Windows updates need to force-update files as well. Can I get a complete list of the platforms affected by this so I can know which processes and toolchains to investigate? 

Status: NEW → ASSIGNED
I've done some comparisons of the repackaged jar files on Windows, using the ja locale for Google as my in all cases:
* compared original 1.5.0.4 builds (zip and installer);
* compared repackaged 1.5.0.4 builds (zip and installer);
* compared original 1.5.0.4 builds with repackaged builds that were simply unpacked then repacked with no string manipulation in the middle (designated as no-change below)

Results:
* browser.jar files for the original zip and installer builds have different md5sums, but there are no differences between the contents of the two files;
* browser.jar files for the repackaged Google zip and installer builds have different md5sums, but there are no differences between the contents of the two files;
* browser.jar files for the original zip and no-change zip, and the original installer and no-change installer, have different md5sums, but there are no differences between the contents of the two files;

* ja.jar files are identical (md5sum) between the original zip and installer builds;
* ja.jar files for the repackaged Google zip and installer builds have different md5sums, but there are no differences between the contents of the two files;
* ja.jar files for the original zip and no-change zip have different md5sums, but there are no differences between the contents of the two files;
* ja.jar files for the original installer and no-change installer are identical;
(In reply to comment #3)
> * browser.jar files for the repackaged Google zip and installer builds have
> different md5sums, but there are no differences between the contents of the two
> files;
> * browser.jar files for the original zip and no-change zip, and the original
> installer and no-change installer, have different md5sums, but there are no
> differences between the contents of the two files;

What jumps out at me here is that the process is yielding different md5sums for the repackaged zip and installer builds, even when we don't actually change any strings in the interim, and even when the md5sums of the original files are identical (as in the case of ja.jar). 

Since we use the zip builds to generate the MAR files for the installer, this is a possible source of the update errors we're seeing.
Depends on: 342213
I've done some further analysis of the ja.jar files repackaged for the Google zip and installer builds. 

As mentioned previously, the contents of the two zips are identical when unpacked, but the md5sums of the two files differ. Using the Archive::Zip perl module, I compared the actual structure of the two zip files. Internally, the ja.jar files for the installer and zip repacks differed only in the file entry/meta-information that zip keeps for the files we are actively changing during repackaging, and in the zip central directory information for these files. These files are:

ja.jar:
locale/branding/brand.properties
locale/browser-region/region.properties

The meta-information for these two files only differed in two respects:
1) lastModFileDateTime: last modification date/time stamp (makes sense)
2) cdExtraField: not exactly sure what this is, and it's not human-readable (e.g. "UT^E^@^C<9E>^A<AC>DUx^@^@"). One reference I've found suggests that this contains unix filemode bits. Interestingly, this only gets set for files we repackage, i.e. original files in the archive _don't_ have a value set for this.

Perhaps a 'zip -X' would help here when repackaging, although we're stuck with changing the mod time regardless.
Using 'zip -X' wasn't enough to get matching md5sums for the browser.jar and $(LOCALE).jar files between the installer and zip builds. However, I was able to get identical md5sums between the files by touch-ing the changed files with a known date. To do this, I grab a current timestamp in the main perl repackaging script and then pass that timestamp to each individual repackaging process.
Status: ASSIGNED → RESOLVED
Closed: 18 years ago
Resolution: --- → FIXED
Component: Build Config → General
Product: Firefox → Firefox Build System
You need to log in before you can comment on or make changes to this bug.