Manifest files are not created deterministically

RESOLVED FIXED in mozilla31

Status

()

Core
Build Config
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: Georg Koppen, Assigned: Georg Koppen)

Tracking

(Blocks: 1 bug)

24 Branch
mozilla31
x86
All
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 1 obsolete attachment)

(Assignee)

Description

4 years ago
User Agent: Mozilla/5.0 (Windows NT 6.1; rv:17.0) Gecko/20100101 Firefox/17.0 (Beta/Release)
Build ID: 20100101

Steps to reproduce:

We are trying to update the Tor Browser to ESR24 which is built deterministically. While doing so we noticed that the omni.ja in /browser has different SHA256 sums in different builds. It turned out that the browser.manifest contains entries in different order in some builds. We worked around this problem in ESR17 by patching link-manifests.py but that is not available anymore since bug 780561 landed. I tried to reproduce the differences by running JarMaker.py by hand but that did not work.


Actual results:

The order of the content in browser.manifest files was different.


Expected results:

The order of the content in browser.manifest files should always have been the same.
(Assignee)

Updated

4 years ago
OS: Windows 7 → All
I'm sure we have a bug on file for that already. Not for the order, though, which would be fixed as a side effect.

An easy workaround in esr (since we're not going to fix that there) is to build with -j1.
This is very much related to bug 924187
(Assignee)

Comment 3

4 years ago
(In reply to Mike Hommey [:glandium] from comment #1)
> I'm sure we have a bug on file for that already. Not for the order, though,
> which would be fixed as a side effect.
> 
> An easy workaround in esr (since we're not going to fix that there) is to
> build with -j1.

Can you think of any less easy workarounds here? Building with -j4 takes already several hours for all our bundles. Thus, I'd rather like avoiding -j1...
(Assignee)

Comment 4

4 years ago
It seems sorting myregister in JarMaker.py's updateManifest() helps, nevermind then.

Comment 5

4 years ago
While the bug mentions browser.manifest, this is an issue for XPCOM manifests as well. I've been wanting to tackle XPCOM manifest generation/buildlist.py foo now that we're in a moz.build world and we know the set of input manifest files at config time.

There is work around bug 924617 to change how JarMaker.py works. It's all leading to a moz.build-driven jar.mn processing world. We should be able to bake determinism into that.
(Assignee)

Updated

4 years ago
Blocks: 885777
Status: UNCONFIRMED → NEW
Ever confirmed: true
(Assignee)

Comment 6

4 years ago
Created attachment 8388463 [details] [diff] [review]
Generate browser.manifest deterministically

I finally extracted the fix out of our ESR 24 repo and applied it to m-c for review. This covers only the browser.manifest case which bit us in the past.
Attachment #8388463 - Flags: review?(gps)

Updated

4 years ago
Attachment #8388463 - Flags: review?(gps) → review?(mh+mozilla)
Attachment #8388463 - Flags: review?(mh+mozilla) → review+
(Assignee)

Comment 7

4 years ago
Created attachment 8392759 [details] [diff] [review]
patch with updated commit message

Thanks for the review. I just updated the commit message (it references the bug number now as well).
Assignee: nobody → gk
Attachment #8388463 - Attachment is obsolete: true
Status: NEW → ASSIGNED
(Assignee)

Updated

4 years ago
Keywords: checkin-needed
https://hg.mozilla.org/integration/mozilla-inbound/rev/c4473789d528
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/c4473789d528
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla31
You need to log in before you can comment on or make changes to this bug.