942091 - Manifest files are not created deterministically

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Georg Koppen]

User Agent: Mozilla/5.0 (Windows NT 6.1; rv:17.0) Gecko/20100101 Firefox/17.0 (Beta/Release)
Build ID: 20100101

Steps to reproduce:

We are trying to update the Tor Browser to ESR24 which is built deterministically. While doing so we noticed that the omni.ja in /browser has different SHA256 sums in different builds. It turned out that the browser.manifest contains entries in different order in some builds. We worked around this problem in ESR17 by patching link-manifests.py but that is not available anymore since bug 780561 landed. I tried to reproduce the differences by running JarMaker.py by hand but that did not work.


Actual results:

The order of the content in browser.manifest files was different.


Expected results:

The order of the content in browser.manifest files should always have been the same.

Comment 1

[Mike Hommey [:glandium]]

I'm sure we have a bug on file for that already. Not for the order, though, which would be fixed as a side effect.

An easy workaround in esr (since we're not going to fix that there) is to build with -j1.

Comment 2

[Mike Hommey [:glandium]]

This is very much related to bug 924187

Comment 3

[Georg Koppen]

(In reply to Mike Hommey [:glandium] from comment #1)
> I'm sure we have a bug on file for that already. Not for the order, though,
> which would be fixed as a side effect.
> 
> An easy workaround in esr (since we're not going to fix that there) is to
> build with -j1.

Can you think of any less easy workarounds here? Building with -j4 takes already several hours for all our bundles. Thus, I'd rather like avoiding -j1...

Comment 4

[Georg Koppen]

It seems sorting myregister in JarMaker.py's updateManifest() helps, nevermind then.

Comment 5

[Gregory Szorc [:gps]]

While the bug mentions browser.manifest, this is an issue for XPCOM manifests as well. I've been wanting to tackle XPCOM manifest generation/buildlist.py foo now that we're in a moz.build world and we know the set of input manifest files at config time.

There is work around bug 924617 to change how JarMaker.py works. It's all leading to a moz.build-driven jar.mn processing world. We should be able to bake determinism into that.

Comment 6

[Georg Koppen]

Attachment: Generate browser.manifest deterministically

I finally extracted the fix out of our ESR 24 repo and applied it to m-c for review. This covers only the browser.manifest case which bit us in the past.

Comment 7

[Georg Koppen]

Attachment: patch with updated commit message

Thanks for the review. I just updated the commit message (it references the bug number now as well).

Comment 8

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/c4473789d528

Comment 9

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/c4473789d528