526668 - add option to 'unify' to allow files to match if their sorted contents match

Status: VERIFIED FIXED

(Firefox Build System :: General, defect)

Description

[(not currently active) Ted Mielczarek]

We have some files that wind up with the same contents in a slightly different order when you run a parallel build. The 'unify' script currently expects files to have identical contents, so these situations break universal builds easily. I have a patch that adds a --unify-with-sort <regex> option to unify, so we can pass in file patterns and have them special-cased. unify will sort the contents of the files and compare that instead.

Comment 1

[(not currently active) Ted Mielczarek]

Attachment: add --unify-with-sort <regex> to unify, use it in a few places [Checkin: See comment 2 & 5+13 & 14]

Here's the patch. There's an (admittedly ugly) unit test for it included.

Comment 2

[(not currently active) Ted Mielczarek]

Pushed to m-c:
http://hg.mozilla.org/mozilla-central/rev/7378ea1411d8

Comment 3

[(not currently active) Ted Mielczarek]

Comment on attachment 410413 [details] [diff] [review]
add --unify-with-sort <regex> to unify, use it in a few places
[Checkin: See comment 2 & 5+13 & 14]

This needs to land on 1.9.2 if we're going to land bug 519357 (which is a blocker).

Comment 4

[Mike Beltzner [:beltzner, not reading bugmail]]

Comment on attachment 410413 [details] [diff] [review]
add --unify-with-sort <regex> to unify, use it in a few places
[Checkin: See comment 2 & 5+13 & 14]

a191=beltzner, it even comes with tests!

Comment 5

[(not currently active) Ted Mielczarek]

http://hg.mozilla.org/releases/mozilla-1.9.2/rev/7cd2a4716469

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

After running with this config for a couple of days now no failures have been started. The builds look ok. Regarding that this bug is blocking bug 519357 am I right, that the content of the components.list file should be ordered alphabetically? I think so. While checking this file the ordering is fine.

Marking as verified fixed on trunk and 1.9.2.

Comment 7

[Serge Gautherie (:sgautherie)]

http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey2.0/1267690621.1267694362.21111.gz
OS X 10.5 comm-1.9.1 nightly on 2010/03/04 00:17:01
{
/builds/slave/comm-1.9.1-macosx-nightly/build/mozilla/build/macosx/universal/unify: copyIfIdentical: files differ:
  objdir/ppc/mozilla/dist/seamonkey/SeaMonkey.app/Contents/MacOS/chrome/comm.manifest,
  objdir/i386/mozilla/dist/seamonkey/SeaMonkey.app/Contents/MacOS/chrome/comm.manifest
}

Could this land on m-1.9.1 too?

Comment 8

[Serge Gautherie (:sgautherie)]

Comment on attachment 410413 [details] [diff] [review]
add --unify-with-sort <regex> to unify, use it in a few places
[Checkin: See comment 2 & 5+13 & 14]

"approval1.9.1.9=?":
already +'ed in comment 4 but flag not set.

Comment 9

[(not currently active) Ted Mielczarek]

Um, oops. I guess that's my bad.

Comment 10

[(not currently active) Ted Mielczarek]

Serge, if you want this on 1.9.1 feel free to land it, now that it has approval.

Comment 11

[Mike Beltzner [:beltzner, not reading bugmail]]

Comment on attachment 410413 [details] [diff] [review]
add --unify-with-sort <regex> to unify, use it in a few places
[Checkin: See comment 2 & 5+13 & 14]

Let's try again for 1.9.1.10

Comment 12

[Daniel Veditz [:dveditz]]

Comment on attachment 410413 [details] [diff] [review]
add --unify-with-sort <regex> to unify, use it in a few places
[Checkin: See comment 2 & 5+13 & 14]

Approved for 1.9.1.10, a=dveditz for release-drivers

please land this time. Code freeze in a week or so.

Comment 13

[Serge Gautherie (:sgautherie)]

Comment on attachment 410413 [details] [diff] [review]
add --unify-with-sort <regex> to unify, use it in a few places
[Checkin: See comment 2 & 5+13 & 14]


http://hg.mozilla.org/releases/mozilla-1.9.2/rev/ca4501b479e1
(Cv1-192) Fully fix/sync' m-1.9.2.

Comment 14

[Serge Gautherie (:sgautherie)]

Comment on attachment 410413 [details] [diff] [review]
add --unify-with-sort <regex> to unify, use it in a few places
[Checkin: See comment 2 & 5+13 & 14]


http://hg.mozilla.org/releases/mozilla-1.9.1/rev/fa3d83b79848
(Dv1-191) Backport to m-1.9.1 too.
+
http://hg.mozilla.org/releases/mozilla-1.9.1/rev/59e1a6f8fd05
(Ev1-191) Missed part of bug 518641.
+
http://hg.mozilla.org/releases/mozilla-1.9.1/rev/75b15d431177
(Fv1-191) The other missed part of bug 518641.