518641 - "make package-tests" doesn't work right on OS X universal builds

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Jeff Muizelaar [:jrmuizel]]

Error 0 (Unknown error: 0).35957 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/bugs/test_bug427744.html | Should have seen the test plugin
47332 ERROR TEST-UNEXPECTED-FAIL | /tests/layout/generic/test/test_plugin_mouse_coords.html | [SimpleTest/SimpleTest.js, window.onerror] An error occurred - p1.getLastMouseX is not a function
lastScript86495 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_npobject_getters.html | plugin1.pluginFoundElement (NPNVPluginElementNPObject)
86496 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_npobject_getters.html | window.pluginFoundWindow (NPNVWindowNPObject)
86499 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_pluginstream_asfile.html | Test timed out.
86502 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_pluginstream_asfileonly.html | Test timed out.
86505 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_pluginstream_geturl.html | Test timed out.
86508 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_pluginstream_geturlnotify.html | Test timed out.
86509 ERROR TEST-UNEXPECTED-FAIL | (SimpleTest/TestRunner.js) | 4 test timeouts, giving up.
86510 ERROR TEST-UNEXPECTED-FAIL | (SimpleTest/TestRunner.js) | Skipping 159 remaining tests.

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1253812968.1253815919.30710.gz

Comment 1

[Markus Stange [:mstange]]

This happened on the newly-introduced opt machines, which run builds without --enable-logrefcnt.

Comment 2

[(not currently active) Ted Mielczarek]

This is probably a packaging or harness error. Note the first error is "Should have seen the test plugin", so somehow the test plugin is not being loaded in these tests. I'm going to move this over to RelEng for investigation, catlee and I will take a look at it.

Comment 3

[(not currently active) Ted Mielczarek]

I've hidden the OSX opt unittest builds on Firefox-Unittest for the time being.

Comment 4

[(not currently active) Ted Mielczarek]

$ file bin/plugins/Test.plugin/Contents/MacOS/libnptest.dylib 
bin/plugins/Test.plugin/Contents/MacOS/libnptest.dylib: Mach-O dynamically linked shared library ppc

Yeah, that'd be our problem. I guess the test packages just don't currently work right on universal builds. The test plugin that's being packaged is PPC, which clearly doesn't work. On a related note, everything in bin/ is also ppc, including xpcshell. I guess this works because they'll get run in Rosetta (which is kind of insane), and libxul and everything are universal binaries, so that works, but an x86 Firefox trying to load a PPC plugin fails.

Comment 5

[(not currently active) Ted Mielczarek]

Going to have to make 'make package-tests' have a special case for universal builds, just like 'make package'.

Comment 6

[(not currently active) Ted Mielczarek]

Attachment: Step 1: fix unify to not choke on Java .class files

I had to fix this first. The unify script currently chokes on Java .class files, which we apparently have hiding in our test suites somewhere. The reason is that Mach-O fat binaries and Java .class files use the same magic number, and unify detects fat binaries by magic number. This patch fixes unify, and adds some basic unit tests of this behavior.

Comment 7

[(not currently active) Ted Mielczarek]

Attachment: Step 2: unify test packages in postflight makefile

This makes the universal postflight makefile build arch-specific test packages, and then unify them to dist/universal/test-package-stage, very similar to what we do with the app package. It then adds some special-casing to the "make package-tests" target, such that universal builds will just package up the bits that we staged in the postflight makefile.

This produces a plausible-looking test package, with universal binaries. I haven't tried actually running tests on it yet, nor have I tried running "package-tests" on a non-universal build with this patch yet.

Comment 8

[(not currently active) Ted Mielczarek]

Some basic testing of the packaged build + test package show that the tests mentioned in comment 0 pass.

Comment 9

[(not currently active) Ted Mielczarek]

Pushed to m-c:
http://hg.mozilla.org/mozilla-central/rev/5642cd4a7e88
http://hg.mozilla.org/mozilla-central/rev/07837dc569c6

Hopefully this should make the OS X opt unittest runs go green, and we can un-hide them!

Comment 10

[(not currently active) Ted Mielczarek]

jhford hit bustage on a XULRunner universal build. This needs a followup to move the two cp lines here inside the if:
http://mxr.mozilla.org/mozilla-central/source/build/macosx/universal/flight.mk#118

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

The second time a builder tries to unify the packaged tests it fails with
/builds/moz2_slave/mozilla-central-macosx/build/build/macosx/universal/unify: makeUniversal: mkdir: File exists for:
  obj-firefox/ppc/dist/universal/test-package-stage
Three builds have failed like this now, eg
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1254357655.1254360964.4557.gz

flight.mk has a rm -f for the universal app at 
  http://hg.mozilla.org/mozilla-central/file/07837dc569c6/build/macosx/universal/flight.mk#l106
but there's no equivalent for packaged tests.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Step 3. Followup for bustage during 2nd build

http://hg.mozilla.org/mozilla-central/rev/3914654341be

Comment 13

[(not currently active) Ted Mielczarek]

Comment on attachment 403952 [details] [diff] [review]
Step 3. Followup for bustage during 2nd build

Thanks, I noticed that yesterday but didn't have time to fix it. (Along with comment 10.)

Comment 14

[(not currently active) Ted Mielczarek]

Pushed another bustage fix followup:
http://hg.mozilla.org/mozilla-central/rev/ab3ee8fc063d

Comment 15

[(not currently active) Ted Mielczarek]

Comment on attachment 403252 [details] [diff] [review]
Step 1: fix unify to not choke on Java .class files

We need these patches on 1.9.2 in order to enable unit tests on nightly and release builds for OS X. This patch touches some build machinery, but also has unit tests. Should have no impact on the bits we ship.

Comment 16

[(not currently active) Ted Mielczarek]

Comment on attachment 403255 [details] [diff] [review]
Step 2: unify test packages in postflight makefile

We need these patches on 1.9.2 in order to enable unit tests on nightly and release builds for OS X. This patch touches some build machinery, but only in ways that involve the test package. Should have no impact on the bits we ship.

Comment 17

[Mike Beltzner [:beltzner, not reading bugmail]]

Comment on attachment 403252 [details] [diff] [review]
Step 1: fix unify to not choke on Java .class files

a192=beltzner, would prefer if you could land in the morning EST so if it causes problems you can back it out before the PSTers get up.

Comment 18

[Mike Beltzner [:beltzner, not reading bugmail]]

Comment on attachment 403255 [details] [diff] [review]
Step 2: unify test packages in postflight makefile

a192=beltzner, would prefer if you could land in the morning EST so if it causes problems you can back it out before the PSTers get up.

Comment 19

[(not currently active) Ted Mielczarek]

Pushed to 1.9.2:
http://hg.mozilla.org/releases/mozilla-1.9.2/rev/7c4620338422
http://hg.mozilla.org/releases/mozilla-1.9.2/rev/c0b78865c85e
(the latter changeset is a rollup of the "Step 2" patch, the "Step 3" patch, and the bustage fix from comment 14).

Comment 20

[Serge Gautherie (:sgautherie)]

Attachment: (Dv1) flight.mk: formatting nits

Comment 21

[Serge Gautherie (:sgautherie)]

Attachment: (Ev1-CC) Port it to comm-central [Checkin: Comment 23]

Comment 22

[(not currently active) Ted Mielczarek]

Comment on attachment 429398 [details] [diff] [review]
(Dv1) flight.mk: formatting nits

Really, I don't care.

Comment 23

[Serge Gautherie (:sgautherie)]

Comment on attachment 429399 [details] [diff] [review]
(Ev1-CC) Port it to comm-central
[Checkin: Comment 23]


http://hg.mozilla.org/comm-central/rev/e8064a410e5b

Comment 24

[Serge Gautherie (:sgautherie)]

fixed1.9.1.10 in bug 526668...