Closed Bug 598507 Opened 10 years ago Closed 10 years ago

Talos & unittest don't launch on 10.5 for i386+x86_64 universal binaries

Categories

(Testing :: General, defect, blocker)

x86
macOS
defect
Not set
blocker

Tracking

(blocking2.0 beta7+)

RESOLVED FIXED
Tracking Status
blocking2.0 --- beta7+

People

(Reporter: nthomas, Assigned: ted)

References

Details

(Whiteboard: [ETA: 9/23])

Attachments

(4 files)

It's the same error message each time:
  dyld: unknown required load command 0x80000022
which google implies is from trying to launch the 64bit arch which is built to support 10.6.

python run_tests.py --noisy 20100921_1640_config.yml
 in dir /Users/cltbld/talos-slave/mozilla-central_leopard_test-cold/../talos-data/talos/ (timeout 21600 secs)
[snip]
RETURN:s: talos-r3-leopard-001
RETURN:id:20100921125031
RETURN:<a href = "http://hg.mozilla.org/users/nthomas_mozilla.com/mozilla-central/rev/d7a5f68544ba">rev:d7a5f68544ba</a>
talos-r3-leopard-001: 
		Started Tue, 21 Sep 2010 16:40:37
Running test ts_cold: 
		Started Tue, 21 Sep 2010 16:40:37
dyld: unknown required load command 0x80000022
sh: line 1:   312 Trace/BPT trap          ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmp58i_2l/profile getInfo.html > browser_output.txt

I think we call firefox-bin directly and therefore bypass the Info.plist which specifies 32bit on 10.5, 64bit on 10.6. Blocks a beta7 blocker.
Here's a unit test run (mochitest 1/5):
python mochitest/runtests.py --appname=./Minefield.app/Contents/MacOS/firefox-bin --utility-path=bin --extra-profile-file=bin/plugins --certificate-path=certs --autorun --close-when-done --console-level=INFO --symbols-path=symbols --total-chunks=5 --this-chunk=1 --chunk-by-dir=4
 in dir /Users/cltbld/talos-slave/mozilla-central_leopard_test-mochitests-1/build (timeout 1200 secs)
[snip]
dyld: unknown required load command 0x80000022
INFO | runtests.py | Server pid: 306
Timed out while waiting for server startup.
program finished with exit code 1
Are there builds posted somewhere I can test locally (I have 10.5 still). I thought that running firefox-bin would associate with the plist.
Throwing a 
- subprocess.Popen.__init__(self, args, bufsize, executable,
+ subprocess.Popen.__init__(self, ['arch','-i386] + args, bufsize, executable,
in Process.__init__ of automation.py gets you 

python mochitest/runtests.py
--appname=./Minefield.app/Contents/MacOS/firefox-bin --utility-path=bin
--extra-profile-file=bin/plugins --certificate-path=certs --autorun
--close-when-done --console-level=INFO --symbols-path=symbols --total-chunks=5
--this-chunk=1 --chunk-by-dir=4
INFO | runtests.py | Server pid: 458
uncaught exception: 2147746065
from xpcshell I think, which is also universal.
(In reply to comment #2)
> Are there builds posted somewhere I can test locally (I have 10.5 still). I
> thought that running firefox-bin would associate with the plist.

If you launch the binary directly it won't take the plist into account. You have to use "arch" or use a command that can be pointed at the application bundle ("open -W").
Yeah. Unfortunately that's not how any of our harnesses work currently. They all launch the binary directly.

The easiest fix for the unit tests is probably just to change the app path to be the bundle name "Minefield.app", and make runtests.py use "start <bundle>", assuming that that launches it synchronously (and hopefully via exec, so that the PID becomes the PID of firefox-bin).
Ted, would we do that for xpcshell, certutil, etc that all get launched for suites like mochitest ?
blocking2.0: --- → ?
blocking2.0: ? → beta7+
That won't work for those, they're not bundles. That's...unfortunate.
Let's do the arch thing... do we tell the test suite that we want i686, or is it just supposed to figure that out from being run on 10.5?

Presumably we'd want to be able to test i686 and x86-64 on 10.6, so a flag might be worth it in any case.
I guess this means we should just wrap some hacks into the test harnesses. :-/

If (we're on OSX < 10.6):
  cmd = ["arch", "-arch", "i386"] + cmd

to force us to run the 32-bit binary on 10.5. Not the worst thing in the world, and should avoid buildbot changes.
Assignee: nobody → ted.mielczarek
Status: NEW → ASSIGNED
Comment on attachment 477549 [details] [diff] [review]
wrap test harness process execution with 'arch -arch i386' on OS X 10.5

I haven't tested this at all, I'm rebuilding locally so I can do so.
Comment on attachment 477549 [details] [diff] [review]
wrap test harness process execution with 'arch -arch i386' on OS X 10.5

This seems to work on the loaner Talos slave nthomas gave me. (I just copied the changed files from my build to the unpacked test dir on the Talos slave.)
Attachment #477549 - Flags: review?(catlee)
I have to leave shortly, so if someone else wants to land this patch that'd be great. It ought to work in both our existing setup and the new 32/64 universal build setup.
Attachment #477598 - Flags: review?(ted.mielczarek) → review+
Comment on attachment 477598 [details] [diff] [review]
wrap talos process execution with 'arch -arch i386' on osx10.5

I got a set of green runs on 10.5 using this patch, same for 10.6. I'll report the perf results when I get a chance.
Attachment #477549 - Flags: review?(catlee) → review+
Comment on attachment 477549 [details] [diff] [review]
wrap test harness process execution with 'arch -arch i386' on OS X 10.5

Results of a single run of tests on 10.5 and 10.6:

On 10.5:
* mochitest-browser-chrome (in mochitest-other) - 2 test fails
* xpcshell - 7 test fails
* otherwise green

On 10.6:
* xpcshell hit test_punycodeURIs.js - bug 561350
* otherwise green

I'll attach logs for the 10.5 failures.
Two issues in mochitest-browser-chrome:

1, four of these
dyld: unknown required load command 0x80000022
which indicates we missed a spot, or maybe have tests launching utils directly. We also see this in the xpcshell (to follow) but not on any other test.

2, two test failures
TEST-UNEXPECTED-FAIL | chrome://mochikit/content/browser/toolkit/mozapps/plugins/tests/browser_bug435788.js | Should have been a successful install - Got Failed, expected Installed
TEST-UNEXPECTED-FAIL | chrome://mochikit/content/browser/toolkit/mozapps/plugins/tests/browser_bug435788.js | Should have been a successful install - Got Failed, expected Installed
Nearby, but not directly correlated to, the dyld errors.
Whiteboard: [ETA: 9/23]
Issues:

1, 6 of the 'launching the wrong arch' errors
dyld: unknown required load command 0x80000022

2, test fails in crash reporter, update tests, test_nsIProcess.js, and the punycode fail. All except the punycode seem to be from the launching issue.
Running a 2nd and 3rd round, saw
* all the failures noted above
* bug 595100 in mochitest-2/5, known intermittent orange
* in mochitest-chrome on 10.6, appears to be unfiled:
1140 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/chrome/content/xul/templates/tests/chrome/test_tmpl_wheregreaterstring.xul | where - greater string dynamic step 3
1141 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/chrome/content/xul/templates/tests/chrome/test_tmpl_wheregreaterstring.xul | Error is: extra attribute value - got "<label id=\"http://www.some-fictitious-zoo.com/mammals/africanelephant\" value=\"14\"/><label id=\"http://www.some-fictitious-zoo.com/mammals/llama\" value=\"5\"/><label id=\"http://www.some-fictitious-zoo.com/mammals/polarbear\" value=\"20\"/><label id=\"http://www.some-fictitious-zoo.com/mammals/gorilla\" value=\"7\"/>", expected "Same"

Probably want to farm out the update test problem out to another bug, since it's launching a util:
http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/test/unit/head_update.js.in#130
Not sure about the rest.
Let's take all the test failures to a new bug. This bug, as filed, should be fixed with these two patches. The test failures are going to require more finessing, but we should be able to fix them.
Ran my patch through try and it looks ok. I'll land it tomorrow.
Cool. I'll file separate bugs for the test failures.
Pushed the unittest patch to m-c:
http://hg.mozilla.org/mozilla-central/rev/0c0e27537b72

I'll let Alice close this when she lands the Talos change.
Patch for talos is against talos in hg, so this depends on the patches for shifting to talos in hg landing.
Depends on: 556530
* mochitest-browser-chrome is bug 599467
* xpcshell's crashreporter tests is bug 599475
* xpcshell's update tests is bug 599477
* xpcshell's test_nsIProcess.js is bug 599478
Comment on attachment 477598 [details] [diff] [review]
wrap talos process execution with 'arch -arch i386' on osx10.5

Landed in cvs:
 Checking in ffprocess_mac.py;
 /cvsroot/mozilla/testing/performance/talos/ffprocess_mac.py,v  <--   ffprocess_mac.py
 new revision: 1.20; previous revision: 1.19
 done


Landed in hg:
 http://hg.mozilla.org/build/talos/9233db568b3d
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Depends on: 600328
You need to log in before you can comment on or make changes to this bug.