Closed Bug 951834 Opened 11 years ago Closed 8 years ago

Unable to run mozmill tests on OS X.

Categories

(Thunderbird :: Testing Infrastructure, defect)

Product:
Thunderbird
Component:
Testing Infrastructure
Platform:
x86
macOS
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: mconley, Unassigned)

References

Details

STR:

Try either of the following in an objdir on OS X:

make SOLO_TEST=folder-display mozmill-one

this results in this output for me:

Mikes-MacBook-Pro-2:obj-x86_64-apple-darwin12.5.0 mikeconley$ make SOLO_TEST=folder-display mozmill-one
unset PYTHONHOME && cd ./mozilla/_tests/mozmill && MACOSX_DEPLOYMENT_TARGET= \
	/Users/mikeconley/Projects/comm-central/obj-x86_64-apple-darwin12.5.0/mozilla/_tests/mozmill-virtualenv/bin/python runtest.py \
	--test=/Users/mikeconley/Projects/comm-central/mail/test/mozmill/folder-display \
	--binary=../../.././mozilla/dist/Daily.app/ \
	--symbols-path=/Users/mikeconley/Projects/comm-central/obj-x86_64-apple-darwin12.5.0/mozilla/dist/crashreporter-symbols \
	--plugins-path=/Users/mikeconley/Projects/comm-central/obj-x86_64-apple-darwin12.5.0/mozilla/dist/plugins \

Traceback (most recent call last):
  File "runtest.py", line 28, in <module>
    from automation import Automation
  File "/Users/mikeconley/Projects/comm-central/obj-x86_64-apple-darwin12.5.0/mozilla/_tests/mozmill/automation.py", line 33, in <module>
    import mozcrash
ImportError: No module named mozcrash
make: *** [mozmill-one] Error 1


Running make mozmill spawns a TB, but it just sits there doing nothing until the test times out.

This is pretty rough.
Any idea what's going on jcranmer? Or whimboo?
Flags: needinfo?(hskupin)
Flags: needinfo?(Pidgeot18)
Cc'ing a few more TB-Mozmill hackers.
Have you tried running it once more?

I always get the "ImportError: No module named mozcrash" the first time after i've built from scratch (on linux). 
Re-run and it works. (I usually just run one single test at a time.)
Hm, yes - that does improve things a little.

If I run make SOLO_TEST=folder-display mozmill-one I now get a TB coming up, but no activity - it just sits there.

And here's what I get in the terminal:

unset PYTHONHOME && cd ./mozilla/_tests/mozmill && MACOSX_DEPLOYMENT_TARGET= \
	/Users/mikeconley/Projects/comm-central/obj-x86_64-apple-darwin12.5.0/mozilla/_tests/mozmill-virtualenv/bin/python runtest.py \
	--test=/Users/mikeconley/Projects/comm-central/mail/test/mozmill/folder-display \
	--binary=../../.././mozilla/dist/Daily.app/ \
	--symbols-path=/Users/mikeconley/Projects/comm-central/obj-x86_64-apple-darwin12.5.0/mozilla/dist/crashreporter-symbols \
	--plugins-path=/Users/mikeconley/Projects/comm-central/obj-x86_64-apple-darwin12.5.0/mozilla/dist/plugins \

Using profile dir: /Users/mikeconley/Projects/comm-central/obj-x86_64-apple-darwin12.5.0/mozilla/_tests/mozmill/mozmillprofile
2013-12-18 16:24:51.289 thunderbird-bin[7105:707] *** WARNING: Method userSpaceScaleFactor in class NSWindow is deprecated on 10.7 and later. It should not be used in new applications. Use convertRectToBacking: instead.
Warning: unrecognized command line flag -jsbridge
JavaScript error: chrome://global/content/bindings/findbar.xml, line 262: this._browser.finder is undefined
************************************************************
* Call to xpconnect wrapped JSObject produced this error:  *
[Exception... "Component returned failure code: 0x80570016 (NS_ERROR_XPC_GS_RETURNED_FAILURE) [nsIJSCID.getService]"  nsresult: "0x80570016 (NS_ERROR_XPC_GS_RETURNED_FAILURE)"  location: "JS frame :: file:///Users/mikeconley/Projects/comm-central/obj-x86_64-apple-darwin12.5.0/mozilla/dist/Daily.app/Contents/MacOS/components/nsUrlClassifierListManager.js :: PROT_ListManager :: line 82"  data: no]
************************************************************
************************************************************
* Call to xpconnect wrapped JSObject produced this error:  *
[Exception... "Component returned failure code: 0x80570016 (NS_ERROR_XPC_GS_RETURNED_FAILURE) [nsIJSCID.getService]"  nsresult: "0x80570016 (NS_ERROR_XPC_GS_RETURNED_FAILURE)"  location: "JS frame :: file:///Users/mikeconley/Projects/comm-central/obj-x86_64-apple-darwin12.5.0/mozilla/dist/Daily.app/Contents/MacOS/components/nsUrlClassifierListManager.js :: PROT_ListManager :: line 82"  data: no]
************************************************************
unable to create the phishing warden: [Exception... "Component returned failure code: 0x80570016 (NS_ERROR_XPC_GS_RETURNED_FAILURE) [nsIJSCID.getService]"  nsresult: "0x80570016 (NS_ERROR_XPC_GS_RETURNED_FAILURE)"  location: "JS frame :: file:///Users/mikeconley/Projects/comm-central/obj-x86_64-apple-darwin12.5.0/mozilla/dist/Daily.app/Contents/MacOS/components/nsUrlClassifierListManager.js :: PROT_ListManager :: line 82"  data: no]
Ah - I've noticed that the TB window opening up doesn't have the Mozmill or jsbridge extension installed. That's a problem.
Which version of Mozmill are you using?
Flags: needinfo?(hskupin)
We've been lagging in our upgrade. We're still at 1.5.16.
Ouch. You might want to check for compatibility if that harms you here. Not sure which versions we had at this time.
Hm, interesting. I would expect Mozmill incompatibilities to mean inability to run Mozmill at all - regardless of machine or OS.

We are, however, able to run these tests on our build machines, and I just confirmed that they run properly on my Ubuntu box. :/
(In reply to Magnus Melin from comment #3)
> Have you tried running it once more?
> 
> I always get the "ImportError: No module named mozcrash" the first time
> after i've built from scratch (on linux). 

That is bug 903616.
I have determined that the tests are not running on my OS X machine because the mozmillprofile directory has nothing under the extensions folder, where I'd normally expect jsbridge and the mozmill add-on.
Piecing through the mozrunner stuff, it looks like the profile is being created, and then the add-ons *are* being properly copied into it, but then they either get removed, or the entire profile gets overwritten with a new one without the add-ons.
These console errors seem related:

Timestamp: 2013-12-22 12:52:16 PM
Warning: WARN addons.xpi: Add-on is invalid: [Exception... "Component returned failure code: 0x80570015 (NS_ERROR_XPC_CI_RETURNED_FAILURE) [nsIJSCID.createInstance]"  nsresult: "0x80570015 (NS_ERROR_XPC_CI_RETURNED_FAILURE)"  location: "JS frame :: resource://gre/modules/XPIProvider.jsm :: loadManifestFromRDF :: line 908"  data: no]
Source File: resource://gre/modules/XPIProvider.jsm
Line: 908

So it looks like instantiating an nsIRandomGenerator is not possible anymore.

let rng = Cc["@mozilla.org/security/random-generator;1"].
            createInstance(Ci.nsIRandomGenerator);

Works in Firefox, but not in Thunderbird. This is causing XPIProvider.jsm to fail out, and the add-ons are being cleared from the profile.
Yep, this seems to be the root of the problem, at least for me. I can create the nsIRandomGenerator on Linux, but no dice on OS X.
So this is where I reach the limit of my expertise here. I've entered XPCOM/XPConnect land, where I'm far less skilled.

Does anybody know why XPIProvider.jsm can't create an instance of Ci.nsIRandomGenerator?
Ok, spent this afternoon neck deep in gdb, and I've determined that NSS is not initializing when starting TB from the Mozmill harness.

NSPR logging for "pipnss" gave me this:

2045260160[100563070]: nsNSSComponent::ctor
2045260160[100563070]: Beginning NSS initialization
2045260160[100563070]: nsNSSComponent: adding observers
2045260160[100563070]: nsNSSComponent::InitializeNSS
2045260160[100563070]: NSS Initialization beginning
2045260160[100563070]: can not init NSS r/w in /Users/mikeconley/Projects/comm-central/objdir-tb/mozilla/_tests/mozmill/mozmillprofile
2045260160[100563070]: can not init in r/o either
2045260160[100563070]: Unable to Initialize NSS.
2045260160[100563070]: nsNSSComponent: removing observers
2045260160[100563070]: nsNSSComponent::dtor
2045260160[100563070]: nsNSSComponent::ShutdownNSS
2045260160[100563070]: nsNSSComponent::dtor finished

So I *think* something's up with the Mozmill profile directory, and NSS is flipping out about it.
So at the point where NSS fails to initialize, I've determined that the profile directory has the correct read/write permissions (rwxr-xr-x).

Digging even deeper, nss_InitModules is failing when it attempts to load a module with the following spec:

name="PSM Internal Crypto Services" parameters="configdir='/Users/mikeconley/Projects/comm-central/objdir-tb/mozilla/_tests/mozmill/mozmillprofile' certPrefix='' keyPrefix='' secmod='secmod.db' flags=optimizeSpace updatedir='' updateCertPrefix='' updateKeyPrefix='' updateid='' updateTokenDescription=''  manufacturerID='Mozilla.org' libraryDescription='PSM Internal Crypto Services' cryptoTokenDescription='Generic Crypto Services' dbTokenDescription='Software Security Device' cryptoSlotDescription='PSM Internal Cryptographic Services' dbSlotDescription='PSM Private Keys' FIPSSlotDescription='Software Security Device (FIPS)' FIPSTokenDescription='FIPS 140 Cryptographic, Key and Certificate Services' minPS=0" NSS="flags=internal,moduleDB,moduleDBOnly,critical,defaultModDB,internalKeySlot"
I've determined that when running Thunderbird via Mozmill, NSS attempts to load the following (missing) file path when initting NSS:

objdir-tb/mozilla/security/build/libsoftokn3.dylib


When running Thunderbird normally, it loads this (valid) file instead:

objdir-tb/mozilla/dist/Daily.app/Contents/MacOS/libsoftokn3.dylib

So I think that's our answer. Either we need libsoftoken3.dylib needs to show up under objdir-tb/mozilla/security/build/, or Mozmill needs to load TB so that it looks in 
objdir-tb/mozilla/dist/Daily.app/Contents/MacOS/.
The core issue here is bug 879198 - I think it only affects Mac.

I've been working around it with:

rm mozilla/dist/DailyDebug.app/Contents/MacOS/libnss3.dylib && cp /Users/moztest/comm/main/tb/mozilla/security/build/libnss3.dylib mozilla/dist/DailyDebug.app/Contents/MacOS/libnss3.dylib

in the objdir.
Depends on: 879198
Flags: needinfo?(Pidgeot18)
No comments in bug 951834 since Mark's Comment 20.

Is this and bug 951834 still an issue??
Nobody has reported any problems with running mozmill tests locally for a long time. No problems running the tests from the bug description.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.