903034 - B2G desktop mochitests are broken

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Andrew Halberstadt [:ahal]]

They were busted by the changes I made in bug 835930. Fix coming shortly.

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: b2g_desktop

So this patch gets them running for me at least... but I still get a weird JS error when running the start script. I wonder if it's a problem with my environment/how I'm running them.

Jgriffin, do you mind applying and taking a look? I've never actually run b2g desktop before, so I don't have a baseline to compare against :/. Fyi, you'll probably need to manually specify --appname (dist/bin/b2g) and --utility-path (dist/bin) due to no longer using automation.py. I'll fix this when I make the mach target.

Comment 2

[Jonathan Griffin (:jgriffin)]

I get:

JavascriptException: SyntaxError: invalid regular expression flag j
	stacktrace:
	execute_script @runtestsb2g.py, line 237
	inline javascript, line 
	src: "undefined"

Comment 3

[Jonathan Griffin (:jgriffin)]

This is because we're calling execute_script with a filename, instead of the actual script.

Comment 4

[Jonathan Griffin (:jgriffin)]

(In reply to Jonathan Griffin (:jgriffin) from comment #3)
> This is because we're calling execute_script with a filename, instead of the
> actual script.

After fixing this, I get:

JavascriptException: TypeError: mm is null
	stacktrace:
	execute_script @runtestsb2g.py, line 238
	inline javascript, line 41
	src: "mm.addMessageListener("SPPrefService", specialPowersObserver);"

This is because b2g desktop build is not multi-process by default.  Were we loading this same script for b2g desktop tests before the mozprofile conversion?

Comment 5

[Andrew Halberstadt [:ahal]]

Ah, thanks! Heh, the 'a' and 'j' are from /home/ahal or /home/jgriffin. I thought b2gautomation detected whether it was a string or file automatically.

For the second problem, no, I didn't make any changes to that script, so I'm not sure what would be different. I'll take a closer look (at least now I know that it's not something wrong with my environment).

Comment 6

[Jonathan Griffin (:jgriffin)]

So, before the refactor, the desktop execution path (non-OOP) didn't include the message manager code:

http://hg.mozilla.org/mozilla-central/annotate/d04d512c25d9/testing/mochitest/runtestsb2g.py

Comment 7

[Andrew Halberstadt [:ahal]]

Attachment: Patch 1.0 - fix b2g desktop mochitest

I even had the outOfProcess variable defined at the top of the script already. I must have either forgotten to come back to it, or messed up a merge conflict. Anyway thanks for debugging!

With this patch I can get the mochitests to start running, but they shortly thereafter time out with:

31 INFO TEST-PASS | /tests/Harness_sanity/test_SpecialPowersExtension.html | Set property correctly on Xray-wrapped DOM object
32 INFO TEST-PASS | /tests/Harness_sanity/test_SpecialPowersExtension.html | Got property correctly on Xray-wrapped DOM object
33 INFO TEST-INFO | /tests/Harness_sanity/test_SpecialPowersExtension.html |
Profile::SpecialPowersRunTime: 28

34 INFO TEST-PASS | /tests/Harness_sanity/test_SpecialPowersExtension.html | check mock permission prompt
System JS : ERROR chrome://specialpowers/content/SpecialPowersObserverAPI.js:98
                     ReferenceError: Ci is not defined
System JS : ERROR chrome://specialpowers/content/specialpowersAPI.js:551
                     TypeError: crashDumpFiles is null

It could just be that they are broken normally. Jgriffin, if you think this is still caused by my patch, feel free to disregard this review.

Comment 8

[Jonathan Griffin (:jgriffin)]

Comment on attachment 789698 [details] [diff] [review]
Patch 1.0 - fix b2g desktop mochitest

Review of attachment 789698 [details] [diff] [review]:
-----------------------------------------------------------------

Cool, thanks.  The harness-sanity failures are known, and in fact they're not included in b2g.json.

Comment 9

[Andrew Halberstadt [:ahal]]

https://hg.mozilla.org/integration/b2g-inbound/rev/e5c726f690f8

Comment 10

[Andrew Halberstadt [:ahal]]

Oops, didn't mean to resolve.

Comment 11

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/e5c726f690f8

Comment 12

[Yuan Xulei [:yxl] (Leave from May 18)]

I really love mach, but it is still broken on Ubuntu. 

Ran the following command from the root directory of mozilla-central source code:

./mach mochitest-plain base/test/

b2g-desktop got stuck and the console outputted the following error:

 0:06.16 * Call to xpconnect wrapped JSObject produced this error:  *
 0:06.16 [Exception... "Component returned failure code: 0x8000ffff (NS_ERROR_UNEXPECTED) [nsIPrefBranch.getCharPref]"  nsresult: "0x8000ffff (NS_ERROR_UNEXPECTED)"  location: "JS frame :: chrome://browser/content/shell.js :: shell.homeURL :: line 185"  data: no]

The source code related to the error is (https://hg.mozilla.org/mozilla-central/file/44b0383e1063/b2g/chrome/content/shell.js#l196):

  get homeURL() {
    try {
      let homeSrc = Services.env.get('B2G_HOMESCREEN');
      if (homeSrc)
        return homeSrc;
    } catch (e) {}

    return Services.prefs.getCharPref('browser.homescreenURL');
  },

shell.js failed to get the pref value of |browser.homescreenURL|.

I opened the user.js and pref.js in the temporary profile created by mach and found no pref of |browser.homescreenURL|.

I hope these information can help with solving this bug. If any other information is needed, please let me know.

Comment 13

[Jonathan Griffin (:jgriffin)]

You can't use mach to run b2g desktop mochitests; they're run completely different than the standard Firefox tests.

See https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Platform/Automated_testing/Mochitests for directions on how to run them.

Eventually, we'll probably want to update the desktop targets to be b2g aware.

Comment 14

[Yuan Xulei [:yxl] (Leave from May 18)]

Sorry, I opened the wrong bug. But I followed the instruction https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Platform/Automated_testing/Mochitests and still can't get mozilla-central trunk tree work on Ubuntu 12.10 64bit.

Since I want to test on trunk tree, I tried the following steps as the wiki mentioned:

 cd $GECKO_OBJ_DIR
 source _virtualenv/bin/activate
 cd _tests/testing/mochitest
 python runtestsb2g.py --desktop --profile $GAIA/profile-debug --app $GECKO_OBJ_DIR/dist/bin/b2g --test-manifest b2g.json

got errors:

 Traceback (most recent call last):
   ...
   File "/home/yuan/gecko_ime/obj-b2g/_tests/testing/mochitest/runtests.py", line 186, in __init__
     mozinfo.find_and_update_from_json(*dirs)
 AttributeError: 'module' object has no attribute 'find_and_update_from_json'

Then I tried to setup as mozilla-b2g18 trees:
 cd $GECKO_OBJ_DIR
 python $GECKO_SRC_DIR/python/virtualenv/virtualenv.py venv
 source venv/bin/activate
 cd $GECKO_SRC_DIR/testing/marionette/client
 python setup.py develop
 cd $GECKO_OBJ_DIR/_tests/testing/mochitest
 python runtestsb2g.py --desktop --profile $GAIA/profile-debug --app $GECKO_OBJ_DIR/dist/bin/b2g --test-manifest b2g.json

got errors:
 Traceback (most recent call last):
   File "runtestsb2g.py", line 380, in <module>
     main()
   File "runtestsb2g.py", line 375, in main
     run_desktop_mochitests(parser, options)
   File "runtestsb2g.py", line 368, in run_desktop_mochitests
     sys.exit(mochitest.runTests(options, onLaunch=mochitest.startTests))
   File "/home/yuan/gecko_ime/obj-b2g/_tests/testing/mochitest/runtests.py", line 562, in runTests
     self.startWebServer(options)
   File "/home/yuan/gecko_ime/obj-b2g/_tests/testing/mochitest/runtests.py", line 315, in startWebServer
     self.server.start()
   File "/home/yuan/gecko_ime/obj-b2g/_tests/testing/mochitest/runtests.py", line 85, in start
     "xpcshell" + mozinfo.info['bin_suffix'])
   File "/home/yuan/gecko_ime/obj-b2g/venv/lib/python2.7/posixpath.py", line 77, in join
     elif path == '' or path.endswith('/'):
 AttributeError: 'NoneType' object has no attribute 'endswith'

I tried to add --utility-path parameter to runtestsb2g.py to solve this error:
 python runtestsb2g.py --desktop --profile $GAIA/profile-debug --app $GECKO_OBJ_DIR/dist/bin/b2g --test-manifest b2g.json  --utility-path $GECKO_OBJ_DIR/dist/bin/

b2g-desktop started and got suck with a white screen:

INFO | automation.py | Launching: /home/yuan/gecko_ime/obj-b2g/dist/bin/xpcshell -g /home/yuan/gecko_ime/obj-b2g/dist/bin -v 170 -f /home/yuan/gecko_ime/obj-b2g/_tests/testing/mochitest/httpd.js -e "const _PROFILE_PATH = '/tmp/tmp9ZqhyJ'; const _SERVER_PORT = '8888'; const _SERVER_ADDR = '127.0.0.1'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false;" -f ./server.js
Mochitest INFO | runtests.py | Server pid: 8341
INFO | automation.py | Launching: /home/yuan/gecko_ime/obj-b2g/venv/bin/python /home/yuan/gecko_ime/obj-b2g/_tests/testing/mochitest/pywebsocket_wrapper.py -p 9988 -w /home/yuan/gecko_ime/obj-b2g/_tests/testing/mochitest -l /home/yuan/gecko_ime/obj-b2g/_tests/testing/mochitest/websock.log --log-level=debug --allow-handlers-outside-root-dir
Mochitest INFO | runtests.py | Websocket server pid: 8350
Mochitest INFO | runtests.py | Running tests: start.

INFO | automation.py | Launching: /home/yuan/gecko_ime/obj-b2g/dist/bin/certutil -N -d /tmp/tmp9ZqhyJ -f /tmp/tmp9ZqhyJ/.crtdbpw
INFO | automation.py | Launching: /home/yuan/gecko_ime/obj-b2g/dist/bin/certutil -A -i /home/yuan/gecko_ime/build/pgo/certs/pgoca.ca -d /tmp/tmp9ZqhyJ -f /tmp/tmp9ZqhyJ/.crtdbpw -n pgoca -t CT,,
INFO | automation.py | Launching: /home/yuan/gecko_ime/obj-b2g/dist/bin/certutil -A -i /home/yuan/gecko_ime/build/pgo/certs/jartests-object.ca -d /tmp/tmp9ZqhyJ -f /tmp/tmp9ZqhyJ/.crtdbpw -n jartests-object -t CT,,CT
INFO | automation.py | Launching: /home/yuan/gecko_ime/obj-b2g/dist/bin/certutil -A -i /home/yuan/gecko_ime/build/pgo/certs/bug483440-pk10oflo.ca -d /tmp/tmp9ZqhyJ -f /tmp/tmp9ZqhyJ/.crtdbpw -n bug483440-pk10oflo -t CT,,
INFO | automation.py | Launching: /home/yuan/gecko_ime/obj-b2g/dist/bin/pk12util -i /home/yuan/gecko_ime/build/pgo/certs/mochitest.client -w /tmp/tmp9ZqhyJ/.crtdbpw -d /tmp/tmp9ZqhyJ
pk12util: PKCS12 IMPORT SUCCESSFUL
INFO | automation.py | Launching: /home/yuan/gecko_ime/obj-b2g/dist/bin/certutil -A -i /home/yuan/gecko_ime/build/pgo/certs/bug483440-attack2b.ca -d /tmp/tmp9ZqhyJ -f /tmp/tmp9ZqhyJ/.crtdbpw -n bug483440-attack2b -t CT,,
INFO | automation.py | Launching: /home/yuan/gecko_ime/obj-b2g/dist/bin/certutil -A -i /home/yuan/gecko_ime/build/pgo/certs/evintermediate.ca -d /tmp/tmp9ZqhyJ -f /tmp/tmp9ZqhyJ/.crtdbpw -n evintermediate -t CT,,
INFO | automation.py | Launching: /home/yuan/gecko_ime/obj-b2g/dist/bin/certutil -A -i /home/yuan/gecko_ime/build/pgo/certs/evroot.ca -d /tmp/tmp9ZqhyJ -f /tmp/tmp9ZqhyJ/.crtdbpw -n evroot -t CT,,
INFO | automation.py | Launching: /home/yuan/gecko_ime/obj-b2g/dist/bin/certutil -A -i /home/yuan/gecko_ime/build/pgo/certs/bug483440-attack7.ca -d /tmp/tmp9ZqhyJ -f /tmp/tmp9ZqhyJ/.crtdbpw -n bug483440-attack7 -t CT,,
INFO | automation.py | Launching: /home/yuan/gecko_ime/obj-b2g/dist/bin/ssltunnel /tmp/tmp9ZqhyJ/ssltunnel.cfg
INFO | automation.py | SSL tunnel pid: 8360
INFO | automation.py | Launching: /home/yuan/gecko_ime/obj-b2g/dist/bin/b2g -profile /tmp/tmp9ZqhyJ/
INFO | automation.py | Application pid: 8361
Server listening on port 4443 with cert pgo server certificate
1378352378366	Marionette	INFO	MarionetteComponent loaded
ATTENTION: default value of option force_s3tc_enable overridden by environment.
OpenGL version detected: 300
OpenGL version detected: 300
PermissionsTable.jsm: expandPermissions: Unknown Permission: network-httpPermissionsTable.jsm: expandPermissions: Unknown Permission: network-tcpPermissionsInstaller.jsm: 'network-http' is not a valid Webapps permission name.PermissionsInstaller.jsm: 'network-tcp' is not a valid Webapps permission name.OpenGL version detected: 300
###################################### forms.js loaded
############################### browserElementPanning.js loaded
######################## BrowserElementChildPreload.js loaded

Comment 15

[Jonathan Griffin (:jgriffin)]

B2G mochitests explicitly do not work with a debug gaia profile; you must create a "regular" profile.  The MDN page says:

"You need to have manually generated a gaia profile (which you should already have if you've been running a B2G desktop build manually); this must be a non-DEBUG profile"

> Traceback (most recent call last):
>   ...
>   File "/home/yuan/gecko_ime/obj-b2g/_tests/testing/mochitest/runtests.py", line 186, in __init__
>     mozinfo.find_and_update_from_json(*dirs)
> AttributeError: 'module' object has no attribute 'find_and_update_from_json'

Can you include the entire stack trace along with the exact command you used here?

Comment 16

[Jonathan Griffin (:jgriffin)]

Also, the output of 'pip freeze' would be helpful, after running 'source _virtualenv/bin/activate'.

Comment 17

[Andrew Halberstadt [:ahal]]

(In reply to Yuan Xulei [:yxl] from comment #14)
> got errors:
> 
>  Traceback (most recent call last):
>    ...
>    File "/home/yuan/gecko_ime/obj-b2g/_tests/testing/mochitest/runtests.py",
> line 186, in __init__
>      mozinfo.find_and_update_from_json(*dirs)
>  AttributeError: 'module' object has no attribute 'find_and_update_from_json'

This is happening because the python package 'mozinfo' is out of date on your system. Try running 'pip install --upgrade mozinfo'. If that doesn't work, you could create a python virtualenv (https://pypi.python.org/pypi/virtualenv) and install all the required packages (including mozinfo) into there.