Add Mulet support to b2gdesktop Reftest runner

RESOLVED FIXED in mozilla35

Status

RESOLVED FIXED
4 years ago
7 months ago

People

(Reporter: jgriffin, Assigned: armenzg)

Tracking

unspecified
mozilla35
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(8 attachments, 19 obsolete attachments)

7.37 KB, patch
ochameau
: review+
Details | Diff | Splinter Review
523 bytes, patch
jlund
: review+
Details | Diff | Splinter Review
3.06 KB, patch
jlund
: review+
Details | Diff | Splinter Review
16.70 KB, patch
jlund
: review+
Details | Diff | Splinter Review
1.43 KB, patch
jlund
: review+
Details | Diff | Splinter Review
1.33 KB, patch
jlund
: review+
Details | Diff | Splinter Review
12.75 KB, patch
jgriffin
: review+
Details | Diff | Splinter Review
4.47 KB, patch
jlund
: review+
Details | Diff | Splinter Review
(Reporter)

Description

4 years ago
We want to be able to run reftests on Mulet the same way as we do for b2gdesktop...i.e., in side the test-container gaia app.

The current b2gdesktop runner completely fails with Mulet at present.  It looks like there are several reasons for this:

- There are some gaia services that make non-local network connections, so MOZ_DISABLE_NONLOCAL_CONNECTIONS causes us to crash
- Launching the test-container app by setting b2g.system_startup_url fails; we're going to need to launch it via Marionette, which also probably means having to kill the FTU and lockscreen via Marionette
- Mulet doesn't recognize the --screen 800x1000 argument
(Reporter)

Comment 1

4 years ago
Created attachment 8462225 [details] [diff] [review]
WIP -  - Add Mulet support to b2g reftest runner

WIP, this allows the runner to at least launch Mulet with Gaia running in it.
(Reporter)

Updated

4 years ago
Assignee: nobody → jgriffin
(Reporter)

Updated

4 years ago
Depends on: 1043723
(Reporter)

Comment 2

4 years ago
I don't know how to cope with the fact that Mulet doesn't honor the --screen 800x1000 argument.  Alexandre, can we add support for this?  Reftests require this resolution and many will give false positives or negatives when run at the normal Gaia screensize.
Flags: needinfo?(poirot.alex)
Created attachment 8476005 [details] [diff] [review]
Ensure that -chrome URI doesn't prevent window.arguments[0] to be nsICommandLine

The code to handle --screen is in Mulet, but the -chrome chrome://b2g/content/shell.html
arguments goes through another startup codepath which prevent window.arguments[0]
to be set to the nsICommandLine object! It ends up breaking b2g code implementing --screen argument.
Flags: needinfo?(poirot.alex)
(Reporter)

Updated

4 years ago
Attachment #8476005 - Flags: review?(gavin.sharp)
Comment on attachment 8476005 [details] [diff] [review]
Ensure that -chrome URI doesn't prevent window.arguments[0] to be nsICommandLine

This almost certainly breaks Firefox, which doesn't expect its windows to receive a reference to the cmdLine object. If b2g expects that, it needs to adjust, or we need a special case.
Attachment #8476005 - Flags: review?(gavin.sharp) → review-
Created attachment 8480521 [details] [diff] [review]
Fix command line arguments handling on desktop

Given comment 4, it looks like window.arguments[0] being nsICommandLine
is an unexpected behavior...
So instead of fixing that to an expected behavior, I'm now working around this issue.
Attachment #8476005 - Attachment is obsolete: true
Attachment #8480521 - Flags: review?(fabrice)
Attachment #8480521 - Flags: review?(fabrice) → review+
Created attachment 8480704 [details] [diff] [review]
Fix command line arguments handling on desktop

Forgot to add CommandLine.js in package-manifest.in...

https://tbpl.mozilla.org/?tree=Try&rev=e45bd7303b25
Attachment #8480521 - Attachment is obsolete: true
Attachment #8480704 - Flags: review+
Try is finally green!
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/4d991522bbe3
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla35
(Reporter)

Comment 11

4 years ago
Forgot to 'leave-open'; there's still more work to be done.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Reporter)

Updated

4 years ago
Blocks: 943878
(Assignee)

Updated

4 years ago
Assignee: jgriffin → armenzg
(Assignee)

Updated

4 years ago
Depends on: 1053185
(Assignee)

Comment 12

4 years ago
Created attachment 8488673 [details] [diff] [review]
mulet.diff

With jgriffin's patch we get it started, however, we have the lock screen.

Can marionette help me unlock the screen?

For reference, here's what it shows without the patch:
http://people.mozilla.org/~armenzg/sattap/86644eb8.png

wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/alissy@mozilla.com-b1950fdbd196/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tests.zip
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/alissy@mozilla.com-b1950fdbd196/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tar.bz2
tar xvfj firefox-35.0a1.en-US.linux-x86_64.tar.bz2
virtualenv temp_venv
source temp_venv/bin/activate
pip install -e marionette/marionette
python reftest/runreftestb2g.py --desktop --profile=/home/armenzg/repos/branches/gaia/profile --appname=/home/armenzg/moz/tmp/mulet/firefox/firefox-bin --symbols-path=http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/alissy@mozilla.com-b1950fdbd196/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols.zip reftest/tests/layout/reftests/reftest.list
Attachment #8462225 - Attachment is obsolete: true
(Reporter)

Comment 13

4 years ago
Yes, you can use Marionette to unlock the lock screen.

There is already a script executed via Marionette for B2G reftests:  http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/b2g_start_script.js

I think you could adjust this to include code to unlock the lockscreen, namely the 'unlock' method from here:  https://github.com/mozilla-b2g/gaia/blob/master/tests/atoms/gaia_lock_screen.js
(Assignee)

Comment 14

4 years ago
Created attachment 8488849 [details] [diff] [review]
mulet.diff

I added gaia_lock_screen.js temporarily by b2g_desktop.py and I called it like this:
+        self.marionette.import_script(os.path.abspath(
+            os.path.join(__file__, os.path.pardir, "gaia_lock_screen.js")))
+        self.marionette.switch_to_frame()
+        self.marionette.execute_async_script('GaiaLockScreen.unlock()')

I was hoping to try to emulate this code:
https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/gaia_test.py#L682

There is also this that uses executeScript() instead
https://github.com/mozilla-b2g/gaia/blob/master/tests/js-marionette/helper.js#L7

My apologies it is clear on how to proceed. I've not deal much with JavaScript in the past. I should be able to make progress with some pointers.

However, I'm getting this:
1410556331093	Marionette	DEBUG	accepted connection on 127.0.0.1:44664
1410556336101	Marionette	DEBUG	accepted connection on 127.0.0.1:44665
1410556336127	Marionette	INFO	loaded marionette-listener.js
1410556336127	Marionette	INFO	sendToClient: {"from":"0","value":{"browserName":"B2G","browserVersion":"35.0a1","platformName":"LINUX","platformVersion":"35.0a1","handlesAlerts":false,"nativeEvents":false,"rotatable":true,"secureSsl":false,"takesElementScreenshot":true,"takesScreenshot":true,"platform":"LINUX","XULappId":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","appBuildId":"20140911230945","device":"desktop","version":"35.0a1","b2g":true}}, {ec6b34c4-cc4a-4228-9fba-e05353186962}, {ec6b34c4-cc4a-4228-9fba-e05353186962}
1410556336130	Marionette	DEBUG	Got request: setContext, data: {"to":"0","sessionId":{"rotatable":true,"browserVersion":"35.0a1","takesScreenshot":true,"appBuildId":"20140911230945","XULappId":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","secureSsl":false,"platform":"LINUX","browserName":"B2G","version":"35.0a1","device":"desktop","b2g":true,"nativeEvents":false,"platformVersion":"35.0a1","takesElementScreenshot":true,"platformName":"LINUX","handlesAlerts":false},"name":"setContext","parameters":{"value":"chrome"}}, id: {4e2aa646-d696-44dd-8f1c-091b9416e8e0}
1410556336130	Marionette	INFO	sendToClient: {"from":"0","ok":true}, {4e2aa646-d696-44dd-8f1c-091b9416e8e0}, {4e2aa646-d696-44dd-8f1c-091b9416e8e0}
1410556336132	Marionette	INFO	sendToClient: {"from":"0","ok":true}, {268a247d-8b5b-42d9-90f6-b13573756a84}, {268a247d-8b5b-42d9-90f6-b13573756a84}
1410556336133	Marionette	DEBUG	Got request: switchToFrame, data: {"to":"0","sessionId":{"rotatable":true,"browserVersion":"35.0a1","takesScreenshot":true,"appBuildId":"20140911230945","XULappId":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","secureSsl":false,"platform":"LINUX","browserName":"B2G","version":"35.0a1","device":"desktop","b2g":true,"nativeEvents":false,"platformVersion":"35.0a1","takesElementScreenshot":true,"platformName":"LINUX","handlesAlerts":false},"name":"switchToFrame","parameters":{"id":null,"focus":true}}, id: {5ecb34f3-5a79-429e-b5ec-ac340cbebf07}
1410556336234	Marionette	INFO	sendToClient: {"from":"0","ok":true}, {5ecb34f3-5a79-429e-b5ec-ac340cbebf07}, {5ecb34f3-5a79-429e-b5ec-ac340cbebf07}
1410556336238	Marionette	DEBUG	Got request: executeWithCallback, data: {"to":"0","sessionId":{"rotatable":true,"browserVersion":"35.0a1","takesScreenshot":true,"appBuildId":"20140911230945","XULappId":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","secureSsl":false,"platform":"LINUX","browserName":"B2G","version":"35.0a1","device":"desktop","b2g":true,"nativeEvents":false,"platformVersion":"35.0a1","takesElementScreenshot":true,"platformName":"LINUX","handlesAlerts":false},"name":"executeAsyncScript","parameters":{"scriptTimeout":null,"specialPowers":false,"script":"GaiaLockScreen.unlock()","newSandbox":true,"args":[],"filename":"b2g_desktop.py","line":44}}, id: {7585ad9a-0d1c-454c-9805-631b56ea66a0}
1410556336242	Marionette	INFO	sendToClient: {"from":"0","error":{"message":"TypeError: window.wrappedJSObject.SettingsListener.getSettingsLock is not a function","status":17,"stacktrace":"execute_async_script @b2g_desktop.py, line 44\ninline javascript, line 10\nsrc: \"undefined\""}}, {7585ad9a-0d1c-454c-9805-631b56ea66a0}, {7585ad9a-0d1c-454c-9805-631b56ea66a0}
PROCESS-CRASH | b2g_desktop.py | application crashed [unknown top frame]
Crash dump filename: /tmp/tmps6jzGQ/minidumps/1b6fce7d-f369-4d0f-22db530f-198a27dd.dmp
No symbols path given, can't process dump.
MINIDUMP_STACKWALK not set, can't process dump.
Traceback (most recent call last):
  File "reftest/runreftestb2g.py", line 611, in <module>
    sys.exit(main())
  File "reftest/runreftestb2g.py", line 606, in main
    return run_desktop_reftests(parser, options, args)
  File "/home/armenzg/moz/tmp/mulet/reftest/b2g_desktop.py", line 191, in run_desktop_reftests
    sys.exit(reftest.run_tests(args[0], options))
  File "/home/armenzg/moz/tmp/mulet/reftest/b2g_desktop.py", line 99, in run_tests
    self.run_marionette_script()
  File "/home/armenzg/moz/tmp/mulet/reftest/b2g_desktop.py", line 44, in run_marionette_script
    self.marionette.execute_async_script('GaiaLockScreen.unlock()')
  File "/home/armenzg/moz/tmp/mulet/marionette/marionette/marionette.py", line 1253, in execute_async_script
    filename=os.path.basename(frame[0]))
  File "/home/armenzg/moz/tmp/mulet/marionette/marionette/decorators.py", line 35, in _
    return func(*args, **kwargs)
  File "/home/armenzg/moz/tmp/mulet/marionette/marionette/marionette.py", line 638, in _send_message
    self._handle_error(response)
  File "/home/armenzg/moz/tmp/mulet/marionette/marionette/marionette.py", line 686, in _handle_error
    raise errors.JavascriptException(message=message, status=status, stacktrace=stacktrace)
marionette.errors.JavascriptException: JavascriptException: TypeError: window.wrappedJSObject.SettingsListener.getSettingsLock is not a function
stacktrace:
	execute_async_script @b2g_desktop.py, line 44
	inline javascript, line 10
	src: "undefined"
Attachment #8488673 - Attachment is obsolete: true
(Reporter)

Comment 15

4 years ago
Comment on attachment 8488849 [details] [diff] [review]
mulet.diff

Review of attachment 8488849 [details] [diff] [review]:
-----------------------------------------------------------------

I think this is very close, just a chrome/content mismatch here!  Nice work!

::: ../mulet.original/reftest/b2g_desktop.py
@@ +36,5 @@
>      def run_marionette_script(self):
>          self.marionette = Marionette(**self.marionette_args)
>          assert(self.marionette.wait_for_port())
>          self.marionette.start_session()
>          self.marionette.set_context(self.marionette.CONTEXT_CHROME)

The lockscreen is accessible only in content context, so you'll need to remove this line that sets the context to CONTEXT_CHROME (context defaults to content), and then add this line back after you execute the GaiaLockScreen.unlock() method.

@@ +92,5 @@
>              log.info("%s | Application pid: %d",
>                       os.path.basename(__file__),
>                       self.runner.process_handler.pid)
> +            import time
> +            time.sleep(15)

If this is still necessary after fixing the context problem above, we'll have to add code to wait for the homescreen.  See http://mxr.mozilla.org/mozilla-central/source/testing/web-platform/harness/wptrunner/browsers/b2g.py#233 for an example of how this can be done.
(Assignee)

Comment 16

4 years ago
Created attachment 8488862 [details] [diff] [review]
mulet.diff

I was loosing hope and was going to study deeper!

It works (I see the screen being unlocked) and it seems the sleep is not needed.

I'm now hitting:
1410557694318	Marionette	INFO	sendToClient: {"from":"0","error":{"message":"NS_ERROR_FILE_NOT_FOUND: Component returned failure code: 0x80520012 (NS_ERROR_FILE_NOT_FOUND) [nsIXPCComponents_Utils.import]","status":17,"stacktrace":"execute_script @b2g_desktop.py, line 57\ninline javascript, line 100\nsrc: \"Cu.import(\"chrome://reftest/content/reftest.jsm\", reftest);\""}}, {ebb7e00c-beeb-449b-80a3-059b38009f8a}, {ebb7e00c-beeb-449b-80a3-059b38009f8a}
PROCESS-CRASH | b2g_desktop.py | application crashed [unknown top frame]
Crash dump filename: /tmp/tmpNLLpCD/minidumps/05be21c9-af6e-d77c-2a4fd4ca-79da497b.dmp
No symbols path given, can't process dump.
MINIDUMP_STACKWALK not set, can't process dump.
Traceback (most recent call last):
  File "reftest/runreftestb2g.py", line 611, in <module>
    sys.exit(main())
  File "reftest/runreftestb2g.py", line 606, in main
    return run_desktop_reftests(parser, options, args)
  File "/home/armenzg/moz/tmp/mulet/reftest/b2g_desktop.py", line 196, in run_desktop_reftests
    sys.exit(reftest.run_tests(args[0], options))
  File "/home/armenzg/moz/tmp/mulet/reftest/b2g_desktop.py", line 104, in run_tests
    self.run_marionette_script()
  File "/home/armenzg/moz/tmp/mulet/reftest/b2g_desktop.py", line 57, in run_marionette_script
    self.marionette.execute_script(self.test_script)
  File "/home/armenzg/moz/tmp/mulet/marionette/marionette/marionette.py", line 1205, in execute_script
    filename=os.path.basename(frame[0]))
  File "/home/armenzg/moz/tmp/mulet/marionette/marionette/decorators.py", line 35, in _
    return func(*args, **kwargs)
  File "/home/armenzg/moz/tmp/mulet/marionette/marionette/marionette.py", line 638, in _send_message
    self._handle_error(response)
  File "/home/armenzg/moz/tmp/mulet/marionette/marionette/marionette.py", line 686, in _handle_error
    raise errors.JavascriptException(message=message, status=status, stacktrace=stacktrace)
marionette.errors.JavascriptException: JavascriptException: NS_ERROR_FILE_NOT_FOUND: Component returned failure code: 0x80520012 (NS_ERROR_FILE_NOT_FOUND) [nsIXPCComponents_Utils.import]
stacktrace:
	execute_script @b2g_desktop.py, line 57
	inline javascript, line 100
	src: "Cu.import("chrome://reftest/content/reftest.jsm", reftest);"
Attachment #8488849 - Attachment is obsolete: true
(Assignee)

Comment 17

4 years ago
I have managed to run reftests. [1]

We now have to figure out where we should put the gaia lock screen JS code and meanwhile I will work locally on my build environment to generate reftest.jsm when building mulet.

I put the gaia lock screen JS code beside the b2g_desktop.py code. Let me know if that would be the right place.

[1]
* I downloaded https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-emulator/20140912061053/gaia-tests.zip
* I copied reftest.jsm from it
cp b2g_tests/reftest/reftest/chrome/reftest/content/reftest.jsm mulet/reftest/reftest/chrome/reftest/content/
(Reporter)

Comment 18

4 years ago
> I put the gaia lock screen JS code beside the b2g_desktop.py code. Let me know if that would be the 
> right place.

Yes, I think that's fine.
(Assignee)

Comment 19

4 years ago
Created attachment 8489634 [details] [diff] [review]
mulet_reftests.diff

armenzg-thinkpad mozilla-central hg:[default!] $ find obj-dir-mulet/ -name reftest.js
obj-dir-mulet/dist/bin/chrome/reftest/content/reftest.js
armenzg-thinkpad mozilla-central hg:[default!] $ find obj-dir-mulet/ -name gaia_lock_screen.js
obj-dir-mulet/_tests/reftest/gaia_lock_screen.js

Pushed to try:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=ae607740c4f4
Attachment #8488862 - Attachment is obsolete: true
Attachment #8489634 - Flags: feedback?(jgriffin)
(Assignee)

Comment 20

4 years ago
Created attachment 8490106 [details] [diff] [review]
mulet_reftests.diff

This is the proper patch. I pushed something wrong:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=69fa463786c1
Attachment #8489634 - Attachment is obsolete: true
Attachment #8489634 - Flags: feedback?(jgriffin)
Comment on attachment 8490106 [details] [diff] [review]
mulet_reftests.diff

Review of attachment 8490106 [details] [diff] [review]:
-----------------------------------------------------------------

::: layout/tools/reftest/b2g_desktop.py
@@ +61,5 @@
>          env = self.buildBrowserEnv(options, self.profile.profile)
> +
> +        # FIXME
> +        if "MOZ_DISABLE_NONLOCAL_CONNECTIONS" in env:
> +            del env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"]

This will need to be removed for the jobs to be unhidden, seeing as this is a new job, per:
https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#8.29_Must_avoid_patterns_known_to_cause_non_deterministic_failures
(Assignee)

Comment 22

4 years ago
The build seems to generate something that works.

You can reproduce it locally by following the steps below [1]

With regards to mozharness changes, for now it seems that I need to:
1) install marionette
2) use runreftestb2g.py.
3) append --desktop
4) append --profile
The Firefox's mozharness reftest config is not good as is.

On another note, are the mulet mochitest jobs calling the right thing?
They seem to be calling mochitest/runtests.py instead of mochitest/runtestsb2g.py [3]

[1]
Check out a local clone of gaia and make a gaia profile (by running 'make' in 
the gaia dir)
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tests.zip
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tar.bz2
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/jsshell-linux-x86_64.zip
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols.zip
unzip -q firefox-35.0a1.en-US.linux-x86_64.tests.zip
tar xvfj firefox-35.0a1.en-US.linux-x86_64.tar.bz2
virtualenv temp_venv
source temp_venv/bin/activate
pip install -e marionette
python reftest/runreftestb2g.py --desktop --profile=/home/armenzg/repos/branches/gaia/profile --appname=firefox/firefox-bin reftest/tests/layout/reftests/reftest.list

[2]
python scripts/desktop_unittest.py --cfg unittests/linux_unittest.py --reftest-suite reftest --blob-upload-branch mozilla-central --download-symbols ondemand --test-url ~/moz/tmp/mulet_latest/firefox-35.0a1.en-US.linux-x86_64.tests.zip --installer-url ~/moz/tmp/mulet_latest/firefox-35.0a1.en-US.linux-x86_64.tar.bz2 --cfg developer_config.py --jsshell-url ~/moz/tmp/mulet_latest/jsshell-linux-x86_64.zip


python', '-u', '/home/armenzg/repos/mozharness/build/tests/reftest/runreftestb2g.py', '--appname=/home/armenzg/repos/mozharness/build/application/firefox/firefox', '--utility-path=tests/bin', '--extra-profile-file=tests/bin/plugins', '--symbols-path=/home/armenzg/moz/tmp/mulet_latest/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols.zip', 'tests/reftest/tests/layout/reftests/reftest.list'

[3]
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/try_ubuntu64_vm_mulet_test-mochitest-1-bm115-tests1-linux64-build149.txt.gz
10:05:45     INFO - Calling ['/builds/slave/test/build/venv/bin/python', '-u', '/builds/slave/test/build/tests/mochitest/runtests.py', '--total-chunks', '5', '--this-chunk', '1', '--appname=/builds/slave/test/build/application/firefox/firefox', '--utility-path=tests/bin', '--extra-profile-file=tests/bin/plugins', '--symbols-path=https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols.zip', '--certificate-path=tests/certs', '--autorun', '--close-when-done', '--console-level=INFO', '--setpref=webgl.force-enabled=true', '--quiet', '--log-raw=/builds/slave/test/build/blobber_upload_dir/raw_structured_logs.log', '--use-test-media-devices', '--chunk-by-dir=4'] with output_timeout 1000
(Reporter)

Comment 23

4 years ago
> On another note, are the mulet mochitest jobs calling the right thing?
> They seem to be calling mochitest/runtests.py instead of mochitest/runtestsb2g.py [3]

Probably they're not.  Those jobs were setup with people unfamiliar with the way we run mochitests on B2G, so what we have running is the desktop flavor.  This is fine, except that those tests don't run OOP, which we probably want.  So, I think we'll need to move to B2G-style mochitests for Mulet.
(Assignee)

Comment 24

4 years ago
Jordan, I have managed to the most extent be able to use desktop_unittest.py to run mulet reftests, however, there's one step that does not fit properly inside of desktop_unittest.py.
The step requires to checkout gaia and run a "make" command.

What do you suggest doing?
I'm thinking of creating a mulet_unittest.py and have a simplified version of desktop_unittest.py that also checkout gaia.

I'm also considering a few days experiment where the script and config to deal with mulet lives in the tree and the mozharness code does a very setup and lets the script from the tree do the rest.
This would lay ground for a data point for next week's mozharness conversations.

What do you think?
Flags: needinfo?(jlund)
(Reporter)

Comment 25

4 years ago
Can we use b2g_desktop_unittest instead of desktop_unittest?
(Reporter)

Comment 26

4 years ago
(In reply to Jonathan Griffin (:jgriffin) from comment #25)
> Can we use b2g_desktop_unittest instead of desktop_unittest?

The way these tests run is much more similar to b2gdesktop than it is to Firefox.
(Assignee)

Comment 27

4 years ago
Alexandre,
To get this running we need to checkout gaia and make a profile.

Which branch should we test against as we move through the release channels?
Does this look correct?

* m-i, m-c, fx-team, try et trunk trees:
** http://hg.mozilla.org/integration/gaia-central
* m-a
** http://hg.mozilla.org/integration/gaia-2_1
* Mozilla-B2g32-v2.0
** http://hg.mozilla.org/integration/gaia-2_0
and on...

I'm not saying that we will enable it yet on those branches but I want to make sure that we want to follow the right versions versus sticking to gaia-central across trees.
Flags: needinfo?(poirot.alex)
(Reporter)

Comment 28

4 years ago
(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #27)
> Alexandre,
> To get this running we need to checkout gaia and make a profile.
> 
> Which branch should we test against as we move through the release channels?
> Does this look correct?
> 
> * m-i, m-c, fx-team, try et trunk trees:
> ** http://hg.mozilla.org/integration/gaia-central
> * m-a
> ** http://hg.mozilla.org/integration/gaia-2_1
> * Mozilla-B2g32-v2.0
> ** http://hg.mozilla.org/integration/gaia-2_0
> and on...
> 
> I'm not saying that we will enable it yet on those branches but I want to
> make sure that we want to follow the right versions versus sticking to
> gaia-central across trees.

See e.g., http://hg.mozilla.org/build/mozharness/file/0f41dc44292e/scripts/gaia_unit.py.  We have a GaiaTest mixin that automatically clones the correct branch of gaia:  http://hg.mozilla.org/build/mozharness/file/0f41dc44292e/mozharness/mozilla/testing/gaia_test.py  It also has a 'make_gaia' method that builds a gaia profile.
(Assignee)

Comment 29

4 years ago
Created attachment 8492358 [details] [diff] [review]
wip - mulet mozharness

With this patch and command I can get to checkout gaia and make the profile.

python scripts/mulet_unittest.py --cfg b2g/desktop_automation_config.py --blob-upload-branch mozilla-central --download-symbols ondemand --test-url ~/moz/tmp/mulet_latest/firefox-35.0a1.en-US.linux-x86_64.tests.zip --installer-url ~/moz/tmp/mulet_latest/firefox-35.0a1.en-US.linux-x86_64.tar.bz2 --cfg b2g/mulet_config.py --cfg developer_config.py

More next week.
Attachment #8490106 - Attachment is obsolete: true
> The step requires to checkout gaia and run a "make" command.
> 
> What do you suggest doing?
> I'm thinking of creating a mulet_unittest.py and have a simplified version
> of desktop_unittest.py that also checkout gaia.

inheriting from GaiaTest should allow you to do that. however I'm not sure if you need all that it comes with. it looks like there is a GaiaMixin that may do exactly what you want: clone/make

marionette already avails of that directly: http://mxr.mozilla.org/build/source/mozharness/scripts/marionette.py#31

GaiaMixin: http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/gaia.py#12

so you can do what marionette did and add it in to desktop_unittest.py. maybe b2g_desktop_unittest.py is more appropriate like jgriffin suggested?

> 
> I'm also considering a few days experiment where the script and config to
> deal with mulet lives in the tree and the mozharness code does a very setup
> and lets the script from the tree do the rest.
> This would lay ground for a data point for next week's mozharness
> conversations.

sure. depends on how we end up wanting to implement this but it could be a cool experiment :)

does this help?
Flags: needinfo?(jlund)
(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #27)
> Alexandre,
> To get this running we need to checkout gaia and make a profile.
> 
> Which branch should we test against as we move through the release channels?

We should be doing what we are already doing for other b2g projects (b2g-desktop/device/emulator).
(It looks like you already found the code that does that...)

> Does this look correct?

Otherwise, yes at first sight it looks correct.
(Just note that m-a won't stick to 2.1 and will switch to 2.2 someday)
Flags: needinfo?(poirot.alex)
(Assignee)

Comment 32

4 years ago
I have tried various way on how to re-use the right class, however, there are a bunch of issues with the various Gaia related classes.

I'm going to try not do any re-factoring, however, it has to be dealt in the future (bug 1071219).
(Assignee)

Comment 33

4 years ago
Created attachment 8493994 [details] [diff] [review]
wip - mulet mozharness

This can be run like this:
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tests.zip
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tar.bz2
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols.zip
hg clone http://hg.mozilla.org/integration/gaia-central gaia
python scripts/mulet_unittest.py --cfg b2g/generic_config.py --cfg b2g/mulet_config.py --blob-upload-branch mozilla-central --download-symbols ondemand --test-url firefox-35.0a1.en-US.linux-x86_64.tests.zip --installer-url firefox-35.0a1.en-US.linux-x86_64.tar.bz2 --cfg developer_config.py --test-suite reftest --test-manifest tests/layout/reftests/reftest.list

Known issues:
* The gaia checkout is never clobbered
* It does not yet checkout gaia by itself

I'm requesting for feedback so at least I can get an idea that I'm not too off from your expectations.

NOTE: I have tried subclassing many other classes with much pain and little luck. I hope this can meet the expectations.
Attachment #8492358 - Attachment is obsolete: true
Attachment #8493994 - Flags: feedback?(jlund)
(Assignee)

Comment 34

4 years ago
Created attachment 8494000 [details] [diff] [review]
[checked-in] mulet_in_tree_configs.diff

With the previous patch we can "run_file_names" from the tree.
(Assignee)

Comment 35

4 years ago
Created attachment 8494540 [details] [diff] [review]
wip - mozharness_mulet.diff

This patch can now checkout gaia for me. It will need to test other scripts that make use of GaiaTest and GaiaMixin.

Instructions on how to run this locally [1]

Known issues:
* The gaia checkout and the gaia profile are never clobbered
* [FIXED] It does not yet checkout gaia by itself

[1]
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tests.zip
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tar.bz2
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols.zip
python scripts/mulet_unittest.py --cfg b2g/generic_config.py --cfg b2g/mulet_config.py --blob-upload-branch mozilla-central --download-symbols ondemand --test-url firefox-35.0a1.en-US.linux-x86_64.tests.zip --installer-url firefox-35.0a1.en-US.linux-x86_64.tar.bz2 --cfg developer_config.py --test-suite reftest --test-manifest tests/layout/reftests/reftest.list --gaia-repo http://hg.mozilla.org/integration/gaia-central
Attachment #8493994 - Attachment is obsolete: true
Attachment #8493994 - Flags: feedback?(jlund)
Attachment #8494540 - Flags: feedback?(jlund)
(Assignee)

Comment 36

4 years ago
Created attachment 8494686 [details] [diff] [review]
[checked-in] add mulet reftests on Ash
Attachment #8494686 - Flags: review?(jlund)
Comment on attachment 8494686 [details] [diff] [review]
[checked-in] add mulet reftests on Ash

Review of attachment 8494686 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm. suppose we will do a tbpl patch after?
Attachment #8494686 - Flags: review?(jlund) → review+
Comment on attachment 8494540 [details] [diff] [review]
wip - mozharness_mulet.diff

Review of attachment 8494540 [details] [diff] [review]:
-----------------------------------------------------------------

lrgtm :D :D
one high level question is how come you did not do to B2GDesktopTest what MarionetteTest did? I know you mentioned that you were having problems but could you not have just added GaiaMixin to B2GDesktopTest, overwritten make_gaia like you did in mulet_unittest.py, and extended the other mulet_unittest.py methods (run_test and query_abs_dirs) to support the mulet edge case.

On one argument, classes are getting overloaded with logic, on the other, we have too many mixins and subclasses. Not sure I know what we should do so just polling your opinion.

Looks great :)

::: mozharness/mozilla/gaia.py
@@ +17,5 @@
> +     }],
> +    [["--gaia-repo"],
> +    {"action": "store",
> +     "dest": "gaia_repo",
> +     "default": None,

I don't think the 'default' lines are needed. if not given, I think it will default to None

@@ +239,5 @@
> +                # gaia_branch by default is set to default
> +                self.fatal("You're trying to run this outside of buildbot, " \
> +                    "therefore, you need to specify --gaia-repo.")
> +
> +    def pull(self, **kwargs):

looks like marionette and GaiaTest have their own pull action. In which case the preflight_pull here will be called. I guess that's OK? Maybe even a good thing

@@ +250,5 @@
> +        dirs = self.query_abs_dirs()
> +        dest = dirs['abs_gaia_dir']
> +        repo = {}
> +
> +        if self.buildbot_config is not None:

same as 'if self.buildbot_config'.

I think we normally: 'self.buildbot_config and 'properties' in self.buildbot_config:'

@@ +261,5 @@
> +        else:
> +            repo = {
> +                'repo_path': self.config['gaia_repo'],
> +                'revision': 'default',
> +                'branch': self.config['gaia_branch']

should we add 'gaia_branch to preflight_pull like gaia_repo?

::: mozharness/mozilla/testing/gaia_test.py
@@ +76,5 @@
>           "help": "Number of this chunk",
>           }
>      ]] + copy.deepcopy(testing_config_options) + \
> +        copy.deepcopy(blobupload_config_options) + \
> +        copy.deepcopy(gaia_config_options)

good call :)

::: mozharness/mozilla/testing/testbase.py
@@ +113,5 @@
>                                              create_parent_dir=create_parent_dir,
>                                              error_level=error_level,
>                                              exit_code=exit_code)
>  
> +    def query_value(self, key):

I suspect that as you go, this method will make it closer and closer to the internals of mozharness. :)

::: scripts/b2g_desktop_unittest.py
@@ +70,5 @@
>          {'substr': 'Timeout waiting for marionette on port', 'level': ERROR},
>          {'regex': re.compile(r'''(Timeout|NoSuchAttribute|Javascript|NoSuchElement|XPathLookup|NoSuchWindow|StaleElement|ScriptTimeout|ElementNotVisible|NoSuchFrame|InvalidElementState|NoAlertPresent|InvalidCookieDomain|UnableToSetCookie|InvalidSelector|MoveTargetOutOfBounds)Exception'''), 'level': ERROR},
>      ]
>  
> +    def __init__(self, config=None, require_config_file=False):

I think we need to add GaiaMixin to this class no?

@@ +75,2 @@
>          super(B2GDesktopTest, self).__init__(
> +            config_options=self.config_options + copy.deepcopy(config),

if config is None, it will error trying to concat to a list

@@ +99,5 @@
>          self.test_url = c.get('test_url')
>          self.test_manifest = c.get('test_manifest')
>  
> +        suite = self.config['test_suite']
> +        if suite not in self.test_suites:

we want to support tree_config['suite'] or self.config['test_suite'] right? so if it's the former on a given script run, this test will always fatal right?

That is unless we pass suite to self.config *and* self.tree_config every time. but if we are moving to tree_config, I think we should drop self.config['suite'] altogether no?

::: scripts/mulet_unittest.py
@@ +18,5 @@
> +from mozharness.base.transfer import TransferMixin
> +from mozharness.mozilla.gaia import GaiaMixin, gaia_config_options
> +from b2g_desktop_unittest import B2GDesktopTest
> +
> +class MuletUnittest(B2GDesktopTest, GaiaMixin, TransferMixin):

since pull is defined in GaiaMixin, I think we should add GaiaMixin to its definition as I mentioned above

@@ +22,5 @@
> +class MuletUnittest(B2GDesktopTest, GaiaMixin, TransferMixin):
> +    test_suites = ('reftest',)
> +
> +    def __init__(self):
> +        super(MuletUnittest, self).__init__(config=gaia_config_options)

not sure if config is the best name for this. could we add options to the name?

@@ +52,5 @@
> +
> +    def preflight_run_tests(self):
> +        if self.config.get("binary_path"):
> +            self.binary_path = self.config["binary_path"]
> +        assert self.binary_path, \

asserts will skip log placement. I think self.fatal would be better as this is something we can only test at runtime
Attachment #8494540 - Flags: feedback?(jlund) → feedback+
(Assignee)

Comment 39

4 years ago
Comment on attachment 8494686 [details] [diff] [review]
[checked-in] add mulet reftests on Ash

https://hg.mozilla.org/build/buildbot-configs/rev/93e40fcc9464
Attachment #8494686 - Attachment description: add mulet reftests on Ash → [checked-in] add mulet reftests on Ash
(Assignee)

Comment 40

4 years ago
Comment on attachment 8494540 [details] [diff] [review]
wip - mozharness_mulet.diff

Review of attachment 8494540 [details] [diff] [review]:
-----------------------------------------------------------------

> lrgtm :D :D
> one high level question is how come you did not do to B2GDesktopTest what
> MarionetteTest did? I know you mentioned that you were having problems but
> could you not have just added GaiaMixin to B2GDesktopTest, overwritten
> make_gaia like you did in mulet_unittest.py, and extended the other
> mulet_unittest.py methods (run_test and query_abs_dirs) to support the mulet
> edge case.
> 
I believe my original thought was not to mix mulet jobs with b2g jobs too much, however, as you can see we have already set a dependency on b2g desktop.

I would like to request for a review for my first approach so we can land it while I give a shot again at not needing mulet_unittest.py.

> On one argument, classes are getting overloaded with logic, on the other, we
> have too many mixins and subclasses. Not sure I know what we should do so
> just polling your opinion.

In my opinion, the long term solution is moving to writing scripts from the tree.
On the mean time, we should try to have code that makes it easy to write different scripts.

For instance, the marionette script had to redo a lot of functionality since it was not inside of GaiaMixin.
I think what would help here is if we had a clear separation of what goes into a Mixin and move it there.
For instance, GaiaTest should have defined whatever actions is overwriting and put everything else into GaiaMixin.
I did this a bit in my patch but not completely (e.g. query_abs_dir, extract_xre).

I'm also convinced that we should start thinking about using the concept of super reviewers to ensure that we have consistency and increase knowledge to make better architectural designs.
It is an area that we need due to the complexity of mozharness and one that not too many want to go to (more review load).
If we had mozharness modularized (as in proposal) we would have less need for actions, subclassing and mixins which would drop the need for super reviews.

::: mozharness/mozilla/gaia.py
@@ +261,5 @@
> +        else:
> +            repo = {
> +                'repo_path': self.config['gaia_repo'],
> +                'revision': 'default',
> +                'branch': self.config['gaia_branch']

gaia_branch has a default value and that is why I did not query, however, I would not mind adding it to be explicit (we never know if the default would change).

::: scripts/b2g_desktop_unittest.py
@@ +75,2 @@
>          super(B2GDesktopTest, self).__init__(
> +            config_options=self.config_options + copy.deepcopy(config),

I fixed this in a following patch.

@@ +99,5 @@
>          self.test_url = c.get('test_url')
>          self.test_manifest = c.get('test_manifest')
>  
> +        suite = self.config['test_suite']
> +        if suite not in self.test_suites:

In this case, all we're trying to do is make sure that --test-suite $suite is inside of the list of approved suites. In the case of MuletUnittests we currently only support reftests.

::: scripts/mulet_unittest.py
@@ +22,5 @@
> +class MuletUnittest(B2GDesktopTest, GaiaMixin, TransferMixin):
> +    test_suites = ('reftest',)
> +
> +    def __init__(self):
> +        super(MuletUnittest, self).__init__(config=gaia_config_options)

Sure.

@@ +52,5 @@
> +
> +    def preflight_run_tests(self):
> +        if self.config.get("binary_path"):
> +            self.binary_path = self.config["binary_path"]
> +        assert self.binary_path, \

Fixed in next patch.
(Assignee)

Comment 41

4 years ago
Created attachment 8495356 [details] [diff] [review]
mulet_reftests.diff

This patch works on Ash, including for b2g desktop and gaia unit jobs.

As I mentioned, I would like to land as-is and do a bit of work to refactor it properly.
Attachment #8494540 - Attachment is obsolete: true
Attachment #8495356 - Flags: review?(jlund)
(Assignee)

Comment 42

4 years ago
jgriffin, what do we need to do with regards to this comment?

(In reply to Ed Morley [:edmorley] from comment #21)
> Comment on attachment 8490106 [details] [diff] [review]
> mulet_reftests.diff
> 
> Review of attachment 8490106 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: layout/tools/reftest/b2g_desktop.py
> @@ +61,5 @@
> >          env = self.buildBrowserEnv(options, self.profile.profile)
> > +
> > +        # FIXME
> > +        if "MOZ_DISABLE_NONLOCAL_CONNECTIONS" in env:
> > +            del env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"]
> 
> This will need to be removed for the jobs to be unhidden, seeing as this is
> a new job, per:
> https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#8.
> 29_Must_avoid_patterns_known_to_cause_non_deterministic_failures
Flags: needinfo?(jgriffin)
(Reporter)

Comment 43

4 years ago
(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #42)
> jgriffin, what do we need to do with regards to this comment?
> 
> (In reply to Ed Morley [:edmorley] from comment #21)
> > Comment on attachment 8490106 [details] [diff] [review]
> > mulet_reftests.diff
> > 
> > Review of attachment 8490106 [details] [diff] [review]:
> > -----------------------------------------------------------------
> > 
> > ::: layout/tools/reftest/b2g_desktop.py
> > @@ +61,5 @@
> > >          env = self.buildBrowserEnv(options, self.profile.profile)
> > > +
> > > +        # FIXME
> > > +        if "MOZ_DISABLE_NONLOCAL_CONNECTIONS" in env:
> > > +            del env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"]
> > 
> > This will need to be removed for the jobs to be unhidden, seeing as this is
> > a new job, per:
> > https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#8.
> > 29_Must_avoid_patterns_known_to_cause_non_deterministic_failures

Good question.  The issue is that Gaia makes some remote http requests on startup that we have no way to disable.  AFAICT, these don't impact the tests, but who knows for sure.  I'll pass this back to Ed...Ed, if you feel strongly about this, we'll need to file bugs for devs to implement ways to skip these.
Flags: needinfo?(jgriffin) → needinfo?(emorley)
(Assignee)

Comment 44

4 years ago
Created attachment 8495506 [details] [diff] [review]
[checked-in] mozharness_mulet.diff

I attached an old patch.
Attachment #8495356 - Attachment is obsolete: true
Attachment #8495356 - Flags: review?(jlund)
Attachment #8495506 - Flags: review?(jlund)
(In reply to Jonathan Griffin (:jgriffin) from comment #43)
> > > > +        if "MOZ_DISABLE_NONLOCAL_CONNECTIONS" in env:
> > > > +            del env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"]
> > > 
> > > This will need to be removed for the jobs to be unhidden, seeing as this is
> > > a new job, per:
> > > https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#8.
> > > 29_Must_avoid_patterns_known_to_cause_non_deterministic_failures
> 
> Good question.  The issue is that Gaia makes some remote http requests on
> startup that we have no way to disable.  AFAICT, these don't impact the
> tests, but who knows for sure.  I'll pass this back to Ed...Ed, if you feel
> strongly about this, we'll need to file bugs for devs to implement ways to
> skip these.

Remote requests on startup can still affect the reliability of the job - and worse, by having this disabled we won't detect future regressions for requests not at startup. Given that opt B2G mochitest jobs are successfully running without this disabled (http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/runtestsb2g.py#163) I would like to see at least the opt builds running without undefining MOZ_DISABLE_NONLOCAL_CONNECTIONS, if that's ok?
Flags: needinfo?(emorley)
(Reporter)

Comment 46

4 years ago
Understood.  In that case, can you remove those bits Armen, and file a bug for whichever request it is that causes the test to crash?  I can help direct the bug to the right component if I can see a test log for it.
Comment on attachment 8495506 [details] [diff] [review]
[checked-in] mozharness_mulet.diff

Review of attachment 8495506 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm :) if it works on ash, shweet.

But I am not sure if it is all working. My main concern is that this still uses a 'config' arg in init of scripts/b2g_desktop_unittest.py but scripts/mulet_unittest.py uses 'options' now.

::: configs/b2g/generic_config.py
@@ +1,1 @@
> +# This is a config with generic releng related values 

remind me, what's the plan here? are you going to rip out duplicate items from b2g configs and call this after a buildbot-config patch?

I wonder if we should be creating this file in tree and adding it to self.tree_config rather than within mozharn repo. Apologies if you have explained this already.

::: mozharness/mozilla/testing/gaia_test.py
@@ -154,5 @@
> -
> -        self.clone_gaia(dest, repo,
> -                        use_gaia_json=self.buildbot_config is not None)
> -
> -        super(GaiaTest, self).pull(**kwargs)

what pull was this super calling? looks like we are no longer doing that after your patch.

::: scripts/b2g_desktop_unittest.py
@@ +70,5 @@
>          {'substr': 'Timeout waiting for marionette on port', 'level': ERROR},
>          {'regex': re.compile(r'''(Timeout|NoSuchAttribute|Javascript|NoSuchElement|XPathLookup|NoSuchWindow|StaleElement|ScriptTimeout|ElementNotVisible|NoSuchFrame|InvalidElementState|NoAlertPresent|InvalidCookieDomain|UnableToSetCookie|InvalidSelector|MoveTargetOutOfBounds)Exception'''), 'level': ERROR},
>      ]
>  
> +    def __init__(self, config=[], require_config_file=False):

I think because of implentation of python, we have to assign config with [] outside of of the method signature or spurious things can happen: http://stackoverflow.com/questions/9887180/the-value-of-an-empty-list-in-function-parameter-example-here

also, I'm guessing this is supposed to be 'options' now after your change in mulet_unittest.py

::: scripts/mulet_unittest.py
@@ +55,5 @@
> +        # of download-and-extract()
> +        if self.config.get("binary_path"):
> +            self.binary_path = self.config["binary_path"]
> +
> +        if not self.binary_path:

should we do this instead incase we have self.binary_path already set?

if not self.binary_path:
    if self.config.get("binary_path"):
        self.binary_path = self.config["binary_path"]
    else:
        self.fatal("Use --binary-path as it is needed for _query_abs_dir().")

it might not matter. the above is the idiom we normally do with self attrs and self.config. doesn't block
Attachment #8495506 - Flags: review?(jlund) → review+
(Assignee)

Updated

4 years ago
Attachment #8494000 - Flags: review?(jlund)
(Assignee)

Comment 48

4 years ago
Comment on attachment 8495506 [details] [diff] [review]
[checked-in] mozharness_mulet.diff

Review of attachment 8495506 [details] [diff] [review]:
-----------------------------------------------------------------

> But I am not sure if it is all working. My main concern is that this still
> uses a 'config' arg in init of scripts/b2g_desktop_unittest.py but
> scripts/mulet_unittest.py uses 'options' now.
> 
You're right. This was fixed on a posterior push to Ash. I have kept on working on a proper refactoring for this.

::: configs/b2g/generic_config.py
@@ +1,1 @@
> +# This is a config with generic releng related values 

The plain is experiment with the idea that *all* if not most releng values could be contained on a generic config. Eventually, once we move things into the tree, I would like to make all scripts to point to such universal generic config and then specialize for the specific platform. This is just one step towards it.
I don't see this config in the tree since a lot of these values might be needed before the download-and-extract step happens.
I don't currently see the value of having this in the tree as of now, however, I'm happy to discuss it further now or later down the road.

::: mozharness/mozilla/testing/gaia_test.py
@@ -154,5 @@
> -
> -        self.clone_gaia(dest, repo,
> -                        use_gaia_json=self.buildbot_config is not None)
> -
> -        super(GaiaTest, self).pull(**kwargs)

For some reason I recall seeing the tools repo being checked out, however, I can't see "repos" being specified for marionete, b2g desktop or gaia unit.
I will review it again if I do need this.

::: scripts/b2g_desktop_unittest.py
@@ +70,5 @@
>          {'substr': 'Timeout waiting for marionette on port', 'level': ERROR},
>          {'regex': re.compile(r'''(Timeout|NoSuchAttribute|Javascript|NoSuchElement|XPathLookup|NoSuchWindow|StaleElement|ScriptTimeout|ElementNotVisible|NoSuchFrame|InvalidElementState|NoAlertPresent|InvalidCookieDomain|UnableToSetCookie|InvalidSelector|MoveTargetOutOfBounds)Exception'''), 'level': ERROR},
>      ]
>  
> +    def __init__(self, config=[], require_config_file=False):

That is insane!! That is a rather absurd behaviour of python.

I was thinking of fixing, however, after some thought, we never call an __init__ method of a script more than once.
Do you think I should still fix it?

::: scripts/mulet_unittest.py
@@ +55,5 @@
> +        # of download-and-extract()
> +        if self.config.get("binary_path"):
> +            self.binary_path = self.config["binary_path"]
> +
> +        if not self.binary_path:

What do you think of this instead? A bit cleaner; I think.

    def preflight_run_tests(self):
        # This is if we don't run all actions since we set this inside
        # of download-and-extract()
        if not self.binary_path:
            if self.config.get("binary_path"):
                self.binary_path = self.config["binary_path"]
            else:
                self.fatal("Use --binary-path as it is needed for _query_abs_dir().")
https://hg.mozilla.org/mozilla-central/rev/4282122a081e
Status: REOPENED → RESOLVED
Last Resolved: 4 years ago4 years ago
Resolution: --- → FIXED
(Assignee)

Updated

4 years ago
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Whiteboard: [leave open]
Attachment #8494000 - Flags: review?(jlund) → review+
(Assignee)

Updated

4 years ago
Attachment #8494000 - Attachment description: mulet_in_tree_configs.diff → [checked-in] mulet_in_tree_configs.diff
(Assignee)

Comment 50

4 years ago
Created attachment 8497039 [details] [diff] [review]
[checked-in] bconfigs_enable_mulet_on_try.diff
Attachment #8497039 - Flags: review?(jlund)
(Assignee)

Comment 51

4 years ago
Comment on attachment 8495506 [details] [diff] [review]
[checked-in] mozharness_mulet.diff

https://hg.mozilla.org/build/mozharness/rev/b59a1a06dd1e
Attachment #8495506 - Attachment description: mozharness_mulet.diff → [checked-in] mozharness_mulet.diff
(Assignee)

Comment 52

4 years ago
You can now run mulet through mozharness.
You can try it out with:
python scripts/mulet_unittest.py --cfg b2g/generic_config.py --cfg b2g/mulet_config.py --test-suite reftest --test-manifest tests/layout/reftests/reftest.list --blob-upload-branch ash --download-symbols ondemand --installer-url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/ash-linux64-mulet/1412005020/firefox-35.0a1.en-US.linux-x86_64.tar.bz2 --test-url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/ash-linux64-mulet/1412005020/firefox-35.0a1.en-US.linux-x86_64.tests.zip --cfg developer_config.py --gaia-repo ~/repos/branches/gaia-central

(In reply to Jonathan Griffin (:jgriffin) from comment #46)
> Understood.  In that case, can you remove those bits Armen, and file a bug
> for whichever request it is that causes the test to crash?  I can help
> direct the bug to the right component if I can see a test log for it.

I have seen this happen locally [1]

While on Ash I see this instead [2]
https://tbpl.mozilla.org/php/getParsedLog.php?id=49121431&tree=Ash&full=1#error0

[1]
http://hg.mozilla.org/mozilla-central/file/6eb7121059fb/netwerk/base/src/nsSocketTransport2.cpp#l1184

16:38:06     INFO -  FATAL ERROR: Non-local network connections are disabled and a connection attempt to inapp-pay-test.paas.allizom.org (63.245.215.73) was made.
16:38:06     INFO -  You should only access hostnames available via the test networking proxy (if running mochitests) or from a test-specific httpd.js server (if running xpcshell tests). Browser services should be disabled or redirected to a local server.
16:38:06     INFO -  ###!!! [Child][MessageChannel] Error: Channel closing: too late to send/recv, messages will be lost
16:38:06     INFO -  [Child 8101] WARNING: pipe error (3): Connection reset by peer: file /builds/slave/ash-l64-mulet-0000000000000000/build/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 451
16:38:06     INFO -  ###!!! [Child][MessageChannel] Error: Channel error: cannot send/recv
16:38:06  WARNING -  PROCESS-CRASH | b2g_desktop.pyc | application crashed [unknown top frame]
16:38:06     INFO -  Crash dump filename: /tmp/tmpc07zn1/minidumps/3b46a9ed-d7b0-b362-782e876c-53ad2a11.dmp
16:38:06     INFO -  No symbols path given, can't process dump.
16:38:06     INFO -  Traceback (most recent call last):
16:38:06     INFO -    File "runreftestb2g.py", line 611, in <module>
16:38:06     INFO -      sys.exit(main())
16:38:06     INFO -    File "runreftestb2g.py", line 606, in main
16:38:06     INFO -      return run_desktop_reftests(parser, options, args)
16:38:06     INFO -    File "/home/armenzg/repos/mozharness/build/tests/reftest/b2g_desktop.py", line 187, in run_desktop_reftests
16:38:06     INFO -      sys.exit(reftest.run_tests(args[0], options))
16:38:06     INFO -    File "/home/armenzg/repos/mozharness/build/tests/reftest/b2g_desktop.py", line 95, in run_tests
16:38:06     INFO -      self.run_marionette_script()
16:38:06     INFO -    File "/home/armenzg/repos/mozharness/build/tests/reftest/b2g_desktop.py", line 44, in run_marionette_script
16:38:06     INFO -      self.marionette.start_session()
16:38:06     INFO -    File "/home/armenzg/repos/mozharness/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 818, in start_session
16:38:06     INFO -      self.session = self._send_message('newSession', 'value')
16:38:06     INFO -    File "/home/armenzg/repos/mozharness/build/venv/local/lib/python2.7/site-packages/marionette/decorators.py", line 35, in _
16:38:06     INFO -      return func(*args, **kwargs)
16:38:06     INFO -    File "/home/armenzg/repos/mozharness/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 615, in _send_message
16:38:06     INFO -      response = self.client.send(message)
16:38:06     INFO -    File "/home/armenzg/repos/mozharness/build/venv/local/lib/python2.7/site-packages/marionette_transport/transport.py", line 85, in send
16:38:06     INFO -      self.connect()
16:38:06     INFO -    File "/home/armenzg/repos/mozharness/build/venv/local/lib/python2.7/site-packages/marionette_transport/transport.py", line 67, in connect
16:38:06     INFO -      self.sock.connect((self.addr, self.port))
16:38:06     INFO -    File "/usr/lib/python2.7/socket.py", line 224, in meth
16:38:06     INFO -      return getattr(self._sock,name)(*args)
16:38:06     INFO -  socket.error: [Errno 111] Connection refused

[2]
https://tbpl.mozilla.org/php/getParsedLog.php?id=49121431&tree=Ash&full=1#error0
09:44:19     INFO -  Traceback (most recent call last):
09:44:19     INFO -    File "runreftestb2g.py", line 611, in <module>
09:44:19     INFO -      sys.exit(main())
09:44:19     INFO -    File "runreftestb2g.py", line 606, in main
09:44:19     INFO -      return run_desktop_reftests(parser, options, args)
09:44:19     INFO -    File "/builds/slave/test/build/tests/reftest/b2g_desktop.py", line 187, in run_desktop_reftests
09:44:19     INFO -      sys.exit(reftest.run_tests(args[0], options))
09:44:19     INFO -    File "/builds/slave/test/build/tests/reftest/b2g_desktop.py", line 95, in run_tests
09:44:19     INFO -      self.run_marionette_script()
09:44:19     INFO -    File "/builds/slave/test/build/tests/reftest/b2g_desktop.py", line 45, in run_marionette_script
09:44:19     INFO -      self._unlockScreen()
09:44:19     INFO -    File "/builds/slave/test/build/tests/reftest/b2g_desktop.py", line 39, in _unlockScreen
09:44:19     INFO -      self.marionette.execute_async_script('GaiaLockScreen.unlock()')
09:44:19     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 1269, in execute_async_script
09:44:19     INFO -      filename=os.path.basename(frame[0]))
09:44:19     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/decorators.py", line 35, in _
09:44:19     INFO -      return func(*args, **kwargs)
09:44:19     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 638, in _send_message
09:44:19     INFO -      self._handle_error(response)
09:44:19     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 686, in _handle_error
09:44:19    ERROR -      raise errors.JavascriptException(message=message, status=status, stacktrace=stacktrace)
09:44:19    ERROR -  marionette.errors.JavascriptException: JavascriptException: TypeError: this.states.instance is null
09:44:19     INFO -  stacktrace:
09:44:19     INFO -  	@app://system.gaiamobile.org/js/lockscreen_window_manager.js, line 245
09:44:19    ERROR - Return code: 1
Flags: needinfo?(jgriffin)
(Assignee)

Comment 53

4 years ago
The code used for the previous comment is this:
https://hg.mozilla.org/projects/ash/rev/6a0c55df4061

Which is the same as attachment 8490106 [details] [diff] [review] but by removing this:
::: layout/tools/reftest/b2g_desktop.py
@@ +61,5 @@
>          env = self.buildBrowserEnv(options, self.profile.profile)
> +
> +        # FIXME
> +        if "MOZ_DISABLE_NONLOCAL_CONNECTIONS" in env:
> +            del env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"]
(Assignee)

Comment 54

4 years ago
jlund, I can't seem to find a way to mark a test job to not be run by default on try.
I can see "try_by_default" for build jobs but not for build jobs.

I could also hide mulet reftests on try until they're fixed.
Flags: needinfo?(jlund)
(Reporter)

Comment 55

4 years ago
I'll file a bug for #1.  #2 looks like a timing issue...the slave is probably slower than your machine, and possibly the lock screen hasn't appeared by the time we are trying to unlock it.  Does the problem on ash disappear if you sleep(15) before unlocking the lockscreen?  If so, we probably need to add some code to wait for the lock screen to appear before we attempt to unlock it.
Flags: needinfo?(jgriffin)
(Reporter)

Updated

4 years ago
Depends on: 1053163
(Assignee)

Comment 56

4 years ago
Created attachment 8497650 [details] [diff] [review]
mulet_reftests.diff

With this patch we can now hit my local issue:
https://tbpl.mozilla.org/php/getParsedLog.php?id=49211733&tree=Ash&full=1#error0

I guess we just have to wait now for bug 1053163.

Meanwhile let's get the try support so developers can fix it on try (attachment 8497039 [details] [diff] [review]).
Comment on attachment 8497039 [details] [diff] [review]
[checked-in] bconfigs_enable_mulet_on_try.diff

Review of attachment 8497039 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm but can we make this a trychooser option? looking at this patch, I assume this will run by default if we -all on try. Which might upset sheriffs as, correct me if I am wrond, we shouldn't have any tests that are perma fail running by default on try.
(Assignee)

Comment 58

4 years ago
I'm aiming at hiding it since I don't know of a way to not trigger it by default.
Merged to production, and deployed.
(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #58)
> I'm aiming at hiding it since I don't know of a way to not trigger it by
> default.

if you can not find a way to trigger it only via trychooser, I can only see an example for builds[1], hiding wfm. Sorry to stall you on this. Just trying to avoid adding unneeded jobs

[1] https://bug691177.bugzilla.mozilla.org/attachment.cgi?id=695029
Flags: needinfo?(jlund)
(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #54)
> jlund, I can't seem to find a way to mark a test job to not be run by
> default on try.
> I can see "try_by_default" for build jobs but not for build jobs.
> 
> I could also hide mulet reftests on try until they're fixed.

well that's embarrassing. just seeing this comment now. looks like I've been one step behind all along.
Attachment #8497039 - Flags: review?(jlund) → review+
(Assignee)

Comment 62

4 years ago
Comment on attachment 8497039 [details] [diff] [review]
[checked-in] bconfigs_enable_mulet_on_try.diff

I did not find a way to disable a specific builder.
I only found a way to disable a whole platform by adding "try_by_default": False (e.g. u'linux64-mulet': ['Ubuntu VM 12.04 x64 Mulet try-nondefault'],).

I'm landing as-is and I hope to hide it on time once it shows up on tbpl.
https://hg.mozilla.org/build/buildbot-configs/rev/3bfd982c2211
Attachment #8497039 - Attachment description: bconfigs_enable_mulet_on_try.diff → [checked-in] bconfigs_enable_mulet_on_try.diff
Merged to production, and deployed.
(Assignee)

Comment 64

4 years ago
Comment on attachment 8497650 [details] [diff] [review]
mulet_reftests.diff

I think we should land this while it does not get bitrotten.
Attachment #8497650 - Flags: review?(jgriffin)
(Assignee)

Comment 65

4 years ago
Created attachment 8500612 [details] [diff] [review]
[checked-in] [buildbot-configs] enable mulet reftests on Cedar

This will allow developers to know the current state of mulet reftests without first having to push anything to try.
Attachment #8500612 - Flags: review?(jlund)
(Reporter)

Comment 66

4 years ago
Comment on attachment 8497650 [details] [diff] [review]
mulet_reftests.diff

Review of attachment 8497650 [details] [diff] [review]:
-----------------------------------------------------------------

::: layout/tools/reftest/b2g_desktop.py
@@ +117,5 @@
>          # Turn off the locale picker screen
>          prefs["browser.firstrun.show.localepicker"] = False
> +        # FIXME: these cause Gaia not to launch
> +        #prefs["b2g.system_startup_url"] = "app://test-container.gaiamobile.org/index.html"
> +        #prefs["b2g.system_manifest_url"] = "app://test-container.gaiamobile.org/manifest.webapp"

Have we done a try run to make sure this doesn't break reftest on b2gdesktop?  I'm afraid the above lines might do that.
(Assignee)

Updated

4 years ago
Attachment #8497650 - Flags: review?(jgriffin)
(Assignee)

Comment 67

4 years ago
(In reply to Jonathan Griffin (:jgriffin) from comment #66)
> Comment on attachment 8497650 [details] [diff] [review]
> mulet_reftests.diff
> 
> Review of attachment 8497650 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: layout/tools/reftest/b2g_desktop.py
> @@ +117,5 @@
> >          # Turn off the locale picker screen
> >          prefs["browser.firstrun.show.localepicker"] = False
> > +        # FIXME: these cause Gaia not to launch
> > +        #prefs["b2g.system_startup_url"] = "app://test-container.gaiamobile.org/index.html"
> > +        #prefs["b2g.system_manifest_url"] = "app://test-container.gaiamobile.org/manifest.webapp"
> 
> Have we done a try run to make sure this doesn't break reftest on
> b2gdesktop?  I'm afraid the above lines might do that.

My bad. We will know tomorrow:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=3e3714577db2
(Assignee)

Comment 68

4 years ago
It does affect the b2g desktop jobs.
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=3e3714577db2
https://treeherder.mozilla.org/ui/logviewer.html#?job_id=2253625&repo=try

 15:52:03 INFO - FATAL ERROR: Non-local network connections are disabled and a connection attempt to inapp-pay-test.paas.allizmozcrash INFO | Downloading symbols from: https://ftp-ssl.mozilla.org/pub/mozilla.org/b2g/try-builds/armenzg@mozilla.com-3e3714577db2/try-linux64_gecko/b2g-35.0a1.en-US.linux-x86_64.crashreporter-symbols.zip 

jgriffin, what do you suggest I try next? Or leave this until bug 1053163 is fixed?

I've considered adding a --mulet parameter to the harness and use it for mulet jobs, hence not affecting the b2g desktop jobs.
Flags: needinfo?(jgriffin)
(Reporter)

Comment 69

4 years ago
Since this is just on cedar for now (or try) I think we should allow network connections for the purpose of greening up the tests themselves.   After bug 1053163 is addressed, we can look at disabling network connections again.
Flags: needinfo?(jgriffin)
Comment on attachment 8500612 [details] [diff] [review]
[checked-in] [buildbot-configs] enable mulet reftests on Cedar

Review of attachment 8500612 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/b2g_config.py
@@ +2242,5 @@
>  exclude_suites(('linux32_gecko', 'ubuntu32_vm-b2gdt'), 'cedar', ('gaia-ui-test',), True, True)
>  exclude_suites(('macosx64_gecko', 'mountainlion-b2gdt'), 'cedar', ('gaia-ui-test',), True, True)
>  exclude_suites(('emulator', 'ubuntu64_vm-b2g-lg-emulator'), 'cedar', ('gaia-ui-test',), True, True)
>  
>  # Enable mulet reftests on Ash and Try

# and cedar
Attachment #8500612 - Flags: review?(jlund) → review+
Committed code from this bug has been rolled out to production.
(Assignee)

Comment 72

4 years ago
Created attachment 8501770 [details] [diff] [review]
[m-c] patch to run mulet reftest: 1) allow external connections, 2) do not comment out gaia prefs and 3) add sleep

I want to see if I can create a patch that will not turn b2g desktop reftest jobs red for Cedar (like attachment 8497650 [details] [diff] [review] does):
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=dd38ffbef696

I added back allowing external connections.
(Assignee)

Updated

4 years ago
Depends on: 1080123
(Assignee)

Comment 73

4 years ago
Created attachment 8511304 [details] [diff] [review]
[m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) add sleep

I'm back to this. I'm adding a --mulet option to runreftestb2g.py/b2g_desktop.py to allow us to have the necessary code to run mulet reftests without affecting the b2g desktop jobs.

Pushing this change to try:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=24fee2a32d3c

Also pushed without permitting external connections to see where we are:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=109394c6b860

STR:
python scripts/mulet_unittest.py --cfg b2g/generic_config.py \
--cfg b2g/mulet_config.py --test-suite reftest --test-manifest tests/layout/reftests/reftest.list \
--blob-upload-branch ash --download-symbols ondemand \
--installer-url TBD \
--test-url TBD \
--cfg developer_config.py --gaia-repo /path/to/gaia/centra/checkout

NOTE: Check-out gaia-central if you don't want the script to be checking out every time.
(Assignee)

Comment 74

4 years ago
Created attachment 8512212 [details] [diff] [review]
[m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) add sleep

Updated patch.
Latest run on try looks good:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=07296ac81420 (pending desktop jobs)

There's some clean up of imports that are not used.
Attachment #8497650 - Attachment is obsolete: true
Attachment #8501770 - Attachment is obsolete: true
Attachment #8511304 - Attachment is obsolete: true
Attachment #8512212 - Flags: review?(jgriffin)
Comment on attachment 8512212 [details] [diff] [review]
[m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) add sleep

Review of attachment 8512212 [details] [diff] [review]:
-----------------------------------------------------------------

::: layout/tools/reftest/b2g_desktop.py
@@ +67,5 @@
> +
> +        if self.build_type == "mulet":
> +            # FIXME
> +            if "MOZ_DISABLE_NONLOCAL_CONNECTIONS" in env:
> +                del env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"]

Is this still needed now that bug 1053163's PR landed?
If it is needed, could we make it mulet+debug? (Presuming it's green on opt like other B2G)
(Assignee)

Comment 76

4 years ago
(In reply to Ed Morley [:edmorley] from comment #75)
> Comment on attachment 8512212 [details] [diff] [review]
> [m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external
> connections, 3) do not add gaia prefs and 4) add sleep
> 
> Review of attachment 8512212 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: layout/tools/reftest/b2g_desktop.py
> @@ +67,5 @@
> > +
> > +        if self.build_type == "mulet":
> > +            # FIXME
> > +            if "MOZ_DISABLE_NONLOCAL_CONNECTIONS" in env:
> > +                del env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"]
> 
> Is this still needed now that bug 1053163's PR landed?
> If it is needed, could we make it mulet+debug? (Presuming it's green on opt
> like other B2G)

I have pushed separately to determine that:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=5fe267a972e5
(Assignee)

Comment 77

4 years ago
All jobs are green except for the mulet reftest ones:
* Log: http://ftp.mozilla.org/pub/mozilla.org/b2g/try-builds/armenzg@mozilla.com-5fe267a972e5/try-linux64-mulet/try_ubuntu64_vm-mulet_test-reftest-bm117-tests1-linux64-build40.txt.gz
* Changes: https://hg.mozilla.org/try/rev/07296ac81420

I will run it locally and see what is going on.

16:59:19     INFO -  [3086] WARNING: Cannot parse integer value '800x1000' for --screen: 'glib warning', file /builds/slave/try-l64-mulet-0000000000000000/build/toolkit/xre/nsSigHandlers.cpp, line 129
16:59:19     INFO -  (firefox:3086): Gtk-WARNING **: Cannot parse integer value '800x1000' for --screen
16:59:19     INFO -  Xlib:  extension "RANDR" missing on display ":0".
16:59:19     INFO -  1414454351593	addons.xpi	WARN	Error loading bootstrap.js for reftest@mozilla.org: Error opening input stream (invalid filename?)
16:59:19     INFO -  1414454351594	addons.xpi	WARN	Add-on reftest@mozilla.org is missing bootstrap method install
16:59:19     INFO -  1414454351624	addons.xpi	WARN	Disabling foreign installed add-on ubufox@ubuntu.com in app-system-share
16:59:19     INFO -  1414454351631	addons.xpi	WARN	Add-on reftest@mozilla.org is missing bootstrap method startup
...
16:59:19     INFO -  1414454359735	Marionette	INFO	sendToClient: {"from":"0","error":{"message":"TypeError: doc is undefined","status":17,"stacktrace":"execute_script @b2g_desktop.py, line 57\ninline javascript, line 280\nsrc: \"undefined\""}}, {ae7ae623-0f8f-4b41-85a5-df8df50a4da7}, {ae7ae623-0f8f-4b41-85a5-df8df50a4da7}
16:59:19     INFO -  Traceback (most recent call last):
16:59:19     INFO -    File "runreftestb2g.py", line 633, in <module>
16:59:19     INFO -      sys.exit(main())
16:59:19     INFO -    File "runreftestb2g.py", line 628, in main
16:59:19     INFO -      return run_desktop_reftests(parser, options, args)
16:59:19     INFO -    File "/builds/slave/test/build/tests/reftest/b2g_desktop.py", line 208, in run_desktop_reftests
16:59:19     INFO -      sys.exit(reftest.run_tests(args[0], options))
16:59:19     INFO -    File "/builds/slave/test/build/tests/reftest/b2g_desktop.py", line 105, in run_tests
16:59:19     INFO -      self.run_marionette_script()
16:59:19     INFO -    File "/builds/slave/test/build/tests/reftest/b2g_desktop.py", line 57, in run_marionette_script
16:59:19     INFO -      self.marionette.execute_script(self.test_script)
16:59:19     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 1221, in execute_script
16:59:19     INFO -      filename=os.path.basename(frame[0]))
16:59:19     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/decorators.py", line 36, in _
16:59:19     INFO -      return func(*args, **kwargs)
16:59:19     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 638, in _send_message
16:59:19     INFO -      self._handle_error(response)
16:59:19     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 686, in _handle_error
16:59:19    ERROR -      raise errors.JavascriptException(message=message, status=status, stacktrace=stacktrace)
16:59:19    ERROR -  marionette.errors.JavascriptException: JavascriptException: TypeError: doc is undefined
16:59:19     INFO -  stacktrace:
16:59:19     INFO -  	execute_script @b2g_desktop.py, line 57
16:59:19     INFO -  	inline javascript, line 280
16:59:19     INFO -  	src: "undefined"
(Assignee)

Comment 78

4 years ago
I spotted the issue. I will put up a new patch.
(Assignee)

Comment 79

4 years ago
Created attachment 8512740 [details] [diff] [review]
[m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) add sleep

I had forgotten to adjust the in-tree mozharness config to switch from --desktop to --mulet.

The new patch being tested:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=caf13ef7ff7c
Attachment #8512212 - Attachment is obsolete: true
Attachment #8512212 - Flags: review?(jgriffin)
(Assignee)

Comment 80

4 years ago
Created attachment 8512949 [details] [diff] [review]
m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) add sleep

I attached the wrong patch. This is the right one.

We now can run lots of reftests, however, we time out after 2 hours.

I think we have these options:
- We probably have to break the reftests into various chunks
- Determine why we run so slowly
- Run on faster VMs (not sure if we can)

https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=caf13ef7ff7c&showall=1
https://treeherder.mozilla.org/ui/logviewer.html#?job_id=2757778&repo=try
Attachment #8512740 - Attachment is obsolete: true
Attachment #8512949 - Flags: review?(jgriffin)
(Reporter)

Comment 81

4 years ago
Comment on attachment 8512949 [details] [diff] [review]
m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) add sleep

Review of attachment 8512949 [details] [diff] [review]:
-----------------------------------------------------------------

::: layout/tools/reftest/b2g_desktop.py
@@ +45,5 @@
>          self.marionette.start_session()
> +        if self.build_type == "mulet":
> +            # XXX: bug 1043699 - unlocking takes too long on EC2
> +            import time
> +            time.sleep(15)

We already know how to get around this problem, see http://dxr.mozilla.org/mozilla-central/source/testing/web-platform/harness/wptrunner/browsers/b2g.py#236
Attachment #8512949 - Flags: review?(jgriffin) → review+
(Assignee)

Comment 82

4 years ago
Created attachment 8514467 [details] [diff] [review]
[checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen

Pushed to try:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=787ab3d5bb8e

I will append a patch for buildbot-configs to chunkify.

I added this to the patch to test chunking:
--- a/testing/config/mozharness/linux_mulet_config.py
+++ b/testing/config/mozharness/linux_mulet_config.py
@@ -1,13 +1,11 @@
 # This is used by mozharness' mulet_unittest.py
 config = {
     # testsuite options
     "reftest_options": [
         "--mulet", "--profile=%(gaia_profile)s",
-        "--appname=%(application)s", "%(test_manifest)s",
-        "--this-chunk", "1",
-        "--total-chunk", "6"
+        "--appname=%(application)s", "%(test_manifest)s"
     ],
     "run_file_names": {
         "reftest": "runreftestb2g.py",
     },
 }
Attachment #8512949 - Attachment is obsolete: true
(Assignee)

Comment 83

4 years ago
Created attachment 8514470 [details] [diff] [review]
[b-c] add 4 chunks for mulet reftests
Attachment #8514470 - Flags: review?(jlund)
(Assignee)

Comment 84

4 years ago
Comment on attachment 8514467 [details] [diff] [review]
[checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen

I want to land this once I get a review for the recent changes I made.

For 6 chunks we get a 33 minute run for the push I made.

I will update my b-c configs to have 6 chunks instead of 4.

Chunk 1/6:
https://treeherder.mozilla.org/ui/logviewer.html#?job_id=2833982&repo=try
Attachment #8514467 - Flags: review?(jgriffin)
(Assignee)

Comment 85

4 years ago
Created attachment 8514959 [details] [diff] [review]
[checked-in] [b-c] add 6 chunks for mulet reftests
Attachment #8514959 - Flags: review?(jlund)
(Reporter)

Comment 86

4 years ago
Comment on attachment 8514467 [details] [diff] [review]
[checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen

Review of attachment 8514467 [details] [diff] [review]:
-----------------------------------------------------------------

Lgtm, thanks!
Attachment #8514467 - Flags: review?(jgriffin) → review+
Comment on attachment 8514959 [details] [diff] [review]
[checked-in] [b-c] add 6 chunks for mulet reftests

Review of attachment 8514959 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm :)

I'm assuming we must wait for https://bug1043699.bugzilla.mozilla.org/attachment.cgi?id=8514467 to land?

do we need a tbpl patch to recognize the chunking?
Attachment #8514959 - Flags: review?(jlund) → review+
Comment on attachment 8514470 [details] [diff] [review]
[b-c] add 4 chunks for mulet reftests

Review of attachment 8514470 [details] [diff] [review]:
-----------------------------------------------------------------

I'm assuming this is obsolete now based of newer patch.

If you going to iterate by landing 4 chunks and then 6 chunks later and you need a review, feel free to r+ this.
Attachment #8514470 - Flags: review?(jlund)
(Assignee)

Updated

4 years ago
Attachment #8514959 - Attachment description: [b-c] add 4 chunks for mulet reftests → [b-c] add 6 chunks for mulet reftests
(Assignee)

Updated

4 years ago
Attachment #8514470 - Attachment is obsolete: true
(Assignee)

Comment 89

4 years ago
Comment on attachment 8500612 [details] [diff] [review]
[checked-in] [buildbot-configs] enable mulet reftests on Cedar

Checked-in a while ago.
Attachment #8500612 - Attachment description: [buildbot-configs] enable mulet reftests on Cedar → [checked-in] [buildbot-configs] enable mulet reftests on Cedar
(Assignee)

Comment 90

4 years ago
Comment on attachment 8514959 [details] [diff] [review]
[checked-in] [b-c] add 6 chunks for mulet reftests

Waiting for a reconfig.
http://hg.mozilla.org/build/buildbot-configs/rev/f6d6c9302c55
Attachment #8514959 - Attachment description: [b-c] add 6 chunks for mulet reftests → [checked-in] [b-c] add 6 chunks for mulet reftests
(Assignee)

Comment 91

4 years ago
We don't need for the mulet change to land first to run the reconfig.
However, I will be landing before EOD assuming I see no new surprises in my last try test:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=5258a11ecc3c
(Assignee)

Comment 92

4 years ago
Comment on attachment 8514467 [details] [diff] [review]
[checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen

https://hg.mozilla.org/integration/mozilla-inbound/rev/6921bd616ff1
Attachment #8514467 - Attachment description: [m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) wait for screen → [checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) wait for screen
(Assignee)

Comment 94

4 years ago
Comment on attachment 8514467 [details] [diff] [review]
[checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen

I landed the wrong patch.
I will land the right one tomorrow after I get the right thing pushed to try.
Attachment #8514467 - Attachment description: [checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) wait for screen → [m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) wait for screen
Comment on attachment 8514467 [details] [diff] [review]
[checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen

(This patch no longer allows external connections, so removing the mention of that)
Attachment #8514467 - Attachment description: [m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) wait for screen → [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen

Updated

4 years ago
Keywords: leave-open
Whiteboard: [leave open]
(Assignee)

Comment 96

4 years ago
Thanks for updating the comment for the patch.

Pushing the right patch:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=e55985ad7e90
(Assignee)

Comment 98

4 years ago
Comment on attachment 8514467 [details] [diff] [review]
[checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen

https://hg.mozilla.org/integration/mozilla-inbound/rev/dfd2eb97d840
Attachment #8514467 - Attachment description: [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen → [checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen
(Assignee)

Comment 100

4 years ago
Unfortunately something has changed since comment 84 (2014-10-30) [1][2]

Who knows how we can disable the checking for updates? It is breaking the rule of hitting the network. [3]

I can reproduce the issue locally. The Mulet starts up, however, it aborts soon after. [4]

Allowing non-local connections makes it work again (even though this is not what we want to do).
I've pushed it to try so we can know where we stand once we fix the checking for updates.
https://tbpl.mozilla.org/?tree=Try&rev=be3fb02d83bd&show_all=1

[1]
Results from comment 84:
https://treeherder.mozilla.org/ui/logviewer.html#?job_id=2833982&repo=try
13:55:45     INFO - TinderboxPrint: reftest<br/>1733/<em class="testfail">120</em>/78
Log:
http://ftp.mozilla.org/pub/mozilla.org/b2g/try-builds/armenzg@mozilla.com-787ab3d5bb8e/try-linux64-mulet/try_ubuntu64_vm-mulet_test-reftest-bm113-tests1-linux64-build55.txt.gz

[2] Chunkified results:
https://tbpl.mozilla.org/?tree=Try&rev=a89990bc0b83&show_all=1

[3]
09:18:10     INFO -  UpdatePrompt: Forcing update check
09:18:10     INFO -  *** AUS:SVC Checker: checkForUpdates, force: true
09:18:10     INFO -  *** AUS:SVC getLocale - getting locale from file: resource://gre/update.locale, locale: en-US
09:18:10     INFO -  *** AUS:SVC Checker:getUpdateURL - update URL: https://aus4.mozilla.org/update/3/Firefox/36.0a1/20141104131517/%PRODUCT_DEVICE%/en-US/default/Linux%203.13.0-39-generic%20(GTK%202.24.23)/default/default/update.xml?force=1
09:18:10     INFO -  *** AUS:SVC gCanCheckForUpdates - able to check for updates
09:18:10     INFO -  *** AUS:SVC recordInHealthReport - updateCheckStart - 0
09:18:10     INFO -  *** AUS:SVC Checker:checkForUpdates - sending request to: https://aus4.mozilla.org/update/3/Firefox/36.0a1/20141104131517/%PRODUCT_DEVICE%/en-US/default/Linux%203.13.0-39-generic%20(GTK%202.24.23)/default/default/update.xml?force=1
09:18:10     INFO -  FATAL ERROR: Non-local network connections are disabled and a connection attempt to aus4.mozilla.org (63.245.217.219) was made.
09:18:10     INFO -  You should only access hostnames available via the test networking proxy (if running mochitests) or from a test-specific httpd.js server (if running xpcshell tests). Browser services should be disabled or redirected to a local server.

[4]
NOTE: If you already have a gaia repo you can point to it like I do

* Installers from 2014-11-05:
python scripts/mulet_unittest.py --cfg b2g/generic_config.py --cfg b2g/mulet_config.py --test-suite reftest --test-manifest tests/layout/reftests/reftest.list  --blob-upload-branch try --download-symbols ondemand --installer-url http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-a89990bc0b83/try-linux64-mulet/firefox-36.0a1.en-US.linux-x86_64.tar.bz2 --test-url http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-a89990bc0b83/try-linux64-mulet/firefox-36.0a1.en-US.linux-x86_64.tests.zip --gaia-repo ~/repos/branches/gaia-central  --cfg developer_config.py --total-chunks 6 --this-chunk 1
(Assignee)

Comment 101

4 years ago
ochameau: who knows how we can disable the checking for updates? It is breaking the rule of hitting the network. [3]

Side note, we need to watch for this change that *could* be affecting mulet reftests (probably not related):
https://bugzilla.mozilla.org/show_bug.cgi?id=1093691#c19
Flags: needinfo?(poirot.alex)
(Assignee)

Updated

4 years ago
Depends on: 1094369
(Assignee)

Updated

4 years ago
Depends on: 1094362
(Reporter)

Comment 102

4 years ago
(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #101)
> ochameau: who knows how we can disable the checking for updates? It is
> breaking the rule of hitting the network. [3]
> 
> Side note, we need to watch for this change that *could* be affecting mulet
> reftests (probably not related):
> https://bugzilla.mozilla.org/show_bug.cgi?id=1093691#c19

Can you help identify someone to help with this, gwagner?
Flags: needinfo?(anygregor)
(Assignee)

Comment 103

4 years ago
I have filed two bugs (bug 1043699 and bug 1094369) which (IIUC) need dev attention.

I've also pushed a try push with 25 chunks instead of 6 since we're timing out after two hours.
I wonder what changed since in comment 84 we were having 30 minutes run.
https://tbpl.mozilla.org/?tree=Try&rev=1c38b6313fa3&show_all=1
I think we have to wait for the dep bugs to be fixed first before doing the estimate but it might be good to have a data point for reference.

In one sense I think this bug is done wrt to making the harness work, however, I will wait to hear from jgriffing/ochameau as to if more help is needed from me.
Alex, can you help out here?
Flags: needinfo?(anygregor) → needinfo?(lissyx+mozillians)
(Assignee)

Comment 105

4 years ago
I have landed a change on Cedar to allow seeing the jobs run by disabling the network block [1]
Once we have a fix for bug 1094362 we can back the change out on Cedar.

Once we fix the reftests locally (bug 1094369) please let me know so we can determine how many chunks we need (as of _now_ I think we will need 15 or more) [2].

[1] http://hg.mozilla.org/projects/cedar/rev/bfe35dbdef31
https://tbpl.mozilla.org/?tree=Cedar&jobname=Mulet&show_all=1

[2]
https://tbpl.mozilla.org/?tree=Try&rev=1c38b6313fa3&show_all=1
Armen, is it possible you are being hit by the same thing that made Mnw perma fail ?
Patch for bug 1092941 should help if it's the case.

Or I misread why I'm being asked for help here. If that's the case, please forgive and tell me what's wrong and how I can try to help.
Flags: needinfo?(lissyx+mozillians) → needinfo?(armenzg)
(Assignee)

Comment 107

4 years ago
(In reply to Alexandre LISSY :gerard-majax from comment #106)
> Armen, is it possible you are being hit by the same thing that made Mnw
> perma fail ?
> Patch for bug 1092941 should help if it's the case.
> 
> Or I misread why I'm being asked for help here. If that's the case, please
> forgive and tell me what's wrong and how I can try to help.

After chatting on IRC, it seems that for Mulet mochitest we disabled the network block [1] (bug 1066044).
We should undo that.

We first need to disable the updates, we can then look at which failures we have.

[1] https://hg.mozilla.org/mozilla-central/rev/3538768a23f4#l7.1
Flags: needinfo?(armenzg)
It's really unclear why mulet ends up having updates.
We try really hard to disable it in multiple places, and especially, here, for reftest:
  http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/reftest-preferences.js#17
Aren't we using these prefs when running reftests on mulet??
Flags: needinfo?(poirot.alex)
(Assignee)

Comment 109

4 years ago
The prefs are being ignored since we force the update.

Let's move the check for update issue to bug 1094362.
(Assignee)

Comment 110

4 years ago
The harness work is done in here.

To make the tests go green please follow up in bug 1094362 and bug 1094369.
Status: REOPENED → RESOLVED
Last Resolved: 4 years ago4 years ago
Resolution: --- → FIXED
Depends on: 1142928
Depends on: 1142990
No longer depends on: 1142928
No longer depends on: 1142990
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.