1192124 - [meta] To fix marionette orange on emu-x86-kk, TaskCluster

Status: RESOLVED FIXED

(Taskcluster :: Services, defect, P2)

Comment 1

[Hsin-Yi Tsai (she/her) | away [:hsinyi]]

(In reply to Hsin-Yi Tsai [:hsinyi] from comment #0)
> 

recent try: https://treeherder.allizom.org/#/jobs?repo=try&revision=b1cff608589c

Comment 2

[Edgar Chen [:edgar]]

I also see following error in the log,

marionette.py: error: --test-packages-url option requires an argument

Maybe it is because b2g_emulator_x86_base.yml misses defining "test_package" [1][2].

[1] https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/builds/b2g_emulator_base.yml#30
[2] https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/mach_commands.py#354-360

Comment 3

[Edgar Chen [:edgar]]

Quick try: https://treeherder.allizom.org/#/jobs?repo=try&revision=cd9a73a6a6bb

Comment 4

[Edgar Chen [:edgar]]

(In reply to Edgar Chen [:edgar][:echen] from comment #3)
> Quick try:
> https://treeherder.allizom.org/#/jobs?repo=try&revision=cd9a73a6a6bb

Still having another error in marionette test:
12:47:55    ERROR - Marionette exited with return code 1: harness failures
12:47:55    ERROR - # TBPL FAILURE #

Comment 5

[Hsin-Yi Tsai (she/her) | away [:hsinyi]]

Hi Andrew,

I was suggested by Dylan that you are the person that I can get in touch with to start diagnosing these suites. Could you please help here?

For your reference, here's a recent try commit that Mn tests on emu-x86-kk are orange: https://treeherder.allizom.org/#/jobs?repo=try&revision=4d30caf50fa6

Comment 6

[Andrew Halberstadt [:ahal]]

In the future, please try to be more specific and provide as much information as possible.

Here is a stack trace pulled out of one of the Mn logs:
> Traceback (most recent call last):
>   File "/home/worker/build/tests/marionette/marionette/runtests.py", line 64, in <module>
>     cli()
>   File "/home/worker/build/tests/marionette/marionette/runtests.py", line 59, in cli
>     runner = startTestRunner(runner_class, args)
>   File "/home/worker/build/tests/marionette/marionette/runtests.py", line 38, in startTestRunner
>     runner.run_tests(tests)
>   File "/home/worker/build/venv/local/lib/python2.7/site-packages/marionette/runner/base.py", line 795, in run_tests
>     self.run_test_sets()
>   File "/home/worker/build/venv/local/lib/python2.7/site-packages/marionette/runner/base.py", line 1024, in run_test_sets
>     self.run_test_set(self.tests)
>   File "/home/worker/build/venv/local/lib/python2.7/site-packages/marionette/runner/base.py", line 1003, in run_test_set
>     self.run_test(test['filepath'], test['expected'], test['test_container'])
>   File "/home/worker/build/venv/local/lib/python2.7/site-packages/marionette/runner/base.py", line 973, in run_test
>     self.launch_test_container()
>   File "/home/worker/build/venv/local/lib/python2.7/site-packages/marionette/runner/base.py", line 713, in launch_test_container
>     }""", script_timeout=60000)
>   File "/home/worker/build/venv/local/lib/python2.7/site-packages/marionette_driver/marionette.py", line 1685, in execute_async_script
>     rv = self._send_message("executeAsyncScript", body, key="value")
>   File "/home/worker/build/venv/local/lib/python2.7/site-packages/marionette_driver/decorators.py", line 36, in _
>     return func(*args, **kwargs)
>   File "/home/worker/build/venv/local/lib/python2.7/site-packages/marionette_driver/marionette.py", line 716, in _send_message
>     self._handle_error(resp)
>   File "/home/worker/build/venv/local/lib/python2.7/site-packages/marionette_driver/marionette.py", line 770, in _handle_error
>     raise errors.lookup(error)(message, stacktrace=stacktrace)
> marionette_driver.errors.ScriptTimeoutException: ScriptTimeoutException: timed out
> Exception in thread ProcessReader (most likely raised during interpreter shutdown):
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
>   File "/usr/lib/python2.7/threading.py", line 504, in run
>   File "/home/worker/build/venv/local/lib/python2.7/site-packages/mozprocess/processhandler.py", line 909, in _read
>   File "/usr/lib/python2.7/Queue.py", line 177, in get
>   File "/usr/lib/python2.7/threading.py", line 258, in wait
> <type 'exceptions.TypeError'>: 'NoneType' object is not callable
> Exception in thread ProcessReader (most likely raised during interpreter shutdown):
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
>   File "/usr/lib/python2.7/threading.py", line 504, in run
>   File "/home/worker/build/venv/local/lib/python2.7/site-packages/mozprocess/processhandler.py", line 909, in _read
>   File "/usr/lib/python2.7/Queue.py", line 177, in get
>   File "/usr/lib/python2.7/threading.py", line 258, in wait
> <type 'exceptions.TypeError'>: 'NoneType' object is not callable

It looks like this call to execute_script_sync is timing out, but I have no idea why:
https://dxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/runner/base.py#676

Comment 7

[Andrew Halberstadt [:ahal]]

So here's the script that's timing out:
https://dxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/runner/base.py#761

It looks like it's trying to load the test-container app, which is a dummy app used by mochitests. Here are some ideas:
1. Maybe Mnw isn't using this app or setting it up properly
2. Maybe it was written for emulator ics arm and there is some difference (e.g timing) with emulator x86

I don't have an x86 build currently, and on this wifi I likely won't for quite some time, so I'll take a closer look when I get back from the work week. Needinfo'ing myself to remember.

Comment 8

[Hsin-Yi Tsai (she/her) | away [:hsinyi]]

Set to P2 since this whole suite isn't running on emu-ics

Comment 9

[Andrew Halberstadt [:ahal]]

Sorry, I've been told not to investigate b2g problems anymore :/. Though I'm happy to answer more specific questions if comment 7 doesn't help.

Comment 10

[Pete Moore [:pmoore][:pete]]

Moving to TaskCluster component...