807050 - schedule mozharness b2g emulator reftest on cedar

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Aki Sasaki (not active)]

per bug 774535 comment 64.

Comment 1

[Andrew Halberstadt [:ahal]]

Once this is up and running I have a patch that resulted in a nearly green reftest run (though it is a few weeks out of date). It disables roughly half of the tests and results in ~20 minute jobs with 8 chunks.

It might be better to simply start with layout/reftests/reftest-sanity/reftest.list for now though.

Comment 2

[Aki Sasaki (not active)]

That's currently hardcoded here:
http://hg.mozilla.org/build/mozharness/file/8de53d8a6437/scripts/b2g_emulator_unittest.py#l230

Do we want to make that a config setting?

Comment 3

[Andrew Halberstadt [:ahal]]

That should only happen if self.test_manifest doesn't get set here: http://hg.mozilla.org/build/mozharness/file/8de53d8a6437/scripts/b2g_emulator_unittest.py#l126

Comment 4

[Aki Sasaki (not active)]

Attachment: b2g reftests

testing in staging atm.

Comment 5

[Aki Sasaki (not active)]

Currently getting this:

14:10:25     INFO -  restarting B2G
14:10:25     INFO -  waiting for system-message-listener-ready...
14:10:25     INFO -  done
14:10:25     INFO -  'cp' not found, but 'dd' was found as a replacement
14:10:25     INFO -  args: ['../bin/xpcshell', '-g', '/home/cltbld/talos-slave/test/build/xpcshell', '-v', '170', '-f', '/home/cltbld/talos-slave/test/build/tests/reftest/reftest/components/httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmpRa95Rj';const _SERVER_PORT = '8888'; const _SERVER_ADDR ='10.0.2.2';", '-f', '/home/cltbld/talos-slave/test/build/tests/reftest/server.js']
14:10:25  WARNING -  TEST-UNEXPECTED-FAIL | [Errno 8] Exec format error | Exception caught while running tests.

Comment 6

[Jonathan Griffin (:jgriffin)]

This seems like the same xpcshell problem we were getting with mochitest earlier.  Does the staging environment using the latest mozharness configs?

Comment 7

[Aki Sasaki (not active)]

Attachment: b2g reftests, in 6 chunks, with a fixed manifest path

Tested in staging.
We're still hitting the harness error when running the tests, but we should be able to debug on cedar.

Comment 8

[Aki Sasaki (not active)]

(In reply to Jonathan Griffin (:jgriffin) from comment #6)
> This seems like the same xpcshell problem we were getting with mochitest
> earlier.  Does the staging environment using the latest mozharness configs?

We're using build/mozharness, yes.  And I'm using the latest cedar build.

Comment 9

[Andrew Halberstadt [:ahal]]

That error happens when you use a b2g-built version of xpcshell instead of a desktop one.

Comment 10

[Aki Sasaki (not active)]

We're downloading the xpcshell_url http://runtime-binaries.pvt.build.mozilla.org/tooltool/sha512/d4297e762649b174070a33d039fd062edd9f29a751650f0508327a6cf366b3a35fe24e7cd0f7b728d74f7d15399f9c1adc5b178e5803a3a66bfce7a8dcd62daa

which I think is the right one.

I do see this error:

14:26:05     INFO -  Android Debug Bridge version 1.0.29
14:26:09     INFO -  Traceback (most recent call last):
14:26:09     INFO -    File "runreftestb2g.py", line 537, in main
14:26:09     INFO -      retVal = reftest.startWebServer(options)
14:26:09     INFO -    File "runreftestb2g.py", line 326, in startWebServer
14:26:09     INFO -      retVal = self.server.start()
14:26:09     INFO -    File "/home/cltbld/talos-slave/test/build/tests/reftest/remotereftest.py", line 186, in start
14:26:09     INFO -      self._process = self._automation.Process([xpcshell] + args, env = env)
14:26:09     INFO -    File "/home/cltbld/talos-slave/test/build/tests/reftest/automation.py", line 179, in __init__
14:26:09     INFO -      universal_newlines, startupinfo, creationflags)
14:26:09     INFO -    File "/usr/lib/python2.6/subprocess.py", line 595, in __init__
14:26:09     INFO -      errread, errwrite)
14:26:09     INFO -    File "/usr/lib/python2.6/subprocess.py", line 1092, in _execute_child
14:26:09     INFO -      raise child_exception
14:26:09     INFO -  OSError: [Errno 8] Exec format error
14:26:09     INFO -  'cp' not found, but 'dd' was found as a replacement
14:26:09     INFO -  waiting for system-message-listener-ready...

Hm, I should just attach the log.

Comment 11

[Aki Sasaki (not active)]

Attachment: reftest-4 log

Comment 12

[Andrew Halberstadt [:ahal]]

(In reply to Aki Sasaki [:aki] from comment #10)
> 14:26:09     INFO -    File "/usr/lib/python2.6/subprocess.py", line 1092,
> in _execute_child
> 14:26:09     INFO -      raise child_exception
> 14:26:09     INFO -  OSError: [Errno 8] Exec format error
> 14:26:09     INFO -  'cp' not found, but 'dd' was found as a replacement

Yep, this is the exact same traceback I get when I don't specify --xre-path properly on my local machine (keep in mind it has to be the directory containing the binary).

I see:
'--xre-path', '/home/cltbld/talos-slave/test/build/xpcshell'

Are we sure the new config isn't extracting the above URL somewhere else?

Comment 13

[Aki Sasaki (not active)]

(In reply to Andrew Halberstadt [:ahal] from comment #12)
> (In reply to Aki Sasaki [:aki] from comment #10)
> > 14:26:09     INFO -    File "/usr/lib/python2.6/subprocess.py", line 1092,
> > in _execute_child
> > 14:26:09     INFO -      raise child_exception
> > 14:26:09     INFO -  OSError: [Errno 8] Exec format error
> > 14:26:09     INFO -  'cp' not found, but 'dd' was found as a replacement
> 
> Yep, this is the exact same traceback I get when I don't specify --xre-path
> properly on my local machine (keep in mind it has to be the directory
> containing the binary).
> 
> I see:
> '--xre-path', '/home/cltbld/talos-slave/test/build/xpcshell'
> 
> Are we sure the new config isn't extracting the above URL somewhere else?

14:23:21     INFO - Running command: ['unzip', '-q', '-o', '/home/cltbld/talos-slave/test/build/d4297e762649b174070a33d039fd062edd9f29a751650f0508327a6cf366b3a35fe24e7cd0f7b728d74f7d15399f9c1adc5b178e5803a3a66bfce7a8dcd62daa'] in /home/cltbld/talos-slave/test/build/xpcshell

So it's extracting it in the right place.  (I wish we had verbose unzip here.)

xpcshell should be in bin/xpcshell.

Comment 14

[Jonathan Griffin (:jgriffin)]

Sorry, my fault.  I fixed this for mochitest:

http://hg.mozilla.org/build/mozharness/file/13531beb7ac6/scripts/b2g_emulator_unittest.py#l189

but not for reftest.  Will post a patch.

Comment 15

[Jonathan Griffin (:jgriffin)]

Attachment: specify the 'bin' folder for xpcshell,

Comment 16

[Jonathan Griffin (:jgriffin)]

Comment on attachment 676828 [details] [diff] [review]
specify the 'bin' folder for xpcshell,

http://hg.mozilla.org/build/mozharness/rev/1e2d7b3fe9cc

Comment 17

[[e-mail] (:bhearsum)]

Comment on attachment 676774 [details] [diff] [review]
b2g reftests, in 6 chunks, with a fixed manifest path

Review of attachment 676774 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/b2g_config.py
@@ +157,5 @@
>                      ],
>                  },
> +                'reftest-1': {
> +                    'extra_args': [
> +                        '--cfg', 'b2g/emulator_prod_config.py',

Sorry, can you remind me why "prod" config is appropriate here even though this ends up being used in staging, too?

Comment 18

[Aki Sasaki (not active)]

(In reply to Ben Hearsum [:bhearsum] from comment #17)
> Sorry, can you remind me why "prod" config is appropriate here even though
> this ends up being used in staging, too?

http://hg.mozilla.org/build/mozharness/file/b676ea834692/configs/b2g/emulator_prod_config.py

This config file works in staging as well; I've been using it there for testing without modifications.  There are no keys, no production vs staging graphserver/upload locations/reporting differences/etc.  The word "prod" here differentiates it from a standalone command line config, as opposed to specifying it isn't appropriate for staging.

Comment 19

[Aki Sasaki (not active)]

Comment on attachment 676774 [details] [diff] [review]
b2g reftests, in 6 chunks, with a fixed manifest path

http://hg.mozilla.org/build/buildbot-configs/rev/1e01d75025af

Comment 20

[[e-mail] (:bhearsum)]

In production.