Closed Bug 807050 Opened 7 years ago Closed 7 years ago

schedule mozharness b2g emulator reftest on cedar

Categories

(Release Engineering :: Applications: MozharnessCore, defect)

All
Linux
defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: aki, Assigned: aki)

References

Details

(Whiteboard: [mozharness][unittest])

Attachments

(3 files, 1 obsolete file)

Once this is up and running I have a patch that resulted in a nearly green reftest run (though it is a few weeks out of date). It disables roughly half of the tests and results in ~20 minute jobs with 8 chunks.

It might be better to simply start with layout/reftests/reftest-sanity/reftest.list for now though.
That's currently hardcoded here:
http://hg.mozilla.org/build/mozharness/file/8de53d8a6437/scripts/b2g_emulator_unittest.py#l230

Do we want to make that a config setting?
Attached patch b2g reftests (obsolete) — Splinter Review
testing in staging atm.
Currently getting this:

14:10:25     INFO -  restarting B2G
14:10:25     INFO -  waiting for system-message-listener-ready...
14:10:25     INFO -  done
14:10:25     INFO -  'cp' not found, but 'dd' was found as a replacement
14:10:25     INFO -  args: ['../bin/xpcshell', '-g', '/home/cltbld/talos-slave/test/build/xpcshell', '-v', '170', '-f', '/home/cltbld/talos-slave/test/build/tests/reftest/reftest/components/httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmpRa95Rj';const _SERVER_PORT = '8888'; const _SERVER_ADDR ='10.0.2.2';", '-f', '/home/cltbld/talos-slave/test/build/tests/reftest/server.js']
14:10:25  WARNING -  TEST-UNEXPECTED-FAIL | [Errno 8] Exec format error | Exception caught while running tests.
This seems like the same xpcshell problem we were getting with mochitest earlier.  Does the staging environment using the latest mozharness configs?
Tested in staging.
We're still hitting the harness error when running the tests, but we should be able to debug on cedar.
Attachment #676731 - Attachment is obsolete: true
Attachment #676774 - Flags: review?(bhearsum)
(In reply to Jonathan Griffin (:jgriffin) from comment #6)
> This seems like the same xpcshell problem we were getting with mochitest
> earlier.  Does the staging environment using the latest mozharness configs?

We're using build/mozharness, yes.  And I'm using the latest cedar build.
That error happens when you use a b2g-built version of xpcshell instead of a desktop one.
We're downloading the xpcshell_url http://runtime-binaries.pvt.build.mozilla.org/tooltool/sha512/d4297e762649b174070a33d039fd062edd9f29a751650f0508327a6cf366b3a35fe24e7cd0f7b728d74f7d15399f9c1adc5b178e5803a3a66bfce7a8dcd62daa

which I think is the right one.

I do see this error:

14:26:05     INFO -  Android Debug Bridge version 1.0.29
14:26:09     INFO -  Traceback (most recent call last):
14:26:09     INFO -    File "runreftestb2g.py", line 537, in main
14:26:09     INFO -      retVal = reftest.startWebServer(options)
14:26:09     INFO -    File "runreftestb2g.py", line 326, in startWebServer
14:26:09     INFO -      retVal = self.server.start()
14:26:09     INFO -    File "/home/cltbld/talos-slave/test/build/tests/reftest/remotereftest.py", line 186, in start
14:26:09     INFO -      self._process = self._automation.Process([xpcshell] + args, env = env)
14:26:09     INFO -    File "/home/cltbld/talos-slave/test/build/tests/reftest/automation.py", line 179, in __init__
14:26:09     INFO -      universal_newlines, startupinfo, creationflags)
14:26:09     INFO -    File "/usr/lib/python2.6/subprocess.py", line 595, in __init__
14:26:09     INFO -      errread, errwrite)
14:26:09     INFO -    File "/usr/lib/python2.6/subprocess.py", line 1092, in _execute_child
14:26:09     INFO -      raise child_exception
14:26:09     INFO -  OSError: [Errno 8] Exec format error
14:26:09     INFO -  'cp' not found, but 'dd' was found as a replacement
14:26:09     INFO -  waiting for system-message-listener-ready...

Hm, I should just attach the log.
Attached file reftest-4 log
(In reply to Aki Sasaki [:aki] from comment #10)
> 14:26:09     INFO -    File "/usr/lib/python2.6/subprocess.py", line 1092,
> in _execute_child
> 14:26:09     INFO -      raise child_exception
> 14:26:09     INFO -  OSError: [Errno 8] Exec format error
> 14:26:09     INFO -  'cp' not found, but 'dd' was found as a replacement

Yep, this is the exact same traceback I get when I don't specify --xre-path properly on my local machine (keep in mind it has to be the directory containing the binary).

I see:
'--xre-path', '/home/cltbld/talos-slave/test/build/xpcshell'

Are we sure the new config isn't extracting the above URL somewhere else?
Blocks: 807125
(In reply to Andrew Halberstadt [:ahal] from comment #12)
> (In reply to Aki Sasaki [:aki] from comment #10)
> > 14:26:09     INFO -    File "/usr/lib/python2.6/subprocess.py", line 1092,
> > in _execute_child
> > 14:26:09     INFO -      raise child_exception
> > 14:26:09     INFO -  OSError: [Errno 8] Exec format error
> > 14:26:09     INFO -  'cp' not found, but 'dd' was found as a replacement
> 
> Yep, this is the exact same traceback I get when I don't specify --xre-path
> properly on my local machine (keep in mind it has to be the directory
> containing the binary).
> 
> I see:
> '--xre-path', '/home/cltbld/talos-slave/test/build/xpcshell'
> 
> Are we sure the new config isn't extracting the above URL somewhere else?

14:23:21     INFO - Running command: ['unzip', '-q', '-o', '/home/cltbld/talos-slave/test/build/d4297e762649b174070a33d039fd062edd9f29a751650f0508327a6cf366b3a35fe24e7cd0f7b728d74f7d15399f9c1adc5b178e5803a3a66bfce7a8dcd62daa'] in /home/cltbld/talos-slave/test/build/xpcshell

So it's extracting it in the right place.  (I wish we had verbose unzip here.)

xpcshell should be in bin/xpcshell.
Sorry, my fault.  I fixed this for mochitest:

http://hg.mozilla.org/build/mozharness/file/13531beb7ac6/scripts/b2g_emulator_unittest.py#l189

but not for reftest.  Will post a patch.
Attachment #676828 - Flags: review?(ahalberstadt) → review+
Comment on attachment 676774 [details] [diff] [review]
b2g reftests, in 6 chunks, with a fixed manifest path

Review of attachment 676774 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/b2g_config.py
@@ +157,5 @@
>                      ],
>                  },
> +                'reftest-1': {
> +                    'extra_args': [
> +                        '--cfg', 'b2g/emulator_prod_config.py',

Sorry, can you remind me why "prod" config is appropriate here even though this ends up being used in staging, too?
Attachment #676774 - Flags: review?(bhearsum) → review+
(In reply to Ben Hearsum [:bhearsum] from comment #17)
> Sorry, can you remind me why "prod" config is appropriate here even though
> this ends up being used in staging, too?

http://hg.mozilla.org/build/mozharness/file/b676ea834692/configs/b2g/emulator_prod_config.py

This config file works in staging as well; I've been using it there for testing without modifications.  There are no keys, no production vs staging graphserver/upload locations/reporting differences/etc.  The word "prod" here differentiates it from a standalone command line config, as opposed to specifying it isn't appropriate for staging.
Comment on attachment 676774 [details] [diff] [review]
b2g reftests, in 6 chunks, with a fixed manifest path

http://hg.mozilla.org/build/buildbot-configs/rev/1e01d75025af
Attachment #676774 - Flags: checked-in+
In production.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
Component: General Automation → Mozharness
You need to log in before you can comment on or make changes to this bug.