schedule mozharness b2g emulator reftest on cedar

RESOLVED FIXED

Status

Release Engineering
Mozharness
RESOLVED FIXED
5 years ago
3 years ago

People

(Reporter: aki, Assigned: aki)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [mozharness][unittest])

Attachments

(3 attachments, 1 obsolete attachment)

(Assignee)

Description

5 years ago
per bug 774535 comment 64.
Once this is up and running I have a patch that resulted in a nearly green reftest run (though it is a few weeks out of date). It disables roughly half of the tests and results in ~20 minute jobs with 8 chunks.

It might be better to simply start with layout/reftests/reftest-sanity/reftest.list for now though.
(Assignee)

Comment 2

5 years ago
That's currently hardcoded here:
http://hg.mozilla.org/build/mozharness/file/8de53d8a6437/scripts/b2g_emulator_unittest.py#l230

Do we want to make that a config setting?
That should only happen if self.test_manifest doesn't get set here: http://hg.mozilla.org/build/mozharness/file/8de53d8a6437/scripts/b2g_emulator_unittest.py#l126
(Assignee)

Comment 4

5 years ago
Created attachment 676731 [details] [diff] [review]
b2g reftests

testing in staging atm.
(Assignee)

Comment 5

5 years ago
Currently getting this:

14:10:25     INFO -  restarting B2G
14:10:25     INFO -  waiting for system-message-listener-ready...
14:10:25     INFO -  done
14:10:25     INFO -  'cp' not found, but 'dd' was found as a replacement
14:10:25     INFO -  args: ['../bin/xpcshell', '-g', '/home/cltbld/talos-slave/test/build/xpcshell', '-v', '170', '-f', '/home/cltbld/talos-slave/test/build/tests/reftest/reftest/components/httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmpRa95Rj';const _SERVER_PORT = '8888'; const _SERVER_ADDR ='10.0.2.2';", '-f', '/home/cltbld/talos-slave/test/build/tests/reftest/server.js']
14:10:25  WARNING -  TEST-UNEXPECTED-FAIL | [Errno 8] Exec format error | Exception caught while running tests.
This seems like the same xpcshell problem we were getting with mochitest earlier.  Does the staging environment using the latest mozharness configs?
(Assignee)

Comment 7

5 years ago
Created attachment 676774 [details] [diff] [review]
b2g reftests, in 6 chunks, with a fixed manifest path

Tested in staging.
We're still hitting the harness error when running the tests, but we should be able to debug on cedar.
Attachment #676731 - Attachment is obsolete: true
Attachment #676774 - Flags: review?(bhearsum)
(Assignee)

Comment 8

5 years ago
(In reply to Jonathan Griffin (:jgriffin) from comment #6)
> This seems like the same xpcshell problem we were getting with mochitest
> earlier.  Does the staging environment using the latest mozharness configs?

We're using build/mozharness, yes.  And I'm using the latest cedar build.
That error happens when you use a b2g-built version of xpcshell instead of a desktop one.
(Assignee)

Comment 10

5 years ago
We're downloading the xpcshell_url http://runtime-binaries.pvt.build.mozilla.org/tooltool/sha512/d4297e762649b174070a33d039fd062edd9f29a751650f0508327a6cf366b3a35fe24e7cd0f7b728d74f7d15399f9c1adc5b178e5803a3a66bfce7a8dcd62daa

which I think is the right one.

I do see this error:

14:26:05     INFO -  Android Debug Bridge version 1.0.29
14:26:09     INFO -  Traceback (most recent call last):
14:26:09     INFO -    File "runreftestb2g.py", line 537, in main
14:26:09     INFO -      retVal = reftest.startWebServer(options)
14:26:09     INFO -    File "runreftestb2g.py", line 326, in startWebServer
14:26:09     INFO -      retVal = self.server.start()
14:26:09     INFO -    File "/home/cltbld/talos-slave/test/build/tests/reftest/remotereftest.py", line 186, in start
14:26:09     INFO -      self._process = self._automation.Process([xpcshell] + args, env = env)
14:26:09     INFO -    File "/home/cltbld/talos-slave/test/build/tests/reftest/automation.py", line 179, in __init__
14:26:09     INFO -      universal_newlines, startupinfo, creationflags)
14:26:09     INFO -    File "/usr/lib/python2.6/subprocess.py", line 595, in __init__
14:26:09     INFO -      errread, errwrite)
14:26:09     INFO -    File "/usr/lib/python2.6/subprocess.py", line 1092, in _execute_child
14:26:09     INFO -      raise child_exception
14:26:09     INFO -  OSError: [Errno 8] Exec format error
14:26:09     INFO -  'cp' not found, but 'dd' was found as a replacement
14:26:09     INFO -  waiting for system-message-listener-ready...

Hm, I should just attach the log.
(Assignee)

Comment 11

5 years ago
Created attachment 676787 [details]
reftest-4 log
(In reply to Aki Sasaki [:aki] from comment #10)
> 14:26:09     INFO -    File "/usr/lib/python2.6/subprocess.py", line 1092,
> in _execute_child
> 14:26:09     INFO -      raise child_exception
> 14:26:09     INFO -  OSError: [Errno 8] Exec format error
> 14:26:09     INFO -  'cp' not found, but 'dd' was found as a replacement

Yep, this is the exact same traceback I get when I don't specify --xre-path properly on my local machine (keep in mind it has to be the directory containing the binary).

I see:
'--xre-path', '/home/cltbld/talos-slave/test/build/xpcshell'

Are we sure the new config isn't extracting the above URL somewhere else?
(Assignee)

Updated

5 years ago
Blocks: 807125
(Assignee)

Comment 13

5 years ago
(In reply to Andrew Halberstadt [:ahal] from comment #12)
> (In reply to Aki Sasaki [:aki] from comment #10)
> > 14:26:09     INFO -    File "/usr/lib/python2.6/subprocess.py", line 1092,
> > in _execute_child
> > 14:26:09     INFO -      raise child_exception
> > 14:26:09     INFO -  OSError: [Errno 8] Exec format error
> > 14:26:09     INFO -  'cp' not found, but 'dd' was found as a replacement
> 
> Yep, this is the exact same traceback I get when I don't specify --xre-path
> properly on my local machine (keep in mind it has to be the directory
> containing the binary).
> 
> I see:
> '--xre-path', '/home/cltbld/talos-slave/test/build/xpcshell'
> 
> Are we sure the new config isn't extracting the above URL somewhere else?

14:23:21     INFO - Running command: ['unzip', '-q', '-o', '/home/cltbld/talos-slave/test/build/d4297e762649b174070a33d039fd062edd9f29a751650f0508327a6cf366b3a35fe24e7cd0f7b728d74f7d15399f9c1adc5b178e5803a3a66bfce7a8dcd62daa'] in /home/cltbld/talos-slave/test/build/xpcshell

So it's extracting it in the right place.  (I wish we had verbose unzip here.)

xpcshell should be in bin/xpcshell.
Sorry, my fault.  I fixed this for mochitest:

http://hg.mozilla.org/build/mozharness/file/13531beb7ac6/scripts/b2g_emulator_unittest.py#l189

but not for reftest.  Will post a patch.
Created attachment 676828 [details] [diff] [review]
specify the 'bin' folder for xpcshell,
Attachment #676828 - Flags: review?(ahalberstadt)
Attachment #676828 - Flags: review?(ahalberstadt) → review+
Comment on attachment 676828 [details] [diff] [review]
specify the 'bin' folder for xpcshell,

http://hg.mozilla.org/build/mozharness/rev/1e2d7b3fe9cc
Attachment #676828 - Flags: checked-in+
Blocks: 800025
Comment on attachment 676774 [details] [diff] [review]
b2g reftests, in 6 chunks, with a fixed manifest path

Review of attachment 676774 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/b2g_config.py
@@ +157,5 @@
>                      ],
>                  },
> +                'reftest-1': {
> +                    'extra_args': [
> +                        '--cfg', 'b2g/emulator_prod_config.py',

Sorry, can you remind me why "prod" config is appropriate here even though this ends up being used in staging, too?
Attachment #676774 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 18

5 years ago
(In reply to Ben Hearsum [:bhearsum] from comment #17)
> Sorry, can you remind me why "prod" config is appropriate here even though
> this ends up being used in staging, too?

http://hg.mozilla.org/build/mozharness/file/b676ea834692/configs/b2g/emulator_prod_config.py

This config file works in staging as well; I've been using it there for testing without modifications.  There are no keys, no production vs staging graphserver/upload locations/reporting differences/etc.  The word "prod" here differentiates it from a standalone command line config, as opposed to specifying it isn't appropriate for staging.
(Assignee)

Comment 19

5 years ago
Comment on attachment 676774 [details] [diff] [review]
b2g reftests, in 6 chunks, with a fixed manifest path

http://hg.mozilla.org/build/buildbot-configs/rev/1e01d75025af
Attachment #676774 - Flags: checked-in+
In production.
(Assignee)

Updated

5 years ago
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering

Updated

3 years ago
Component: General Automation → Mozharness
You need to log in before you can comment on or make changes to this bug.