774535 - Write mozharness script for B2G versions of desktop unittests

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Jonathan Griffin (:jgriffin)]

This can probably be implemented as b2g_unittest.py, depending on the work being done in bug 650880.  This should allow us to run xpcshell, mochitest, and reftest on B2G emulators.

Comment 1

[Jonathan Griffin (:jgriffin)]

I'll provide instructions on running mochitest, reftest, and webapi tests locally.

Comment 2

[Jonathan Griffin (:jgriffin)]

Aki, I've made a wiki page for these instructions, at https://wiki.mozilla.org/Auto-tools/B2G/Running_Emulator_Tests.  This has instructions for WebAPI tests and mochitests; instructions for reftests coming soon.

There are a few pieces of work on our side which I will file bugs for blocking this one, but I think there's enough there to get you started.

If you want to run an emulator mozharness script locally, you can use one of the nightly emulator builds at https://releases.mozilla.com/b2g/latest/; ping me for credentials if you need them.

Comment 3

[Jonathan Griffin (:jgriffin)]

https://wiki.mozilla.org/Auto-tools/B2G/Running_Emulator_Tests#Reftests has been updated with reftest instructions.

Comment 4

[Aki Sasaki (not active)]

Attachment: (untested) b2g emulator tests: webapi

This adds webapi emulator testing support to marionette.py per https://wiki.mozilla.org/Auto-tools/B2G/Running_Emulator_Tests#WebAPI_Tests .

This needs testing, which would be easier with the --install-gecko support + emulators on the test machines.  In theory this could land beforehand, but we can't switch it on til then anyway.

Comment 5

[Aki Sasaki (not active)]

Attachment: (untested) b2g emulator tests: webapi, with more user-friendly defaults

Rather than fatal() if 'install' is in the actions list, just skip it.

Comment 6

[Aki Sasaki (not active)]

Ooc, how do we clean up the emulator environment after we're done / before we start?  (i.e., remove whatever files --install-gecko added.)

Comment 7

[Jonathan Griffin (:jgriffin)]

(In reply to Aki Sasaki [:aki] from comment #6)
> Ooc, how do we clean up the emulator environment after we're done / before
> we start?  (i.e., remove whatever files --install-gecko added.)

I don't believe there's any need, since the next test run will overwrite them.  The emulator which will get installed on the slaves will come with a pre-built gecko; I don't see any reason to restore that version at the end of a test run, since we'll never be interested in testing that version.

Comment 8

[Aki Sasaki (not active)]

If we rely on files to overwrite on copy, then any file moves/renames/removals might leave old file cruft behind.  It would probably be best practice to add some cleanup, but if you're not worried we can make that low priority.

Comment 9

[Jonathan Griffin (:jgriffin)]

I think we can add that to the --install-gecko step.  It copies files to only one directory; we can delete that directory before copying in order to ensure a clean copy exists during test.

Comment 10

[Aki Sasaki (not active)]

Attachment: (untested) b2g emulator tests: webapi, with config files

3rd iteration.
I have the mochitest/reftest stuff in an unfinished state; I'll try to get them to a testable state.

Comment 11

[Jonathan Griffin (:jgriffin)]

Comment on attachment 664725 [details] [diff] [review]
(untested) b2g emulator tests: webapi, with config files

Review of attachment 664725 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good except for one minor point:

::: configs/marionette/windows_config.py
@@ -2,5 @@
>  
>  config = {
>      # marionette options
>      "type": "browser",
> -    "marionette_address": "localhost:2828",

We'll actually need two separate sets of config files for Marionette; one for desktop Firefox tests (the current versions) and one for B2G emulator tests (the new versions).

Comment 12

[Aki Sasaki (not active)]

(In reply to Jonathan Griffin (:jgriffin) from comment #11)
> Comment on attachment 664725 [details] [diff] [review]
> (untested) b2g emulator tests: webapi, with config files
> 
> Review of attachment 664725 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Looks good except for one minor point:
> 
> ::: configs/marionette/windows_config.py
> @@ -2,5 @@
> >  
> >  config = {
> >      # marionette options
> >      "type": "browser",
> > -    "marionette_address": "localhost:2828",
> 
> We'll actually need two separate sets of config files for Marionette; one
> for desktop Firefox tests (the current versions) and one for B2G emulator
> tests (the new versions).

I think splinter reviews don't like hg cp.

The raw diff says:

diff --git a/configs/marionette/windows_config.py b/configs/marionette/windows_emulator_config.py
copy from configs/marionette/windows_config.py
copy to configs/marionette/windows_emulator_config.py
--- a/configs/marionette/windows_config.py
+++ b/configs/marionette/windows_emulator_config.py

Comment 13

[Jonathan Griffin (:jgriffin)]

(In reply to Aki Sasaki [:aki] from comment #12)
> I think splinter reviews don't like hg cp.
> 
> The raw diff says:
> 
> diff --git a/configs/marionette/windows_config.py
> b/configs/marionette/windows_emulator_config.py
> copy from configs/marionette/windows_config.py
> copy to configs/marionette/windows_emulator_config.py
> --- a/configs/marionette/windows_config.py
> +++ b/configs/marionette/windows_emulator_config.py

Ah quite so! Sorry for the confusion!  The emulator versions will need a "type" parameter of "b2g", otherwise they look fine.

Comment 14

[Aki Sasaki (not active)]

(In reply to Jonathan Griffin (:jgriffin) from comment #13)
> Ah quite so! Sorry for the confusion!  The emulator versions will need a
> "type" parameter of "b2g", otherwise they look fine.

Fixed, thanks for the feedback!

Comment 15

[Aki Sasaki (not active)]

Attachment: b2g emulator tests: webapi, with ability to install emulator

Thinking about landing this to avoid bitrot.

Comment 16

[Aki Sasaki (not active)]

Attachment: wip: b2g unit tests.

Not there yet.
With luck, I'll be able to spend most of tomorrow on this.

Comment 17

[Jonathan Griffin (:jgriffin)]

Comment on attachment 666794 [details] [diff] [review]
b2g emulator tests: webapi, with ability to install emulator

Review of attachment 666794 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/marionette.py
@@ +183,5 @@
>          cmd = [python, '-u', os.path.join(dirs['abs_marionette_dir'],
>                                            'runtests.py')]
> +        if self.config.get('emulator'):
> +            cmd.extend(self._build_arg('--emulator', self.config['emulator']))
> +            cmd.extend(self._build_arg('--install-gecko', self.installer_path))

We've changed --install-gecko to --gecko-path, sorry for the switch.  I'm not sure which files self.installer_path will point to, do you know?  --gecko-path expects the contents of $OBJDIR/dist/b2g.  I guess we can adjust that path as needed depending on the file structure at self.installer_path.

Comment 18

[Aki Sasaki (not active)]

(In reply to Jonathan Griffin (:jgriffin) from comment #17)
> We've changed --install-gecko to --gecko-path, sorry for the switch.

No worries, easy enough to change.

> I'm
> not sure which files self.installer_path will point to, do you know?
> --gecko-path expects the contents of $OBJDIR/dist/b2g.  I guess we can
> adjust that path as needed depending on the file structure at
> self.installer_path.

Right now it'll be whatever zipfile is downloaded from the ics build.
I guess we have to extract and point to the extracted directory.

I've landed this with the --gecko-path fix.

Comment 19

[Aki Sasaki (not active)]

Comment on attachment 666794 [details] [diff] [review]
b2g emulator tests: webapi, with ability to install emulator

http://hg.mozilla.org/build/mozharness/rev/82b0bfed2771

Comment 20

[Aki Sasaki (not active)]

Attachment: fixes for webapi test

This should allow for the .tar.gz that the ics b2g gecko build uploads, create a 'gecko' directory under work_dir that we'll untar into, and extracts the installer into that directory.

We may need to adjust the --gecko-path if we need to specify a subdirectory of gecko/ , but that should hopefully be easy enough.

Comment 21

[Aki Sasaki (not active)]

Attachment: b2g_emulator_unittest.py

Still pretty rough, but getting closer.

Comment 22

[Aki Sasaki (not active)]

For b2g_emulator_unittest.py's config files, I'm thinking configs/unittest/b2g_emulator_mochitest.py and b2g_emulator_reftest.py would make sense.  We don't have to split by platform or branch currently.

Comment 23

[Jonathan Griffin (:jgriffin)]

Comment on attachment 667274 [details] [diff] [review]
fixes for webapi test

Review of attachment 667274 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks.  I'll check this in since Aki is PTO.

Comment 24

[Jonathan Griffin (:jgriffin)]

Comment on attachment 667274 [details] [diff] [review]
fixes for webapi test

http://hg.mozilla.org/build/mozharness/rev/192ee18cb135

Comment 25

[Jonathan Griffin (:jgriffin)]

@ahal is going to look at the unittest mozharness script specifically in a reftest context.

Comment 26

[Andrew Halberstadt [:ahal]]

Attachment: incremental wip patch

This patch correctly configures the reftest parameters, xpcshell, tests and emulator. The emulator gets launched but it then hits a "failed to get nsJSRuntimeService!" error. I think it might have something to do with my jury-rigged xpcshell.zip

Comment 27

[Jonathan Griffin (:jgriffin)]

Attachment: New prod config for WebAPI tests

Comment 28

[Jonathan Griffin (:jgriffin)]

Attachment: Updated WebAPI mozharness script

This works fine for me locally.

Comment 29

[Andrew Halberstadt [:ahal]]

Comment on attachment 670063 [details] [diff] [review]
Updated WebAPI mozharness script

Review of attachment 670063 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good. Didn't get a chance to test it locally, but I don't see anything wrong.

::: scripts/marionette.py
@@ +160,5 @@
>              tar = self.query_exe('tar', return_type='list')
>              self.run_command(tar + ['zxf', self.installer_path],
>                               cwd=dirs['abs_gecko_dir'],
>                               error_list=TarErrorList,
>                               halt_on_failure=True)

I know this isn't part of this patch, but another way of doing this is to pass the gecko url to "installer_url" and making sure the install step runs. You'd need another minor change that I made in my patch for it to work though. Anyway either I should change my script to do it this way or vice versa.

Comment 30

[Andrew Halberstadt [:ahal]]

Attachment: B2G emulator unittest mozharness script

This patch sets up things for running B2G unit tests on emulators. However the reftest harness doesn't get installed properly when using --gecko-path and the mochitest harness is waiting for a devicemanager fix to work it's way down the pipeline. We might want to work on bug 800111 in the meantime.

Comment 31

[Andrew Halberstadt [:ahal]]

Comment on attachment 670120 [details] [diff] [review]
B2G emulator unittest mozharness script

I should mention in order to test I put up an emulator, xpcshell dir, tests.zip and gecko build up on a local at 127.0.1.1/b2g and ran
python scripts/b2g_emulator_unittest.py --cfg configs/b2g/emulator_test_config.py

Comment 32

[Jonathan Griffin (:jgriffin)]

(In reply to Andrew Halberstadt [:ahal] from comment #29)
> ::: scripts/marionette.py
> @@ +160,5 @@
> >              tar = self.query_exe('tar', return_type='list')
> >              self.run_command(tar + ['zxf', self.installer_path],
> >                               cwd=dirs['abs_gecko_dir'],
> >                               error_list=TarErrorList,
> >                               halt_on_failure=True)
> 
> I know this isn't part of this patch, but another way of doing this is to
> pass the gecko url to "installer_url" and making sure the install step runs.
> You'd need another minor change that I made in my patch for it to work
> though. Anyway either I should change my script to do it this way or vice
> versa.

I landed this as-is since it's working, and didn't want to take the time to change the code.

http://hg.mozilla.org/build/mozharness/rev/c169f6e9f8cb
http://hg.mozilla.org/build/mozharness/rev/9f0b27b474cb

Comment 33

[Andrew Halberstadt [:ahal]]

Attachment: B2G emulator unittest mozharness script v2

I still haven't been able to get a successful run due to the many mochitest and reftest breakages that happened over the week, but this patch sets everything up properly and shouldn't need to be changed at all when they get unbroken. I propose checking it in but not enabling it in production until some better local testing is conducted.

I'm flagging Armen for feedback due to the change in testbase.py. I'm 99% sure that self.workdir is a typo since it isn't defined anywhere and every other method uses abs_dirs['abs_work_dir'], but it has the potential to break existing automation. I'd be fine with working around it in the b2g emulator script if the risk is too high.

Comment 34

[Jonathan Griffin (:jgriffin)]

Comment on attachment 670816 [details] [diff] [review]
B2G emulator unittest mozharness script v2

Review of attachment 670816 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good, I just have a few suggestions/questions.

::: configs/b2g/emulator_prod_config.py
@@ +4,5 @@
> +    # mozharness options
> +    "application": "b2g",
> +    "emulator_url": "http://runtime-binaries.pvt.build.mozilla.org/tooltool/sha512/69cba761fc84f8db3b5f536c60027b6515acd5b3084156c67319bdb8e18b06170aedff64333692a80ea1ef8f9c5bdc6fc688b559703b59b5071aebb1bd6ffddf",
> +    "installer_url": "http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cedar-ics_armv7a_gecko/1349452518/b2g-18.0a1.en-US.eabi-arm.tar.gz",
> +    "tests_url": "http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cedar-ics_armv7a_gecko/1349452518/b2g-18.0a1.en-US.eabi-arm.tests.zip",

We shouldn't specify installer_url or tests_url in the production config; they get supplied by buildbot.

::: mozharness/mozilla/testing/testbase.py
@@ +158,5 @@
>          """Generic download+unzip.
>          This is hardcoded to halt on failure.
>          We should probably change some other methods to call this."""
> +        dirs = self.query_abs_dirs()
> +        zipfile = self.download_file(url, parent_dir=dirs['abs_work_dir'],

This might affect mozharness/mozilla/testing/talos.py; have you verified it's harmless there?

::: scripts/b2g_emulator_unittest.py
@@ +62,5 @@
> +        {"action": "store",
> +         "dest": "adb_path",
> +         "default": os.getenv('ADB_PATH'),
> +         "help": "Path to adb",
> +        }],

adb is always in the emulator package, so I think we should just calculate the path to adb in query_abs_dirs, rather than having to pass it as an argument

@@ +83,5 @@
> +    virtualenv_modules = [
> +        { 'manifestparser': os.path.join(mozbase_dir, 'manifestdestiny') },
> +        { 'mozhttpd': os.path.join(mozbase_dir, 'mozhttpd') },
> +        { 'mozinfo': os.path.join(mozbase_dir, 'mozinfo') },
> +        { 'mozinstall': os.path.join(mozbase_dir, 'mozinstall') },

mozinstall now depends on mozfile, so we should install it here too (I need to update the Marionette script for this as well).

Comment 35

[Jeff Hammel]

> @@ +83,5 @@
> > +    virtualenv_modules = [
> > +        { 'manifestparser': os.path.join(mozbase_dir, 'manifestdestiny') },
> > +        { 'mozhttpd': os.path.join(mozbase_dir, 'mozhttpd') },
> > +        { 'mozinfo': os.path.join(mozbase_dir, 'mozinfo') },
> > +        { 'mozinstall': os.path.join(mozbase_dir, 'mozinstall') },
> 
> mozinstall now depends on mozfile, so we should install it here too (I need to update the Marionette script for this as well).

So we don't have to tackle this in this bug, but we should figure out a way to get away from specifying dependencies verbosely.  As jgriffin points out in this case, it will break whenever dependencies change.  I also believe but could be wrong that you will have to specify virtualenv modules in unrolled dependency order (see e.g. https://github.com/mozilla/mozbase/blob/master/setup_development.py for an example of unrolling dependencies).  While setuptools is mostly kinda lousy, it does do dependencies correctly and I'd much rather lean on this than hand-roll our deps in order in a bunch of different places.

If we don't want to depend on setuptools dependency resolution (for whatever reason) we should probably move towards a script that can unroll an arbitrary set of deps for us, probably built on pip

Comment 36

[Andrew Halberstadt [:ahal]]

(In reply to Jonathan Griffin (:jgriffin) from comment #34)
> ::: mozharness/mozilla/testing/testbase.py
> @@ +158,5 @@
> >          """Generic download+unzip.
> >          This is hardcoded to halt on failure.
> >          We should probably change some other methods to call this."""
> > +        dirs = self.query_abs_dirs()
> > +        zipfile = self.download_file(url, parent_dir=dirs['abs_work_dir'],
> 
> This might affect mozharness/mozilla/testing/talos.py; have you verified
> it's harmless there?

It looks like talos.py won't be a problem (see http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/testing/talos.py#125). But yes, it could cause problems for something else which is why I cc'ed Armen. I'm convinced it's a bug, but I can defer if we are still unsure.

Comment 37

[Andrew Halberstadt [:ahal]]

Attachment: B2G emulator unittest - address review comments (inc)

Comment 38

[Jonathan Griffin (:jgriffin)]

Comment on attachment 670895 [details] [diff] [review]
B2G emulator unittest - address review comments (inc)

Review of attachment 670895 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm, thanks!

Comment 39

[Andrew Halberstadt [:ahal]]

Comment on attachment 670895 [details] [diff] [review]
B2G emulator unittest - address review comments (inc)

http://hg.mozilla.org/build/mozharness/rev/0c7d04dcb74a

I'd be very surprised if the abs_work_dir changed caused any problems, but I'll keep a close eye on things and be ready to back out just in case.

Comment 40

[Armen [:armenzg]]

(In reply to Jonathan Griffin (:jgriffin) from comment #34)
> Comment on attachment 670816 [details] [diff] [review]
> B2G emulator unittest mozharness script v2
> 
> Review of attachment 670816 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Looks good, I just have a few suggestions/questions.
> 
> ::: configs/b2g/emulator_prod_config.py
> @@ +4,5 @@
> > +    # mozharness options
> > +    "application": "b2g",
> > +    "emulator_url": "http://runtime-binaries.pvt.build.mozilla.org/tooltool/sha512/69cba761fc84f8db3b5f536c60027b6515acd5b3084156c67319bdb8e18b06170aedff64333692a80ea1ef8f9c5bdc6fc688b559703b59b5071aebb1bd6ffddf",
> > +    "installer_url": "http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cedar-ics_armv7a_gecko/1349452518/b2g-18.0a1.en-US.eabi-arm.tar.gz",
> > +    "tests_url": "http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cedar-ics_armv7a_gecko/1349452518/b2g-18.0a1.en-US.eabi-arm.tests.zip",
> 
> We shouldn't specify installer_url or tests_url in the production config;
> they get supplied by buildbot.
> 
What jgriffin says.

> ::: mozharness/mozilla/testing/testbase.py
> @@ +158,5 @@
> >          """Generic download+unzip.
> >          This is hardcoded to halt on failure.
> >          We should probably change some other methods to call this."""
> > +        dirs = self.query_abs_dirs()
> > +        zipfile = self.download_file(url, parent_dir=dirs['abs_work_dir'],
> 
> This might affect mozharness/mozilla/testing/talos.py; have you verified
> it's harmless there?
>
Code wise it looks correct and harmless.
On another note, talos on mozharness only runs on Cedar.

Comment 41

[Jeff Hammel]

> > ::: mozharness/mozilla/testing/testbase.py
> > @@ +158,5 @@
> > >          """Generic download+unzip.
> > >          This is hardcoded to halt on failure.
> > >          We should probably change some other methods to call this."""
> > > +        dirs = self.query_abs_dirs()
> > > +        zipfile = self.download_file(url, parent_dir=dirs['abs_work_dir'],
> > 
> > This might affect mozharness/mozilla/testing/talos.py; have you verified
> > it's harmless there?
> >
> Code wise it looks correct and harmless.
> On another note, talos on mozharness only runs on Cedar.

Not sure what only running on cedar has to do with this code-block?  Could you elaborate please?

FWIW, I don't believe this will cause any problems with mozharness/mozilla/testing/talos.py ; however, I have been wrong before ;)

Comment 42

[Armen [:armenzg]]

(In reply to Jeff Hammel [:jhammel] from comment #41)
> > > ::: mozharness/mozilla/testing/testbase.py
> > > @@ +158,5 @@
> > > >          """Generic download+unzip.
> > > >          This is hardcoded to halt on failure.
> > > >          We should probably change some other methods to call this."""
> > > > +        dirs = self.query_abs_dirs()
> > > > +        zipfile = self.download_file(url, parent_dir=dirs['abs_work_dir'],
> > > 
> > > This might affect mozharness/mozilla/testing/talos.py; have you verified
> > > it's harmless there?
> > >
> > Code wise it looks correct and harmless.
> > On another note, talos on mozharness only runs on Cedar.
> 
> Not sure what only running on cedar has to do with this code-block?  Could
> you elaborate please?
> 
> FWIW, I don't believe this will cause any problems with
> mozharness/mozilla/testing/talos.py ; however, I have been wrong before ;)

I was meaning that the only branch were we could use any problems would be on Cedar since Cedar is the only branch running mozharness-talos AFAIK.

Comment 43

[Andrew Halberstadt [:ahal]]

If the emulator/gecko builds you will be using are going to track m-c the checked in b2g_emulator_unittest script should be good to go.

Otherwise if the emulator/gecko builds are going to be tracking aurora this is blocked on 
https://bugzilla.mozilla.org/show_bug.cgi?id=800110 (mochitest only) and
https://bugzilla.mozilla.org/show_bug.cgi?id=800641 (reftest and mochitest)

landing in aurora.

Comment 44

[Jonathan Griffin (:jgriffin)]

For now we're running these on cedar, which is based off of m-c.  After that works well, we'll promote to m-c.  The last stage will be getting them working on aurora.

Comment 45

[Andrew Halberstadt [:ahal]]

So reftests still have a problem with the extensions getting installed, but mochitests work fine with the script that got checked in. Is this bug good to be closed?

Comment 46

[Andrew Halberstadt [:ahal]]

To be clear the reftest problem isn't specific to mozharness, so when that issue is solved it should start working with mozharness again.

Comment 47

[Jonathan Griffin (:jgriffin)]

(In reply to Andrew Halberstadt [:ahal] from comment #45)
> So reftests still have a problem with the extensions getting installed, but
> mochitests work fine with the script that got checked in. Is this bug good
> to be closed?

I don't think we should close this until we actually have this running in production.  There are a few remaining items, AFAICT, such as providing the correct xulrunner path in the config files.

Comment 48

[Jonathan Griffin (:jgriffin)]

Attachment: Handle xpcshell_url

Comment 49

[Jonathan Griffin (:jgriffin)]

Comment on attachment 673016 [details] [diff] [review]
Handle xpcshell_url

http://hg.mozilla.org/build/mozharness/rev/cdef9ff6fef6

I think we can turn on mochitests on cedar now, to see what the initial results look like.  The current set of tests is relatively small, so we can probably start out with just 3 chunks.

Comment 50

[Aki Sasaki (not active)]

Attachment: fix b2g emulator configs

Invalid action run-marionette not in ['clobber', 'read-buildbot-config', 'download-and-extract', 'create-virtualenv', 'install', 'run-tests']!

Comment 51

[Aki Sasaki (not active)]

Attachment: b2g mochitest log

It looks like the b2g mochitests are hanging without output.

Comment 52

[Aki Sasaki (not active)]

Comment on attachment 673423 [details] [diff] [review]
fix b2g emulator configs

http://hg.mozilla.org/build/mozharness/rev/ce22913dc3c8

Comment 53

[Jonathan Griffin (:jgriffin)]

(In reply to Aki Sasaki [:aki] from comment #51)
> Created attachment 673449 [details]
> b2g mochitest log
> 
> It looks like the b2g mochitests are hanging without output.

This is an unfortunate case of bug 791406.  The real error, after running it locally, is:

17:13:11     INFO -  runtestsb2g.py: error: unable to find --run-only-tests file 'b2g.json'

This happened because b2g.json isn't being packaged into tests.zip.  I'll have a patch momentarily.

Comment 54

[Jonathan Griffin (:jgriffin)]

The latest error in the mochitest run is this:

16:29:55     INFO -  args: ['/home/cltbld/talos-slave/test/build/xpcshell/bin/xpcshell', '-g', '/home/cltbld/talos-slave/test/build/xpcshell/bin', '-v', '170', '-f', './httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmpyAfOXi';const _SERVER_PORT = '8888'; const _SERVER_ADDR = '10.0.2.2';\n                     const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false;", '-f', './server.js']
16:29:55     INFO -  TEST-UNEXPECTED-FAIL | [Errno 8] Exec format error | Exception caught while running tests.

It looks like the xpcshell that we're downloading isn't the right version for linux32; can you verify aki?

Comment 55

[Aki Sasaki (not active)]

Hm, 

file xpcshell
xpcshell: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped

Yeah, looks like it's 64bit.

Comment 56

[Aki Sasaki (not active)]

http://runtime-binaries.pvt.build.mozilla.org/tooltool/sha512/7c349ae926492bd67bd861f23013d5dd1ad14966bc5da314c6036ddd8db53c119db909a6a22718a0794c56d1f2d8c1742be067489f29e2066d9a5eee823b403a

Comment 57

[Jonathan Griffin (:jgriffin)]

Attachment: Use linux32 version of xpcshell from tooltool,

Comment 58

[Jonathan Griffin (:jgriffin)]

Comment on attachment 674080 [details] [diff] [review]
Use linux32 version of xpcshell from tooltool,

http://hg.mozilla.org/build/mozharness/rev/6e9aab653f25

Comment 59

[Jonathan Griffin (:jgriffin)]

Attachment: Fix result handling in b2g unittest script

I've landed this already so we can see the effects on cedar this weekend; I'll land any followups as needed.  hg.mozilla.org/build/mozharness/rev/e4eb9965d085a

Comment 60

[Aki Sasaki (not active)]

Comment on attachment 675853 [details] [diff] [review]
Fix result handling in b2g unittest script

I think we should move DesktopUnittestOutputParser into mozharness.mozilla.testing.* if we're going to inherit from it, but I'm ok with this for now.

Comment 61

[Aki Sasaki (not active)]

Attachment: move DesktopUnittestOutputParser to mozharness.mozilla.testing.unittest

Comment 62

[Jonathan Griffin (:jgriffin)]

Comment on attachment 676323 [details] [diff] [review]
move DesktopUnittestOutputParser to mozharness.mozilla.testing.unittest

Review of attachment 676323 [details] [diff] [review]:
-----------------------------------------------------------------

Cool!

Comment 63

[Aki Sasaki (not active)]

Comment on attachment 676323 [details] [diff] [review]
move DesktopUnittestOutputParser to mozharness.mozilla.testing.unittest

http://hg.mozilla.org/build/mozharness/rev/28fb45d30900

Comment 64

[Andrew Halberstadt [:ahal]]

Just to note, the reftest extension not getting installed issue got fixed by jgriffin last week, so we can start work on getting those up on cedar as well. Let me know if there's anything I can do.

Comment 65

[Aki Sasaki (not active)]

Oh?
This is the first I heard of this.
Filing a bug and writing a patch.

Comment 66

[Jonathan Griffin (:jgriffin)]

This is done and in production.  Any followups can be filed as new bugs.