892569 - Run remote C++ unit tests from test package

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Dan Minor [:dminor]]

+++ This bug was initially created as a clone of Bug #811409 +++

This is to track work getting remote c++ unit tests running from the test packages.

Comment 1

[Dan Minor [:dminor]]

I hit following failures consistently when running on my pandaboard:
TestTXMgr | test failed with return code 9
TestPlainTextSerializer | test failed with return code 1
TestNativeXMLHttpRequest | test failed with return code 1
TestAudioEventTimeline | Couldn't create nsIXMLHttpRequest instance!
mediapipeline_unittest | test crashed

I plan to spend a bit of time to see if these are easily fixable, otherwise I think the best approach is to file individual bugs and modify the cpp unittest harness to provide a way to skip tests.

Comment 2

[(not currently active) Ted Mielczarek]

We can add manifest support to the harness to make that work if need be, or we can just ifdef those tests in moz.build files as a quick workaround. I bet someone would be interested in that mediapipeline_unittest crash if you can give them instructions on how to reproduce it!

Comment 3

[Dan Minor [:dminor]]

Attachment: Patch to add cppunittest support for pandas.

Ash run here: https://tbpl.mozilla.org/?tree=Ash&rev=6d2ef5f37053 seems to be no more broken than the previous ash run: https://tbpl.mozilla.org/?tree=Ash&rev=0e12c4cc159f.

Comment 4

[Kim Moir [:kmoir] ET]

Comment on attachment 809855 [details] [diff] [review]
Patch to add cppunittest support for pandas.

Ran green on my master.

I assume you need buidlbot patches to schedule these on cedar too, if so I have them already on my master.

Comment 5

[Dan Minor [:dminor]]

Thanks, https://hg.mozilla.org/build/mozharness/rev/5f7089936a19.

Comment 6

[[e-mail] (:bhearsum)]

This broke when it got merged to production today because of the mozinfo installation:
https://tbpl.mozilla.org/php/getParsedLog.php?id=28469167&tree=Mozilla-Inbound

I'm not sure if we need a newer mozinfo or something else.

Comment 7

[(not currently active) Ted Mielczarek]

Is this another case of "using a mozbase package from a local install and not the test package"? I know we saw this bustage elsewhere.

Comment 8

[Aki Sasaki (not active)]

Yeah.

     virtualenv_modules = [
         'mozpoolclient',
         'mozcrash',
-        {'name': 'mozdevice', 'url': os.path.join('tests', os.path.join('mozbase', 'mozdevice'))}
+        {'name': 'mozdevice', 'url': os.path.join('tests', os.path.join('mozbase', 'mozdevice'))},
+        'mozinfo',
+        'mozlog',
+        'mozprocess'
     ]

That means mozdevice comes from the in-tree test zip but everything else comes from puppetagain.  I imagine we want to get everything (except maybe mozpoolclient?) from the test zip.

Comment 9

[Dan Minor [:dminor]]

Backed this out here: https://hg.mozilla.org/build/mozharness/rev/f4c6df13618a

I'll update my patch to pull everything in from the tests package.

Comment 10

[Jeff Hammel]

(In reply to Aki Sasaki [:aki] from comment #8)
> Yeah.
> 
>      virtualenv_modules = [
>          'mozpoolclient',
>          'mozcrash',
> -        {'name': 'mozdevice', 'url': os.path.join('tests',
> os.path.join('mozbase', 'mozdevice'))}
> +        {'name': 'mozdevice', 'url': os.path.join('tests',
> os.path.join('mozbase', 'mozdevice'))},
> +        'mozinfo',
> +        'mozlog',
> +        'mozprocess'
>      ]
> 
> That means mozdevice comes from the in-tree test zip but everything else
> comes from puppetagain.  I imagine we want to get everything (except maybe
> mozpoolclient?) from the test zip.

Certainly all the mozbase packages, I would assume

Comment 11

[Kim Moir [:kmoir] ET]

Sorry I didn't see this when I tested in staging Dan.  I commented out the other test builders so the tests would run faster on my master and just ran the cppunitest ones which ran green.

Comment 12

[Dan Minor [:dminor]]

Kim, no problem, it turns out it actually showed up in my Ash run as well, which I really should have noticed.

Comment 13

[Dan Minor [:dminor]]

Attachment: Patch to add cppunittest support for pandas.

Kim, this is running well locally, but would probaby benefit from a run in your staging environment. Thanks!

Comment 14

[Kim Moir [:kmoir] ET]

Comment on attachment 811246 [details] [diff] [review]
Patch to add cppunittest support for pandas.

So I'm seeing the same error the sheriffs saw earlier Friday morning when running in staging for mochitests and friends

AttributeError: 'module' object has no attribute 'find_and_update_from_json' when running the non-cpp unittests such as mochitests.

This is running the tests m-c revision 1a2d9a04ffb2 and it's associated apk
http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android/1380297706/
 
The installed mozinfo is still 0.5 which doesn't have the new code.

I tried adding 
 {'name': 'mozinfo', 'url': os.path.join('tests', 'mozbase', 'mozinfo')}

to the script but it still fails for many tests.  This is because the mozharness scripts don't upgrade if the dependency is already satisfied with an earlier version, not sure if the default behavior in this case should be to upgrade.

Comment 15

[Kim Moir [:kmoir] ET]

dminor suggested not not pull mozcrash and mozprocess from the tests.zip in the script.  I tried that and the non-cpp tests worked, but the cpp one didn't.

My understanding from looking at the mozharness code that installs the virtual env is that there is no concept of dependency resolution within version constraints.  We don't specify the version of the package to install, just check that a package is installed. And if there is a newer version, we don't upgrade.  Which is problematic if you want to install from a newer package in the tests.zip.  So they only way to install a newer version is to make it available on the mozilla python packages repo.  Aki, is that an accurate description?

Comment 16

[Aki Sasaki (not active)]

(In reply to Kim Moir [:kmoir] from comment #15)
> My understanding from looking at the mozharness code that installs the
> virtual env is that there is no concept of dependency resolution within
> version constraints.

I don't understand what you mean by this.
If you specify

    mozinfo==0.6

as your module, it will require that version.  If you specify

    mozinfo>=0.5

it will find a version of mozinfo that matches that version.  If you specify

    mozinfo

it will find any version of mozinfo.  |pip install| will find missing dependencies and install them from pypi (or from puppetagain, if we're doing things right with our --find-links and --no-index options) if possible.

> We don't specify the version of the package to
> install, just check that a package is installed.

True, though we usually clobber the virtualenv along with everything else.  Do we need to add a clobber here?

> And if there is a newer
> version, we don't upgrade.

That's true if you only specify |mozinfo| as above, and don't clobber the virtualenv at the beginning.

> Which is problematic if you want to install from
> a newer package in the tests.zip.  So they only way to install a newer
> version is to make it available on the mozilla python packages repo.  Aki,
> is that an accurate description?

See above.  I think puppetagain only has 0.5 as its latest currently.

Comment 17

[Kim Moir [:kmoir] ET]

Oh okay thanks Aki for the explanation.  So Dan you could try specifying versions you need from the test package.

Comment 18

[Dan Minor [:dminor]]

Bug 921596 will attempt to fix the versioning problems by pulling all mozbase modules from the test package. If it looks like that can't be fixed in a reasonable amount of time, then we can always try specifying the version like kmoir suggests.

Comment 19

[Dan Minor [:dminor]]

Attachment: Patch to add cpp unittest support

Looks like bug 921596 is doing ok in production, so maybe this can be landed again.

Comment 20

[Dan Minor [:dminor]]

Thanks, pushed here: https://hg.mozilla.org/build/mozharness/rev/b2f6b2fa10cd

Comment 21

[Justin Wood (:Callek)]

merged to production mozharness

Comment 22

[Dan Minor [:dminor]]

We're running these from the test package on pandas.

Additional work would be required to run on tegras (since they are not based on mozharness) but I don't think it makes sense to do that work at this point, since the plan is to replace them with either emulators or pandas in the near term.