Run remote C++ unit tests from test package

RESOLVED FIXED

Status

Release Engineering
General Automation
RESOLVED FIXED
5 years ago
4 years ago

People

(Reporter: dminor, Assigned: dminor)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 2 obsolete attachments)

(Assignee)

Description

5 years ago
+++ This bug was initially created as a clone of Bug #811409 +++

This is to track work getting remote c++ unit tests running from the test packages.
(Assignee)

Updated

5 years ago
Depends on: 893085
See Also: → bug 811409
Product: mozilla.org → Release Engineering
(Assignee)

Comment 1

4 years ago
I hit following failures consistently when running on my pandaboard:
TestTXMgr | test failed with return code 9
TestPlainTextSerializer | test failed with return code 1
TestNativeXMLHttpRequest | test failed with return code 1
TestAudioEventTimeline | Couldn't create nsIXMLHttpRequest instance!
mediapipeline_unittest | test crashed

I plan to spend a bit of time to see if these are easily fixable, otherwise I think the best approach is to file individual bugs and modify the cpp unittest harness to provide a way to skip tests.
We can add manifest support to the harness to make that work if need be, or we can just ifdef those tests in moz.build files as a quick workaround. I bet someone would be interested in that mediapipeline_unittest crash if you can give them instructions on how to reproduce it!
(Assignee)

Comment 3

4 years ago
Created attachment 809855 [details] [diff] [review]
Patch to add cppunittest support for pandas.

Ash run here: https://tbpl.mozilla.org/?tree=Ash&rev=6d2ef5f37053 seems to be no more broken than the previous ash run: https://tbpl.mozilla.org/?tree=Ash&rev=0e12c4cc159f.
Assignee: nobody → dminor
Status: NEW → ASSIGNED
Attachment #809855 - Flags: review?(kmoir)

Comment 4

4 years ago
Comment on attachment 809855 [details] [diff] [review]
Patch to add cppunittest support for pandas.

Ran green on my master.

I assume you need buidlbot patches to schedule these on cedar too, if so I have them already on my master.
Attachment #809855 - Flags: review?(kmoir) → review+
(Assignee)

Comment 5

4 years ago
Thanks, https://hg.mozilla.org/build/mozharness/rev/5f7089936a19.
This broke when it got merged to production today because of the mozinfo installation:
https://tbpl.mozilla.org/php/getParsedLog.php?id=28469167&tree=Mozilla-Inbound

I'm not sure if we need a newer mozinfo or something else.
Is this another case of "using a mozbase package from a local install and not the test package"? I know we saw this bustage elsewhere.

Comment 8

4 years ago
Yeah.

     virtualenv_modules = [
         'mozpoolclient',
         'mozcrash',
-        {'name': 'mozdevice', 'url': os.path.join('tests', os.path.join('mozbase', 'mozdevice'))}
+        {'name': 'mozdevice', 'url': os.path.join('tests', os.path.join('mozbase', 'mozdevice'))},
+        'mozinfo',
+        'mozlog',
+        'mozprocess'
     ]

That means mozdevice comes from the in-tree test zip but everything else comes from puppetagain.  I imagine we want to get everything (except maybe mozpoolclient?) from the test zip.
(Assignee)

Comment 9

4 years ago
Backed this out here: https://hg.mozilla.org/build/mozharness/rev/f4c6df13618a

I'll update my patch to pull everything in from the tests package.

Comment 10

4 years ago
(In reply to Aki Sasaki [:aki] from comment #8)
> Yeah.
> 
>      virtualenv_modules = [
>          'mozpoolclient',
>          'mozcrash',
> -        {'name': 'mozdevice', 'url': os.path.join('tests',
> os.path.join('mozbase', 'mozdevice'))}
> +        {'name': 'mozdevice', 'url': os.path.join('tests',
> os.path.join('mozbase', 'mozdevice'))},
> +        'mozinfo',
> +        'mozlog',
> +        'mozprocess'
>      ]
> 
> That means mozdevice comes from the in-tree test zip but everything else
> comes from puppetagain.  I imagine we want to get everything (except maybe
> mozpoolclient?) from the test zip.

Certainly all the mozbase packages, I would assume
Sorry I didn't see this when I tested in staging Dan.  I commented out the other test builders so the tests would run faster on my master and just ran the cppunitest ones which ran green.
(Assignee)

Comment 12

4 years ago
Kim, no problem, it turns out it actually showed up in my Ash run as well, which I really should have noticed.
(Assignee)

Comment 13

4 years ago
Created attachment 811246 [details] [diff] [review]
Patch to add cppunittest support for pandas.

Kim, this is running well locally, but would probaby benefit from a run in your staging environment. Thanks!
Attachment #809855 - Attachment is obsolete: true
Attachment #811246 - Flags: review?(kmoir)
Comment on attachment 811246 [details] [diff] [review]
Patch to add cppunittest support for pandas.

So I'm seeing the same error the sheriffs saw earlier Friday morning when running in staging for mochitests and friends

AttributeError: 'module' object has no attribute 'find_and_update_from_json' when running the non-cpp unittests such as mochitests.

This is running the tests m-c revision 1a2d9a04ffb2 and it's associated apk
http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android/1380297706/
 
The installed mozinfo is still 0.5 which doesn't have the new code.

I tried adding 
 {'name': 'mozinfo', 'url': os.path.join('tests', 'mozbase', 'mozinfo')}

to the script but it still fails for many tests.  This is because the mozharness scripts don't upgrade if the dependency is already satisfied with an earlier version, not sure if the default behavior in this case should be to upgrade.
dminor suggested not not pull mozcrash and mozprocess from the tests.zip in the script.  I tried that and the non-cpp tests worked, but the cpp one didn't.

My understanding from looking at the mozharness code that installs the virtual env is that there is no concept of dependency resolution within version constraints.  We don't specify the version of the package to install, just check that a package is installed. And if there is a newer version, we don't upgrade.  Which is problematic if you want to install from a newer package in the tests.zip.  So they only way to install a newer version is to make it available on the mozilla python packages repo.  Aki, is that an accurate description?
Flags: needinfo?(aki)

Comment 16

4 years ago
(In reply to Kim Moir [:kmoir] from comment #15)
> My understanding from looking at the mozharness code that installs the
> virtual env is that there is no concept of dependency resolution within
> version constraints.

I don't understand what you mean by this.
If you specify

    mozinfo==0.6

as your module, it will require that version.  If you specify

    mozinfo>=0.5

it will find a version of mozinfo that matches that version.  If you specify

    mozinfo

it will find any version of mozinfo.  |pip install| will find missing dependencies and install them from pypi (or from puppetagain, if we're doing things right with our --find-links and --no-index options) if possible.

> We don't specify the version of the package to
> install, just check that a package is installed.

True, though we usually clobber the virtualenv along with everything else.  Do we need to add a clobber here?

> And if there is a newer
> version, we don't upgrade.

That's true if you only specify |mozinfo| as above, and don't clobber the virtualenv at the beginning.

> Which is problematic if you want to install from
> a newer package in the tests.zip.  So they only way to install a newer
> version is to make it available on the mozilla python packages repo.  Aki,
> is that an accurate description?

See above.  I think puppetagain only has 0.5 as its latest currently.
Flags: needinfo?(aki)
Oh okay thanks Aki for the explanation.  So Dan you could try specifying versions you need from the test package.
(Assignee)

Comment 18

4 years ago
Bug 921596 will attempt to fix the versioning problems by pulling all mozbase modules from the test package. If it looks like that can't be fixed in a reasonable amount of time, then we can always try specifying the version like kmoir suggests.
Depends on: 921596
(Assignee)

Updated

4 years ago
Attachment #811246 - Flags: review?(kmoir)
(Assignee)

Comment 19

4 years ago
Created attachment 820326 [details] [diff] [review]
Patch to add cpp unittest support

Looks like bug 921596 is doing ok in production, so maybe this can be landed again.
Attachment #811246 - Attachment is obsolete: true
Attachment #820326 - Flags: review?(kmoir)

Updated

4 years ago
Attachment #820326 - Flags: review?(kmoir) → review+
(Assignee)

Comment 20

4 years ago
Thanks, pushed here: https://hg.mozilla.org/build/mozharness/rev/b2f6b2fa10cd
merged to production mozharness
(Assignee)

Updated

4 years ago
Depends on: 931926
(Assignee)

Updated

4 years ago
Depends on: 937637
(Assignee)

Updated

4 years ago
Depends on: 949536
(Assignee)

Updated

4 years ago
Depends on: 949538
(Assignee)

Comment 22

4 years ago
We're running these from the test package on pandas.

Additional work would be required to run on tegras (since they are not based on mozharness) but I don't think it makes sense to do that work at this point, since the plan is to replace them with either emulators or pandas in the near term.
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.