1065583 - Mach / Mozharness should execute Mozrunner unit tests with a real binary

Status: REOPENED

(Testing :: Mozbase, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

In February this year I added a bunch of new mozrunner tests via bug 960084, which are only getting executed when we run the mozbase tests with a binary specified. While this has been fixed on bug 966441 comment 27 for cedar, we do not run the tests for inbound and other integration branches:

> test_run_interactive (test_interactive.MozrunnerInteractiveTestCase)
> Bug 965183: Run process in interactive mode and call wait() ... skipped 'No binary has been specified.'

We should really enable those tests to ensure that mozrunner is stable enough in terms of the binaries we are producing, and that no regressions are introduced e.g. in Firefox. For example here see the upcoming large changes for the signed OS X builds via bug 1047728.

So what's needed is something like:

    1.11      "mozbase_options": [
    1.12 +        "-b", "%(binary_path)s"
    1.13      ],

Ben, would RelEng be able to do us the favor and get this enabled?

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Chris, I'm not sure how much time it would take to enable that option, but it would be a huge step for our unit tests in mozbase. Can we get that enabled please?

Comment 2

[Jonathan Griffin (:jgriffin)]

This is dependent on splitting mozbase unit tests into a separate test job, bug 966441.

Comment 3

[Chris AtLee [:catlee]]

Sorry, I'm not sure where these tests are specified. Where do you need this change to be made?

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ok, so running the mozrunner tests with a real binary will be hard now given that those tests are getting no longer run as part of the build process but via the python-test task.

Gregory, any idea how we could ensure to get those additional tests for mozrunner executed?

Comment 5

[Gregory Szorc [:gps]]

Uh, create a TaskCluster task that runs with the binary produced by the build task? This would basically be copying what we do for mochitest, wpt, etc.

Comment 6

[Andrew Halberstadt [:ahal]]

Where's this python-test task? I'm not seeing it on treeherder anywhere yet.

But I think what Henrik means is that it is now running under the 'lint' platform or similar and does not depend on any builds. Personally I'd prefer just using the system firefox or downloading the latest release. There's no reason it needs to be a build from the same changeset. In fact, if the build has a startup crash or something, then the mozrunner tests will fail even though the problem doesn't lie with mozrunner itself. So it's arguably even better to use a known stable build.

That being said, splitting the mozbase tests out of the python-test task is likely a good idea anyway. That way we can only run them if something under testing/mozbase has changed.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Oh, I was wrong with that! For local testing we have the "mach python-test" command, but as it looks like we still run the mozbase unit tests as part of the build. So we would still have to change the line where we invoke those tests and specify a binary which would satisfy us.

In comment 0 I mentioned what has to be done but as it looks like the configuration has been moved. Not sure where it is located right now.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Well, I think we have enough information on this bug to not have to mark it as incomplete. Someone just has to get this done.

A question I have is how python tests are run. I assume that those do not require a build, and as such we won't have a binary available? If that is the case I do not see a way how to run those mozbase tests with a real binary.

Maja, do you know?

Comment 9

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

This shows how mozbase tests are run: https://hg.mozilla.org/mozilla-central/raw-file/05bfa2831c0ba4a26fa72328ffe6a99aba9c356a/taskcluster/ci/source-test/python-tests.yml

I don't know whether this is the only place/way mozbase tests are run.

Re binary: What about using the system firefox, as suggested in Comment 6?

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Oh right. I think downloading a stable build in case of no system default version can be found, sounds reasonable.

Would there be any side-effect for the python-tests job beside having network access to download a build? I mean if the download fails, we shouldn't have to error but print a warning and continue without it.

For local tests as run with mach we could use the built binary.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Btw with the changes on bug 1003417 we got the mozbase unit tests running as a separate job on Taskcluster for Linux.