899704 - Investigate running mochitest plain tests in webapp runtime

Status: RESOLVED WONTFIX

(Firefox Graveyard :: Webapp Runtime, defect, P1)

Description

[Marco Castelluccio [:marco]]

The webapprt-test-content basically runs plain mochitests inside a webapp window. It'd be good to run all the plain mochitests in the webapp runtime so that we have greater test coverage.
Not every test will pass in the webapp runtime, we need to study them. Some of them will rightly fail because the behavior of the browser is different from the behavior of the webapp runtime. We need to have a blacklist of these failing tests.

Comment 1

[Marco Castelluccio [:marco]]

Any news about your investigation?

Comment 2

[cmtalbert]

(In reply to Marco Castelluccio [:marco] from comment #1)
> Any news about your investigation?

So sorry, I guess I never replied. I looked into it, got it running, and saw all the various failures that came from it. And then I got overrun with b2g and other stuff. So sorry :-(  

I think it will work to run tests that way, but we'll want to manifest out the failing tests and file bugs on getting them fixed. Now that we have the manifests for mochitests it will be much easier to do that.

Comment 3

[Marco Castelluccio [:marco]]

(In reply to Clint Talbert ( :ctalbert ) from comment #2)
> (In reply to Marco Castelluccio [:marco] from comment #1)
> > Any news about your investigation?
> 
> So sorry, I guess I never replied. I looked into it, got it running, and saw
> all the various failures that came from it. And then I got overrun with b2g
> and other stuff. So sorry :-(  
> 
> I think it will work to run tests that way, but we'll want to manifest out
> the failing tests and file bugs on getting them fixed. Now that we have the
> manifests for mochitests it will be much easier to do that.

Should I just add |skip-if = buildapp == "webapprt"| (or something similar) to each test that fails?

Comment 4

[Marco Castelluccio [:marco]]

Is there a mozinfo attribute that specifies whether the test is being run in the webapp runtime or in the browser? (I guess buildapp is always "browser")

Comment 5

[cmtalbert]

Yes, you would add that, but we likely have to add this logic to the manifest parser because I don't think the manifest parser knows that these conditions are set. Jmaher will have more informtaion.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

adding:
skip-if = buildapp == "webapprt"

will work if we run this with a separate build.  I am not sure how webapprt relates to a normal desktop build.

Either way, we have all the mochitests in mochitest.ini files now and later this week we will be filtering tests at runtime via the test harness instead of at buildtime.

Right now there is no specific mozinfo attribute that I know of for webapprt, in bug 943326, we are adding better support for os=='b2g', maybe doing something similar for webapprt would be a good idea!

Comment 7

[Marco Castelluccio [:marco]]

(In reply to Joel Maher (:jmaher) from comment #6)
> adding:
> skip-if = buildapp == "webapprt"
> 
> will work if we run this with a separate build.  I am not sure how webapprt
> relates to a normal desktop build.
> 
> Either way, we have all the mochitests in mochitest.ini files now and later
> this week we will be filtering tests at runtime via the test harness instead
> of at buildtime.
> 
> Right now there is no specific mozinfo attribute that I know of for
> webapprt, in bug 943326, we are adding better support for os=='b2g', maybe
> doing something similar for webapprt would be a good idea!

We don't run it with a separate build, we build Firefox and the webapprt together. So I think buildapp is always "browser".

I guess filtering at runtime is the only viable way.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

how do you know you are in webapprt?  is there a preferences, a different attribute in the browser?

Comment 9

[Marco Castelluccio [:marco]]

Currently when we run the webapprt content tests, we have the testPrefix ("webapprt_") to decide which tests to run.

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

if we encounter a webapprt_* filename, what happens?  are these run as a separate suite (I can't find evidence of this on tbpl) or do we do something different with the harness?

Comment 11

[Marco Castelluccio [:marco]]

(In reply to Joel Maher (:jmaher) from comment #10)
> if we encounter a webapprt_* filename, what happens?  are these run as a
> separate suite (I can't find evidence of this on tbpl) or do we do something
> different with the harness?

We aren't yet running these tests on tbpl (we're going to add webapprt chrome tests in bug 920569).

A couple of relevant links:
http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#136
http://mxr.mozilla.org/mozilla-central/source/testing/testsuite-targets.mk#168

So webapprt content tests are run just like mochitests. The first difference is that the mochitest server will only select tests that start with the "webapprt_" prefix. The second difference is that the tests are run using the webapprt-stub executable.

Comment 12

[Myk Melez [:myk] [@mykmelez]]

Per bug 1238079, we're going to disable the desktop web runtime and remove it from the codebase, so we won't fix these bugs in it.