Investigate running mochitest plain tests in webapp runtime

RESOLVED WONTFIX

Status

Firefox Graveyard
Webapp Runtime
P1
normal
RESOLVED WONTFIX
5 years ago
2 years ago

People

(Reporter: marco, Unassigned)

Tracking

Details

(Reporter)

Description

5 years ago
The webapprt-test-content basically runs plain mochitests inside a webapp window. It'd be good to run all the plain mochitests in the webapp runtime so that we have greater test coverage.
Not every test will pass in the webapp runtime, we need to study them. Some of them will rightly fail because the behavior of the browser is different from the behavior of the webapp runtime. We need to have a blacklist of these failing tests.
(Reporter)

Updated

5 years ago
OS: Linux → All
Hardware: x86_64 → All
Priority: -- → P1
(Reporter)

Comment 1

4 years ago
Any news about your investigation?
Flags: needinfo?(ctalbert)

Comment 2

4 years ago
(In reply to Marco Castelluccio [:marco] from comment #1)
> Any news about your investigation?

So sorry, I guess I never replied. I looked into it, got it running, and saw all the various failures that came from it. And then I got overrun with b2g and other stuff. So sorry :-(  

I think it will work to run tests that way, but we'll want to manifest out the failing tests and file bugs on getting them fixed. Now that we have the manifests for mochitests it will be much easier to do that.
Flags: needinfo?(ctalbert)
(Reporter)

Comment 3

4 years ago
(In reply to Clint Talbert ( :ctalbert ) from comment #2)
> (In reply to Marco Castelluccio [:marco] from comment #1)
> > Any news about your investigation?
> 
> So sorry, I guess I never replied. I looked into it, got it running, and saw
> all the various failures that came from it. And then I got overrun with b2g
> and other stuff. So sorry :-(  
> 
> I think it will work to run tests that way, but we'll want to manifest out
> the failing tests and file bugs on getting them fixed. Now that we have the
> manifests for mochitests it will be much easier to do that.

Should I just add |skip-if = buildapp == "webapprt"| (or something similar) to each test that fails?
Flags: needinfo?(ctalbert)
(Reporter)

Comment 4

4 years ago
Is there a mozinfo attribute that specifies whether the test is being run in the webapp runtime or in the browser? (I guess buildapp is always "browser")

Comment 5

4 years ago
Yes, you would add that, but we likely have to add this logic to the manifest parser because I don't think the manifest parser knows that these conditions are set. Jmaher will have more informtaion.
Flags: needinfo?(ctalbert) → needinfo?(jmaher)
adding:
skip-if = buildapp == "webapprt"

will work if we run this with a separate build.  I am not sure how webapprt relates to a normal desktop build.

Either way, we have all the mochitests in mochitest.ini files now and later this week we will be filtering tests at runtime via the test harness instead of at buildtime.

Right now there is no specific mozinfo attribute that I know of for webapprt, in bug 943326, we are adding better support for os=='b2g', maybe doing something similar for webapprt would be a good idea!
Flags: needinfo?(jmaher)
(Reporter)

Comment 7

4 years ago
(In reply to Joel Maher (:jmaher) from comment #6)
> adding:
> skip-if = buildapp == "webapprt"
> 
> will work if we run this with a separate build.  I am not sure how webapprt
> relates to a normal desktop build.
> 
> Either way, we have all the mochitests in mochitest.ini files now and later
> this week we will be filtering tests at runtime via the test harness instead
> of at buildtime.
> 
> Right now there is no specific mozinfo attribute that I know of for
> webapprt, in bug 943326, we are adding better support for os=='b2g', maybe
> doing something similar for webapprt would be a good idea!

We don't run it with a separate build, we build Firefox and the webapprt together. So I think buildapp is always "browser".

I guess filtering at runtime is the only viable way.
how do you know you are in webapprt?  is there a preferences, a different attribute in the browser?
(Reporter)

Comment 9

4 years ago
Currently when we run the webapprt content tests, we have the testPrefix ("webapprt_") to decide which tests to run.
if we encounter a webapprt_* filename, what happens?  are these run as a separate suite (I can't find evidence of this on tbpl) or do we do something different with the harness?
(Reporter)

Comment 11

4 years ago
(In reply to Joel Maher (:jmaher) from comment #10)
> if we encounter a webapprt_* filename, what happens?  are these run as a
> separate suite (I can't find evidence of this on tbpl) or do we do something
> different with the harness?

We aren't yet running these tests on tbpl (we're going to add webapprt chrome tests in bug 920569).

A couple of relevant links:
http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#136
http://mxr.mozilla.org/mozilla-central/source/testing/testsuite-targets.mk#168

So webapprt content tests are run just like mochitests. The first difference is that the mochitest server will only select tests that start with the "webapprt_" prefix. The second difference is that the tests are run using the webapprt-stub executable.
Blocks: 1111077
Per bug 1238079, we're going to disable the desktop web runtime and remove it from the codebase, so we won't fix these bugs in it.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → WONTFIX
(Assignee)

Updated

2 years ago
Product: Firefox → Firefox Graveyard
You need to log in before you can comment on or make changes to this bug.