986163 - Webapp Runtime tests don't work anymore

Status: RESOLVED FIXED

(Testing :: Mochitest, defect, P1)

Description

[Marco Castelluccio [:marco]]

This is a really recent regression (it should be max 2-3 days).

I can't run both webapprt-test-chrome and webapprt-test-content.

TEST-UNEXPECTED-FAIL | (browser-test.js) | No tests to run. Did you pass an invalid --test-path?

Comment 1

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

It's probably my fault. Do we run these tests on tinderbox? How do I run them locally?

Comment 2

[Marco Castelluccio [:marco]]

(In reply to Bill McCloskey (:billm) from comment #1)
> It's probably my fault. Do we run these tests on tinderbox? How do I run
> them locally?

Yes, after a quick look I strongly suspect bug 938019.

These tests aren't yet run on tinderbox.
You can run them via mach:
./mach webapprt-test-chrome
./mach webapprt-test-content

In your patch I noticed:
- The tests' prefixes are wrong (webapprt-test-chrome tests' prefix is "browser_", webapprt-test-content tests' prefix is "webapprt_")
- I think you've only taken webapprt-test-chrome into account.

Comment 3

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

OK, it's pretty easy to fix the webapprt-test-chrome tests. I'm having a harder time figuring out the content tests though. Do we have manifest files for those tests?

If we don't, it should be easy to disable bug 938019 for webapprt tests. But it would be nice to fix it properly.

Comment 4

[(not currently active) Ted Mielczarek]

Unfortunately I don't actually know anything about webapprt tests.

Comment 5

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Attachment: webapprt-fix

Not sure who should review this. Ted, please pass it on to someone else if you like.

As far as I can tell, the webapprtChrome tests have their own manifests. The webapprtContent tests piggyback on the mochitest-plain manifests and use a different filename prefix. This complicates bug 983867, because we can no longer assume that any test in a manifest that doesn't pass the isTest condition (which just looks at the prefix) is a mistake. I think eventually we want webapprtContent tests to use separate manifests. For now, though, I think this patch moves us in the right direction: it gets everything working as far as I can tell, and the webapprt tests are using manifests--just not in quite the way we want.

Marco, can you see if this works for you? I'm getting failures in some of the tests, but I'm guessing that's not the fault of this patch.

Comment 6

[Marco Castelluccio [:marco]]

(In reply to Bill McCloskey (:billm) from comment #5)
> As far as I can tell, the webapprtChrome tests have their own manifests. The
> webapprtContent tests piggyback on the mochitest-plain manifests and use a
> different filename prefix. This complicates bug 983867, because we can no
> longer assume that any test in a manifest that doesn't pass the isTest
> condition (which just looks at the prefix) is a mistake. I think eventually
> we want webapprtContent tests to use separate manifests. For now, though, I
> think this patch moves us in the right direction: it gets everything working
> as far as I can tell, and the webapprt tests are using manifests--just not
> in quite the way we want.
> 
> Marco, can you see if this works for you? I'm getting failures in some of
> the tests, but I'm guessing that's not the fault of this patch.

Yes, it works!

The webapprt-test-content takes more time to start (because it logs a lot of warnings like "Warning: test_plugin_scroll_invalidation.html from manifest /home/marco/Documents/FD/src/obj-x86_64-unknown-linux-gnu/_tests/testing/mochitest/tests/widget/tests/mochitest.ini is not a valid test"), but it isn't a problem.

Notice that in the future we plan to run all the mochitest plain tests as webapprtContent tests (bug 899704).

Comment 7

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/ca4eb9580260

Comment 8

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/ca4eb9580260