consider running wpt tests on b2g in automation

RESOLVED WONTFIX

Status

Testing
web-platform-tests
RESOLVED WONTFIX
2 years ago
2 years ago

People

(Reporter: bkelly, Unassigned)

Tracking

(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

2 years ago
One of our long term goals is to move more testing to wpt instead of mochitests.  This is great because it means we can improve compatibility with other browsers that run the same tests.

Right now, however, we face the challenge that wpt does not get us coverage across all our shipped platforms.  We've made great strides here and now have opt/debug and e10s/non-e10s.

It would be great to have wpt tests running on b2g in automation, however, since b2g is dramatically different in how it handles apps and security.  I don't think we have any other platforms that exercise app ids in our principal code or the IPC security infrastructure.

We are writing a lot of wpt tests for ServiceWorker now, so this effects SW in particular.  For example, see bug 1204596 comment 6.

I'm tentatively marking this as ServiceWorkers-v2 blocking, but realize it may not be practical to actually block the release on this.  Its more of a wish. :-)

Updated

2 years ago
Blocks: 1194860
Blocks: 1207265
No longer blocks: 1131322
(Reporter)

Comment 1

2 years ago
Jonathon, James, what are the odds this could happen in Q1 of 2016?  We'd like to make a push for service workers on b2g, but many of our tests are in wpt.  It would help a lot of if we could run them on b2g. (Note, we expect many of them to fail there at the moment.)
Flags: needinfo?(jgriffin)
Flags: needinfo?(james)
Ben, we have quite a bit on our plate already for Q1. Would it be ok if we just did service worker tests from WPT?

Limiting the scope gives us a better chance of being able to do this next quarter (but no promises). I am meeting with jgriffin later today and we can discuss. Will update the bug then. N-i to remind myself.
Flags: needinfo?(jgriffin)
Flags: needinfo?(james)
Flags: needinfo?(dburns)
(Reporter)

Comment 3

2 years ago
(In reply to David Burns :automatedtester from comment #2)
> Ben, we have quite a bit on our plate already for Q1. Would it be ok if we
> just did service worker tests from WPT?
> 
> Limiting the scope gives us a better chance of being able to do this next
> quarter (but no promises). I am meeting with jgriffin later today and we can
> discuss. Will update the bug then. N-i to remind myself.

For my purposes this would be fine.  It feels like the hard part is infrastructure stuff, though.

I would personally be happy if we got a small subset of wpt tests running on b2g with most disabled.  Then we can enable our tests as we fix service workers for b2g.
So there are a few of components to this work:

1) Teaching wptrunner to launch the B2G emulators
2) Running tests on the emulators once they are launched
3) Getting the tests to give stable results

1) should in theory not be that hard since the framework for creating new targets already exists and we previously had this working on B2G devices. But my expectations around B2G are that things will take longer than anticipated for some reason. 2) is something that was working on devices in the 1.4 era. I don't know to what extent the test app we used then is still usable in modern times. 3) has historically been problematic on every new platform we've targeted, and is the part where limiting the scope to fewer tests is worthwhile so that, for example, we aren't held up on serviceworker tests trying to get navigation tests to pass.

So I think that the idea of limiting the scope can help to get this done, but there are many other factors that could contrive to make this task either relatively straightforward or unexpectedly difficult.
Ben,

I have spoken to jgriffin and we will organise a meeting with you next week so we can discuss this more in depth and come up with a plan for getting what you need.
Flags: needinfo?(dburns)
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.