899707 - Automatically run webapprt tests

Status: RESOLVED WONTFIX

(Firefox Graveyard :: Webapp Runtime, defect, P1)

Description

[Marco Castelluccio [:marco]]

We should run webapprt-test-content and webapprt-test-chrome automatically for each changeset.

Comment 1

[Myk Melez [:myk] [@mykmelez]]

Note this policy for enabling new test suites on TBPL: 

  https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy

I think we should focus on enabling webapprt-test-chrome.  We'll want to start running content tests as well, but it isn't clear that webapprt-test-content is the right way to do so.  I'd rather get the regular content tests up and running against the runtime, if possible.

Comment 2

[Marco Castelluccio [:marco]]

I agree, we should start with webappr-test-chrome tests that are also more meaningful at the moment.
What should we do to enable them?

Comment 3

[Myk Melez [:myk] [@mykmelez]]

cc:ing a few folks for confirmation, but I think the first step is to start running the tests automatically on each checkin while leaving them hidden in TBPL.  Then we'll want to go through the checklist on the aforementioned page and confirm that the tests meet the requirements for unhiding.  And finally we'll file a bug requesting they be unhidden.

But I'm not sure exactly what to do to start running the tests automatically.  Perhaps one of the cc:ed folks can point us in the right direction!

Comment 4

[Ed Morley [:emorley]]

The A-team (#ateam , auto-tools@moco) and release engineering (#releng , release@moco) are your best bet to speak to :-)

Comment 5

[Ed Morley [:emorley]]

Few questions, the answers to which will help releng:
* Which platforms?
* Opt and debug?
* How are the tests run?
* Are these going to be wanted just on trunk trees, or release branches (eg b2g18, aurora, beta) too initially? 

I would imagine releng would enable per-push tests just for Cedar initially, until they are greened up, then they can be enabled for other trees.

Comment 6

[Marco Castelluccio [:marco]]

(In reply to Ed Morley [:edmorley UTC+1] from comment #5)
> Few questions, the answers to which will help releng:
> * Which platforms?

Windows, Linux, Mac.

> * Opt and debug?

Yes.

> * How are the tests run?

They are a special kind of mochitest, they can be run with a mach command: |mach webapprt-test-chrome| and |mach webapprt-test-content|.

> * Are these going to be wanted just on trunk trees, or release branches (eg
> b2g18, aurora, beta) too initially? 

I think initially on trunk trees.

> I would imagine releng would enable per-push tests just for Cedar initially,
> until they are greened up, then they can be enabled for other trees.

That'd be great!

Comment 7

[Ed Morley [:emorley]]

Filed bug 920569 for the initial releng parts (ping release@moco if you need traction on that bug). We'll eventually want to hang other bugs off of this one, for adding support to trychooser, TBPL, enabling on more trees than Cedar etc.

Comment 8

[Myk Melez [:myk] [@mykmelez]]

Per bug 1238079, we're going to disable the desktop web runtime and remove it from the codebase, so we won't fix these bugs in it.