1029183 - make loop functional tests run from one command

Status: RESOLVED INCOMPLETE

(Hello (Loop) :: General, defect, P3)

Comment 1

[:shell escalante]

Hi Nils,

how realistic is this to find people with the knowledge and have it prioritized for Fx37?  sounds like we need A-team help. :)

Comment 3

[Nils Ohlmeier [:drno]]

I'm going to evaluate the remaining work today with Richard. Hopefully after that I'll have a better understanding of how much work is left for this.

Comment 4

[Nils Ohlmeier [:drno]]

Unfortunately it turned out that we can't run this on TBPL, because you can run only code from mozilla central on TBPL (with very exceptions).
So the current plan is to get this in the mean time running in the WebRTC QA lab (bug 1095170).

Comment 5

[Dan Mosedale (:dmosedale, :dmose)]

So without understanding more about that policy, it's hard to triangulate on exactly what should happen.
Do you have a link?

I've had some discussions with the a-team in the past, and had not until now heard of an objection to running it there, but perhaps I didn't make it clear enough what I was hoping to do.

That said, I think there's a reasonable chance it'll make sense to specifically request such an exception, so I'd recommend we explore more on that front.

That said, I quite like the idea of getting it running elsewhere in the meantime.

Comment 6

[cmtalbert]

There are two aspects to running a job in the buildbot automation:
1. Sheriffing policies (if you expect the sheriffs to sheriff these builds for you) [1]
2. The networking issue: because we've been bitten many times in the past by tests that depended on external networking behaving in strange ways, we generally frown on (and sometimes disallow) tests that depend on systems external to the build and test automation network from running in our automation.

Historically, everything you wanted to see on our automation dashboard of choice (tbpl) had to adhere to both these policies. Nowadays, we have TreeHerder which liberates us from the restriction that in order to be visible, all automation has to run in buildbot. You can effectively have automation running anywhere and still see it on our dashboard (i.e. Treeherder). This means that you can have automation that does not adhere to these two rules but still have it visible (it also means that you're sheriffing it yourself, until we figure out what the non-buildbot hosted sheriffing policies should be).

In the future, I want to build a system where we allow services end points to be easily integrated into our existing automation and to be sheriffable from Treeherder. That future starts with how we approach jobs like this one, where we can start figuring out how to do some of these things by prototyping what is possible and porting the best practices from that back into the core automation systems.

So, it is entirely possible to run this in the webRTC QA lab set up, have it reporting to Treeherder so everyone can see it.  We need to figure out a set of things to make this (and every other services automation project) successful:
1. How do we associate the test with the external server system that it needs. Or even better, is there any way to spin up the server locally for testing? What kind of manifest/capabilities are required to make that successful?
2. How can we make it *very* clear from the failure modes of these tests when there is a problem with the server aspect versus the client? This way it is clear how to find the right party to fix the issue - which will be necessary for sheriffing
3. How do we trigger these tests? Do we trigger upon a server change? A client change? How do we track down issues caused by both sides updating simultaneously? 
4. How do we keep tests like this from becoming another source of intermittent failures? What best practices can we adopt?

[1] https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy

Comment 7

[Maire Reavy [:mreavy]]

Moving this to P2 based on our new priority definitions.

Comment 8

[:shell escalante]

moving to 38 based on time - can consider uplifting if done early/hits bar/low risk

Comment 9

[:shell escalante]

is there anything else we want or should we start moving on the breakdown here?

Comment 10

[Nils Ohlmeier [:drno]]

Lets redefine the scope here. I don't think that we actually want a test execution on Tbpl any more (at least not in the short or middle term):
- the Loop client (standalone and conversation client) depend on TokBox client SDK which requires talking to TB servers, which is forbidden on Tbpl
- we have a test execution outside of Tbpl running here http://10.252.73.218:8080/view/hello/job/hello-e2e-marionette/

A clean execution just from 'mach', which might still have a dependency on talking to the Internet (see above TokBox client SDK), would still be nice, but not essential.

Comment 11

[Dan Mosedale (:dmosedale, :dmose)]

Maybe the user story now wants to look like this?

* write a python script to configure a known tag of the server
** if not already bootstrapped, execute the bootstrap process to be written in bug 1112786
** read tag to pull from JSON file in functional test dir
** clone or update an existing server repo
** do "npm install" ("make install?") and any other necessary config setup
** execute the functional tests once

nils: what do you think?

Comment 12

[Nils Ohlmeier [:drno]]

(In reply to Dan Mosedale (:dmose) - needinfo? me for response from comment #11)
> Maybe the user story now wants to look like this?
> 
> * write a python script to configure a known tag of the server
> ** if not already bootstrapped, execute the bootstrap process to be written
> in bug 1112786
> ** read tag to pull from JSON file in functional test dir
> ** clone or update an existing server repo
> ** do "npm install" ("make install?") and any other necessary config setup
> ** execute the functional tests once
> 
> nils: what do you think?

Yes that sounds reasonable and doable.

Comment 13

[Dan Mosedale (:dmosedale, :dmose)]

For the same reasoning described at https://bugzilla.mozilla.org/show_bug.cgi?id=1065889#c6 I'm no longer so convinced that this wants to be a mach command.  I'll curious to hear Standard8's thoughts in response to that comment before deciding what to do here.

Comment 14

[Mark Banner (:standard8)]

Support for Hello/Loop has been discontinued.

https://support.mozilla.org/kb/hello-status

Hence closing the old bugs. Thank you for your support.