1071762 - Loop functional tests need stub FxA server so we can put these in Tbpl

Status: RESOLVED INCOMPLETE

(Hello (Loop) :: General, defect, P4)

Description

[Dan Mosedale (:dmosedale, :dmose)]

Loop functional tests currently depend on running against the live FxA server.  Since Tbpl can't afford to have intermittent failures caused by network connectivity issues, we need to do something different.

I know MattN has written some mock code that supports his existing mochitests.  I'm not sure whether that's a good fit to be extended into something that could support a full end-to-end call.  MattN?

If that's not the case, perhaps the FxA folks have some advice (or even code?) that we could use for this...

Comment 1

[Matthew N. [:MattN]]

I used the standard mochitest/xpcshell method of using SJS files. Mozilla developers are already familiar with it and it's documented on MDN so that's why I chose it. The disadvantage is that if your stub server is incorrect then you may not be testing the correct behaviour.

My understanding is that Firefox sync has its own harness that doesn't run on TBPL but does have network access which means it can test against real servers. Perhaps that's something Loop could do too?

Comment 2

[Dan Mosedale (:dmosedale, :dmose)]

It would be in some ways ideal to do what we're doing with loop-server, which is actually to test against a local, isolated copy of the real thing.  Whether that's actually practical in this case, I have no idea; needinfo-ing ckarloff for his thoughts here.

The main reason for stubbing to avoid networks is that they tend to be unreliable in ways that cause intermittent failures.  Secondarily, it's also useful to be able to do development and run tests when disconnected from the net (eg on Muni or airplanes).

That said, it's also conceivable that TreeHerder will change the desired constraints around networking in tests, so if it's not practicaly to do without a network, we could look into that more...

Comment 3

[Chris Karlof [:ckarlof]]

It's going to be difficult to mock the FxA server for functional tests, but if you're set up to test against a local Loop server, you could also have use a local FxA setup for the functional tests. The wrinkle is that FxA is actually a set of servers, and getting all of them running is not trivial. However, we have a project fxa-dev to make provisioning FxA for dev/testing environments easier: 

https://github.com/mozilla/fxa-dev

Perhaps we could take advantage of fxa-dev here?

Comment 4

[Richard Pappalardo [:rpapa][:rpappalardo]]

I created a mock fxa auth server for running marionette js tests on b2g. Don't know if that's sufficient for what you're trying to do, but the code is here:
https://github.com/mozilla-b2g/gaia/tree/master/apps/system/fxa/test/marionette/lib

Comment 5

[Cosmin Malutan, [:cosmin-malutan]]

This issue is tagged with softvision in trello dashboard so we can at-least have an input here.

For the sync testruns, we use the production server, and we monitor the testruns separately from TBPL.
We run the tests after the build has been published on ftp, and if we find regressions we file bugs afterwards. I know that this is not that effective as running the tests on commit but on the other hand we use the production FxA server and we might find problems on the server-side too, as it happened for sync(TPS).

If you guys want to run the tests outside of TBPL and use Jenkins-CI we might help, otherwise mocking the FxA server is way out of our area of expertise.

Comment 6

[:shell escalante]

Moving these to blocking-loop="tech-debt" so they can be triaged against other tech debt bugs.  When we take a tech-debt bug to work on - just mark it "firefox-backlog"+ and give priority.  Then we'll pull it during iteration planning (or factor it into workload if someone is already working on).

Comment 7

[Mark Banner (:standard8)]

Support for Hello/Loop has been discontinued.

https://support.mozilla.org/kb/hello-status

Hence closing the old bugs. Thank you for your support.