Closed Bug 1065889 Opened 11 years ago Closed 9 years ago

Loop functional tests should download and setup loop-server

Categories

(Hello (Loop) :: General, defect, P3)

Product:
Hello (Loop)
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE
Blocking Flags:
backlog tech-debt

People

(Reporter: aoprea, Assigned: aoprea)

References

Details

(Whiteboard: [testing][tech-debt])

User Story

As a developer, I run functional tests using 'mach marionette-test browser/components/loop/test/functional/manifest.ini' even if I've never pulled and configured a server from git:

* if LOOP_SERVER in the environment points to a directory, use it without updating to run the tests
* otherwise, look for an arbitrary well-known directory (maybe OBJDIR/loop-server?) containing a git repo
* if not found, git clone from the appropriate tag
* use an appropriate config file that is sufficient to make a live call through the tokbox network (maybe this file wants to live and be maintained in the loop-server repo?).  Need to figure out what keys we can use for this, given that it'll be publicly checked in.
Running loop functional tests require a local loop-server running. When starting the tests if loop-server is not preset it should automatically be cloned and setup for use.
Whiteboard: [loop-fn-test]
Most importantly, we need to decide what version of the server makes sense to test against. Since these are nightly builds, I could imagine us using the latest master of loop-server, as this would help ensure that no changes on one side break the other. There are, of course, pros and cons to coupling things that tightly. We could also just do "latest version deployed to prod", since that's what nightly actually runs against for users. Thoughts? I found a suggestion in some old notes that the mozharness Python package could do some of the lifting to read a JSON file containing the loop-server revision that we want to use and then pull it from git. There is apparently some prior art here in gaia. Ask in #ateam to find out more about that (maybe jlal knows?).
User Story: (updated)
We try not to break loop-server master, but it happened to be the case in the past. As these functional tests are meant to detect problems, I believe we should run latest master, though. Latest version deployed to prod will not let us detect problems earlier in the cycle, but that would break less often. I believe we should start with testing against master and if that causes troubles, then we can move to the latest tag for instance.
backlog: --- → Fx37?
backlog: Fx37? → Fx37+
Priority: -- → P2
can you add details to the user story as to what the work is - so the person picking up can go with it.
Flags: needinfo?(dmose)
Shell, I'll add an approximate technical checklist once I get back in January. Sorry for the delay.
backlog: Fx37+ → Fx38+
Whiteboard: [loop-fn-test] → [loop-fn-test][tech-debt]
Priority: P2 → P3
Short status update: the functional test here: http://10.252.73.218:8080/view/hello/job/hello-e2e-marionette/ clones the loop-server repo before the test execution. And even if we add this feature to the test setup functions we should have an option to skip this step in case either the developer or the test executor has its own version of loop server already checked out. In my opinion this is mostly a convenience thing for people who are not familiar with Loop development.
I think I generally agree with comment 5, however, I think it's more than just a convenience thing. It's actually a barrier to a few important things: * making it practical for other Mozillians to help out ** in the case of the marionette team, debugging Marionette problems in situ (with requiring reduced test cases) ** in the case of other contributors: making it hard for them to tell if their changes have broken our basic call functionality * helping us bootstrap new contributors once we start add "functional tests" to our "definition of done". I think, however, that we don't necessarily have to have this integrated into the running of the functional test. It probably just wants to be either a standalone Python script or a mach command, named something like "bootstrap-functional-test". I'm not really convinced we want this to be a mach command, as I can imagine wanting to run this in places like the loop-client repo as part of the loop-server dev process. Admittedly, we could somehow include mach in the sync to other places, but it just seems like adding this dependency is probably more trouble than it's worth. I'd love to hear Standard8's thoughts here before nailing down how to get started, so giving him a needing...
Flags: needinfo?(dmose) → needinfo?(standard8)
I've just had a think about this, and I realised the critical part here is "contributors". When you start to focus on that, then the issue becomes that setting up the loop-server to run is difficult for new contributors - it requires getting access to keys and things which isn't all that easy. Additionally, the tests are currently run against the OpenTok servers - i.e. over the internet. Given those, I'm tempted to say that we should actually make the default be "run against the dev server". This shouldn't be too difficult I believe we should be able to point the tests directly at the dev server (I believe I landed that support a while ago). If we want to make the default to use the standalone files stored locally (Rather than the loop-client on the dev server), then something like bug 1128979 coupled with some url re-writing (e.g. http://dev/123456789 -> http://localhost/123456789) should make it work. When we get to the stage of stubbing out the OpenTok servers (or something appropriate), then we can revisit what we do with the defaults.
backlog: Fx38+ → tech-debt
Flags: needinfo?(standard8)
OS: Mac OS X → All
Hardware: x86 → All
Blocks: 1230204
Rank: 35
Whiteboard: [loop-fn-test][tech-debt] → [testing][tech-debt]
Support for Hello/Loop has been discontinued. https://support.mozilla.org/kb/hello-status Hence closing the old bugs. Thank you for your support.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.