1092239 - Investigate automation for the Standalone page

Status: RESOLVED FIXED

(Hello (Loop) :: General, defect, P2)

Description

[u279076]

This bug is to track standing up some basic automation for the Standalone page in various browsers across all platforms. The basic idea will be to load a https://hello.firefox.com/#call/<id> URL and verify the page loads. In supported browsers (Opera, Chrome, Firefox) we should see the Start Call page. In unsupported browsers (IE, Safari) we should see the Unsupported Browser page.

I'll attach current screenshots of both for reference.

Comment 1

[u279076]

Attachment: Page for supported browsers

Comment 2

[u279076]

Attachment: Page for unsupported browsers

Comment 3

[u279076]

Stephen and I sat down this morning to talk through some of the initial scope. Flagging him so he can add his notes/questions from that meeting.

Comment 4

[Stephen Donner [:stephend] Not actively reading bugmail]

From Web QA's side (just listing here the technical aspects + capabilities, for now):

Capabilities:
* through our Sauce Labs partner, we've got access to all of the platforms + combinations listed on https://saucelabs.com/platforms
* in-house, too (if we had to, had priorities, resources, etc.) through our Selenium Grid (in the QA lab in MTV), https://github.com/mozilla/moz-grid-config and our various nodes (https://wiki.mozilla.org/QA/Execution/Web_Testing/Automation/Selenium_Grid) could be updated to include other browsers (like Aurora/nightly)

Testing Content:
* given the browser/OS configuration capabilities above, we "should" be able to:
** negative-testing use-cases (unsupported browsers):
1) load a https://hello.firefox.com/#call/[callIDString] URL (perhaps without the [callIDString]?, and 2) assert on the "incompatible browser" message across the various unsupported clients
** positive-testing use-case (supported browsers):
1) load a https://hello.firefox.com/#call/[callIDString] URL (again, perhaps at the level asked for here, a genuine callIDString value isn't needed?) and 2) assert on the DOM content (without testing functionality itself)

(Though, looking at the page source for the in-Firefox experience, it's a bunch of browser-chrome level JS scripts - Marionette can surely get at this -- it's then "rendered" content, right?  rather than webpage-derived DOM, but I'm not sure about WebDriver, and Python vs. JS etc.)

Comment 5

[Nils Ohlmeier [:drno]]

Stephen this the API documentation I showed you earlier: http://docs.services.mozilla.com/loop/apis.html

Comment 6

[Nils Ohlmeier [:drno]]

So it looks like you need to register a dummy URL and get Hawk session token first like this:

curl -v -d 'simplePushURL=https://push.services.mozilla.com/update/thisIsAnIvalidLink' https://loop.services.mozilla.com/v0/registration

The reply will have a header in it with something like this:

Hawk-Session-Token: 8c68ddd9a0cd800dde0e1afa1c45338e265c78f5ad48b418bc61.....

Then you need to break that token into pieces and turn into an Auth header like this:

Authorization:	Hawk id="570bc60130b20a18444f7203d25226e5bb778da9e902a1d0.....", ts="1414789336", nonce="kcpcNtQ97EA=", hash="PAfN1XfNj/l/9hQ5Izw0eg4kPDaEQ6Elt.....", mac="Qw65LHNe6gIXj1QTs6obWeQLlGWurlHI...."

And add that header to a request like this:

curl -v -d '{"callerId":""}' https://loop.services.mozilla.com/call-url/

Which then returns you something like this:

{"callUrl":"https://hello.firefox.com/#call/odCMBS6...","callToken":"odCMBS6m...","call_url":"https://hello.firefox.com/#call/odCMBS6...","expiresAt":1417381342}

Comment 7

[Nils Ohlmeier [:drno]]

The API documentation has some basic explanation in it how to do Hawk authentication: http://docs.services.mozilla.com/loop/apis.html#derive-hawk-credentials-from-the-hawk-session-token

Comment 8

[Nils Ohlmeier [:drno]]

Actually I just realized forget my previous comments.
You should be able to call any https://hello.firefox.com/#call/<id> URL and it should always load the same page no matter if the ID exists or not. Only if you click the start button the ID get verified (there is a bug open to change that behavior).
So you could simply load any random or fixed URL and always should get either the error page for unsupported browser or the proper start call page on supported browsers.

Comment 9

[Richard Pappalardo [:rpapa][:rpappalardo]]

You can also use httpie (easier than curl): 
https://github.com/jakubroztocil/httpie
which has Hawk Auth support. (requests-hawk plugin)

Example:
http POST https://loop.stage.mozaws.net/registration simplePushURL=http://perdu.com
see:  https://docs.services.mozilla.com/loop/apis.html#post-registration

http POST https://loop.stage.mozaws.net/call-url calleeName=foobar --auth-type hawk --auth "sessionToken:"
see: https://docs.services.mozilla.com/loop/apis.html#post-call-url

see: Alexis' blog post:
http://blog.notmyidea.org/whats-hawk-and-how-to-use-it.html

Comment 10

[Mark Banner (:standard8)]

From the client side, we've already had some thoughts around testing. Bug 976134 may be useful for background info, but let me try and clarify a few things:

> (Though, looking at the page source for the in-Firefox experience, it's a
> bunch of browser-chrome level JS scripts - Marionette can surely get at this
> -- it's then "rendered" content, right?  rather than webpage-derived DOM,
> but I'm not sure about WebDriver, and Python vs. JS etc.)

The Link-clicker UI (aka standalone UI), which is what this bug is about, is hosted on a web server and is entirely standalone, save for accessing the relevant loop and tokbox servers for actual call setup. This can obviously be viewed on any web server, and any browser.

The in-Firefox experience is a mixture of chrome and content. We already have a functional test for this, based in Marionette, however it relies on accessing real servers and a bunch of other currently non-automated setup which is why it isn't run automatically just yet (and there's a few other issues, for which bugs are filed). The test can be seen here:

http://mxr.mozilla.org/mozilla-central/source/browser/components/loop/test/functional/

(In reply to Nils Ohlmeier [:drno] from comment #8)
> Actually I just realized forget my previous comments.
> So you could simply load any random or fixed URL and always should get
> either the error page for unsupported browser or the proper start call page
> on supported browsers.

That's correct. We might get move to an "expired" view, but that would still help.


Some more notes:

- The Link-clicker UI parts of Loop share some files with the desktop side.
- For deploying to the server, the relevant files for the link clicker are extracted to the loop-client repository on github (this makes it easier for server ops & also deploying/tagging without affecting the rest of m-c):

https://github.com/mozilla/loop-client

- We have automated unit tests for Link-clicker and the desktop which are run in Marionette:

http://mxr.mozilla.org/mozilla-central/source/browser/components/loop/test/ (desktop-local, shared, standalone directories).

- We want to be able to run these in at least Chrome, to reduce the possibility of issues with not finding faults until deployment time or later.
- Our idea was to set up the loop-client repository to:
-- Be able to run the unit tests in Firefox in Marionette on every push, probably using Travis (as we've used it before)
-- Extend the tests to run with Chrome, also on Travis (I think there's work ongoing for Chrome + Marionette)
-- Push results back to treeherder if possible

It would be good to think that if we're doing automated testing of the whole page, then we could also be incorporating running the unit tests into that work.


Most of this work hasn't been done yet, as we've just not had time. I'm happy to discuss more and look at other solutions if we need to.

Comment 11

[Stephen Donner [:stephend] Not actively reading bugmail]

Thanks, Mark; would there be value in doing a short-term test-automation solution in Python/Selenium WebDriver, run through Sauce Labs, in the interim (re: comment 10), or should we get together soon and plan a combined testing strategy which pulls in the unittests + reports to Treeherder, too?

Comment 12

[Mark Banner (:standard8)]

Are all the potential solutions using selenium/webdriver? If so then I think we should do whatever makes most sense. We've already got unit tests in marionette but we've got the code for selenium/firefox driver as well so porting shouldn't be difficult.

From the sounds of it, whatever functional test solution we come up with, we should be able to do unittests very easily.

I'm hoping to have a bit more time before the end of the year, as a result I was thinking of looking at how much it takes to set up unittests in Travis with chrome.

I would definitely like to get something on at least one non-ff platform before the end of the year.

Comment 13

[:shell escalante]

going to put under meta bug for automation

Comment 14

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Mark Banner (:standard8) from comment #12)
> Are all the potential solutions using selenium/webdriver? If so then I think
> we should do whatever makes most sense. We've already got unit tests in
> marionette but we've got the code for selenium/firefox driver as well so
> porting shouldn't be difficult.
> 
> From the sounds of it, whatever functional test solution we come up with, we
> should be able to do unittests very easily.
> 
> I'm hoping to have a bit more time before the end of the year, as a result I
> was thinking of looking at how much it takes to set up unittests in Travis
> with chrome.
> 
> I would definitely like to get something on at least one non-ff platform
> before the end of the year.

The ones our team know and can/have readily done, yes.  The ones Bob Silverberg has brought up and are running in our Jenkins (currently at http://selenium.qa.mtv2.mozilla.com:8080/view/Hello/).

Definitely want to keep all of our efforts coordinate and well-scoped, so results and the work put in are valuable, so happy to (and we should, soon) meet to talk specifically about the mid-to-longer term automation goals/plans

Comment 15

[Mark Banner (:standard8)]

clearing ni, I think Shell's already put this under a meta bug.

Comment 16

[Ian Bicking (:ianbicking)]

Tests are running now; any improvements will be described in new/followup bugs