976134 - get functional tests running with link-clickers on Chrome

Status: RESOLVED INCOMPLETE

(Hello (Loop) :: Client, defect, P5)

Description

[Adam Roach [:abr]]

We need to implement end-to-end system tests for the standalone web-page UI (the one the "call me" link clicker receives). Ultimately, these tests need to encompass Firefox Desktop, Firefox Android, Firefox OS, Chrome Desktop, and Chrome Android.

Comment 1

[Dan Mosedale (:dmosedale, :dmose)]

For this, the thing that probably makes the most sense is a test suite the uses the webdriver API and wire protocol.  See bug 976114 for most of the reasoning.

Marionette does this for at least Firefox Desktop and FirefoxOS today; not sure about Android.  I believe Chrome has or will have something equivalent built in as well.  

David, do you know any more details about the various platforms mentioned above that are (or will be) in play with implementations?

Comment 2

[David Burns :automatedtester]

(In reply to Dan Mosedale (:dmose) from comment #1)
> 
> Marionette does this for at least Firefox Desktop and FirefoxOS today; not
> sure about Android.  I believe Chrome has or will have something equivalent
> built in as well.  
> 

Chrome is supported by the Chromium team

We don't have Firefox for Android support, the ground work is there and done but we havent done the work to get tests running on TBPL but B2G and Desktop support is there. The support on desktop may have a few bugs but we can get them sorted.

Hope that helps.

Comment 3

[Dan Mosedale (:dmosedale, :dmose)]

Thanks does help; thanks, David.

A few more questions: 

* is there a timeline planned for Firefox Android support?

https://sites.google.com/a/chromium.org/chromedriver/getting-started/getting-started---android

It current says that ChromeDriver supports Chrome 30+ in Android as a beta (ie not all APIs implemented).

https://sites.google.com/a/chromium.org/chromedriver/getting-started

Seems to suggest that Chrome on desktop is already supported.  Presumably if that support has problems, we can just fall back to WebDriver classic until they get fixed.

Do you have any information or links that would confirm or deny the information above?

Thanks!

Comment 4

[Maire Reavy [:mreavy]]

Since this is a MLP bug, I need this to have an owner.  Dan -- can I give this to you?

Comment 5

[David Burns :automatedtester]

(In reply to Dan Mosedale (:dmose) from comment #3)
> Thanks does help; thanks, David.
> 
> A few more questions: 
> 
> * is there a timeline planned for Firefox Android support?
> 

Not currently, mainly because no one has asked for it. as I said in comment 2 the ground work is there, it just needs finishing off and putting on to TBPL. 


> https://sites.google.com/a/chromium.org/chromedriver/getting-started/getting-
> started---android
> 
> It current says that ChromeDriver supports Chrome 30+ in Android as a beta
> (ie not all APIs implemented).
> 
> https://sites.google.com/a/chromium.org/chromedriver/getting-started
> 

Note that it only works on rooted android devices, presumably to get around the security and privacy people.

https://sites.google.com/a/chromium.org/chromedriver/getting-started/getting-started---android


> Seems to suggest that Chrome on desktop is already supported.  Presumably if
> that support has problems, we can just fall back to WebDriver classic until
> they get fixed.

In this world its just webdriver, no classic. Their desktop support is based on the OSS project and the WebDriver Specification.

http://dvcs.w3.org/hg/webdriver/raw-file/tip/webdriver-spec.html

Hope that helps

Comment 6

[Dan Mosedale (:dmosedale, :dmose)]

(In reply to David Burns :automatedtester from comment #5)
>
> In this world its just webdriver, no classic. Their desktop support is based
> on the OSS project and the WebDriver Specification.
> 
> http://dvcs.w3.org/hg/webdriver/raw-file/tip/webdriver-spec.html
> 
> Hope that helps

Meaning that we'd just use the open-source Selenium python web-driver bindings, which would incidentally use the webdriver protocol directly to Chrome's built-in support under the hood?

So, assuming the above is correct (but even if it's not) given that we have some HTML/CSS/JS code (the main conversation Tab, if you're running Chrome), which we're hoping to execute in SocialAPI context in Firefox, what Python bindings make the most sense to choose for those tests?

We're also going to need to be testing pure Firefox chrome stuff (eg pressing Firefox buttons).

Will it be possible to use a single python binding everywhere?

Comment 7

[Dan Mosedale (:dmosedale, :dmose)]

And by "SocialAPI context" above, I meant in a SocialAPI chat window iframe.

Comment 8

[David Burns :automatedtester]

(In reply to Dan Mosedale (:dmose) from comment #6)
> (In reply to David Burns :automatedtester from comment #5)
> >
> > In this world its just webdriver, no classic. Their desktop support is based
> > on the OSS project and the WebDriver Specification.
> > 
> > http://dvcs.w3.org/hg/webdriver/raw-file/tip/webdriver-spec.html
> > 
> > Hope that helps
> 
> Meaning that we'd just use the open-source Selenium python web-driver
> bindings, which would incidentally use the webdriver protocol directly to
> Chrome's built-in support under the hood?
> 

That is correct. This is still a bit off in timescale but that is our end goal. To people using webdriver the change will be invisible, it will just appear to work.

> So, assuming the above is correct (but even if it's not) given that we have
> some HTML/CSS/JS code (the main conversation Tab, if you're running Chrome),
> which we're hoping to execute in SocialAPI context in Firefox, what Python
> bindings make the most sense to choose for those tests?
> 

WebDriver

> We're also going to need to be testing pure Firefox chrome stuff (eg
> pressing Firefox buttons).
> 

Marionette === WebDriver++. The Chrome stuff will require the use of Marionette but both WebDriver and Marionette will share dependencies, I just don't want to give web developers and testers (our biggest audience) a massive footgun by allowing them to work with the browser chrome.

> Will it be possible to use a single python binding everywhere?

Everywhere unless you need to access browser chrome. You could share sessions between marionette and webdriver if you need to step out of content into Chrome. Using WebDriver will satisfy the 95% of our users.

Comment 9

[Dan Mosedale (:dmosedale, :dmose)]

(In reply to David Burns :automatedtester from comment #8)

> > So, assuming the above is correct (but even if it's not) given that we have
> > some HTML/CSS/JS code (the main conversation Tab, if you're running Chrome),
> > which we're hoping to execute in SocialAPI context in Firefox, what Python
> > bindings make the most sense to choose for those tests?
> > 
> 
> WebDriver


Presumably, we're going to need at least one chrome hook to start a bunch of our tests: click the social API provider button in chrome to drop down the iframe.  I assume there's no way to do this from WebDriver?

If there is a way to do this from WebDriver, how do we select the SocialAPI panels (some of which are dynamically created) to test them?

> 
> > We're also going to need to be testing pure Firefox chrome stuff (eg
> > pressing Firefox buttons).
> > 
> 
> Marionette === WebDriver++. The Chrome stuff will require the use of
> Marionette but both WebDriver and Marionette will share dependencies, I just
> don't want to give web developers and testers (our biggest audience) a
> massive footgun by allowing them to work with the browser chrome.

Would it make sense to spin-off a bug about including making SocialAPI tests be more generally available?

> > Will it be possible to use a single python binding everywhere?
> 
> Everywhere unless you need to access browser chrome. You could share
> sessions between marionette and webdriver if you need to step out of content
> into Chrome. Using WebDriver will satisfy the 95% of our users.

By sharing sessions, the idea would be to import both Marionette and WebDriver into a single script, and when we needed chrome privs, we'd use Marionette on the same session handle for that stuff?

Comment 10

[David Burns :automatedtester]

(In reply to Dan Mosedale (:dmose) from comment #9)
> (In reply to David Burns :automatedtester from comment #8)
> 
> > > So, assuming the above is correct (but even if it's not) given that we have
> > > some HTML/CSS/JS code (the main conversation Tab, if you're running Chrome),
> > > which we're hoping to execute in SocialAPI context in Firefox, what Python
> > > bindings make the most sense to choose for those tests?
> > > 
> > 
> > WebDriver
> 
> 
> Presumably, we're going to need at least one chrome hook to start a bunch of
> our tests: click the social API provider button in chrome to drop down the
> iframe.  I assume there's no way to do this from WebDriver?
> 
> If there is a way to do this from WebDriver, how do we select the SocialAPI
> panels (some of which are dynamically created) to test them?

If you need to interact with the browser chrome you're not going to get it from Chrome and WebDriver. We have added Chrome access to Marionette as a way to centralise all our tools (Like WebDriver and Mozmill). If you just need to handle dynamically added content then webdriver can handle that. And by WebDriver I mean your tests need to handle this situation


> 
> > 
> > > We're also going to need to be testing pure Firefox chrome stuff (eg
> > > pressing Firefox buttons).
> > > 
> > 
> > Marionette === WebDriver++. The Chrome stuff will require the use of
> > Marionette but both WebDriver and Marionette will share dependencies, I just
> > don't want to give web developers and testers (our biggest audience) a
> > massive footgun by allowing them to work with the browser chrome.
> 
> Would it make sense to spin-off a bug about including making SocialAPI tests
> be more generally available?

Sure, if you put how you want to interact with all the different browsers we can then set realistic expectations on what each can or can't do. I am happy in some circumstances to interface with the WebDriver teams in other vendors to see if we can come to some middle ground if what we need isnt available.

> 
> > > Will it be possible to use a single python binding everywhere?
> > 
> > Everywhere unless you need to access browser chrome. You could share
> > sessions between marionette and webdriver if you need to step out of content
> > into Chrome. Using WebDriver will satisfy the 95% of our users.
> 
> By sharing sessions, the idea would be to import both Marionette and
> WebDriver into a single script, and when we needed chrome privs, we'd use
> Marionette on the same session handle for that stuff?

Exactly that. Since they are both going to be using the same transportation mechanism you won't have to worry that the connections are done in different ways.

Comment 11

[Dan Mosedale (:dmosedale, :dmose)]

OK, the SocialAPI bug still needs to be spun off.  For now, I think the thing to do, since people are working on the in-browser code first, is to start there (i.e. in bug 976114).

Based on the above conversation, I think the right plan is to mostly focus on Marionette, and then mix in WebDriver when we need to test the conversation-in-a-tab code so that those tests will Just Work in Chrome.

Comment 12

[Maire Reavy [:mreavy]]

We want to do this quickly after MLP.

Comment 13

[:shell escalante]

need to group all system test bugs together to review, prioritize, and break-down.

Comment 14

[Maire Reavy [:mreavy]]

Mark -- Do you think you could a bug breakdown for this?  Or do you need more info?

Comment 15

[Mark Banner (:standard8)]

I *think* Dan is working on this elsewhere...

Comment 16

[:shell escalante]

Hey Dan, is this actually a dupe of all the functional test work you're doing now?  so do we still need this meta bug for breakdown or can we close it?

Comment 17

[Dan Mosedale (:dmosedale, :dmose)]

This is not really a dupe, no.  The other bugs focussed on automated testing are really about getting things started on Firefox.  This bug might reasonably be re-titled, though, so I'll do that.

Comment 18

[Maire Reavy [:mreavy]]

I really want this, but I don't think we have to have this in the Fx34 cycle.

Dan -- If you disagree, please let me know.

Comment 19

[Dan Mosedale (:dmosedale, :dmose)]

I don't think this can happen before the nightly -> aurora uplift.  It's conceivable it could happen later than that, however.

Comment 20

[Dan Mosedale (:dmosedale, :dmose)]

Actually, I think that's gonna depend pretty heavily on some interop changes that :AutomatedTester recently started working on, but I'd be a little surprised if they were scheduled to be ready soon enough.

Comment 21

[Dan Mosedale (:dmosedale, :dmose)]

I _think_ that the easiest way forward is likely to still be the one I mentioned in comment 11:

> Based on the above conversation, I think the right plan is to mostly focus on Marionette, and
> then mix in WebDriver when we need to test the conversation-in-a-tab code so that those tests
> will Just Work in Chrome.

That said, I think that it depends on some work that :AutomatedTester has in progress, and he may also have new thoughts on whether that's the right strategy or not.  So I think the first thing to do here is have a conversation with him.

Comment 22

[Dan Mosedale (:dmosedale, :dmose)]

There are plenty of other platforms to target, but I would expect Chrome desktop to be the one that makes the most sense to target first, given it's broad distribution, and similar UX to firefox desktop (for which there is already a test written).

Comment 23

[(not currently active) Ted Mielczarek]

FYI the Chrome team has setup a Chrome/Firefox WebRTC interop test in their automation, the details there might be useful:
http://googletesting.blogspot.se/2014/08/chrome-firefox-webrtc-interop-test-pt-1.html

We're also interested in Firefox/Chrome interop testing as part of bug 1064253 (not immediately, but in the future), so I'd certainly be interested in comparing notes there.

Comment 24

[:shell escalante]

Nils - is this on your radar...or has progress been started?

Comment 25

[Nils Ohlmeier [:drno]]

I can add it to my todo list. 
I only fear it that we still quite a backlog before accomplishing the end goal:
- downloading Google Chrome on a continuous base (hopefully easy)
- starting and controlling two browsers from Marionette (no idea how hard this)
- start one Firefox and one Chrome and control them.
- put all the pieces together.

Comment 26

[Maire Reavy [:mreavy]]

Moving this to P2 based on our new priority definitions.

Comment 27

[Maire Reavy [:mreavy]]

Moving this to Fx38? given the scope -- and a later release may make more sense (again, because of the scope, not because we don't want this).

Comment 28

[:shell escalante]

Nils comment shows a lot blocking this still - so lower in tech debt

Comment 29

[Ian Bicking (:ianbicking)]

Dan: does this still fit with our testing plan?

Comment 30

[Mark Banner (:standard8)]

(In reply to Ian Bicking (:ianb) from comment #29)
> Dan: does this still fit with our testing plan?

Yes, I totally want this. We should be having cross-browser tests automated - for one thing it'd simplify the manual testing needed whenever we upgrade the SDK, and it'd also provide us protection if we broke Chrome somehow (or they changed an API for example.

It does depend on some other work though, so I've added a couple of things to the dependencies.

Comment 31

[Mark Banner (:standard8)]

Support for Hello/Loop has been discontinued.

https://support.mozilla.org/kb/hello-status

Hence closing the old bugs. Thank you for your support.