Closed Bug 976134 Opened 11 years ago Closed 8 years ago

get functional tests running with link-clickers on Chrome

Categories

(Hello (Loop) :: Client, defect, P5)

defect

Tracking

(Not tracked)

RESOLVED INCOMPLETE
backlog tech-debt

People

(Reporter: abr, Unassigned)

References

Details

(Whiteboard: [tech-debt])

We need to implement end-to-end system tests for the standalone web-page UI (the one the "call me" link clicker receives). Ultimately, these tests need to encompass Firefox Desktop, Firefox Android, Firefox OS, Chrome Desktop, and Chrome Android.
For this, the thing that probably makes the most sense is a test suite the uses the webdriver API and wire protocol. See bug 976114 for most of the reasoning. Marionette does this for at least Firefox Desktop and FirefoxOS today; not sure about Android. I believe Chrome has or will have something equivalent built in as well. David, do you know any more details about the various platforms mentioned above that are (or will be) in play with implementations?
(In reply to Dan Mosedale (:dmose) from comment #1) > > Marionette does this for at least Firefox Desktop and FirefoxOS today; not > sure about Android. I believe Chrome has or will have something equivalent > built in as well. > Chrome is supported by the Chromium team We don't have Firefox for Android support, the ground work is there and done but we havent done the work to get tests running on TBPL but B2G and Desktop support is there. The support on desktop may have a few bugs but we can get them sorted. Hope that helps.
Thanks does help; thanks, David. A few more questions: * is there a timeline planned for Firefox Android support? https://sites.google.com/a/chromium.org/chromedriver/getting-started/getting-started---android It current says that ChromeDriver supports Chrome 30+ in Android as a beta (ie not all APIs implemented). https://sites.google.com/a/chromium.org/chromedriver/getting-started Seems to suggest that Chrome on desktop is already supported. Presumably if that support has problems, we can just fall back to WebDriver classic until they get fixed. Do you have any information or links that would confirm or deny the information above? Thanks!
Flags: needinfo?(dburns)
Since this is a MLP bug, I need this to have an owner. Dan -- can I give this to you?
Assignee: nobody → dmose
(In reply to Dan Mosedale (:dmose) from comment #3) > Thanks does help; thanks, David. > > A few more questions: > > * is there a timeline planned for Firefox Android support? > Not currently, mainly because no one has asked for it. as I said in comment 2 the ground work is there, it just needs finishing off and putting on to TBPL. > https://sites.google.com/a/chromium.org/chromedriver/getting-started/getting- > started---android > > It current says that ChromeDriver supports Chrome 30+ in Android as a beta > (ie not all APIs implemented). > > https://sites.google.com/a/chromium.org/chromedriver/getting-started > Note that it only works on rooted android devices, presumably to get around the security and privacy people. https://sites.google.com/a/chromium.org/chromedriver/getting-started/getting-started---android > Seems to suggest that Chrome on desktop is already supported. Presumably if > that support has problems, we can just fall back to WebDriver classic until > they get fixed. In this world its just webdriver, no classic. Their desktop support is based on the OSS project and the WebDriver Specification. http://dvcs.w3.org/hg/webdriver/raw-file/tip/webdriver-spec.html Hope that helps
Flags: needinfo?(dburns)
(In reply to David Burns :automatedtester from comment #5) > > In this world its just webdriver, no classic. Their desktop support is based > on the OSS project and the WebDriver Specification. > > http://dvcs.w3.org/hg/webdriver/raw-file/tip/webdriver-spec.html > > Hope that helps Meaning that we'd just use the open-source Selenium python web-driver bindings, which would incidentally use the webdriver protocol directly to Chrome's built-in support under the hood? So, assuming the above is correct (but even if it's not) given that we have some HTML/CSS/JS code (the main conversation Tab, if you're running Chrome), which we're hoping to execute in SocialAPI context in Firefox, what Python bindings make the most sense to choose for those tests? We're also going to need to be testing pure Firefox chrome stuff (eg pressing Firefox buttons). Will it be possible to use a single python binding everywhere?
Flags: needinfo?(dburns)
And by "SocialAPI context" above, I meant in a SocialAPI chat window iframe.
(In reply to Dan Mosedale (:dmose) from comment #6) > (In reply to David Burns :automatedtester from comment #5) > > > > In this world its just webdriver, no classic. Their desktop support is based > > on the OSS project and the WebDriver Specification. > > > > http://dvcs.w3.org/hg/webdriver/raw-file/tip/webdriver-spec.html > > > > Hope that helps > > Meaning that we'd just use the open-source Selenium python web-driver > bindings, which would incidentally use the webdriver protocol directly to > Chrome's built-in support under the hood? > That is correct. This is still a bit off in timescale but that is our end goal. To people using webdriver the change will be invisible, it will just appear to work. > So, assuming the above is correct (but even if it's not) given that we have > some HTML/CSS/JS code (the main conversation Tab, if you're running Chrome), > which we're hoping to execute in SocialAPI context in Firefox, what Python > bindings make the most sense to choose for those tests? > WebDriver > We're also going to need to be testing pure Firefox chrome stuff (eg > pressing Firefox buttons). > Marionette === WebDriver++. The Chrome stuff will require the use of Marionette but both WebDriver and Marionette will share dependencies, I just don't want to give web developers and testers (our biggest audience) a massive footgun by allowing them to work with the browser chrome. > Will it be possible to use a single python binding everywhere? Everywhere unless you need to access browser chrome. You could share sessions between marionette and webdriver if you need to step out of content into Chrome. Using WebDriver will satisfy the 95% of our users.
Flags: needinfo?(dburns)
(In reply to David Burns :automatedtester from comment #8) > > So, assuming the above is correct (but even if it's not) given that we have > > some HTML/CSS/JS code (the main conversation Tab, if you're running Chrome), > > which we're hoping to execute in SocialAPI context in Firefox, what Python > > bindings make the most sense to choose for those tests? > > > > WebDriver Presumably, we're going to need at least one chrome hook to start a bunch of our tests: click the social API provider button in chrome to drop down the iframe. I assume there's no way to do this from WebDriver? If there is a way to do this from WebDriver, how do we select the SocialAPI panels (some of which are dynamically created) to test them? > > > We're also going to need to be testing pure Firefox chrome stuff (eg > > pressing Firefox buttons). > > > > Marionette === WebDriver++. The Chrome stuff will require the use of > Marionette but both WebDriver and Marionette will share dependencies, I just > don't want to give web developers and testers (our biggest audience) a > massive footgun by allowing them to work with the browser chrome. Would it make sense to spin-off a bug about including making SocialAPI tests be more generally available? > > Will it be possible to use a single python binding everywhere? > > Everywhere unless you need to access browser chrome. You could share > sessions between marionette and webdriver if you need to step out of content > into Chrome. Using WebDriver will satisfy the 95% of our users. By sharing sessions, the idea would be to import both Marionette and WebDriver into a single script, and when we needed chrome privs, we'd use Marionette on the same session handle for that stuff?
Flags: needinfo?(dburns)
(In reply to Dan Mosedale (:dmose) from comment #9) > (In reply to David Burns :automatedtester from comment #8) > > > > So, assuming the above is correct (but even if it's not) given that we have > > > some HTML/CSS/JS code (the main conversation Tab, if you're running Chrome), > > > which we're hoping to execute in SocialAPI context in Firefox, what Python > > > bindings make the most sense to choose for those tests? > > > > > > > WebDriver > > > Presumably, we're going to need at least one chrome hook to start a bunch of > our tests: click the social API provider button in chrome to drop down the > iframe. I assume there's no way to do this from WebDriver? > > If there is a way to do this from WebDriver, how do we select the SocialAPI > panels (some of which are dynamically created) to test them? If you need to interact with the browser chrome you're not going to get it from Chrome and WebDriver. We have added Chrome access to Marionette as a way to centralise all our tools (Like WebDriver and Mozmill). If you just need to handle dynamically added content then webdriver can handle that. And by WebDriver I mean your tests need to handle this situation > > > > > > We're also going to need to be testing pure Firefox chrome stuff (eg > > > pressing Firefox buttons). > > > > > > > Marionette === WebDriver++. The Chrome stuff will require the use of > > Marionette but both WebDriver and Marionette will share dependencies, I just > > don't want to give web developers and testers (our biggest audience) a > > massive footgun by allowing them to work with the browser chrome. > > Would it make sense to spin-off a bug about including making SocialAPI tests > be more generally available? Sure, if you put how you want to interact with all the different browsers we can then set realistic expectations on what each can or can't do. I am happy in some circumstances to interface with the WebDriver teams in other vendors to see if we can come to some middle ground if what we need isnt available. > > > > Will it be possible to use a single python binding everywhere? > > > > Everywhere unless you need to access browser chrome. You could share > > sessions between marionette and webdriver if you need to step out of content > > into Chrome. Using WebDriver will satisfy the 95% of our users. > > By sharing sessions, the idea would be to import both Marionette and > WebDriver into a single script, and when we needed chrome privs, we'd use > Marionette on the same session handle for that stuff? Exactly that. Since they are both going to be using the same transportation mechanism you won't have to worry that the connections are done in different ways.
Flags: needinfo?(dburns)
OK, the SocialAPI bug still needs to be spun off. For now, I think the thing to do, since people are working on the in-browser code first, is to start there (i.e. in bug 976114). Based on the above conversation, I think the right plan is to mostly focus on Marionette, and then mix in WebDriver when we need to test the conversation-in-a-tab code so that those tests will Just Work in Chrome.
Priority: -- → P3
Blocks: 994274
No longer blocks: 972024
We want to do this quickly after MLP.
Whiteboard: [tech-risk]
Priority: P3 → P5
Whiteboard: [tech-risk] → [tech-risk][p=?, s=U32]I
Target Milestone: --- → mozilla33
Priority: P5 → P2
Whiteboard: [tech-risk][p=?, s=U32]I → [tech-risk][p=?, p=?]
need to group all system test bugs together to review, prioritize, and break-down.
Whiteboard: [tech-risk][p=?, p=?] → [tech-risk][p=2, investigation]
Mark -- Do you think you could a bug breakdown for this? Or do you need more info?
Assignee: dmose → standard8
I *think* Dan is working on this elsewhere...
Assignee: standard8 → dmose
Target Milestone: mozilla33 → mozilla34
Priority: P2 → P1
Hey Dan, is this actually a dupe of all the functional test work you're doing now? so do we still need this meta bug for breakdown or can we close it?
Flags: needinfo?(dmose)
This is not really a dupe, no. The other bugs focussed on automated testing are really about getting things started on Firefox. This bug might reasonably be re-titled, though, so I'll do that.
Flags: needinfo?(dmose)
Summary: [meta] Implement Loop standalone-page system tests → [meta] get functional tests running with link-clickers on browsers other than Firefox desktop
I really want this, but I don't think we have to have this in the Fx34 cycle. Dan -- If you disagree, please let me know.
Target Milestone: mozilla34 → mozilla35
I don't think this can happen before the nightly -> aurora uplift. It's conceivable it could happen later than that, however.
Actually, I think that's gonna depend pretty heavily on some interop changes that :AutomatedTester recently started working on, but I'd be a little surprised if they were scheduled to be ready soon enough.
I _think_ that the easiest way forward is likely to still be the one I mentioned in comment 11: > Based on the above conversation, I think the right plan is to mostly focus on Marionette, and > then mix in WebDriver when we need to test the conversation-in-a-tab code so that those tests > will Just Work in Chrome. That said, I think that it depends on some work that :AutomatedTester has in progress, and he may also have new thoughts on whether that's the right strategy or not. So I think the first thing to do here is have a conversation with him.
Summary: [meta] get functional tests running with link-clickers on browsers other than Firefox desktop → [meta] get functional tests running with link-clickers on Chrome
There are plenty of other platforms to target, but I would expect Chrome desktop to be the one that makes the most sense to target first, given it's broad distribution, and similar UX to firefox desktop (for which there is already a test written).
Summary: [meta] get functional tests running with link-clickers on Chrome → get functional tests running with link-clickers on Chrome
Whiteboard: [tech-risk][p=2, investigation] → [tech-debt][p=2, investigation]
backlog: --- → Fx36+
FYI the Chrome team has setup a Chrome/Firefox WebRTC interop test in their automation, the details there might be useful: http://googletesting.blogspot.se/2014/08/chrome-firefox-webrtc-interop-test-pt-1.html We're also interested in Firefox/Chrome interop testing as part of bug 1064253 (not immediately, but in the future), so I'd certainly be interested in comparing notes there.
backlog: Fx36+ → Fx37+
Target Milestone: mozilla35 → ---
Nils - is this on your radar...or has progress been started?
Assignee: dmose → nobody
Flags: needinfo?(drno)
I can add it to my todo list. I only fear it that we still quite a backlog before accomplishing the end goal: - downloading Google Chrome on a continuous base (hopefully easy) - starting and controlling two browsers from Marionette (no idea how hard this) - start one Firefox and one Chrome and control them. - put all the pieces together.
Flags: needinfo?(drno)
Moving this to P2 based on our new priority definitions.
Priority: P1 → P2
Moving this to Fx38? given the scope -- and a later release may make more sense (again, because of the scope, not because we don't want this).
backlog: Fx37+ → Fx38?
Nils comment shows a lot blocking this still - so lower in tech debt
Priority: P2 → P5
backlog: Fx38? → tech-debt
Whiteboard: [tech-debt][p=2, investigation] → [tech-debt][needs marionette work]
Rank: 55
Dan: does this still fit with our testing plan?
Flags: needinfo?(dmose)
(In reply to Ian Bicking (:ianb) from comment #29) > Dan: does this still fit with our testing plan? Yes, I totally want this. We should be having cross-browser tests automated - for one thing it'd simplify the manual testing needed whenever we upgrade the SDK, and it'd also provide us protection if we broke Chrome somehow (or they changed an API for example. It does depend on some other work though, so I've added a couple of things to the dependencies.
Depends on: 1238562, 1071990
Flags: needinfo?(dmose)
Whiteboard: [tech-debt][needs marionette work] → [tech-debt]
Support for Hello/Loop has been discontinued. https://support.mozilla.org/kb/hello-status Hence closing the old bugs. Thank you for your support.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.