1294151 - New versions of Firefox break Selenium testing suites too often.

Status: RESOLVED INVALID

(Firefox :: General, defect)

Description

[Stephanie Hobson [:shobson]]

New versions of Firefox break Selenium testing suites too often.

At a conference last week I had two separate developers tell me they have stopped testing in Firefox because this happens to them too frequently.

I have experienced this myself and have run tests only in Chrome when Firefox fails.

If we want people to do automated testing in Firefox it needs to be easy and dependable :/

Comment 1

[:Gijs (he/him)]

Can you elaborate on what "break" means?

Is Firefox::General really the best place for this bug, that is, maybe this is an issue with the webdriver or whatever that marionette/selenium component thing is called? While I work on Fx frontend, I have no idea about the things selenium needs/wants here and why they would break. If it's any help, we're not breaking anything on purpose. :-\

Maybe dburns can help get this bug into productive waters. :-)

Comment 2

[Stephanie Hobson [:shobson]]

Break for me means: Usually when I run tests I see the Firefox windows pop open and the website load and the page interacted with. When they're "broken" I see a series of Firefox windows pop open but the website never loads in any of them. All the tests time out. 

After an update for Selenium comes out the tests all pass again.

Comment 3

[David Burns :automatedtester]

One the steps that my team is doing is to ship Marionette as our WebDriver replacement. 47 was particularly bad with Extension signing about to ship and then didnt and things were unprepared causing the crashes.

Marionette is being built from the WebDriver W3C specification as are the one's from Google for Chrome and MS for Edge. This means it is run on every push to Mozilla-* trees. Any change to Firefox/Gecko gets tested to make sure we don't break it.

With Selenium 3 Marionette has now replaced FirefoxDriver as the default for driving Firefox. There is going to be a bit of beta testing until we are fully compliant (which is something we are working hard towards)

As for moving this into productive waters I am not sure what more we can do with this other than continue along the route we are going with Marionette. I also suggest, if you have the capacity, to try run the Selenium 3 beta against your code in parallel and raise bugs in testing::marionette for anything that doesn't work the same as in FirefoxDriver.

Comment 4

[David Burns :automatedtester]

Are you happy for this bug to be closed with the details in comment 3 or can you give us some more concrete examples that we can make something actionable.

Comment 5

[Stephanie Hobson [:shobson]]

I don't have any concrete suggestions. I just wanted to make sure people know we are losing devs over this.

Excited to hear about Marionette. Hopefully that will make a difference.

Comment 6

[Andreas Tolfsen ❲:ato❳]

It’s an unavoidable fact that FirefoxDriver is not compatible with e10s and that it will not under any circumstances be considered for add-on signing.  It is also maintained by the Selenium project and not Mozilla.  This has been known to both the Firefox product team and Selenium for quite some time.

As David mentions, Marionette is the intended replacement for FirefoxDriver, although implementing a slightly modernised protocol defined by the W3C compared to the one used by the Selenium drivers.  This causes momentary problems as Marionette is currently the only spec-compliant implementation, and the Selenium client binding APIs have not been fully updated to support the spec yet.

https://developer.mozilla.org/en-US/docs/Mozilla/QA/Marionette/WebDriver/status outlines the progress we are making on Marionette and what remains for us to reach spec conformance.  There is also https://developer.mozilla.org/en-US/docs/Mozilla/QA/Marionette/WebDriver on how to get started using Marionette through geckodriver (https://github.com/mozilla/geckodriver), our HTTP frontend that communicates with Selenium.  Unfortunately this only works with Selenium 3 beta clients.