731158 - Mozmill test failure /testPreferences/testPreferredLanguage.js | could not find element Link: Gruppi

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect)

Description

[Maniac Vlad Florin (:vladmaniac)]

Build ID:
Build identifier: Mozilla/5.0 (X11; Linux i686; rv:13.0) Gecko/20120227 Firefox/13.0a1
---------------------------------------------------------------------------------------
Platform: All Firefox branches
---------------------------------------------------------------------------------------
Mozmill Version: Mozmill 1.5.9
---------------------------------------------------------------------------------------
Error:
([object Object])@resource://mozmill/modules/controller.js:877 ()@resource://mozmill/modules/frame.js -> file:///c:/users/mozilla/appdata/local/temp/tmp6vp5zh.mozmill-tests/tests/functional/testPreferences/testPreferredLanguage.js:79 ((function () {controller.open("about:blank");prefs.openPreferencesDialog(controller, prefDialogCallback);controller.open("http://www.google.com/");controller.waitForPageLoad();if (browserLocale == "it") {controller.assertNode(new elementslib.Link(controller.tabs.activeTab, "Zaloguj"));controller.assertNode(new elementslib.Link(controller.tabs.activeTab, "Grupy"));controller.assertNode(new elementslib.Link(controller.tabs.activeTab, "Szukanie zaawansowane"));} else {controller.assertNode(new elementslib.Link(controller.tabs.activeTab, "Accedi"));controller.assertNode(new elementslib.Link(controller.tabs.activeTab, "Gruppi"));controller.assertNode(new elementslib.Link(controller.tabs.activeTab, "Ricerca avanzata"));}}))@resource://mozmill/modules/frame.js:557 ([object Object])@resource://mozmill/modules/frame.js:626 ([object Object])@resource://mozmill/modules/frame.js:669 ("c:\\users\\mozilla\\appdata\\local\\temp\\tmp6vp5zh.mozmill-tests\\tests\\functional\\testPreferences\\testPreferredLanguage.js")@resource://mozmill/modules/frame.js:506 ("c:\\users\\mozilla\\appdata\\local\\temp\\tmp6vp5zh.mozmill-tests\\tests\\functional\\testPreferences\\testPreferredLanguage.js")@resource://mozmill/modules/frame.js:681 ((function (filename, invokedFromIDE) {var runner = new Runner(new Collector, invokedFromIDE);runner.runTestFile(filename);runner.end();return true;}),[object Proxy])@resource://jsbridge/modules/server.js:179 ("5e20e624-615e-11e1-b7d8-005056f2858a",(function (filename, invokedFromIDE) {var runner = new Runner(new Collector, invokedFromIDE);runner.runTestFile(filename);runner.end();return true;}),[object Proxy])@resource://jsbridge/modules/server.js:183 
---------------------------------------------------------------------------------------
First failure date: 2012-02-23
---------------------------------------------------------------------------------------
Results dashboard:
http://mozmill-release.blargon7.com/#/functional/failure?branch=13.0&platform=All&from=2012-02-23&to=2012-02-28&test=%2FtestPreferences%2FtestPreferredLanguage.js&func=testPreferredLanguage.js%3A%3AtestSetLanguages

Comment 1

[Maniac Vlad Florin (:vladmaniac)]

Attachment: disable v1.0 [landed]

This test should use a local test page - its failing sometimes because of the google page. 

The failure can be manually reproduced but I cannot find a fix for it, test related. 

The test fails on all Firefox default and aurora branches since 2012/02/23, but it also can be reproduced on beta and release branch. 
I recommend disabling this test and decide whether we should have our own test page and this test refactored. 

Please debate your ideas on the bug

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Are you sure the link element still exists? Has its name been changed, or replaced by another element?

Comment 3

[Maniac Vlad Florin (:vladmaniac)]

(In reply to Henrik Skupin (:whimboo) [away 02/17 - 02/26] from comment #2)
> Are you sure the link element still exists? Has its name been changed, or
> replaced by another element?

As I said, the failure is intermittent so the link exists sometimes, and sometimes it cannot be found.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Then we should use waitForElement before asserting that the element exists.

Comment 5

[Maniac Vlad Florin (:vladmaniac)]

(In reply to Henrik Skupin (:whimboo) [away 02/17 - 02/26] from comment #4)
> Then we should use waitForElement before asserting that the element exists.

It does not help at all. There will be a timeout failure in waiting for that specific element. 

http://pastebin.mozilla.org/1494559

Comment 7

[u279076]

Comment on attachment 601227 [details] [diff] [review]
disable v1.0 [landed]

Patch is fine however I am going to hold off check-in so we can use it as a training check-in for Vlad's commit access tomorrow.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Can we please get this checked-in? We don't really want to wait until someone has commit privs. We have to disable broken tests immediately. Thanks.

Comment 9

[Maniac Vlad Florin (:vladmaniac)]

(In reply to Henrik Skupin (:whimboo) from comment #8)
> Can we please get this checked-in? We don't really want to wait until
> someone has commit privs. We have to disable broken tests immediately.
> Thanks.

Anthony is not around at this time, so I guess no one besides you can check it in at this moment

Comment 10

[u279076]

Comment on attachment 601227 [details] [diff] [review]
disable v1.0 [landed]

Agreed that we can't wait any longer for check-ins. Vlad, once you get your commit access squared away we will practice using whatever is in the current queue.

Patch has now been landed where it fails:
http://hg.mozilla.org/qa/mozmill-tests/rev/36e75bd8fbe9 (default)
http://hg.mozilla.org/qa/mozmill-tests/rev/0210192ecf02 (mozilla-aurora)

Comment 11

[u279076]

Test also fails for 10.0.3 ESR today:
http://mozmill-release.blargon7.com/#/functional/report/e438d6e3916b2b636037d77445184f64

Patch does not apply cleanly, please provide an ESR skip patch.

Comment 12

[Maniac Vlad Florin (:vladmaniac)]

Attachment: disable v1.0 (esr) [landed]

Adding skip test patch on esr branch - I don't know why it did not apply cleanly, locally it did for me

This new patch is for mozilla-esr10 branch exclusively, as requested

Comment 13

[u279076]

Comment on attachment 602831 [details] [diff] [review]
disable v1.0 (esr) [landed]

Landed:
http://hg.mozilla.org/qa/mozmill-tests/rev/236966250704 (mozilla-esr10)

Comment 14

[u279076]

Comment on attachment 601227 [details] [diff] [review]
disable v1.0 [landed]

Disabled the test on remaining branches as this failed today with Beta. It's not disabled everywhere.

Landed:
http://hg.mozilla.org/qa/mozmill-tests/rev/aca4c2481e87 (mozilla-beta)
http://hg.mozilla.org/qa/mozmill-tests/rev/4262ee1a05c3 (mozilla-release)

Comment 15

[u279076]

Comment on attachment 601227 [details] [diff] [review]
disable v1.0 [landed]

Also on mozilla-1.9.2...

Landed:
http://hg.mozilla.org/qa/mozmill-tests/rev/272a63fb3dd4 (mozilla-1.9.2)

Comment 16

[Maniac Vlad Florin (:vladmaniac)]

What can we do to fix this test judging by the fact that this is a Google issue? 
We'd want to fix it as soon as possible as part of our target to provide fixes for skipped tests.

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Some questions:
1. In such a case does Google load the Italian version of the page?
2. Are the other links we are referencing present?

Comment 18

[Maniac Vlad Florin (:vladmaniac)]

(In reply to Henrik Skupin (:whimboo) from comment #17)
> Some questions:
> 1. In such a case does Google load the Italian version of the page?
Yes, the Italian version is loaded, but sometimes the page looks different, probably a bad redirect  
> 2. Are the other links we are referencing present?
The other links are available afaik 

Are you suggesting to just test for another link ?

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Yes, just use another link which will indeed be there all the time.

Comment 20

[Maniac Vlad Florin (:vladmaniac)]

Attachment: fix v1.0 [landed:default]

Proposed initial fix, which passes for all branches. 
Since google changed, we changed the test to search for a link which is always present

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 604937 [details] [diff] [review]
fix v1.0 [landed:default]

Looks better.

Comment 22

[u279076]

Comment on attachment 604937 [details] [diff] [review]
fix v1.0 [landed:default]

Looks fine to me.

Comment 23

[u279076]

Comment on attachment 604937 [details] [diff] [review]
fix v1.0 [landed:default]

Landed:
http://hg.mozilla.org/qa/mozmill-tests/rev/af518eba0d7b (default)

Comment 24

[u279076]

If this passes on default, I'll go ahead and land on other branches.

Comment 25

[Dave Hunt [:davehunt] [he/him] ⌚BST]

It appears that this is still disabled on mozilla-aurora, mozilla-beta, mozilla-release, and mozilla-esr10

Comment 26

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Dave Hunt (:davehunt) from comment #25)
> It appears that this is still disabled on mozilla-aurora, mozilla-beta,
> mozilla-release, and mozilla-esr10

How can this be? The fix landed in March this year and should have been settled down to probably release meanwhile. Anthony, haven't you merged those skip changes whenever we had a version bump? If yes, there should be more than this single one probably we should fix. Dave, have you seen something else?

Comment 27

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Not sure what actually happened here, but the code changes have been merged except that the skipped lines have been removed. We just have to unskip those tests on older branches and have to find out why that happened.

Comment 28

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: unskip patch

Dave or Anthony, whoever comes first please review this patch.

Comment 29

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So the merge went wrong from default to mozilla-aurora and the skip lines have been re-added:

http://hg.mozilla.org/qa/mozmill-tests/rev/e070324ab322

Anthony, when you do merges and you are unclear please always consult the bug and appropriate dashboard results. Usually those unskip patches have to be forwarded to the next branch. The only exception are remote tests for the discovery pane so far.

Comment 30

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Transplanted original patch to ESR10:
http://hg.mozilla.org/qa/mozmill-tests/rev/299061be6cdf

Comment 31

[u279076]

Comment on attachment 639581 [details] [diff] [review]
unskip patch

Patch looks fine to me.

Note that with skipped tests, I err on the side of caution. If a test is skipped in Firefox 15 Nightly, I assume it needs to continue to be disabled when it merges, unless the comment specifically indicates the test is fine for Aurora. It's far better, in my opinion, to have to re-enable unintentionally skipped tests then to have to disable unintentionally enabled tests which are failing.

Perhaps we need a more descriptive commenting scheme for skipped blocks so that it is more evident.

Comment 32

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #31)
> Note that with skipped tests, I err on the side of caution. If a test is
> skipped in Firefox 15 Nightly, I assume it needs to continue to be disabled
> when it merges, unless the comment specifically indicates the test is fine
> for Aurora. It's far better, in my opinion, to have to re-enable

That's true but a different scenario. In this case the test wasn't marked as skip anymore. The final patch removed the skip lines. See your comment 23 with the changeset landed. 

Pushed to remaining branches:
http://hg.mozilla.org/qa/mozmill-tests/rev/0381a4a8ff08 (aurora)
http://hg.mozilla.org/qa/mozmill-tests/rev/9448e8b6ca1f (beta)
http://hg.mozilla.org/qa/mozmill-tests/rev/e4d1ddec0276 (release)

Vlad, please check all the Litmus tests and re-enable those for mozmill if necessary.

Comment 33

[Maniac Vlad Florin (:vladmaniac)]

There is no Litmus work to be done here as the test is no longer available. 

I did: 
1. Browsed 'manage testcases' section in Litmus and found no test under Options/Preferences to match this bug
2. Browsed our mozmill repo, took the litmus id from the map and searched - no results 

https://litmus.mozilla.org/single_result.cgi?id=8322 

In conclusion, dismissing the litmus flag

Comment 34

[Maniac Vlad Florin (:vladmaniac)]

It seems that we have the test for 3.6 branch 

https://litmus.mozilla.org/show_test.cgi?id=8322

Comment 35

[Maniac Vlad Florin (:vladmaniac)]

Following Litmus tests are not disabled 

aurora: https://litmus.mozilla.org/show_test.cgi?id=15786
release: https://litmus.mozilla.org/show_test.cgi?id=55779
beta: https://litmus.mozilla.org/show_test.cgi?id=64133

We need to re-enable for esr

Comment 36

[Maniac Vlad Florin (:vladmaniac)]

Enabled for mozmill group and edited the testcase to point that 
https://litmus.mozilla.org/show_test.cgi?id=40884

Comment 37

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Henrik Skupin (:whimboo) from comment #26)
> (In reply to Dave Hunt (:davehunt) from comment #25)
> > It appears that this is still disabled on mozilla-aurora, mozilla-beta,
> > mozilla-release, and mozilla-esr10
> 
> How can this be? The fix landed in March this year and should have been
> settled down to probably release meanwhile. Anthony, haven't you merged
> those skip changes whenever we had a version bump? If yes, there should be
> more than this single one probably we should fix. Dave, have you seen
> something else?

This is the only one I've seen.