710347 - Failure in testPasswordManager :: testPasswordNotSaved

Status: VERIFIED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect)

Description

[u279076]

This just started happening recently on Nightly and Aurora though intermittently. It appears to be Mac only.

Test: testPasswordManager :: testPasswordNotSaved.js :: testPasswordNotSaved()
Fail: Pane has been changed - expected 'panePrivacy'

Comment 1

[Maniac Vlad Florin (:vladmaniac)]

Assigning to Alex

Comment 2

[Alex Lakatos[:AlexLakatos]]

Attachment: disable test: default/aurora [checked-in]

I could not figure out the issue. I'm still investigating, but skipping the test for now, as to our policy.

Comment 3

[u279076]

Comment on attachment 581694 [details] [diff] [review]
disable test: default/aurora [checked-in]

Landed:
http://hg.mozilla.org/qa/mozmill-tests/rev/43597e45052b (default)
http://hg.mozilla.org/qa/mozmill-tests/rev/dfabcd7f5ecf (mozilla-aurora)

Comment 4

[Alex Lakatos[:AlexLakatos]]

I ran this today on Lion, 100 times for each branch. And out of that it only failed once on Nightly. Can we unskip the test?

Comment 5

[u279076]

(In reply to Alex Lakatos[:AlexLakatos] from comment #4)
> I ran this today on Lion, 100 times for each branch. And out of that it only
> failed once on Nightly. Can we unskip the test?

Were you able to reproduce the failure locally before this test was skipped?

Comment 6

[Alex Lakatos[:AlexLakatos]]

(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #5)
> Were you able to reproduce the failure locally before this test was skipped?
It was failing more than once during 200 runs. But not a lot more. I could not reproduce the failure when running the test once, instead of running it in a for loop.

Comment 7

[u279076]

Then I would say the situation has not improved, and since there is no change in Firefox code we can't just re-enable the test. Until we can figure out why it intermittently fails we cannot re-enable this test.

Comment 8

[Maniac Vlad Florin (:vladmaniac)]

Please check out bug 671382. I'm working to fix the paneId setter right now.
Taking all the failures into consideration, I'll double the efforts with Alex's help in finding a robust fix.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Alex, do you have an update for us?

Comment 10

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Any updates?

Comment 11

[Alex Lakatos[:AlexLakatos]]

It's passing now, but I'm still running it over and over to determine if we are safe to re-enable it.

Comment 12

[Alex Lakatos[:AlexLakatos]]

Ran the test 100 times on it's own on Win XP, Ubuntu and Mac. It did not fail once.

Ran a full functional testrun with the test enabled, three times, and it did not fail. Here are the reports:
XP:
http://mozmill-crowd.blargon7.com/#/functional/report/c67fb1fc2ea8384105b843e32c18cf47
http://mozmill-crowd.blargon7.com/#/functional/report/c67fb1fc2ea8384105b843e32c194353
http://mozmill-crowd.blargon7.com/#/functional/report/c67fb1fc2ea8384105b843e32c19d31e

Ubuntu:
http://mozmill-crowd.blargon7.com/#/functional/report/c67fb1fc2ea8384105b843e32c1b4cb3
http://mozmill-crowd.blargon7.com/#/functional/report/c67fb1fc2ea8384105b843e32c1c2e46
http://mozmill-crowd.blargon7.com/#/functional/report/c67fb1fc2ea8384105b843e32c1ce95a

Mac:
http://mozmill-crowd.blargon7.com/#/functional/report/44fc451c2e6b66b62172f2e13e05b865
http://mozmill-crowd.blargon7.com/#/functional/report/44fc451c2e6b66b62172f2e13e05a494
http://mozmill-crowd.blargon7.com/#/functional/report/44fc451c2e6b66b62172f2e13e057979

Comment 13

[Alex Lakatos[:AlexLakatos]]

Based on Comment 12, I think we're safe to re-enable it for default and go from there. Henrik, what do you think?

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ok, lets try that. Landed on default as:
http://hg.mozilla.org/qa/mozmill-tests/rev/1397c0ccc72a

Alex, please check which branches are affected. I assume all of the supported ones. If it sticks green we can transplant to older branches by mid of this week.

Comment 15

[Alex Lakatos[:AlexLakatos]]

(In reply to Henrik Skupin (:whimboo) from comment #14)
> Ok, lets try that. Landed on default as:
> http://hg.mozilla.org/qa/mozmill-tests/rev/1397c0ccc72a
> 
> Alex, please check which branches are affected. I assume all of the
> supported ones. If it sticks green we can transplant to older branches by
> mid of this week.
Latest run in CI passes.
http://mozmill-ci.blargon7.com/#/functional/report/9acea0c923846813ba8ce41994000af5

This needs to land on aurora, beta, release and esr10.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Pushed:
http://hg.mozilla.org/qa/mozmill-tests/rev/c55ad3bc7803 (aurora)
http://hg.mozilla.org/qa/mozmill-tests/rev/e9af10605968 (beta)
http://hg.mozilla.org/qa/mozmill-tests/rev/b570a5a1027c (release)
http://hg.mozilla.org/qa/mozmill-tests/rev/799e387e806a (esr10)

Please update all the Litmus tests which are disabled.

Comment 17

[Alex Lakatos[:AlexLakatos]]

https://litmus.mozilla.org/show_test.cgi?id=55632
https://litmus.mozilla.org/show_test.cgi?id=63986
https://litmus.mozilla.org/show_test.cgi?id=15639

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+1]]

An update for the litmus testcase of the esr10 branch is still missing.

Comment 19

[Alex Lakatos[:AlexLakatos]]

I'm not seeing an esr branch in litmus. There are several tests for the aurora branch, but I can't see an esr branch.

Comment 20

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This is Firefox 10.0!

Comment 21

[Alex Lakatos[:AlexLakatos]]

Edited this a while back but seems I forgot to flip the flag.
https://litmus.mozilla.org/show_test.cgi?id=40737