922967 - Test failure "Identity is unknown - 'verifiedIdentity' should equal 'unknownIdentity'" in /testSecurity/testSecurityNotification.js

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect, P2)

Description

[mario garbi]

This started happening recently on all platforms with different localizations of Beta 25 as well as en-US.

I tried to reproduce it locally but was not able.

Failure reports:
http://mozmill-release.blargon7.com/#/remote/report/6ec6776efe900da3fd2b64a750770d89
http://mozmill-release.blargon7.com/#/remote/report/6ec6776efe900da3fd2b64a75098c5b2
http://mozmill-release.blargon7.com/#/remote/report/6ec6776efe900da3fd2b64a7509d388c
http://mozmill-release.blargon7.com/#/remote/report/6ec6776efe900da3fd2b64a7509e3ae2

Comment 1

[mario garbi]

I also tried to reproduce it on 'mm-win-8-32-2' running both single tests using mozmill -t and testrun_remote.py script. I was not able to reproduce it as can be seen in the reports:

http://mozmill-crowd.blargon7.com/#/remote/report/6ec6776efe900da3fd2b64a750bde5bb
http://mozmill-crowd.blargon7.com/#/remote/report/6ec6776efe900da3fd2b64a750be0151
http://mozmill-crowd.blargon7.com/#/remote/report/6ec6776efe900da3fd2b64a750be285e

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

What fails in detail given by the stack information from the tests?

Comment 3

[mario garbi]

The value for "identityBox.getNode().className" did not changed after loading the "unsecure HTTP site" at line 35 and so remained "verifiedIdentity". This caused a failure at line 38 that can be seen in the link below:
 
http://hg.mozilla.org/qa/mozmill-tests/file/541b0cb3957ff33b7b7991e0ccde4b6e80e77af7/tests/remote/testSecurity/testSecurityNotification.js#l38

I suspect this was caused by a 404 when loading the page at line 35 because the waitForPageLoad() was not triggered. The fact that this is not reproducible now indicates a possible network issue that got fixed in the meanwhile. I will look over the failing machine as soon as it reproduces again to better understand what happens.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

With the upgrade to Mozmill 2.0 we should not fetch those broken page loads. Lets see if it happens again and we get more information.

Comment 5

[mario garbi]

Happened again today on Linux Ubuntu 13.04 (x86) with Firefox 26.0a2:
http://mozmill-daily.blargon7.com/#/remote/report/6ec6776efe900da3fd2b64a750eb2aac

Comment 6

[Cosmin Malutan, [:cosmin-malutan]]

Happend mac, with Esr24 to
http://mozmill-daily.blargon7.com/#/remote/report/6d6f6a58b02eeffc06eafa8bea109c6f

Comment 7

[Andreea Matei [:AndreeaMatei]]

We should skip this if it fails that much on Beta especially.

Comment 8

[Cosmin Malutan, [:cosmin-malutan]]

Attachment: skip.patch

Here is the skip patch for beta.

I tried to reproduce this failure on affected machines yesterday and today again, without success.
http://mozmill-crowd.blargon7.com/#/remote/report/6d6f6a58b02eeffc06eafa8bea4a696d
http://mozmill-crowd.blargon7.com/#/remote/report/6d6f6a58b02eeffc06eafa8bea4a5bf6

Most likely it fails because waitforpagetoload passes before the page loads at line 
http://hg.mozilla.org/qa/mozmill-tests/file/b3f578c0111d/tests/remote/testSecurity/testSecurityNotification.js#l36
As Henrik said on comment 4 we shouldn't see those failures again once we have mozmill-2.0 on ci

Comment 9

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 814340 [details] [diff] [review]
skip.patch

Review of attachment 814340 [details] [diff] [review]:
-----------------------------------------------------------------

Disabled: 
http://hg.mozilla.org/qa/mozmill-tests/rev/b5a65140e3dc (beta)

We still want to disable on all affected branches as I saw a failure on esr24 as well today.

Also, please try to reproduce on the remote machines or watch a few runs there. Today reproduced quite often, it might be an issue with the page, but we cannot know for sure until we catch it.

Comment 10

[Cosmin Malutan, [:cosmin-malutan]]

Attachment: skip_esr17.patch

Here is the skip patch for esr17, the previous patch applies cleanly on  aurora, beta, release, esr24

Comment 11

[Cosmin Malutan, [:cosmin-malutan]]

Nightly is affected too 
http://mozmill-daily.blargon7.com/#/remote/report/6d6f6a58b02eeffc06eafa8bea329953
Esr 17 is affected too
http://mozmill-daily.blargon7.com/#/remote/report/6d6f6a58b02eeffc06eafa8bea27b11a
Esr 24 is affected too 
http://mozmill-daily.blargon7.com/#/remote/report/6d6f6a58b02eeffc06eafa8bea4f7515

Comment 12

[Andrei Eftimie]

Ok, lets skip this until we have mozmill 2.0 running in CI:

Disabled:
http://hg.mozilla.org/qa/mozmill-tests/rev/15c8233978b8 (default)
http://hg.mozilla.org/qa/mozmill-tests/rev/3a5f2c543803 (mozilla-aurora)
http://hg.mozilla.org/qa/mozmill-tests/rev/2da3a2a39254 (mozilla-release)
http://hg.mozilla.org/qa/mozmill-tests/rev/e8aa8bc990cb (mozilla-esr24)
http://hg.mozilla.org/qa/mozmill-tests/rev/9899ffda642d (mozilla-esr17)

Comment 13

[Andrei Eftimie]

Comment on attachment 814347 [details] [diff] [review]
skip_esr17.patch

Review of attachment 814347 [details] [diff] [review]:
-----------------------------------------------------------------

Landed as noted above.

I see that the skip in the manifest file for bug 708491 should have been taken out a good while ago.
We should when unskipping as a simple backout would again disable the test.

We should do these things separately to avoid confusion.

Comment 14

[Cosmin Malutan, [:cosmin-malutan]]

Attachment: patch_v1.0.patch

The event-listener that updates this element 
https://developer.mozilla.org/en-US/docs/XPCOM_Interface_Reference/nsIWebProgressListener#onSecurityChange()
it's  called after a security transition and after a "load" event. Also the UI get's updated asynchronous.
Waiting for the UI to change will be more safe than only checking it after a pageload.

  
Reports:
http://mozmill-crowd.blargon7.com/#/remote/report/bb6e314abd8f1e30e8af61ecd4b757f9
http://mozmill-crowd.blargon7.com/#/remote/report/bb6e314abd8f1e30e8af61ecd4b7c1ea
http://mozmill-crowd.blargon7.com/#/remote/report/bb6e314abd8f1e30e8af61ecd4b76032

Comment 15

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 820946 [details] [diff] [review]
patch_v1.0.patch

Review of attachment 820946 [details] [diff] [review]:
-----------------------------------------------------------------

Looks like a good fix.
Lets prove that with our daily testruns before enabling on all branches.

http://hg.mozilla.org/qa/mozmill-tests/rev/e99ceebcfe34 (default)

Comment 16

[Cosmin Malutan, [:cosmin-malutan]]

This is still failing, it failed today 4 times on Aurora, I could't reproduce it. I would say to disable this on Nightly again, until we get a better understanding of why is failing. 

http://mozmill-daily.blargon7.com/#/remote/report/d17690a112360a2b3155acaa276fa987
http://mozmill-daily.blargon7.com/#/remote/report/d17690a112360a2b3155acaa2772b3ad
http://mozmill-daily.blargon7.com/#/remote/report/d17690a112360a2b3155acaa27727f2b
http://mozmill-daily.blargon7.com/#/remote/report/d17690a112360a2b3155acaa2772df08

Comment 17

[Andrei Eftimie]

Lets not skip it yet if it only failed 4 times today.

If it hasn't failed since we landed the fix, and we can't reproduce it manually/locally, I'd say to leave it in and monitor failures closely.

Please update the bug if/when this fails again, ATM we don't have a TOP failures for remote tests, so it is harder to see them all in the dashboard.

Comment 18

[Andrei Eftimie]

I have managed to reproduce this locally on a Win8 VM while testing the environment for 1.5.24.1
http://mozmill-crowd.blargon7.com/#/remote/report/d17690a112360a2b3155acaa2784f6a2

Will try to see if its further reproducile

Comment 19

[Cosmin Malutan, [:cosmin-malutan]]

This failed a lot this week so I suggest we backout the last patch as it did't fixed the problem.

Comment 20

[Andrei Eftimie]

Ok lets get this skipped again, fails roughly 10 times per day.

Comment 21

[Andrei Eftimie]

Backed out:

http://hg.mozilla.org/qa/mozmill-tests/rev/17b2cd9bd872 (default)
http://hg.mozilla.org/qa/mozmill-tests/rev/8afbeba6b6f4 (mozilla-aurora)

Comment 22

[Cosmin Malutan, [:cosmin-malutan]]

An update here, I had ran this test for 600 times today and I couldn't reproduce it, we might want to re-enable it.

Comment 23

[Andreea Matei [:AndreeaMatei]]

Reenabled on default, if all goes well will do the rest tomorrow:
http://hg.mozilla.org/qa/mozmill-tests/rev/8f1bd29492eb

Comment 24

[Andreea Matei [:AndreeaMatei]]

Re-enabled:
http://hg.mozilla.org/qa/mozmill-tests/rev/69a2e8fafb45 (aurora)
http://hg.mozilla.org/qa/mozmill-tests/rev/1832161a6a5e (beta)
http://hg.mozilla.org/qa/mozmill-tests/rev/9841ecc9e6da (release)
http://hg.mozilla.org/qa/mozmill-tests/rev/7bf8efb8f799 (esr24)

Hopefully now it works fine with 2.0.