1003083 - [zh-cn] Test failure "controller.waitForPageLoad(URI=https://addons.mozilla.org/zh-CN/firefox/licenses/5.txt, readyState=complete) " in testSecurityNotification.js

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect, P2)

Description

[Andreea Matei [:AndreeaMatei]]

This also started on Saturday for Aurora zh-cn, reproduces on all platforms.

Affected line:
http://hg.mozilla.org/qa/mozmill-tests/file/mozilla-aurora/firefox/tests/remote/testSecurity/testSecurityNotification.js#l39

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So this is reproducible? If yes, please get on it. We had so many waitForPageLoad() failures we were never able to understand. Now we would have a fantastic chance!

Comment 2

[Andreea Matei [:AndreeaMatei]]

It's not reproducing locally for me, but I'll try to investigate it on a remote machine.
Might be a network issue as we're using remote pages.

Comment 3

[Andreea Matei [:AndreeaMatei]]

Yep, it's something with the network as the page isn't loading.

Comment 4

[Andrei Eftimie]

Another couple of failures (OSX):
http://mozmill-daily.blargon7.com/#/remote/report/0bd22723d1fdbb446269f765b9249e4d
http://mozmill-daily.blargon7.com/#/remote/report/0bd22723d1fdbb446269f765b90cef74

Comment 5

[Andrei Eftimie]

Failed 6 times today.

Comment 6

[Andrei Eftimie]

Has a 12-25% reproducibility rate on daily ci testruns with zh-CN builds. (based on reports from the last days)

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This is remote. Maybe we should use one of the boxes to run the tests on. I did more than 200 runs locally but wasn't able to see the failure.

Comment 8

[Cosmin Malutan, [:cosmin-malutan]]

An update here, when it fails with 
>controller.waitForPageLoad(URI=https://addons.mozilla.org/zh-CN/firefox/licenses/5.txt, readyState=complete)
it fails because it can't find the page. When it fails with:
>controller.waitForPageLoad(URI=http://www.firefox.com.cn/, readyState=loading)
it fails because the page takes too long to load, that's due to latency probably coused by the fact that data center that delivers this content is in China. For the second case increasing the timeout for waitForPageLoad should be enough.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Cosmin Malutan from comment #8)
> An update here, when it fails with 
> >controller.waitForPageLoad(URI=https://addons.mozilla.org/zh-CN/firefox/licenses/5.txt, readyState=complete)
> it fails because it can't find the page

Interesting. Can you please check which events are triggered here? Therefore please enable the dump statements in window.js especially for the error case. We should have a 404 here, so not sure why we get a complete state.

> >controller.waitForPageLoad(URI=http://www.firefox.com.cn/, readyState=loading)
> it fails because the page takes too long to load, that's due to latency
> probably coused by the fact that data center that delivers this content is
> in China. For the second case increasing the timeout for waitForPageLoad
> should be enough.

No, we shouldn't increase the timeout but fix the real underlying issue! No-one would expect to wait that long to see the loading of the page finished. Please also enable debugging in Firefox, and check the debug output. Maybe you even want to create a HTTP log, so we can see which servers are getting hit.

Comment 10

[Daniel Gherasim]

Failed 3 times today on Windows 8 64 & 32, & Windows 8.1 64 with Aurora zh-CN
http://mozmill-daily.blargon7.com/#/remote/report/7a69d3e8424ecf8642bb6566e46b4261
http://mozmill-daily.blargon7.com/#/remote/report/7a69d3e8424ecf8642bb6566e46bcaad
http://mozmill-daily.blargon7.com/#/remote/report/7a69d3e8424ecf8642bb6566e46b40fa

Comment 11

[Daniel Gherasim]

Failed again today on Firefox 31.0a2 Aurora zh-CN on mm-ub-1204-64-2.
http://mozmill-daily.blargon7.com/#/remote/report/7a69d3e8424ecf8642bb6566e475f5c5

Comment 12

[Daniel Gherasim]

Still lot of failures with the same locale.
http://mozmill-daily.blargon7.com/#/remote/report/7a69d3e8424ecf8642bb6566e48823be
http://mozmill-daily.blargon7.com/#/remote/report/7a69d3e8424ecf8642bb6566e48827ed
http://mozmill-daily.blargon7.com/#/remote/report/7a69d3e8424ecf8642bb6566e4886b1c
http://mozmill-daily.blargon7.com/#/remote/report/7a69d3e8424ecf8642bb6566e48986a3

I've accesssed http://www.firefox.com.cn/ & it took about 40 sec to load.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Please file a bug so we can make that faster. It's a bit way too long to load that page.

Comment 14

[Daniel Gherasim]

Lots of failures accessing http://www.firefox.com.cn/ this week with that locale. 
That happens as we access unsecure page: http://mozilla.org/ which get's redirected to http://www.firefox.com.cn/.

I suggest to change the test to visit http://mozqa.com/ as unsecure page so we don't get retirects depending on locale.

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to daniel.gherasim from comment #14)
> That happens as we access unsecure page: http://mozilla.org/ which get's
> redirected to http://www.firefox.com.cn/.

Does that also happen when you run the zh-cn locale and manually open that page? If that is the case I would agree, also given that we always wanted to utilize mozqa.com.

Comment 16

[Daniel Gherasim]

Attachment: replace_mozilla_org_with_mozqa_com.patch

Did that manually & it took about 25 seconds to load.
Running the test locally I can reproduce this 20/30 times. 
Also filed bug 1014549 to reduce loading time on that page.

Comment 17

[Andrei Eftimie]

Comment on attachment 8427016 [details] [diff] [review]
replace_mozilla_org_with_mozqa_com.patch

Review of attachment 8427016 [details] [diff] [review]:
-----------------------------------------------------------------

We should try changing all urls to mozqa.com

As said inline, we still lack a verified cert on mozqa.com
But It would still be better to change at least 2 out of 3 urls to our own domain.

::: firefox/tests/remote/testSecurity/testSecurityNotification.js
@@ +8,5 @@
>  var { assert, expect } = require("../../../../lib/assertions");
>  
>  const TEST_DATA = [
>    // Invalid cert page
>    "https://summitbook.mozilla.org",

We can use https://ssl-md5.mozqa.com/

@@ +10,5 @@
>  const TEST_DATA = [
>    // Invalid cert page
>    "https://summitbook.mozilla.org",
>    // Secure page
>    "https://addons.mozilla.org/licenses/5.txt",

Unfortunately I don't think we have an externally verified cert on mozqa.com

Comment 18

[Andrei Eftimie]

Actually these are the domains that will be used by mozqa.com once we make the switch (see bug 991103):
> www.mozqa.com
> ssl-ov.mozqa.com
> ssl-selfsigned.mozqa.com
> ssl-dv.mozqa.com
> summitbook.mozilla.org
> ssl-md5.mozqa.com
> ssl-ss-unknownissuer.mozqa.com

Right now summitbook.mozilla.org responds from a different IP, so I'm not sure how this domain will work.

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Andrei Eftimie from comment #18)
> Actually these are the domains that will be used by mozqa.com once we make
> the switch (see bug 991103):

That's not true. All of them are currently active and get used.

Comment 20

[Daniel Gherasim]

Attachment: replace_mozilla_org_with_mozqa_com_v2.patch

Changed the links. Also the https://addons.mozilla.org/licenses/5.txt link to avoid the 'redirect' based on localized browser.

We can keep an eye on this & maybe whis will fix the controller.waitForPageLoad with readyState=complete problem.

Comment 21

[Andrei Eftimie]

Comment on attachment 8429843 [details] [diff] [review]
replace_mozilla_org_with_mozqa_com_v2.patch

Review of attachment 8429843 [details] [diff] [review]:
-----------------------------------------------------------------

I'm still confused as to why I see different IP's for different mozqa.com subdomains if they are resolved from the same host.

::: firefox/tests/remote/testSecurity/testSecurityNotification.js
@@ +8,5 @@
>  var { assert, expect } = require("../../../../lib/assertions");
>  
>  const TEST_DATA = [
>    // Invalid cert page
> +  "https://ssl-md5.mozqa.com",

As Henrik said since https://summitbook.mozilla.org is under our control, we should leave this as it was, we won't need to change the error code then.

@@ +13,2 @@
>    // Secure page
> +  "https://addons.mozilla.org/en-US/licenses/5.txt",

This looks like an easy fix for the chinese timeouts we're seeing. I'd rather go with this as a small change if none of the domains we control fulfill this tests requirements. We're having lots of failures because of this URL.

Comment 22

[Daniel Gherasim]

Attachment: replace_mozilla_org_with_mozqa_com_v2.1.patch

Comment 23

[Andrei Eftimie]

Comment on attachment 8431494 [details] [diff] [review]
replace_mozilla_org_with_mozqa_com_v2.1.patch

Review of attachment 8431494 [details] [diff] [review]:
-----------------------------------------------------------------

This should stop all these failures on the CN domain. Local test completion time comes to 5s down from 15s with this change.
I'm landing this across branches. I'll make sure to check on Monday that we didn't have any other failures related to this.

Landed:
http://hg.mozilla.org/qa/mozmill-tests/rev/21ad90017cde (default)
http://hg.mozilla.org/qa/mozmill-tests/rev/e2993b5918f6 (mozilla-aurora)
http://hg.mozilla.org/qa/mozmill-tests/rev/bacf5f6e75e5 (mozilla-beta)

Comment 24

[Andrei Eftimie]

In the future I would still like to change https://addons.mozilla.org/licenses/5.txt to a domain under our control under mozqa.com

Comment 25

[Henrik Skupin [:whimboo][⌚️UTC+1]]

What's up with ESR24? Isn't it affected, Daniel?

(In reply to Andrei Eftimie from comment #24)
> In the future I would still like to change
> https://addons.mozilla.org/licenses/5.txt to a domain under our control
> under mozqa.com

How will you do this when mozqa.com has no EV certificate. But if the test specifically doesn't need one, I'm happy to change it to DV or OV.

Comment 26

[Daniel Gherasim]

(In reply to Henrik Skupin (:whimboo) from comment #25)
> What's up with ESR24? Isn't it affected, Daniel?
> 

ESR24 doesn't have localized builds but it's also affected by the waitForPageLoad on https://addons.mozilla.org/licenses/5.txt . 

Patch applies cleanly on ESR24 too so it can be landed.

Comment 27

[Andrei Eftimie]

Transplanted:
https://hg.mozilla.org/qa/mozmill-tests/rev/3d1b85dc935f (mozilla-esr24)

Comment 28

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to daniel.gherasim from comment #26)
> ESR24 doesn't have localized builds

Why you think so? While we don't have nightly builds for localizations, we still have releases for all supported locales.