945156 - Test failure "'masthead' element has not been found - got 'false'" in /testToolbar/testStopReloadButtons.js

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect, P1)

Description

[Cosmin Malutan, [:cosmin-malutan]]

Failed on Windows 7 x86, with Nightly fr.
I couldn't reproduce it locally, also is a remote test.

http://mozmill-daily.blargon7.com/#/remote/report/b99421c0f132c68dec1548288a69ed35

Comment 1

[Cosmin Malutan, [:cosmin-malutan]]

Aurora is also affected:
Failed on Windows 7 x64, with Aurora en-US
http://mozmill-daily.blargon7.com/#/remote/report/b99421c0f132c68dec1548288a72be23

Comment 2

[Andrei Eftimie]

We've just changed this test in bug 944334 a couple of days ago after the remote site has changed its source code.

I had to think a bit on which element should be used in the test.
We previously used an element named '#footer-right', which might imply that the element might be located near the end of the document.

However the comment above mentioned that the element should be near the beginning of the document:
> // Even an element at the top of a page shouldn't exist when we hit the stop
> // button extremely fast

So I changed the test to check the '#masthead' element, which should be universally available as it contains branding information and is located near the beginning of the document.

It could be that the comment referencing an _early_ element is outdated and we should really reference an element near the end of the document.

If we'll see this failure again, maybe we should amend the change, and check another element.
In that case we should also change the comment.

Comment 3

[Cosmin Malutan, [:cosmin-malutan]]

Ok so this failed for over 30 times, Mario will you look in to this?

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Yes, this comment is wrong and as of now we should use an element which gets loaded with a bit of delay. So anything at the end of the document should make sense. Otherwise lets change the URL to eg. SUMO, which in my experience is slower than other domains.

Comment 5

[mario garbi]

Attachment: skip.patch

A skip patch for the recent failures, I will have a fix patch as soon as possible.

Comment 6

[mario garbi]

Attachment: colophonElement.patch

The issue here was that the new element "masthead" was right at the top of the page and was getting loaded before we pushed the stop button due to the increase in performance of the browser.
Found the bottom element that is the equivalent of "foot" element (which we used before we switched to "masthead") and this should fix the problem.

Comment 7

[mario garbi]

I have tested the previous fix and we seem to load the page too fast so that even an element from the bottom of the page will get loaded:
http://mozmill-crowd.blargon7.com/#/remote/report/f8efdd634ceea4902aaa0c29573c9384

I will look into changing the URL to a page that loads slower as Henrik suggested.

Comment 8

[mario garbi]

Attachment: updateTestURL.patch

The previous colophon patch was not working as intended when I tested it on the CI machines since the page we used was getting loaded so fast that even elements at the bottom were loaded instantly. 
I have updated the remote page used to https://blog.mozilla.org/sumo as Henrik suggested and updated the element we check for to 'nav-below' which is in the footer of the page.

http://mozmill-crowd.blargon7.com/#/remote/report/f8efdd634ceea4902aaa0c295743a5bd

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Just to correct that, I have not suggested the SUMO blog but the SUMO website!

Comment 10

[mario garbi]

https://support.mozilla.org/en-US/home is loading faster than the initial page we had, the blog is the only page I found that takes a second to load. Our pages are too fast now with all the performance improvements.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We don't have any performance improvements in mozmill 1.5.24, which could have been caused reporting those failures. So that is not the reason here, but we no longer go through the proxy for any Mozilla owned domains, and support.mozilla.org (63.245.217.50) is in that IP range of 63.*.*.*.

Comment 12

[Andrei Eftimie]

To have control over the `remote` page shouldn't we better use one of our own pages?

I think we might be able to engineer a page that takes some time to fully load the DOM (either a really long DOM structure, maybe having some inline js that blocks rendering the DOM).

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+2]]

That's what we finally want to have, yes. But the first tries weren't that successful. I think there is already a bug about it.

Comment 14

[mario garbi]

I have created a local page that works perfectly with our test but I couldn't find the bug Henrik mentioned in comment 13. I will create a new bug and upload the test so that we can get this issue fixed.

Comment 15

[mario garbi]

Attachment: stopReloadLocalPage.patch

We are depending on the testcase data patch here. The test has also been moved to functional since it's no longer remote.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Comment on attachment 8342341 [details] [diff] [review]
stopReloadLocalPage.patch

Review of attachment 8342341 [details] [diff] [review]:
-----------------------------------------------------------------

For the first real review please do a 'hg mv' instead of moving the file over yourself. As you should have read in the refactoring bug from Andreea this will destroy the history of the file.

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Why don't we skip that test test when it mostly constantly fails for our daily tests? Not sure why this hasn't been uplifted to P1 yet.

Comment 18

[Andrei Eftimie]

Comment on attachment 8341078 [details] [diff] [review]
skip.patch

Review of attachment 8341078 [details] [diff] [review]:
-----------------------------------------------------------------

It fails about 9 times a day, so yeah, lets skip it.

Skipped:
http://hg.mozilla.org/qa/mozmill-tests/rev/a8a47da05503 (default)
http://hg.mozilla.org/qa/mozmill-tests/rev/dc63126bfcaf (mozilla-aurora)
http://hg.mozilla.org/qa/mozmill-tests/rev/0f6dcf6ea5c6 (mozilla-beta)
http://hg.mozilla.org/qa/mozmill-tests/rev/734055acd8f8 (mozilla-release)
http://hg.mozilla.org/qa/mozmill-tests/rev/4aff53dd1a03 (mozilla-esr24)

Comment 19

[Cosmin Malutan, [:cosmin-malutan]]

I'll take this since it was unassigned.

Comment 20

[Cosmin Malutan, [:cosmin-malutan]]

Attachment: patch v2.0

Here is the patch to make use of slow loading page from mozqa.com
I'll bring the reports in a second

Comment 21

[Andrei Eftimie]

Comment on attachment 8453771 [details] [diff] [review]
patch v2.0

Review of attachment 8453771 [details] [diff] [review]:
-----------------------------------------------------------------

Other than the mentioned item this looks fine to me.

Thanks

::: firefox/tests/remote/testToolbar/testStopReloadButtons.js
@@ +45,5 @@
>    controller.click(stopButton);
>  
>    // Even an element at the top of a page shouldn't exist when we hit the stop
>    // button extremely fast
> +  var masthead = new elementslib.ID(controller.tabs.activeTab, "content");

Please also rename the variable to `content` and since we're here update the retrieval method to `findElement`

Comment 22

[Cosmin Malutan, [:cosmin-malutan]]

Attachment: patch v2.1

I fixed those nits.

Comment 23

[Cosmin Malutan, [:cosmin-malutan]]

Comment on attachment 8453789 [details] [diff] [review]
patch v2.1

http://mozmill-crowd.blargon7.com/db/b4d880bccb8d9ea0732eac8cd85e7215
This fails on windows, I'll have to investigate why.

Comment 24

[Cosmin Malutan, [:cosmin-malutan]]

Attachment: patch v2.2

On windows it fails due to some settings:
http://stackoverflow.com/questions/7414056/ob-flush-flush-not-working-from-wamp2-on-windows-7#answer-7414219
I checked it on staging and it works there even on Windows so might be related to proxy settings.


PASSES
OSX http://mozmill-crowd.blargon7.com/#/remote/report/b4d880bccb8d9ea0732eac8cd85e63ab
LINUX http://mozmill-crowd.blargon7.com/#/remote/report/b4d880bccb8d9ea0732eac8cd852d1da

Comment 25

[Cosmin Malutan, [:cosmin-malutan]]

On windows I had to disable the AVG antivirus, it was blocking the page rendering before it was fully loaded. It works both on staging and master without any change.

WINDOWS http://mozmill-crowd.blargon7.com/db/b4d880bccb8d9ea0732eac8cd85fb54a

Comment 26

[Andrei Eftimie]

Comment on attachment 8453835 [details] [diff] [review]
patch v2.2

Review of attachment 8453835 [details] [diff] [review]:
-----------------------------------------------------------------

Landed:
http://hg.mozilla.org/qa/mozmill-tests/rev/43cd9f3ec99e (default)

Comment 27

[Andrei Eftimie]

We'll want to backport this.
Cosmin please check if the patch applies without conflicts and test that it works.

Thanks for the work here and with the slow loading page!

Comment 28

[Andrei Eftimie]

Cosmin is on PTO.

The patch applies cleanly on all branches.
I've run a full testrun on all of them. All is green.

https://hg.mozilla.org/qa/mozmill-tests/rev/ef8809f01fd7 (mozilla-aurora)
https://hg.mozilla.org/qa/mozmill-tests/rev/7ab5a09432c5 (mozilla-beta)
https://hg.mozilla.org/qa/mozmill-tests/rev/b30c39369cf6 (mozilla-release)
https://hg.mozilla.org/qa/mozmill-tests/rev/8b14c4cd8572 (mozilla-esr31)
https://hg.mozilla.org/qa/mozmill-tests/rev/72586d877be6 (mozilla-esr24)

Comment 29

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Hurray! Lets hope this was the last fix we needed for this test. Thanks

Comment 30

[Andrei Eftimie]

(In reply to Henrik Skupin (:whimboo) [away 07/19 - 08/01] from comment #29)
> Hurray! Lets hope this was the last fix we needed for this test. Thanks

With wptserve this test can become a functional one!

Comment 31

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Indeed. Even better. So lets revise all of our remote tests once that big change in Mozmill has been shipped.