991736 - Disable flash tests for Windows due to crashes

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect, P1)

Description

[Andreea Matei [:AndreeaMatei]]

We had a lot of crashes lately on vista machines, with esr24. I investigated this and managed to submit the crash report which showed it's due to bug 980938.

Unfortunately, we have 4 tests related to flash. At first I though it's just testFlash_SWFVideoURL (bug 991706 comment 1), but retested with the others enabled and we still fail. The problem is Flash.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)]

Can you reproduce this locally outside of SCL3? If yes, that would be very very helpful!!!

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)]

Also do you have a crash report here? Do you know which test or sequence of tests have to be executed here to get this crash? Any minimized testcase?

Comment 3

[Andreea Matei [:AndreeaMatei]]

This is the crash I got submitted:
https://crash-stats.mozilla.com/report/index/8707e012-29e6-40e5-bec4-5722d2140403

I will have to try locally tomorrow, on mm-win-vista-32-2 I narrowed the tests down and I could reproduce it with just having only testFlash_SWFVideoURL in the manifest, testrun with 1 iteration and 2 entities. Can be intermittent sometimes, but rarely, it mostly crashes.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)]

Here the actual file:
http://hg.mozilla.org/qa/mozmill-tests/file/mozilla-esr24/firefox/tests/endurance/testFlash_SWFVideoURL/test1.js

What happens if you add a delay after the waitForPageLoad call? I wonder if that could be a race condition for the Flash plugin when opening pages with contained Flash content too frequently.

Comment 5

[Andreea Matei [:AndreeaMatei]]

We've decided to skip all Flash tests since we've seen also crashes with remote testrun (we have testPrivateBrowsing/testFlashCookie.js) on Beta and Aurora. 
I will continue to investigate, work on a minimized testcase and try to reproduce it locally on Vista.

Comment 6

[Andreea Matei [:AndreeaMatei]]

Attachment: skip patch

Patch for default. I'll check backporting.

Comment 7

[Andreea Matei [:AndreeaMatei]]

Applies cleanly down to esr24, which has a fuzz. So we're good.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)]

As we have seen the problem might be with the new version of Flash. Andreea is investigating and testnig the former version. If that fixes the problem we will go back. So lets not land the patches for now.

Comment 9

[Andreea Matei [:AndreeaMatei]]

Attachment: testcaseFlash.js

I could reproduce it with this testcase on vista-32-2, against ESR24 and Nightly.
We decided to skip the tests over the weekend.

It also reproduced with an older Flash version (12.0.0.44 which was upgraded on Monday).

Tried the testcase on Windows 8.1 but it doesn't fail there.

Comment 10

[Andrei Eftimie]

Comment on attachment 8401792 [details] [diff] [review]
skip patch

Review of attachment 8401792 [details] [diff] [review]:
-----------------------------------------------------------------

As discussed on IRC, lets disable these tests:
http://hg.mozilla.org/qa/mozmill-tests/rev/260ad8949f03 (default)
http://hg.mozilla.org/qa/mozmill-tests/rev/7bb88ca2938e (mozilla-aurora)
http://hg.mozilla.org/qa/mozmill-tests/rev/4496fd8c997e (mozilla-beta)
http://hg.mozilla.org/qa/mozmill-tests/rev/725ccd2faca4 (mozilla-release)
http://hg.mozilla.org/qa/mozmill-tests/rev/0ea4bb6efc4c (mozilla-esr24)

Comment 11

[Andreea Matei [:AndreeaMatei]]

I've tested today on mm-win-vista-32-4 machine, running the testcase as well as the 4 endurance tests on a patched repository to unskip them. I can reproduce with latest env.

But I figured since this issue started on Friday 28th March (because only then we had esr24 since Sunday), what else could have bben changed on the machines?
We had 2.0.6 released on production on Monday 24th March. So I tested with 2.0.5 env and I can't reproduce the crash anymore. 

I'll go next through the changes we had in 2.0.6 to see what could be affecting this.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)]

Well, that could be the reason. But it's not that the crash wasn't present before.  We simply haven't detected it with Mozmill before. This is a new feature in 2.0.6. So it looks like we have to do the same as for the functional tests and skip the flash based endurance tests for FIrefox 24 on Vista?

Andreea, can you please test a debug version of latest Flash if you can reproduce this crash? We weren't able to do that with the clear cookie crash. Such kind of information would be very helpful.

Comment 13

[Andreea Matei [:AndreeaMatei]]

Hm, curiously, it didn't reproduce with the debug version:
http://www.adobe.com/support/flashplayer/downloads.html 
- Download the Windows Flash Player 12 Plugin content debugger (for Netscape-compatible browsers) (EXE, 17.34MB)

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)]

So exactly the same behavior as for the other crash. Maybe we can install the debug version on the affected nodes for now?

Comment 15

[Andreea Matei [:AndreeaMatei]]

Yeah, I think it's a good idea in order to be able to enable the tests again. And also keep looking for the reason why we don't crash on the local machines.

Comment 16

[Andrei Eftimie]

Flash has been upgraded to 13.x on all Production windows nodes and Vista+ have all received the debug version in bug 993433.

I've ran these tests on a Vista machine.
I haven't experienced any crashes. I think we can unskip them.

http://mozmill-crowd.blargon7.com/#/functional/report/7ab760e27012969ae02e3b9e41dae08c
http://mozmill-crowd.blargon7.com/#/endurance/report/7ab760e27012969ae02e3b9e41dbaa94
http://mozmill-crowd.blargon7.com/#/remote/report/7ab760e27012969ae02e3b9e41dcfe61

Comment 17

[Andrei Eftimie]

Backed out:
http://hg.mozilla.org/qa/mozmill-tests/rev/e95df5638820 (default)
http://hg.mozilla.org/qa/mozmill-tests/rev/3b4f892a6346 (mozilla-aurora)
http://hg.mozilla.org/qa/mozmill-tests/rev/2464cba00c5c (mozilla-beta)
http://hg.mozilla.org/qa/mozmill-tests/rev/eaed17515120 (mozilla-release)
http://hg.mozilla.org/qa/mozmill-tests/rev/537368c031cb (mozilla-esr24)

I changed the flags to fixed. Not sure if this is the case since we use the flash debug version to workaround this crash.

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)]

Yeah, that's fine. We only have to remember to install the debug version on Windows until the crash has been fixed.