938029 - Test failure "Java(TM) Platform SE 7 U45 state is Enabled in about:plugins - got 'false'" in testAddons/testPluginDisableAffectsAboutPlugins.js

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect)

Description

[Andreea Matei [:AndreeaMatei]]

Started failing last night, reproduced constantly, with esr 24.1.1, all platforms.

It's reproducing for us as well if we put the affected plugin first in the list. 

Cosmin, please prepare a skip patch if a fix is not quickly found.

Comment 1

[Cosmin Malutan, [:cosmin-malutan]]

Attachment: patch_v1.0

This one failed because the add-on is blocklisted, so the string that is compared in the about:plugins was different. Also blocklisted add-ons cannot be set to always-activate, only to ask-to-activate state in order to be enabled.
I fixed that as it was reproducible on my windows box, where I had only one affected plugin as such :D
I will check if it affects other versions too.

Reports:
http://mozmill-crowd.blargon7.com/#/functional/report/d7f12eb72275b30155ba68b8bc0438f2
http://mozmill-crowd.blargon7.com/#/functional/report/d7f12eb72275b30155ba68b8bc031670
http://mozmill-crowd.blargon7.com/#/functional/report/d7f12eb72275b30155ba68b8bc00750b
http://mozmill-crowd.blargon7.com/#/functional/report/21341f02f219acb032f628de85ff70ca
http://mozmill-crowd.blargon7.com/#/functional/report/d7f12eb72275b30155ba68b8bc004903
http://mozmill-crowd.blargon7.com/#/functional/report/21341f02f219acb032f628de85ff1d56

Comment 2

[Andreea Matei [:AndreeaMatei]]

Dave, is it normal for us to keep this plugin in an old version? I think it should be updated on all machines.

Comment 3

[Cosmin Malutan, [:cosmin-malutan]]

Hi, as a small update, I check both that the plugin is enabled and not disabled so it won't regress in bug 924797, here it takes the first enabled plugin so on some machines it might be affected on others the affected plugin might be the second one and we will never know. Also if we update the plugins the test will fail again when another plugin is outdated.
I would go with this fix and perhaps file a new bug if is needed, for a test where to check that no plugin is broken.

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Andreea Matei [:AndreeaMatei] from comment #2)
> Dave, is it normal for us to keep this plugin in an old version? I think it
> should be updated on all machines.

Which old version are you referring to? As far as I can tell update 45 is the current version. We do try to keep all nodes up to date with the current versions of Java, but with as many as we have this is a time-consuming process.

(In reply to Cosmin Malutan from comment #3)
> Hi, as a small update, I check both that the plugin is enabled and not
> disabled so it won't regress in bug 924797, here it takes the first enabled
> plugin so on some machines it might be affected on others the affected
> plugin might be the second one and we will never know. Also if we update the
> plugins the test will fail again when another plugin is outdated.
> I would go with this fix and perhaps file a new bug if is needed, for a test
> where to check that no plugin is broken.

Why is this plugin blocked in ESR24 but not other builds?

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

That's something I wonder myself. Pulling in feature and branch QA owners.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

When I check the Java plugin blocking state at https://addons.mozilla.org/en-US/firefox/blocked/p457 it doesn't say that Java 7 update 45 is block listed. So is that version getting blocklisted accidentally?

Comment 7

[u279076]

Java 7u45 should not be blocked as per bug 914690 comment 80. However for some versions of Firefox this is not being honoured (25.0 and 24.1.1esr so far) starting yesterday. I've filed bug 938698 to investigate further.

Comment 8

[Matt Wobensmith [:mwobensmith][:matt:]]

When we ran the automated tests for 24.1.0esr, we didn't see errors.

http://mozmill-ondemand.blargon7.com/#/functional/reports?branch=24.1&platform=All&from=2013-10-22&to=2013-10-24

However, a manual inspection of both 24.1.0esr and 24.1.1esr show the latest Java 7u45 plugin as blocked. So it appears that we likely have a real FF issue here to contend with.

Once we get that sorted, we would then ask the question - why did this test suite pass before? What has changed? 

Also, in the current auto test run for 24.1.1, it's not 100% failing. That's another mystery to solve - I'd think it'd consistently all pass or all fail.

http://mozmill-ondemand.blargon7.com/#/functional/reports?branch=24.1&platform=All&from=2013-11-12&to=2013-11-13

Comment 9

[Matt Wobensmith [:mwobensmith][:matt:]]

OK, the dependent bug has been resolved.

From what we learned (see the other bug) the issue is related to a bad blocklist.xml. At some point, FF pings us and retrieves a new one, and that resolves the problem.

That means that the tests should be failing, consistently, but they haven't been. Someone should investigate why that is.

Comment 10

[Cosmin Malutan, [:cosmin-malutan]]

(In reply to Matt Wobensmith from comment #8)
> When we ran the automated tests for 24.1.0esr, we didn't see errors.
> However, a manual inspection of both 24.1.0esr and 24.1.1esr show the latest
> Java 7u45 plugin as blocked. So it appears that we likely have a real FF
> issue here to contend with.
> Also, in the current auto test run for 24.1.1, it's not 100% failing. That's
> another mystery to solve - I'd think it'd consistently all pass or all fail.

It didn't failed because here we don't test if addons are blocklisted, but we try to disable them and enable them back, so if there is another valid addon installed and if it happens to be the first one in list the test will pass.

Comment 11

[Matt Wobensmith [:mwobensmith][:matt:]]

How do you explain that some configs passed while others failed, for 24.1.1?

Comment 12

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 831501 [details] [diff] [review]
patch_v1.0

Review of attachment 831501 [details] [diff] [review]:
-----------------------------------------------------------------

We don't want this fix giving that we want to catch this kind of issue in the future, not silently pass.
We might want to create a new test that handles the plugins in another way, check if they're blocklisted or not, take one randomly or check all and so on.

Comment 13

[Andreea Matei [:AndreeaMatei]]

Fixed in bug 938698.

Comment 14

[Cosmin Malutan, [:cosmin-malutan]]

Attachment: Test_passing_with_addon_blocklisted.png

(In reply to Matt Wobensmith from comment #11)
> How do you explain that some configs passed while others failed, for 24.1.1?
Hi, here is an example of an situation where the test passes with an plugin blocklisted, because the first enabled plugin is not. 
As I said, in this particular test we test only the first enabled plugin which might be enabled.
On other ci nodes we might have more plugins installed, that's why some might pass. 
http://hg.mozilla.org/qa/mozmill-tests/file/a09dd08a9e9d/tests/functional/testAddons/testPluginDisableAffectsAboutPlugins.js#l34

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I think we should enhance this test so we do not pick a blocklisted plugin. This would always cause this failure and makes the test unstable.

Comment 16

[Andreea Matei [:AndreeaMatei]]

I thought we want to catch when a plugin is blocklisted, usually we don't have those installed and might be a firefox bug like this time.

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+1]]

For blocklisting checks we should cover all topics in a separate test. This has really nothing to do with this bug. We simply cannot use such a blocklisted add-on. Probably we might have to also update other tests for it.

Comment 18

[Cosmin Malutan, [:cosmin-malutan]]

Attachment: patch_v2.0

Enhanced the test to don't pick any blocklisted plugins.
Reports:
http://mozmill-crowd.blargon7.com/#/functional/report/2f1ca72938985fc4b989b7efcc552ade
http://mozmill-crowd.blargon7.com/#/functional/report/2f1ca72938985fc4b989b7efcc5573bc
http://mozmill-crowd.blargon7.com/#/functional/report/2f1ca72938985fc4b989b7efcc554d2d
http://mozmill-crowd.blargon7.com/#/functional/report/2f1ca72938985fc4b989b7efcc5587df
http://mozmill-crowd.blargon7.com/#/functional/report/2f1ca72938985fc4b989b7efcc5596c3
http://mozmill-crowd.blargon7.com/#/functional/report/2f1ca72938985fc4b989b7efcc55a6d1

Comment 19

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 8333926 [details] [diff] [review]
patch_v2.0

Review of attachment 8333926 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good. 
http://hg.mozilla.org/qa/mozmill-tests/rev/f4856cd7319a (default)

We also need backporting for this.

Comment 20

[Cosmin Malutan, [:cosmin-malutan]]

Attachment: patch_v2.0 [Esr17]

Here is the patch for Esr17, for all the other branches, the previous patch applies.

Comment 21

[Andrei Eftimie]

Comment on attachment 8334500 [details] [diff] [review]
patch_v2.0 [Esr17]

Review of attachment 8334500 [details] [diff] [review]:
-----------------------------------------------------------------

Transplanted:
http://hg.mozilla.org/qa/mozmill-tests/rev/196af585cc44 (mozilla-aurora)
http://hg.mozilla.org/qa/mozmill-tests/rev/44873366206c (mozilla-beta)
http://hg.mozilla.org/qa/mozmill-tests/rev/2df67c6f630c (mozilla-release)
http://hg.mozilla.org/qa/mozmill-tests/rev/4de7a855a8e7 (mozilla-esr24)

Looks good, landed:
http://hg.mozilla.org/qa/mozmill-tests/rev/4a83f7749553 (mozilla-esr17)