Test failure "Java(TM) Platform SE 7 U45 state is Enabled in about:plugins - got 'false'" in testAddons/testPluginDisableAffectsAboutPlugins.js

RESOLVED FIXED

Status

Mozilla QA
Mozmill Tests
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: AndreeaMatei, Assigned: cosmin)

Tracking

unspecified

Firefox Tracking Flags

(firefox25 fixed, firefox26 fixed, firefox27 fixed, firefox28 fixed, firefox-esr17 fixed, firefox-esr24 fixed)

Details

(Whiteboard: [mozmill-test-failure], URL)

Attachments

(3 attachments, 1 obsolete attachment)

(Reporter)

Description

4 years ago
Started failing last night, reproduced constantly, with esr 24.1.1, all platforms.

It's reproducing for us as well if we put the affected plugin first in the list. 

Cosmin, please prepare a skip patch if a fix is not quickly found.
(Assignee)

Comment 1

4 years ago
Created attachment 831501 [details] [diff] [review]
patch_v1.0

This one failed because the add-on is blocklisted, so the string that is compared in the about:plugins was different. Also blocklisted add-ons cannot be set to always-activate, only to ask-to-activate state in order to be enabled.
I fixed that as it was reproducible on my windows box, where I had only one affected plugin as such :D
I will check if it affects other versions too.

Reports:
http://mozmill-crowd.blargon7.com/#/functional/report/d7f12eb72275b30155ba68b8bc0438f2
http://mozmill-crowd.blargon7.com/#/functional/report/d7f12eb72275b30155ba68b8bc031670
http://mozmill-crowd.blargon7.com/#/functional/report/d7f12eb72275b30155ba68b8bc00750b
http://mozmill-crowd.blargon7.com/#/functional/report/21341f02f219acb032f628de85ff70ca
http://mozmill-crowd.blargon7.com/#/functional/report/d7f12eb72275b30155ba68b8bc004903
http://mozmill-crowd.blargon7.com/#/functional/report/21341f02f219acb032f628de85ff1d56
Assignee: nobody → cosmin.malutan
Status: NEW → ASSIGNED
Attachment #831501 - Flags: review?(andreea.matei)
(Reporter)

Comment 2

4 years ago
Dave, is it normal for us to keep this plugin in an old version? I think it should be updated on all machines.
(Assignee)

Comment 3

4 years ago
Hi, as a small update, I check both that the plugin is enabled and not disabled so it won't regress in bug 924797, here it takes the first enabled plugin so on some machines it might be affected on others the affected plugin might be the second one and we will never know. Also if we update the plugins the test will fail again when another plugin is outdated.
I would go with this fix and perhaps file a new bug if is needed, for a test where to check that no plugin is broken.
Flags: needinfo?(dave.hunt)
(In reply to Andreea Matei [:AndreeaMatei] from comment #2)
> Dave, is it normal for us to keep this plugin in an old version? I think it
> should be updated on all machines.

Which old version are you referring to? As far as I can tell update 45 is the current version. We do try to keep all nodes up to date with the current versions of Java, but with as many as we have this is a time-consuming process.

(In reply to Cosmin Malutan from comment #3)
> Hi, as a small update, I check both that the plugin is enabled and not
> disabled so it won't regress in bug 924797, here it takes the first enabled
> plugin so on some machines it might be affected on others the affected
> plugin might be the second one and we will never know. Also if we update the
> plugins the test will fail again when another plugin is outdated.
> I would go with this fix and perhaps file a new bug if is needed, for a test
> where to check that no plugin is broken.

Why is this plugin blocked in ESR24 but not other builds?
Flags: needinfo?(dave.hunt)
That's something I wonder myself. Pulling in feature and branch QA owners.
When I check the Java plugin blocking state at https://addons.mozilla.org/en-US/firefox/blocked/p457 it doesn't say that Java 7 update 45 is block listed. So is that version getting blocklisted accidentally?
Java 7u45 should not be blocked as per bug 914690 comment 80. However for some versions of Firefox this is not being honoured (25.0 and 24.1.1esr so far) starting yesterday. I've filed bug 938698 to investigate further.
Depends on: 938698
When we ran the automated tests for 24.1.0esr, we didn't see errors.

http://mozmill-ondemand.blargon7.com/#/functional/reports?branch=24.1&platform=All&from=2013-10-22&to=2013-10-24

However, a manual inspection of both 24.1.0esr and 24.1.1esr show the latest Java 7u45 plugin as blocked. So it appears that we likely have a real FF issue here to contend with.

Once we get that sorted, we would then ask the question - why did this test suite pass before? What has changed? 

Also, in the current auto test run for 24.1.1, it's not 100% failing. That's another mystery to solve - I'd think it'd consistently all pass or all fail.

http://mozmill-ondemand.blargon7.com/#/functional/reports?branch=24.1&platform=All&from=2013-11-12&to=2013-11-13
No longer depends on: 938698
Depends on: 938698
OK, the dependent bug has been resolved.

From what we learned (see the other bug) the issue is related to a bad blocklist.xml. At some point, FF pings us and retrieves a new one, and that resolves the problem.

That means that the tests should be failing, consistently, but they haven't been. Someone should investigate why that is.
(Assignee)

Comment 10

4 years ago
(In reply to Matt Wobensmith from comment #8)
> When we ran the automated tests for 24.1.0esr, we didn't see errors.
> However, a manual inspection of both 24.1.0esr and 24.1.1esr show the latest
> Java 7u45 plugin as blocked. So it appears that we likely have a real FF
> issue here to contend with.
> Also, in the current auto test run for 24.1.1, it's not 100% failing. That's
> another mystery to solve - I'd think it'd consistently all pass or all fail.

It didn't failed because here we don't test if addons are blocklisted, but we try to disable them and enable them back, so if there is another valid addon installed and if it happens to be the first one in list the test will pass.
How do you explain that some configs passed while others failed, for 24.1.1?
(Reporter)

Comment 12

4 years ago
Comment on attachment 831501 [details] [diff] [review]
patch_v1.0

Review of attachment 831501 [details] [diff] [review]:
-----------------------------------------------------------------

We don't want this fix giving that we want to catch this kind of issue in the future, not silently pass.
We might want to create a new test that handles the plugins in another way, check if they're blocklisted or not, take one randomly or check all and so on.
Attachment #831501 - Flags: review?(andreea.matei) → review-
(Reporter)

Comment 13

4 years ago
Fixed in bug 938698.
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
status-firefox-esr24: affected → fixed
Resolution: --- → FIXED
(Assignee)

Comment 14

4 years ago
Created attachment 8333741 [details]
Test_passing_with_addon_blocklisted.png

(In reply to Matt Wobensmith from comment #11)
> How do you explain that some configs passed while others failed, for 24.1.1?
Hi, here is an example of an situation where the test passes with an plugin blocklisted, because the first enabled plugin is not. 
As I said, in this particular test we test only the first enabled plugin which might be enabled.
On other ci nodes we might have more plugins installed, that's why some might pass. 
http://hg.mozilla.org/qa/mozmill-tests/file/a09dd08a9e9d/tests/functional/testAddons/testPluginDisableAffectsAboutPlugins.js#l34
I think we should enhance this test so we do not pick a blocklisted plugin. This would always cause this failure and makes the test unstable.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Reporter)

Comment 16

4 years ago
I thought we want to catch when a plugin is blocklisted, usually we don't have those installed and might be a firefox bug like this time.
For blocklisting checks we should cover all topics in a separate test. This has really nothing to do with this bug. We simply cannot use such a blocklisted add-on. Probably we might have to also update other tests for it.
Status: REOPENED → ASSIGNED
(Assignee)

Comment 18

4 years ago
Created attachment 8333926 [details] [diff] [review]
patch_v2.0

Enhanced the test to don't pick any blocklisted plugins.
Reports:
http://mozmill-crowd.blargon7.com/#/functional/report/2f1ca72938985fc4b989b7efcc552ade
http://mozmill-crowd.blargon7.com/#/functional/report/2f1ca72938985fc4b989b7efcc5573bc
http://mozmill-crowd.blargon7.com/#/functional/report/2f1ca72938985fc4b989b7efcc554d2d
http://mozmill-crowd.blargon7.com/#/functional/report/2f1ca72938985fc4b989b7efcc5587df
http://mozmill-crowd.blargon7.com/#/functional/report/2f1ca72938985fc4b989b7efcc5596c3
http://mozmill-crowd.blargon7.com/#/functional/report/2f1ca72938985fc4b989b7efcc55a6d1
Attachment #831501 - Attachment is obsolete: true
Attachment #8333926 - Flags: review?(andrei.eftimie)
Attachment #8333926 - Flags: review?(andreea.matei)
(Reporter)

Comment 19

4 years ago
Comment on attachment 8333926 [details] [diff] [review]
patch_v2.0

Review of attachment 8333926 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good. 
http://hg.mozilla.org/qa/mozmill-tests/rev/f4856cd7319a (default)

We also need backporting for this.
Attachment #8333926 - Flags: review?(andrei.eftimie)
Attachment #8333926 - Flags: review?(andreea.matei)
Attachment #8333926 - Flags: review+
(Reporter)

Updated

4 years ago
status-firefox28: --- → fixed
(Assignee)

Comment 20

4 years ago
Created attachment 8334500 [details] [diff] [review]
patch_v2.0 [Esr17]

Here is the patch for Esr17, for all the other branches, the previous patch applies.
Attachment #8334500 - Flags: review?(andreea.matei)
(Assignee)

Updated

4 years ago
status-firefox25: --- → affected
status-firefox26: --- → affected
status-firefox27: --- → affected
status-firefox-esr17: --- → affected
status-firefox-esr24: fixed → affected

Comment 21

4 years ago
Comment on attachment 8334500 [details] [diff] [review]
patch_v2.0 [Esr17]

Review of attachment 8334500 [details] [diff] [review]:
-----------------------------------------------------------------

Transplanted:
http://hg.mozilla.org/qa/mozmill-tests/rev/196af585cc44 (mozilla-aurora)
http://hg.mozilla.org/qa/mozmill-tests/rev/44873366206c (mozilla-beta)
http://hg.mozilla.org/qa/mozmill-tests/rev/2df67c6f630c (mozilla-release)
http://hg.mozilla.org/qa/mozmill-tests/rev/4de7a855a8e7 (mozilla-esr24)

Looks good, landed:
http://hg.mozilla.org/qa/mozmill-tests/rev/4a83f7749553 (mozilla-esr17)
Attachment #8334500 - Flags: review?(andreea.matei)
Attachment #8334500 - Flags: review+
Attachment #8334500 - Flags: checkin+

Updated

4 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago4 years ago
status-firefox25: affected → fixed
status-firefox26: affected → fixed
status-firefox27: affected → fixed
status-firefox-esr17: affected → fixed
status-firefox-esr24: affected → fixed
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.