Closed Bug 808824 Opened 12 years ago Closed 11 years ago

Set extensions.blocklist.enabled to false for talos tests

Categories

(Testing :: Talos, defect, P4)

x86_64
macOS
defect

Tracking

(firefox19-)

RESOLVED FIXED
Tracking Status
firefox19 - ---

People

(Reporter: philor, Assigned: philor)

References

Details

Attachments

(1 file)

Because bug 799160, I manually updated the in-tree blocklists on mozilla-central and mozilla-aurora on November 3rd. In the case of OS X 10.7, according to the talos regression finder-and-emailer, that caused an improvement in tp5n "Responsiveness" of between 50 and 60%, and a regression in tp5n "(Main RSS)" of around a percent.

The only likely thing in that update was bug 803152, which made a bunch of versions of Java and Flash click-to-play.

Do those versions of Java and Flash include the ones that we have installed on our test slaves? Do we feel good about that, and want to continue testing tp like our users see it, with Java and Flash mostly disabled?
Ah, 10.8 too, I must have already gone back to ignoring the emails by the time 10.8 hit.
Summary: Verify whether we really blocklisted our own versions of Flash and/or Java on 10.7 → Verify whether we really blocklisted our own versions of Flash and/or Java on 10.7 and 10.8
(In reply to Phil Ringnalda (:philor) from comment #0)
> Do those versions of Java and Flash include the ones that we have installed
> on our test slaves? Do we feel good about that, and want to continue testing
> tp like our users see it, with Java and Flash mostly disabled?

No - these tests should either be run with the latest versions of Java and Flash (unblocked), or we need to set extensions.blocklist.enabled to false in the test environment. I see no value in testing CTP as part of our perf suite.

Including catlee/joduinn/ctalbert to get their eyes on.
Flags: needinfo?(catlee)
talos-r4-lion-003
java version "1.6.0_35"
Java Applet Plug-in 14.4.0 with Firefox Nightly

I don't see Flash ?
talos-mtnlion-r5-002
java version "1.6.0_31"
Java Applet Plug-in 14.3.0 with Firefox Nightly

I don't see flash here either.
(In reply to Alex Keybl [:akeybl] from comment #2)
> (In reply to Phil Ringnalda (:philor) from comment #0)
> > Do those versions of Java and Flash include the ones that we have installed
> > on our test slaves? Do we feel good about that, and want to continue testing
> > tp like our users see it, with Java and Flash mostly disabled?
> 
> No - these tests should either be run with the latest versions of Java and
> Flash (unblocked), or we need to set extensions.blocklist.enabled to false
> in the test environment. I see no value in testing CTP as part of our perf
> suite.

Clint - can you help make a recommendation on which of the above options to take? I believe automated tests are the component most impacted here.
Assignee: nobody → ctalbert
We want to test with Java and Flash enabled, but we also want to control the versions so that our performance test bed remains as noise free as possible.   I think we should be conscious of exactly the versions we are using with testing. For instance, we pull down and install a specific version of Flash but not Java. We should pull down specific versions of both.  

As far as keeping them up to date, I'd recommend doing something that mirrors the train model - keeping the same version for the duration of a release train, and then switching to a new version (if needed) at the end of that train.  That way our tests remain consistent during the cycle, and the version of plugins being used will ride the trains into obsolescence.

When code is tweaked to change the behavior of a certain version (like making a certain version CTP) then we can decide whether to update the plugin version out of band for that branch.  

It's a bit more work, but I think this ideal scenario will get us both a small measure of consistency and enable us to test something akin to what the user actually sees. I know Talos isn't really testing what the user sees, but it does test layout and rendering performance, and since plugins are a huge component of that performance metric, it makes sense to have them configured the way most of our user base has them configured (on).

Since this is all in releng code, I'm not going to be the proper assignee here to make this change, so I recommend we change the assignee to someone familiar with these buildbot scripts who can make these changes without breaking everything.
Flags: needinfo?(catlee)
We're no longer considering this a release issue, since it only impacts plugin performance testing.

catlee - can you own ensuring that the plugins we're using for testing are not blocked? Here's a link with the latest: https://wiki.mozilla.org/Blocklisting/PluginBlocks
Assignee: ctalbert → catlee
So for Java we're using 1.6.0_35 (10.7) / 1.6.0_31 (10.8)

Assuming 1.6.0_35 == Java 1.6 update 35, then for firefox 17 and higher these java versions should be CTP (https://addons.mozilla.org/en-US/firefox/blocked/p188)

For Flash we're using flash32_10_3_183_5.zip for 10.7 and 10.8.

I believe this is covered by this CTP block: https://addons.mozilla.org/en-US/firefox/blocked/p176

Is there an easy way to verify?
Blocks: 843373
No longer blocks: 843373
Not sure how to proceed here.
Flags: needinfo?
Priority: -- → P4
Move it to Testing::Talos with a summary morph to "Set extensions.blocklist.enabled to false for talos tests." Whether or not some angel adds an in-tree thing equivalent to talos.json (or just adds to talos.json) to control which versions of plugins we use, if there's a reason to CTP the version of either that we're using, the blocklist update is not going to wait for releng to deploy a new one, and there's no guarantee that they won't both be CTP for all released versions.
Flags: needinfo?
Assignee: catlee → nobody
Component: Release Engineering: Platform Support → Talos
Product: mozilla.org → Testing
QA Contact: coop
Summary: Verify whether we really blocklisted our own versions of Flash and/or Java on 10.7 and 10.8 → Set extensions.blocklist.enabled to false for talos tests
Version: other → Trunk
here are all the extensions related prefs that are set for talos:
http://hg.mozilla.org/build/talos/file/2ff263006133/talos/PerfConfigurator.py#l255

Should I add in:
extensions.blocklist.enabled=false
Attached patch fixSplinter Review
Yes.
Assignee: nobody → philringnalda
Status: NEW → ASSIGNED
Attachment #791757 - Flags: review?(jmaher)
Comment on attachment 791757 [details] [diff] [review]
fix

Review of attachment 791757 [details] [diff] [review]:
-----------------------------------------------------------------

thanks!
Attachment #791757 - Flags: review?(jmaher) → review+
http://hg.mozilla.org/build/talos/rev/2ddd58d18979

I still need to update talos.zip in inbound, that is likely to happen today.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Depends on: 1545438
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: