Closed Bug 1241336 Opened 5 years ago Closed 5 years ago

Run an e10s A/B experiment on Beta 45 (first experiment)


(Firefox :: General, defect)

45 Branch
Not set



Tracking Status
e10s m8+ ---


(Reporter: vladan, Assigned: Felipe)




(4 files, 1 obsolete file)

We want to run an A/B experiment on Beta 45 on 50% of all profiles (with and without extensions).
Start time: beginning of Monday, Jan 25th GMT
End time: beginning of Thursday, Feb 11th GMT
Maximum duration: 15 days
Sampling: 50% of Beta population (25% + 25%)
Filtered only on profiles that have extended Telemetry enabled (default on Beta)
Attachment #8710228 - Flags: review?(felipc)
Is there some way to determine that e10s is blocked because of a11y now? Maybe experiment state "force-disabled" combined with the the values in Telemetry's A11Y_ histograms?
Flags: needinfo?(felipc)
Note that this experiment doesn't have the extension blacklist yet. I will add it tomorrow.
Depends on: 1241362
Depends on: 1228437, 1224374, 1241296
(In reply to Vladan Djeric (:vladan) -- please needinfo from comment #2)
> Is there some way to determine that e10s is blocked because of a11y now?
> Maybe experiment state "force-disabled" combined with the the values in
> Telemetry's A11Y_ histograms?

The E10S_STATUS histogram will have a value of 4. Possible values are here:
(soon to be added 6 = disabled for certain locales, 7 = disabled for addons)
Flags: needinfo?(felipc)
Duplicate of this bug: 1225496
Depends on: 1241507
Comment on attachment 8710228 [details] [diff] [review]
Experiment code, mostly unchanged.patch

Review of attachment 8710228 [details] [diff] [review]:

I'll give r+ after the addon whitelist is added, but the code here looks good.

::: experiments/e10s-enabled-beta45.1/code/bootstrap.js
@@ +32,5 @@
> +      case null:
> +        // Profile is ineligible if either of the two e10s prefs is non-default
> +        if (Preferences.isSet(OPTIN_PREF) || Preferences.isSet(PREF)) {
> +          TelemetryLog.log(SELF_ID, ["Setting branch to 'user-disabled' because e10s prefs have non-default values."]);
> +          yield Experiments.instance().setExperimentBranch(SELF_ID, "user-disabled");

if optin_pref = true, e10s will be enabled, so user-disabled is a bit of a misnomer here.. i.e., they will be running e10s, but just not due to the experiment.  Not sure how to better tag that, that's up to how the data analysis will be done afterwards.

Although that is a small concern since the optin_pref was never exposed to beta, so only a few users messing with about:config or reusing profiles will fall into this.

::: experiments/e10s-enabled-beta45.1/code/install.rdf
@@ +1,4 @@
> +<?xml version="1.0" encoding="utf-8"?>
> +<RDF xmlns="" xmlns:em="">
> +  <Description about="urn:mozilla:install-manifest">
> +    <em:id></em:id>

very nitpicky, but can you change the name to "e10s-beta45-withaddons", just so that this is more descriptive and easy to differentiate with a quick glance?  The ID in the manifest and bootstrap.js also needs to be changed.

::: experiments/e10s-enabled-beta45.1/manifest.json
@@ +1,3 @@
> +{
> +  "publish"     : true,
> +  "priority"    : 2,

I talked with bsmedberg about this. Since there's another experiment running at the same time (the unified urlbar), and that experiment has a smaller sample size than this one, this one needs to have a lower priority. That one is set to 2, so please set the priority here to 1.

@@ +11,5 @@
> +    "maxActiveSeconds" : 1296000,
> +    "appName"          : ["Firefox"],
> +    "channel"          : ["beta"],
> +    "minVersion"       : "44.0",
> +    "maxVersion"       : "44.0",

need to update min/max Version to 45.
Attachment #8710228 - Flags: review?(felipc) → feedback+
Hello Jason, could you sign this xpi for us?

FWIW, there will be another signing needed soon, but we want to start testing this asap so this version is good enough for this. Hope that is ok!
Attachment #8710492 - Flags: feedback?(jthomas)
Assignee: nobody → felipc
Attached file signed experiment.xpi
signed experiment.xpi
Attachment #8710492 - Flags: feedback?(jthomas)
So I made the changes I mentioned here and published an experiment server at .

With the standard experiment instructions, one can set experiments.manifest.uri to:

To start testing it. The difference from what needs to land is that I set the sample size to 100% and I removed the other running experiment, in order to make it easier to test this one.
It's still an open question the about whether we'll include an add-on blocklist in this experiment or not, so I also landed what we already have: (which is everything except the blocklist)
Filters out users of old versions of LastPass from being part of the experiment
Attachment #8711893 - Flags: review?(wmccloskey)
Comment on attachment 8711893 [details] [diff] [review]
Filter old versions of lastpass

Review of attachment 8711893 [details] [diff] [review]:

::: experiments/e10s-beta45-withaddons/filter.js
@@ +1,5 @@
>  function filter(c) {
> +  let activeAddons = {};
> +  try {
> +    activeAddons = c.telemetryEnvironment.addons.activeAddons;
> +  } catch (e) {}

How about we return false if we get an exception here?
Attachment #8711893 - Flags: review?(wmccloskey) → review+
Depends on: 1243612
Pushed the filtering of LastPass 3.* to the manifest

Once QA signs off I'll push the release_tag on this experiment and close this bug
Attached patch Block users of certain locales (obsolete) — Splinter Review
The in-tree code that does the locale blocking blocks based on the OS's locale, rather than the actual Firefox locale. Not a huge deal (would miss users of these locales if they're running the OS in en-US for example), but also not ideal .. So let's block here in the manifest and let the code be fixed for beta2.
Attachment #8713316 - Flags: review?(wmccloskey)
Blocks: 1243867
Comment on attachment 8713316 [details] [diff] [review]
Block users of certain locales

Actually, forget that. The bidi issue is minor and affects very few people. It will be better to have the in-tree code running to test it instead of filtering out these users
Attachment #8713316 - Flags: review?(wmccloskey)
Attachment #8713316 - Attachment is obsolete: true
QA has signed off on the experiment, so I updated release_tag to point to it:

Final deployment to production will occur in bug 1243867
Closed: 5 years ago
Resolution: --- → FIXED
Blocks: 1244187
You need to log in before you can comment on or make changes to this bug.