1368141 - pref study - plugins.remember_infobar_dismissal

Status: RESOLVED FIXED

(Shield :: Shield Study, defect)

Description

[Benjamin Smedberg]

In bug 1368060 I'm adding a pref to control whether dismissing the plugin infobar is remembered per site. "plugins.remember_infobar_dismissal" I'd like to test this on nightly for a week.

Effects desired:
* reduce the total number of infobars seen
* improve the rate of infobars that aren't ignored

studied using PLUGINS_INFOBAR_SHOWN/BLOCK/ALLOW_DISMISSED

I'm owning and doing analysis. Running this now on this schedule in order to make a decision before uplifts on the 12th.

Branches:
50% pref false
50% pref true

Specifics:
nightly channel, would like to ship to 100% for 7 days starting next Wednesday 31-May
minimum buildid determined by when bug 1368060 lands, hopefully tomorrow's nightly
No other criteria required
No survey required

This study complements the release shield study currently shipping.

NI?rweiss for data review.

Comment 1

[[e-mail]]

We have reconsidered the need for data review for pref flipping studies, and determined they aren't necessary.  Signing off.

Comment 2

[Benjamin Smedberg]

Minimum build ID: 20170531030204

intent-to-ship sent to r-d. I believe this is ready to go. Gregg, do you have the power to deploy this?

Comment 3

[Julien Cristau [:jcristau]]

Signing off for release management.

Comment 4

[Benjamin Smedberg]

Gregg, I have verified that this appears in the normandy UI and the public data: https://self-repair.mozilla.org/api/v1/recipe/

However, nobody appears to be enrolled in this experiment. I looked at my own client as well as incoming data from nightly in the past week, and I see people in the following experiments:

defaultdict(int,
            {(u'[e-mail]', u'active'): 19,
             (u'[e-mail]', u'active'): 2,
             (u'pref-flip-test-nightly-1', u'control'): 31,
             (u'pref-flip-test-nightly-1', u'versionA'): 4,
             (u'pref-flip-test-nightly-1', u'versionB'): 8,
             (u'shield-lazy-client-classify', u'default'): 201576})

I don't know where the error might be, and I can't figure out how to debug this. Any data on debugging/logging would be great (also NI?adriansv since he may have logging/debugging tips)

Comment 5

[Michael Cooper [:mythmon]]

I suspect this might be related to bug 1368807, in which at least some clients aren't seeing Normandy's signatures as valid, and so aren't running recipes.

Comment 6

[Adrian Florinescu [:aflorinescu]]

(In reply to Benjamin Smedberg [:bsmedberg] from comment #4)
> I don't know where the error might be, and I can't figure out how to debug
> this. Any data on debugging/logging would be great (also NI?adriansv since
> he may have logging/debugging tips)

The two things that would be useful for debug (on client side) are: 
- Set the extensions.shield-recipe-client.dev_mode preference to true to run recipes immediately on startup.
- Set the extensions.shield-recipe-client.logging.level preference to 0 to enable more logging.
Start Nightly with a new profile, set the above prefs. and as your recipe targets 100% of Nightly, you should see in Browser console if your recipe is executed or if it's not, why. 
Running the prod Shield on a new profile, I get your recipe executed:

1496735307638	extensions.shield-recipe-client.recipe-runner	INFO	Executing recipe "Preference Experiment: SHIELD Public Infobar Display (Bug 1368141)" (action=preference-experiment)
1496735307639	extensions.shield-recipe-client.preference-experiments	DEBUG	PreferenceExperiments.has(shield-public-infobar-display-bug1368141)
1496735307641	extensions.shield-recipe-client.preference-experiments	DEBUG	PreferenceExperiments.start(shield-public-infobar-display-bug1368141, control)
1496735307641	extensions.shield-recipe-client.preference-experiments	DEBUG	PreferenceExperiments.startObserver(shield-public-infobar-display-bug1368141)

On how to debug the telemetry side, I'm afraid I have nothing, since that area is out of my scope.

Comment 7

[Michael Cooper [:mythmon]]

Although the other debugging may be interesting, after more investigation, bug 1368807 is definitely causing this. No users will have enrolled due to that bug.

We fixed the underlying cause yesterday, which was an expired cert. This explains why Adrian's test succeeded. There is another issue that the old expired cert may be incorrectly cached on clients. There should be a fix going out for this today.

Comment 8

[Osmose [:osmose, :mkelly]]

This experiment's recipe has been disabled on Normandy at bsmedberg's request.