1562212 - The "user-disable" and "exit" telemetry pings are no longer sent after manually removing the study from "about:studies" page

Status: RESOLVED WONTFIX

(Shield :: Shield Study, defect)

Description

[Cosmin Muntean [:cmuntean], Ecosystem QA]

Attachment: no disable and exit pings.gif

[Notes]:

• The issue is also reproducible if the running recipe of an opt-out study is manually disabled (from delivery console) and the experiment becomes completed. The "user-disable" and "exit" telemetry pings are no longer sent in "about:telemetry" page.
• This issue affects all the experiments that send the "user-disable" and "exit" telemetry pings.

[Affected Versions]:

• Firefox 68.0b14
• Nightly 69.0a1

[Affected Platforms]

• All Windows
• All Mac
• All Linux

[Prerequisites]

• Access to NORMANDY Delivery Console https://delivery-console.stage.mozaws.net/ is needed.
• Set the app.normandy.dev_mode preference to true to run recipes immediately on startup.
• Set the app.normandy.logging.level preference to 0 to enable more logging.
• Set the security.content.signature.root_hash preference to DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90.
• Set the preference value for app.normandy.api_url to https://stage.normandy.nonprod.cloudops.mozgcp.net/api/v1.
• Set the services.settings.server preference to https://settings.stage.mozaws.net/v1.
• An opt-out experiment recipe targeting the Beta and Nightly channel was already created and is live. ( eg #678)

[Steps to reproduce]:

1. Open the Beta or Nightly version of the browser with the profile targeted in the prerequisites.
2. Wait for the recipes to be executed on the profile.
3. Navigate to about:studies.
4. Click the "Remove" button.
5. Navigate to "about:telemetry" > "Archived ping data".
6. Observe the latest pings related to the opt-out experiment.

[Expected results]:

• The "user-disable" and "exit" telemetry pings are sent.

[Actual results]:

• The "user-disable" and "exit" telemetry pings are not sent.

[Regression]:

• The issue is not reproducible with the latest Firefox 67.0.4 release. Considering this, using mozregression tools I have managed to find a regression range:
  Last good revision: 0d1fd673085631fce1bad6e3aeeeb4b559c57493
  First bad revision: 998d83fa2ae988509435ba88de2c640f4607f2fa
  Pushlog: link.

From this pushlog it seems that bug 1549192 introduced this issue.

[Additional Notes]:

• Probably this issue is also reproducible if you install an experiment and remove it from "about:addons" page. Unfortunately, I don't have an experiment that is displayed in "about:addons" page in order to verify this scenario.
• Attached a screen recording with the issue.

Comment 1

[Cosmin Muntean [:cmuntean], Ecosystem QA]

Andrew, can you please take a look at this bug? This issue affects all the experiments on Firefox 68 and 69 that send the "user-disable" and "exit" telemetry pings.

Comment 2

[Andrew Swan [:aswan]]

I don't know where a "user-disable" or "exit" telemetry ping comes from. I'm happy to help from the addon manager side of this, but mythmon, can you take a look and see if you can figure out what's going on?

Comment 3

[Michael Cooper [:mythmon]]

This telemetry would normally be generated by Shield Study Utils. A couple ideas that I have

1. Telemetry is being generated, but is delayed. Restarting the browser would force an archive, and might make the telemetry show up.
2. The extension isn't getting a chance to actually send the telemetry because it is unloading before it can do what it needs to.

Given the contents of the revision range in comment 0, number 2 seems likely. Especially this quote from the commit message of 998d83fa2a:

Remove the temptation for api authors to write incorrect code by removing extension.shutdownReason and replacing it with an isAppShutdown boolean passed to shutdown handlers.

If we can find a way to update the add-on to compensate for this, we could deploy an update via Normandy.

It may be useful to see the contents of the browser console's logs when the bug is exhibited.

Comment 4

[Andrew Swan [:aswan]]

Oh yay, out-of-tree code relying on extension internals and is apparently untested:
https://github.com/mozilla/shield-studies-addon-utils/blob/63ce0385cc3601907792aba9fd99e41b002a2aed/webExtensionApis/study/src/index.js#L73-L85

I guess changing the test to !isAppShutdown should basically preserve the current behavior. In non-experimental APIs, this would be done with extension lifecycle events (https://firefox-source-docs.mozilla.org/toolkit/components/extensions/webextensions/lifecycle.html#extension-uninstall-and-update) but see the very last paragraph of that section...

Comment 5

[Cosmin Muntean [:cmuntean], Ecosystem QA]

It may be useful to see the contents of the browser console's logs when the bug is exhibited.

The only log that is displayed in browser console after removing the experiment from "about:studies" page is the following one:

1562057201593 addons.xpi WARN Attempted to remove [e-mail] from app-profile but it was already gone

Here is the screenshot with the browser console.

Comment 6

[Julien Cristau [:jcristau]]

Sounds like the fix for this is out of tree so not planning on changing anything in 68.

Comment 7

[Liz Henry (:lizzard) (relman/hg->git project)]

Andrew is this something you might work on for 70 or even 71? Can you set a priority for the issue? Thanks!

Comment 8

[Andrew Swan [:aswan]]

(In reply to Liz Henry (:lizzard) from comment #7)

Andrew is this something you might work on for 70 or even 71? Can you set a priority for the issue? Thanks!

I don't believe I have access to shield-studies-addon-utils, and I have no idea what/where all the extensions that use it are.
Forwarding the request.

Comment 9

[Michael Cooper [:mythmon]]

This is not a problem with Firefox, per se. This is an incompatibility between Firefox (which changed behavior) and the shield-study-addon-utils. Individual studies that use shield-study-addon-utils will need to be updated for compatibility, but there are no changes to be made to Firefox, since I assume Andrew is not interested in reverting the changes to Firefox.

The addon utils are managed on Github, and the general issue should be filed there. Individual studies are generally managed on Bugzilla, but it is up to the individual study authors to update the study addons.

I consider most of the study utils to be deprecated at this point anyways, because bug 1536658 has landed in Firefox 69. This removes the need for most parts of the study utils. I don't think there is much benefit in fixing this generally in Firefox, and I don't know what the future is of the study utils, so I think we should close this and deal with it on individual studies where needed.