Closed Bug 1433335 Opened 2 years ago Closed 1 year ago

Send telemetry events for each action on an extension

Categories

(Toolkit :: Add-ons Manager, enhancement, P1)

enhancement

Tracking

()

VERIFIED FIXED
mozilla64
Iteration:
64.1 - Sep 14
Tracking Status
firefox64 --- verified

People

(Reporter: andy+bugzilla, Assigned: rpl)

References

(Blocks 1 open bug)

Details

Attachments

(3 files, 5 obsolete files)

This is the bug to pepper the Add-ons Manager code with event based telemetry to track the lifecycle of an add-on. The document for this is at:

https://docs.google.com/document/d/1QVelIRhn1MqIZTF0uxfr1XOxMURFpbeMVgjocyHpT88/edit
Priority: -- → P3
See Also: → 1457226
Here is the data review request related to the "Addons Manager" telemetry events that we were discussing about during the past few weeks.

I'm also going to add you as an additional reviewer on the attached patch, to double check the pieces relevant from a Telemetry point of view (e.g. the  definition of the events in Events.yaml, and any advice, or questions/doubts, that you may have on the usage of these events and their content).
Assignee: nobody → lgreco
Status: NEW → ASSIGNED
Attachment #8989190 - Flags: review?(chutten)
Comment on attachment 8989184 [details]
Bug 1433335 - Send telemetry events for each AddonManager action on an extension.

https://reviewboard.mozilla.org/r/254244/#review261042

::: toolkit/components/telemetry/Events.yaml:70
(Diff revision 1)
> +addonsManager:
> +  install:
> +    description: >
> +      This events are recorded during the install and update flow for extensions and themes,
> +      the value of the event is an install_id shared by the events related to the same install
> +      or update flow.
> +    objects: ["extension", "theme", "other", "unknown"]
> +    methods: ["install", "update"]
> +    extra_keys:
> +      addon_id: A string which identify the extension (when available)
> +      error: The AddonManager error related to an install or update failure.
> +      perms_length: The number of permissions shown to the user in the extension permission doorhanger
> +      source: The source that originally triggered the add-on installation
> +      step: >
> +        The current step in the install or update flow:
> +          - started, postponed, cancelled, failed, completed
> +          - site_warning, site_blocked, install_disabled_warning
> +          - download_started, download_completed, download_failed
> +    notification_emails: ["addons-dev-internal@mozilla.com"]
> +    expiry_version: "68"
> +    record_in_processes: ["main"]
> +    bug_numbers: [1433335]
> +  manage:
> +    description: >
> +      This events are recorded when an installed add-ons is being disable/enabled/uninstalled,
> +      the value of the event is the addon_id (which also allow to correlate multiple events
> +      related to each other).
> +    objects: ["extension", "theme", "other"]
> +    methods: ["disable", "enable", "pending_uninstall", "undo_uninstall", "uninstall"]
> +    extra_keys:
> +      source: The source from which the addon has been updated
> +    notification_emails: ["addons-dev-internal@mozilla.com"]
> +    expiry_version: "68"
> +    record_in_processes: ["main"]
> +    bug_numbers: [1433335]

These is the event definitions that are currently used in the patch.

The value of the install/update events is currently the id of the installation (just an integer which starts from 1 and is incremented on every new install flow), and it is the addon id for the "manage" events.

This way the values of the events can be used, in both the cases, to correlated multiple events related to each other.

The install/update events are sent multiple times for every possible step of the install/update flows, and the `step` extra var is the property where the current step is being included in the events.

The objects allowed for these events are currently the type of the extension: "extension", "theme" and "other" (which is currently meant to include locale, dictionary and plugins).

The install/update events also allow "unknown" as an object, because while the extension is still being downloaded we don't know yet the type of the extension (and we don't know its addon id either).

::: toolkit/components/telemetry/Events.yaml:80
(Diff revision 1)
> +      or update flow.
> +    objects: ["extension", "theme", "other", "unknown"]
> +    methods: ["install", "update"]
> +    extra_keys:
> +      addon_id: A string which identify the extension (when available)
> +      error: The AddonManager error related to an install or update failure.

This error is a string that we get using `AddonManager.errorToString(install.error)`, which convert the error into one of these AddonManager error names: https://searchfox.org/mozilla-central/rev/ed2763bea882619ccb48b0aecc54e523d2bdd2ae/toolkit/mozapps/extensions/AddonManager.jsm#3045-3063

::: toolkit/components/telemetry/Events.yaml:82
(Diff revision 1)
> +    methods: ["install", "update"]
> +    extra_keys:
> +      addon_id: A string which identify the extension (when available)
> +      error: The AddonManager error related to an install or update failure.
> +      perms_length: The number of permissions shown to the user in the extension permission doorhanger
> +      source: The source that originally triggered the add-on installation

the `source` extra var is not yet used in this patch, it is meant to be included in this events as part of Bug 1433334, but I've already added in the changes applied to Events.yaml by this patch (but I'm open to move it to Bug 1433334 if you prefer).

::: toolkit/mozapps/extensions/AddonManager.jsm:3797
(Diff revision 1)
> +    let {installId, method} = trackedInstallData;
> +
> +    let extraVars = {};
> +
> +    if (addonId) {
> +      extraVars.addon_id = this.getTrimmedString(addonId);

Here and at line 3823, the addon id is being trimmed to ensure that it is not longer than 80 chars.
Comment on attachment 8989184 [details]
Bug 1433335 - Send telemetry events for each AddonManager action on an extension.

Hi :chutten,
in comment 3 I've added some comments to highlight the pieces that may be more relevant for a review from a telemetry point of view.
Attachment #8989184 - Flags: review?(chutten)
Comment on attachment 8989184 [details]
Bug 1433335 - Send telemetry events for each AddonManager action on an extension.

https://reviewboard.mozilla.org/r/254246/#review261608

Mostly some nits, and a request that you get an addon person to check the addon-adjacent parts.

::: browser/base/content/browser-addons.js:301
(Diff revision 1)
>        };
>        let secondaryAction = {
>          label: gNavigatorBundle.getString("xpinstallPromptMessage.dontAllow"),
>          accessKey: gNavigatorBundle.getString("xpinstallPromptMessage.dontAllow.accesskey"),
> -        callback: () => {},
> +        callback: () => {
> +          for (let install of installInfo.installs) {

I don't know what this does or means, so you'll probably want someone who does to look at the addon-ish parts of this review :)

::: toolkit/components/telemetry/Events.yaml:82
(Diff revision 1)
> +    methods: ["install", "update"]
> +    extra_keys:
> +      addon_id: A string which identify the extension (when available)
> +      error: The AddonManager error related to an install or update failure.
> +      perms_length: The number of permissions shown to the user in the extension permission doorhanger
> +      source: The source that originally triggered the add-on installation

I would love examples of what sources are expected.

...actually, upon review it appears as though source is mentioned but never actually recorded. You probably shouldn't include it in your definition.

::: toolkit/mozapps/extensions/AddonManager.jsm:3486
(Diff revision 1)
> +
> +  // Numeric id included in the install events to correlate multiple events related
> +  // to the same install or update flow.
> +  nextInstallId: 1,
> +
> +  // Keep track of extra telemetry informations related to the AddonInstall objects:

*information

::: toolkit/mozapps/extensions/AddonManager.jsm:3651
(Diff revision 1)
> +    const length = str.length;
> +
> +    // Trim the string to prevent a flood of warnings messages logged internally by recordEvent,
> +    // the trimmed version is going to be composed by the first 40 chars and the last 37 and 3 dots
> +    // that joins the two parts, to visually indicate that the string has been trimmed.
> +    return `${str.slice(0, 40)}...${str.slice(length - 37, length)}`;

Nice.

::: toolkit/mozapps/extensions/AddonManager.jsm:3658
(Diff revision 1)
> +
> +  /**
> +   * Retrieve the addonId for the given AddonInstall instance.
> +   *
> +   * @param {AddonInstall} install
> +   *        The AddonInstall instance to retrieve the initiation source from.

You're getting the addon id, not the source

::: toolkit/mozapps/extensions/AddonManager.jsm:3684
(Diff revision 1)
> +
> +  /**
> +   * Retrieve the telemetry event object for the given AddonInstall instance.
> +   *
> +   * @param {AddonInstall} install
> +   *        The AddonInstall instance to retrieve the initiation source from.

You're getting an event object, not the source

::: toolkit/mozapps/extensions/AddonManager.jsm:3816
(Diff revision 1)
> +  /**
> +   * Record a manage event for the given addon.
> +   *
> +   * @param {AddonWrapper} addon
> +   *        The AddonWrapper instance.
> +   * @param {object} extra

It's called extraVars

::: toolkit/mozapps/extensions/test/xpcshell/test_addon_manager_telemetry_events.js:47
(Diff revision 1)
> +
> +  await promiseStartupManager();
> +});
> +
> +// Test the basic install and management flows.
> +add_task(async function test_basic_telemetry_events() {

You may want to skip-if these tests if the current version is >= the expiry_version. See the patch on bug 1471204
Attachment #8989184 - Flags: review?(chutten) → review-
Annnd I only now read comment #3 which means my notes about `source` can be ignored. If there's already a patch in place, then that's fine.

...I would still like the definition to include a list of acceptable values, though :)
Comment on attachment 8989184 [details]
Bug 1433335 - Send telemetry events for each AddonManager action on an extension.

https://reviewboard.mozilla.org/r/254246/#review261618

::: toolkit/components/telemetry/Events.yaml:91
(Diff revision 1)
> +          - site_warning, site_blocked, install_disabled_warning
> +          - download_started, download_completed, download_failed
> +    notification_emails: ["addons-dev-internal@mozilla.com"]
> +    expiry_version: "68"
> +    record_in_processes: ["main"]
> +    bug_numbers: [1433335]

Your Data Collection Review Request says you wish to record these on Release, but you did not include release_channel_collection: opt-out in your definitions.

I will perform the review as though you meant for these to be included in the release channel.
Comment on attachment 8989190 [details]
request-data-review-bug-1433335-addonmanager-telemetry-events.md

Note: Please list all of the data collection you're defining in this patch with their categories in your review. I would like to see entries for the extra keys, and a quick list of what user interactions prompt an event recording.

For example: Telemetry events recorded when an extension, theme, or other addon is installed, updated, enabled, disabled, or is being uninstalled. <then a table of what is recorded (source, step, etc.) and what categories they fall under.

You can include this as a separate attachment or inline with a comment.

DATA COLLECTION REVIEW:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes, Events.yaml.

    Is there a control mechanism that allows the user to turn the data collection on and off? 

Yes. Standard Telemetry mechanisms apply.

    If the request is for permanent data collection, is there someone who will monitor the data over time?**

N/A, expires in 68 (last recorded version is 67)

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2.

    Is the data collection request for default-on or default-off?

Default-on

    Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

Nothing new that isn't categorical. It includes the addon-id (which isn't new), the type of addon install error (name of the internal error that may have been raised during certain operations), and some categories (names for install steps and action sources)

    Is the data collection covered by the existing Firefox privacy notice? 

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

Yes.

---
Result: datareview+, once the full list of measurements is provided as noted above.
Flags: needinfo?(lgreco)
Attachment #8989190 - Flags: review?(chutten) → review+
Comment on attachment 8989184 [details]
Bug 1433335 - Send telemetry events for each AddonManager action on an extension.

https://reviewboard.mozilla.org/r/254246/#review261608

> I don't know what this does or means, so you'll probably want someone who does to look at the addon-ish parts of this review :)

hehe, sure thing! I'm not going to land this patch until aswan has also looked into it, and then signed it off from an "Addon Manager" point of view.

I didn't added aswan immediately as a reviewer only because I wanted to double-check with you that I was on the right track with the telemetry events (and then add aswan once I was sure that it was worth for him to start taking a look at the actual patch).

> I would love examples of what sources are expected.
> 
> ...actually, upon review it appears as though source is mentioned but never actually recorded. You probably shouldn't include it in your definition.

yeah, you are definitely right, I should have added these details at least as part of the data review request.

I'm going to attach an addendum to the data review request with additional details about the expected values of the `source` extra vars, and some other information about which user actions are meant to trigger these telemetry events and which are the expected values of the event properties.

> You may want to skip-if these tests if the current version is >= the expiry_version. See the patch on bug 1471204

I've added it (currently it is in a separate patch, on which I'm going to add you as a reviewer, to double-check with you that my tweak on the approach looks good to you).
Comment on attachment 8989184 [details]
Bug 1433335 - Send telemetry events for each AddonManager action on an extension.

https://reviewboard.mozilla.org/r/254246/#review261618

> Your Data Collection Review Request says you wish to record these on Release, but you did not include release_channel_collection: opt-out in your definitions.
> 
> I will perform the review as though you meant for these to be included in the release channel.

Ouch, yeah, that was definitely meant to be there, somehow I lost it while I was tweaking the event definition.

Thanks for catching it, I've added it back in the last version of this patch.
Attachment #8989184 - Flags: review?(aswan)
Attachment #8990069 - Flags: review?(chutten)
Comment on attachment 8990069 [details]
Bug 1433335 - Skip addon manager telemetry events test if expired in the current version.

https://reviewboard.mozilla.org/r/255086/#review261962

Looks good to me
Attachment #8990069 - Flags: review?(chutten) → review+
Comment on attachment 8989184 [details]
Bug 1433335 - Send telemetry events for each AddonManager action on an extension.

https://reviewboard.mozilla.org/r/254246/#review262308

Where is the spec for what we want to gather?  There is a doc linked from the bug but it doesn't appear to match what's implemented here (e.g., this patch instruments a bunch of things like download start, undone uninstalls, etc. that aren't mentioned in the doc)

::: toolkit/mozapps/extensions/AddonManager.jsm:3522
(Diff revision 2)
> +    AddonManager.getAllAddons().then(addons => {
> +      for (let addon of addons) {
> +        if (addon.pendingOperations === AddonManager.PENDING_UNINSTALL) {
> +          this.trackedPendingUninstall.add(addon);
> +        }
> +      }
> +    });
> +  },

It looks like you're calling this at startup?  this will cause the XPIDatabase to load which we don't want to do during startup.  In any case, I don't really understand this code -- that flag shouldn't be set on any installed addons during startup...
Attachment #8989184 - Flags: review?(aswan)
Hi :chutten,
This markdown file is an addendum to the data review request attached as attachment 8989190 [details], and it contains a list (and brief description) for the expected values for:

- the "step" extra var included in the "install" and "update" methods
- the "object" of the telemetry events
- the "source" extra var included in the "install" and "update" methods
- the "source" extra var included in the "enable", "disable", "uninstall" methods

Let me know how it looks to you.

Thanks again for you help and advice.
Flags: needinfo?(lgreco)
Attachment #8990715 - Flags: review?(chutten)
Comment on attachment 8989184 [details]
Bug 1433335 - Send telemetry events for each AddonManager action on an extension.

https://reviewboard.mozilla.org/r/254246/#review262308

Thanks a lot Andrew for the initial feedback! 

The document linked in comment 0 is the one that I used as a starting point, but the current events are now the result of some iterations and discussions on the actual telemetry events definition that goes in Events.yaml (e.g. to make the properties included in these events to be more consistent with each other).

The "download start" wasn't initially part of that spec document, I've introduced it as an alternative way to collect enough information related to the the download time (in that document it was an explicit extra var of "download completed", on the contrary, with the implementation proposed in this patch, it would be a value computed from the timestamp of the events for the "download_started" and "download_completed" steps related to the same install flow).

Nevertheless if we prefer, I can definitely remove the "download_start" step and put back the "download_time" extra var in the "download_completed" step.

The "pending uninstall" / "undo uinstall" wasn't explicitly part of that document, and I've included it because initially I wasn't sure if it was missing because it wasn't needed at all or because we missed it when that spec document has been created.

Currently, I tend to think that we may reduce the detail of the recorded event in this case, and only record the event once the "addon" has been completely uninstalled.

> It looks like you're calling this at startup?  this will cause the XPIDatabase to load which we don't want to do during startup.  In any case, I don't really understand this code -- that flag shouldn't be set on any installed addons during startup...

> It looks like you're calling this at startup?  this will cause the XPIDatabase to load which we don't want to do during startup.  

It is calling it when the addonManager is started, but that is a good point and I didn't liked that part neither.

As briefly decribed above, I'm going to remove this part, and just record a telemetry event once the extension has been really uninstalled.

> In any case, I don't really understand this code -- that flag shouldn't be set on any installed addons during startup...

I was also assuming it initially, but then I noticed (and then actually verified) that if the browser exists immediately (e.g. it crashes or its main process has been killed by the user or from the operating system), the pending operation is completed on the next startup (when any pending uninstall is being completed).

Anyway, as I wrote above, I'm going to remove these events completely for now, if we really need to record these additional parts of the add-on lifecycle we can figure out a better way to handle the above scenario (the "pending operations that are being executed on the next startup" scenarios).
Comment on attachment 8990715 [details]
data-review-addendum-but-1433335.md

This ought to do, thank you.
Attachment #8990715 - Flags: review?(chutten) → review+
Comment on attachment 8989184 [details]
Bug 1433335 - Send telemetry events for each AddonManager action on an extension.

https://reviewboard.mozilla.org/r/254246/#review262584

No problems from the Telemetry end. As before my r+ does not cover the addon parts.
Attachment #8989184 - Flags: review?(chutten) → review+
(In reply to Luca Greco [:rpl] from comment #16)
> Nevertheless if we prefer, I can definitely remove the "download_start" step
> and put back the "download_time" extra var in the "download_completed" step.

I don't have any preference here, I think this is probably a good question for whoever will be working with this data.  (bmiroglio?)

> The "pending uninstall" / "undo uinstall" wasn't explicitly part of that
> document, and I've included it because initially I wasn't sure if it was
> missing because it wasn't needed at all or because we missed it when that
> spec document has been created.

My concern here is just that we're not collecting data without a specific intention to use it somehow.  This is again probably better directed to somebody who will actually be looking at the collected data.
Iteration: --- → 63.4 - Aug 20
Attachment #8990069 - Attachment is obsolete: true
I removed Attachment 8990069 [details] from the patches that are part of this mozreview queue (because "skipping an xpcshell test case on the app versions on which the event is expired" by comparing the expected expired version with the appinfo's version is not going to work and it is not really useful given that in an xpcshell test the appinfo version is a fake one assigned in the test file itself).
Attachment #8995994 - Attachment is obsolete: true
Attachment #8995995 - Attachment is obsolete: true
Attachment #8989184 - Flags: review?(aswan)
Attachment #8995999 - Flags: review?(aswan)
Attachment #8989184 - Flags: review?(aswan)
- Added definitions for the new telemetry events
- Send telemetry events for each AddonManager action on an extension.
- Ensure that telemetry events are sent also for the extension prompts.
Comment on attachment 8989184 [details]
Bug 1433335 - Send telemetry events for each AddonManager action on an extension.

Marking the patches part of the mozreview request as obsolete, in favor of the new patch submitted on Phabricator (attachment 9004556 [details]).
Attachment #8989184 - Attachment is obsolete: true
Attachment #8989184 - Flags: review?(aswan)
Comment on attachment 8995999 [details]
Bug 1433335 - Send AddonManager telemetry events on extension prompts.

Marking the patches part of the mozreview request as obsolete, in favor of the new patch submitted on Phabricator (attachment 9004556 [details]).
Attachment #8995999 - Attachment is obsolete: true
Attachment #8995999 - Flags: review?(aswan)
Depends on: 1433334
Priority: P3 → P1
Iteration: 63.4 - Aug 20 → 64.1 (Sep 14)
Duplicate of this bug: 1201304
Blocks: 1492106
Comment on attachment 9004556 [details]
Bug 1433335 - Send telemetry events for each AddonManager action on an extension. r=aswan!

Andrew Swan [:aswan] has approved the revision.
Attachment #9004556 - Flags: review+
Pushed by luca.greco@alcacoop.it:
https://hg.mozilla.org/integration/autoland/rev/b5fe40154133
Send telemetry events for each AddonManager action on an extension. r=aswan
https://hg.mozilla.org/mozilla-central/rev/b5fe40154133
Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla64
Follows some STRs to verify the behaviors expected from this change.

Open "about:telemetry#events-tab" in a new tab, then execute the following scenarios 
and verify in the about:telemetry tab that the expected "addonsManager" telemetry events have been collected.

### Scenario: "enable and disable featured theme from disco panel"

- open a tab on "about:addons" and click on "Get Add-ons" in the sidebar to
  reach the discovery panel
  
- enable a featured theme (and wait it to be applied on the Firefox UI)

- reload the "about:telemetry" tab to refresh the list of the collected telemetry events

- Expected telemetry events:
  we expect two new telemetry events with category "addonsManager" to be collected:
  - method: disable, object: theme, value: default-theme@mozilla.org,
    extra: {"source": "builtin-theme"}
  - method: enable, object: theme, value: <id of the enabled theme>,
    extra: {"source": "lightweight-theme"}
    
- disable the enabled theme (and wait the Firefox UI to switch back to the built-in theme)

- Expected telemetry events:
  we expect two new telemetry events with category "addonsManager" to be collected:
  - method: uninstall, object: theme, value: <id of the enabled theme>,
    extra: {"source": "lightweight-theme"}
  - method: enable, object: theme, value: default-theme@mozilla.org,
    extra: {"source": "builtin-theme"}

### Scenario: "install featured extension from disco panel"

- open a tab on "about:addons" and click on "Get Add-ons" in the sidebar to reach the discovery panel
  
- install a featured extension and then cancel the installation from the extension permission doorhanger

- reload the "about:telemetry" tab to refresh the list of the collected telemetry events

- Expected telemetry events:
  we expect the following new telemetry events with category "addonsManager" to be collected:
  - method: install, object: unknown, value: <a numeric id auto-incremented on every installation/update>
    extra: {"step": "started", "source": "disco", "method": "amWebAPI"}
  - method: install, object: unknown, value: <same numeric id of the other events related to the same installation>
    extra: {"step": "download_started", "source": "disco", "method": "amWebAPI"}
  - method: install, object: extension, value: <same numeric id ...>
    extra: {"step": "download_completed", "download_time": "2285", "source": "disco", "method": "amWebAPI", 
            "addon_id": <id of the installing extension>}
  - method: install, object: extension, value: <same numeric id ...>
    extra: {"step": "permissions_prompt", "num_perms": "<number of permissions>", "num_origins": "<number of origin permissions", 
            "source": "disco", "method": "amWebAPI", "addon_id": <id of the installing extension>}
  - method: install, object: extension, value: <same numeric id ...>
    extra: {"step": "cancelled", "source": "disco", "method": "amWebAPI", "addon_id": <id of the installing extension>} 

- install a feature extension again, this time confirm the installation from the extension permission doorhanger

- Expected telemetry events:
  we expect all the events received before before the last one with the extra var "step": "cancelled" (but with a numeric value
  incremented, because it is a new installation), plus the following one:
  - method: install, object: extension, value: <new numeric id shared by the new installation>,
    extra: {"step": "completed", "source": "disco", "method": "amWebAPI", "addon_id": <id of the installing extension> }

- uninstall the extension from the disco panel

- Expected telemetry events:
  We expect the following additional "addonsManager" telemetry event:
  - method: uninstall, object: extension, value: <id of the uninstalling extension>,
    extra: {"source": "disco", "method": "amWebAPI"}

### Scenario "install extensions from addons.mozilla.org"

After installing an extension from addons.mozilla.org, we expect telemetry events (with the "addonsManager" category)
similar to the one described above, but with the following extra vars:

- extra.source: "amo"
- extra.method: "installTrigger" (at least until AMO has not switched to the amWebAPI) 

### Scenario "install testpilot extensions"

After installing a testpilot experiment extension from "testpilot.firefox.com", we expect telemetry events
(with the "addonsManager" category) similar to the one described above, but with the following properties in
the extra vars:

- extra.source: "testpilot"
- extra.method: "amWebAPI"
- extra.addon_id: "@testpilot-addon"

for the ones related to the installation of the testpilot extension, and the following extra vars for the
installation of the particular testpilot experiment extension:

- extra.source: "testpilot"
- extra.method: "amWebAPI"
- extra.addon_id: <id of the installing testpilot experiment>

### Scenario "install extension from a third party website"

Open a new tab on "https://perf-html.io", and click on "Install add-on", then disallow the installation
from a third party website from the "site warning" doorhanger that appears.

We expect the following "addonsManager" telemetry events:

- method: install, object: unknown, value: <numeric id>
  extra: {"step": "started", "source": "unknown", "method": "installTrigger"}
- method: install, object: unknown, value: <numeric id>
  extra: {"step": "site_warning", "source": "unknown", "method": "installTrigger"}
- method: install, object: unknown, value: <numeric id>
  extra: {"step": "cancelled", "source": "unknown", "method": "installTrigger"} 

Trigger the installation again, this time allow the installation to proceed, then we expect telemetry
events similar to the ones described for the other successfully-completed installation, but with
the extra.source = "unknown" and extra.method = "installTrigger".

### Scenario "user requests to update the installed extensions"

Install an older version of an extension hosted on AMO by vising the "See all versions" page of
an extension and picking an older version (e.g. https://addons.mozilla.org/en-US/firefox/addon/multi-account-containers/versions/)

Then from "about:addons" click on the "Check for Updates" button and allow the extension update
to be installed from the extension permissions doorhanger.

We expect the same kind of telemetry events collected for an extension installed from AMO,
but the events should have "method" = "update" and an extra var extra.updated_from = "user"
(because the update has been explicitly triggered by the user).

### Scenario "install a dictionary from AMO"

Install a dictionary from addons.mozilla.org (e.g. https://addons.mozilla.org/it/firefox/addon/dizionario-italiano/),
we expect the following "addonsManager" telemetry events:

- method: install, object: unknown, value: <numeric id...>,
  extra: {"step": "started", "source": "amo", "method": "installTrigger"}
- method: install, object: unknown, value: <numeric id...>,
  extra: {"step": "download_started", "source": "amo", "method": "installTrigger"}
- method: install, object: dictionary, value: <numeric id...>,
  extra: {"step": "download_completed", "download_time": "1397", "source": "amo", "method": "installTrigger", "addon_id": "it-IT@dictionaries.addons.mozilla.org"}
- method: install, object: dictionary, value: <numeric id...>,     
  extra: {"step": "completed", "source": "amo", "method": "installTrigger", "addon_id": "it-IT@dictionaries.addons.mozilla.org"} 

### Scenario "disable/enable/uninstall an installed extension after a browser restart"

- Install an extension from addons.mozilla.org
- Restart firefox
- Disable the extension
- Enable the extension
- Uninstall the extension (and reload the about:addons page to make the pending uninstall action to be completed)

We expect the "addonsManager" telemetry events for the extension to include an extra.source = "amo" to be included
in the telemetry events collected after a browser restart.

### Other possible scenarios

There are more possible scenarios that are supposed to trigger the addonsManager telemetry events
(basically most of the scenarios that are going to manage the installed extension or installing new ones),
follows some additional scenario that seems worth (and doable) to be verified manually by QA:

- opening the WebIDE for the first time should install automatically the "adb helper" extension
  (and so we expect telemetry events with extra.source = "webide" to be collected)

- installing and reloading an extension temporarily from "about:debugging#addons" should not trigger any addonsManager
  telemetry events (on purpose), but removing it currently emits the following telemetry event:
  - method: uninstall, object: extension, value: <id of the extension installed temporarily>
    extra: {"source": "temporary-addon"} 

- installing an extension from "about:addons" using the "Install Add-on From File" button,
  should trigger the usual installation telemetry events describe above with the following differences:
  - no events with step "download_*" (because we are installing a local xpi file and there isn't any download)
  - the extra.addon_id var is known starting from the first event (the one with extra.step = "started")
  - all the events have the following extra vars:
    - extra.source: "about:addons"
    - extra.method: "install-from-file"
    
- installing an extension by drag and dropping an xpi file into "about:addons",
  we expect the same kind of telemetry events recorded for "Install Add-on from File", but with the following
  extra vars:
  - extra.source: "about:addons"
  - extra.method: "drag-and-drop"
  
- installing an extension by drag and dropping a local xpi file into a new empty tab,
   we expect the same kind of telemetry event recorded when we drag and drop it in the about:addons tab,
   but with the following extra vars:
   - extra.source: "file-url"
   
- installing an extension by drag and dropping an http url into a new empty tab
  (e.g. https://addons.mozilla.org/firefox/downloads/file/872067/firefox_multi_account_containers-6.0.0-an+fx-linux.xpi),
   we expect the same kind of telemetry events recorded for a succesfully completed installation,
   but with the following extra vars:
   - extra.source: "unknown"
   - extra.method: "link"
This issue is verified as fixed on Firefox 64.0a1(20181008100121) under Win 7 64-bit and Mac OS X 10.13.3.

All the scenarios mentioned above passed the tests.
Status: RESOLVED → VERIFIED
Depends on: 1515697
Regressions: 1590398
Depends on: 1590421
You need to log in before you can comment on or make changes to this bug.