1465703 - Add bookmark and follow_bookmark event telemetry for Savant Shield study

Status: RESOLVED FIXED

(Firefox :: General, enhancement, P1)

Description

[Bianca Danforth [:bdanforth]]

This is one of a series of bugs that together comprise the Savant Shield study (tracking bug 1457226).

This bug will register and record (for the duration of the study only):
* When the user saves a bookmark.
* When the user removes a bookmark.
* When the user opens a bookmark.

Comment 1

[Bianca Danforth [:bdanforth]]

Attachment: Bug 1465703 - Add bookmark and follow_bookmark probes for Savant Shield study;

These probes will register and record (for the duration of the study only):
* When a bookmark is added.
* When a bookmark is removed.
* When a bookmark is visited.

Also moved init of study module to later in startup as an idle task to mitigate performance impact. Previously, this patch was failing the browser/base/content/test/performance/browser_startup.js test.

Note for bookmark added/removed: Using the source argument, we can filter out bulk events like from Sync, import, restore, etc.

Note for bookmark visited: This will also fire for the case when the user visits a page directly that happens to be bookmarked. The user doesn't have to "click" on a bookmark.

Review commit: https://reviewboard.mozilla.org/r/249564/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/249564/

Comment 2

[Bianca Danforth [:bdanforth]]

:pdehaan, could you provide QA review on this patch? Instructions for obtaining the test build are here: https://github.com/biancadanforth/gecko-dev/issues/5

Comment 3

[Bianca Danforth [:bdanforth]]

I neglected to mention that for the event telemetry to show up in `about:telemetry#events-tab`, the study pref `shield.savant.enabled` needs to be set to `true`. Normandy will take care of turning this pref ON and OFF when the study is officially launched.

Comment 4

[Drew Willcoxon :adw]

Comment on attachment 8983736 [details]
Bug 1465703 - Add bookmark and follow_bookmark probes for Savant Shield study;

https://reviewboard.mozilla.org/r/249564/#review256050

Looks good, just some nits.

::: browser/modules/SavantShieldStudy.jsm:197
(Diff revision 1)
> +    this.METHOD_1 = "bookmark";
> +    this.EXTRA_SUBCATEGORY_1 = "feature";
> +    this.METHOD_2 = "follow_bookmark";
> +    this.EXTRA_SUBCATEGORY_2 = "navigation";
> +    // nsINavBookmarksService constant
> +    this.TYPE_BOOKMARK = 1;

Any reason to hardcode 1 instead of using Ci.nsINavBookmarksService.TYPE_BOOKMARK?

::: browser/modules/SavantShieldStudy.jsm:201
(Diff revision 1)
> +    // nsINavBookmarksService constant
> +    this.TYPE_BOOKMARK = 1;
> +  }
> +
> +  init() {
> +    this.observer = {

Using a separate this.observer object is fine, but it would be slightly simpler to pass this BookmarkObserver object directly to add/removeObserver() and implement the observer methods on BookmarkObserver directly.  Up to you.

::: browser/modules/SavantShieldStudy.jsm:206
(Diff revision 1)
> +    this.observer = {
> +      // Ignore "fake" bookmarks created for bookmark tags
> +      skipTags: true,
> +
> +      onItemAdded: (itemID, parentID, index, itemType, uri, title, dateAdded, guid, parentGUID, source) => {
> +        if (itemType === this.TYPE_BOOKMARK && !uri.schemeIs("place")

The implementations of this method and onItemRemoved are very similar.  Have you thought about factoring it out into a single method and calling it from both places?  No big deal, up to you.

Comment 5

[Drew Willcoxon :adw]

Clearing the r? on Marco since I think you wanted one of us to review first come first serve, but feel free to look at this too Marco of course

Comment 6

[Bianca Danforth [:bdanforth]]

Comment on attachment 8983736 [details]
Bug 1465703 - Add bookmark and follow_bookmark probes for Savant Shield study;

https://reviewboard.mozilla.org/r/249564/#review256074

Comment 7

[Bianca Danforth [:bdanforth]]

Comment on attachment 8983736 [details]
Bug 1465703 - Add bookmark and follow_bookmark probes for Savant Shield study;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/249564/diff/1-2/

Comment 8

[Peter deHaan [:pdehaan]]

R+

Me thinks this looks good. I was fiddling w/ some bookmarken and saw the following noise in my about:telemetry tab:


timestamp | category | method          | object | value | extra
----------|----------|-----------------|--------|-------|-------
153193    | savant   | follow_bookmark | open   | null  | {subcategory: "navigation}
284500    | savant   | bookmark        | save   | null  | {subcategory: "feature"}
299093    | savant   | bookmark        | remove | null  | {subcategory: "feature"}

So as long as we're kosher w/ the "follow_bookmark" versus "bookmark" methods and the different subcategories, I think this looks great.

Comment 9

[Bianca Danforth [:bdanforth]]

Hi Peter,

Thank you for pointing this out. This is the expected distinction[1], as a follow_bookmark event means a user has visited a bookmark, and we consider that to be in the broad category of navigation, whereas adding/removing bookmarks is considered using bookmarks as a feature.

[1]:https://docs.google.com/spreadsheets/d/192unlFYdQ_nc5UVaG79YhY2bp6bV7XZm1G4BW-etOLE/edit?usp=sharing

Comment 10

[Bianca Danforth [:bdanforth]]

Comment on attachment 8983736 [details]
Bug 1465703 - Add bookmark and follow_bookmark probes for Savant Shield study;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/249564/diff/2-3/

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7deab88709c7
Add bookmark and follow_bookmark probes for Savant Shield study; r=adw

Comment 12

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/7deab88709c7

Comment 13

[Bianca Danforth [:bdanforth]]

Comment on attachment 8983736 [details]
Bug 1465703 - Add bookmark and follow_bookmark probes for Savant Shield study;

Approval Request Comment
[Feature/Bug causing the regression]: There is no regression. The tracking bug for this study is 1457226.
[User impact if declined]: None. This patch is part of a priority Shield study.
[Is this code covered by automated tests?]: There are no study-specific tests, but all code has been run on the try server for regression testing and has been approved by both a Firefox peer and QA (pdehaan).
[Has the fix been verified in Nightly?]: The study’s functionality has been verified in Nightly.
[Needs manual test from QE? If yes, steps to reproduce]: No.
[List of other uplifts needed for the feature/fix]: This request applies to all patches in this bug. In general, here is a comprehensive list of bugs whose patches need to be uplifted (in this order) to complete the study: 1462725, 1465698, 1465707, 1465703, 1465704, 1465697, 1465685, 1465694
[Is the change risky?]: No.
[Why is the change risky/not risky?]: This is an observational pref-flip study with no branches and no UI. It adds some event telemetry probes in various places and a study module to enable/disable collection. Each patch has been QA tested and R+ by pdehaan.
[String changes made/needed]: None.

Comment 14

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8983736 [details]
Bug 1465703 - Add bookmark and follow_bookmark probes for Savant Shield study;

Telemetry code needed to support the Fx61 Savant work. Approved for 61.0b13.

Comment 15

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/2830389b1876