1497616 - Pocket Personalization V2

Status: RESOLVED FIXED

(Firefox :: New Tab Page, enhancement, P1)

Description

[jonathan koren]

This is a tracking bug for Pocket Personalization v2.

Personalization V2 locally examines the user's browse history to determine the person's topics of interest. Pocket's recommended articles are then reranked according to the person's interest.

Comment 1

[Ed Lee :Mardak]

[Tracking Requested - why for this release]: Pocket wants to run new personalization experiments with release audience in 2018, so we'll need to uplift code to 63 beta before this Thursday to be able to turn on the experiment via shield.

The final landing of the code (preffed off everywhere) to nightly 64 of the code https://github.com/mozilla/activity-stream/pull/4447

Comment 2

[Nan Jiang [:nanj]]

Attachment: data_review_request_pockets.txt

Hey Francois, could you take a look at this patch, please? This adds various metrics for the Pocket Personalization V2 in Activity Stream. More details can be found in the attached form.

Thanks!

Comment 3

[Nan Jiang [:nanj]]

The PR is here https://github.com/mozilla/activity-stream/pull/4447/files#diff-f9d20d6688eef2754298456e5d5b836b

Comment 4

[François Marier [:francois]]

Comment on attachment 9015729 [details]
data_review_request_pockets.txt

> 5) Proposed Measurements
> 
> * Add a new performance ping to record the errors (getRecipeError, generateRecipeExecutorError, createInterestVectorError)
> during model building (Category 2, Bug 1497616)

I'm confused by this one. This doesn't sound like a performance / timing ping, but rather more like a correctness / reliability thing.

Could you please clarify whether you are collecting the _number_ of errors of each of the three types, and not including any error messages or any other data about the errors?

Comment 5

[Nan Jiang [:nanj]]

(In reply to François Marier [:francois] from comment #4)
> Comment on attachment 9015729 [details]
> data_review_request_pockets.txt
> 
> > * Add a new performance ping to record the errors (getRecipeError, generateRecipeExecutorError, createInterestVectorError)
> > during model building (Category 2, Bug 1497616)
> 
> I'm confused by this one. This doesn't sound like a performance / timing
> ping, but rather more like a correctness / reliability thing.
> 
> Could you please clarify whether you are collecting the _number_ of errors
> of each of the three types, and not including any error messages or any
> other data about the errors?

That's correct, those self-explained errors will not have any other messages attached in the payload, simply indicating a certain type of error takes place during the model building.

Comment 6

[François Marier [:francois]]

(In reply to Nan Jiang [:nanj] from comment #5)
> That's correct, those self-explained errors will not have any other messages
> attached in the payload, simply indicating a certain type of error takes
> place during the model building.

Great. Would you mind adding that clarification to the data review request?

The rest looks fine to me and I'll give it an r+.

Comment 7

[Nan Jiang [:nanj]]

Comment on attachment 9015729 [details]
data_review_request_pockets.txt

>1) What questions will you answer with this data?
>
>It will answer various performance related questions for Pocket personalization V2.
>
>* How long it takes to build the whole recommendation model
>* How long it takes to finish the indiviual steps during the model building
>* What errors happen during the model building
>* The total number of records fetched from Places for model building
>
>2) Why answer these?
>
>We want to better understand the performance behaviors for Pocket personalization V2.
>
>3) Alternative methods?
>
>No. 
>
>4) Can you answer it now?
>
>No, it is a new feature.
>
>5) Proposed Measurements
>
>* Add a new performance ping to record the duration of the whole model building (Category 2, Bug 1497616)
>* Add a new performance ping to record the duration of individual steps (getRecipe, executeRecipe, tagging, createInterestVector, calculateRelevanceScore) during model building (Category 2, Bug 1497616)
>* Add a new state ping to record the following errors during model building. Note that all the errors are self-explained, and there is no other message included in this ping (Category 2, Bug 1497616):
>  * GET_RECIPE_ERROR
>  * GENERATE_RECIPE_EXECUTOR_ERROR
>  * CREATE_INTEREST_VECTOR_ERROR
>* Add a new performance ping to record the size of Places output (Category 2, Bug 1497616)
>
>6) How long will this data be collected?  Choose one of the following:
>
>This is scoped to the Pocket personalization V2 shield study.
>
>7) What populations will you measure?
>
>2% of en-US users in release in USA.
>
>8) If this data collection is default on, what is the opt-out mechanism for users?
>
>The user can opt-out the data collection by either disabling the telemetry of Activity Stream or disabling the Firefox telemetry as a whole.
>
>9) Please provide a general description of how you will analyze this data.
>
>General reporting on Redash. Those metrics will be analyzed by Jonathan Koren, Scott Downe, and Nan Jiang.
>
>10) Where do you intend to share the results of your analysis?
>
>This will be shared over Mozilla LDAP via Redash.

Comment 8

[Nan Jiang [:nanj]]

Attachment: data_review_request_pockets.txt

Updated.

Thanks!

Comment 9

[Nan Jiang [:nanj]]

Attachment: data_review_request_pockets.txt

Comment 10

[François Marier [:francois]]

Comment on attachment 9016105 [details]
data_review_request_pockets.txt

1) Is there or will there be **documentation** that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes, in https://github.com/mozilla/activity-stream/blob/master/docs/v2-system-addon/data_events.md.

2) Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, telemetry setting (general in about:preferences or Activity Stream-specific in about:config).

3) If the request is for permanent data collection, is there someone who will monitor the data over time?**

No permanent.

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?  **

Category 1.

5) Is the data collection request for default-on or default-off?

Default ON.

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?

No.

7) Is the data collection covered by the existing Firefox privacy notice?

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data?

No, it's only for the Shield study.

Comment 11

[[github robot]]

Commit pushed to master at https://github.com/mozilla/activity-stream

https://github.com/mozilla/activity-stream/commit/87eccdede902211fb63e6e39b65bc7ecc1e2100b
Fix Bug 1497616 - feat(experiments): Pocket Personalization V2 (#4447)

Comment 12

[Ed Lee :Mardak]

https://hg.mozilla.org/mozilla-central/rev/a58d921ba5a1