1092375 - Heartbeat Telemetry Experiment

Status: RESOLVED INCOMPLETE

(Firefox Health Report Graveyard :: Client: Desktop, defect)

Description

[Tyler Downer [He/Him]]

We will need a telemetry experiment for heartbeat!

Comment 1

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

I claim the experiment addon is ready for Blake's review at  https://github.com/gregglind/firefox-pulse/tree/master/heartbeat-telemetry-experiment-1

Unclear:
- how to best handle 'daily' recurring deploy.  At Telemetry, or here?

Incomplete / blocked
- deployment / monitoring of engagement pages (waiting on contractor)



relevant:  https://bug1092376.bugzilla.mozilla.org/attachment.cgi?id=8526074

Comment 2

[Benjamin Smedberg]

What info did you need from me?

Comment 3

[Blake Winton (:bwinton) (:☕️)]

As a note, I'll be doing the review at https://github.com/gregglind/firefox-pulse/pull/87/files

Comment 4

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

Attachment: Sample collected packet for Heartbeat Upload

Api at:  http://fjord.readthedocs.org/en/latest/hb_api.html

Comment 5

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

1.  UPDATED, review at:  https://github.com/gregglind/firefox-pulse/pull/88/files

Comment 6

[Benjamin Smedberg]

Comment on attachment 8529130 [details]
Sample collected packet for Heartbeat Upload

>{
>  "person_id": "6d5cc889-6fae-3649-8c33-e115b8661640",

What is this and why do we need it?

>  "flow_id": "e3dcba35-3069-f44d-a9c7-341219c4e921",

What is this?

>  "flow_began_ts": 1417017597805,
>  "flow_offered_ts": 1417017597928,
>  "flow_voted_ts": 1417018113949,
>  "flow_engaged_ts": 1417018120585,

What timezone/baseline are these timestamps?

>  "profile_age": 0.6726974652774516,

Age from what point? I would prefer not to collect this in general.

>  "profile_usage": {
>    "total": 0
>  },

What is this?

>  "extra": {
>    "crashes": {
>      "total": 0,
>      "submitted": 0,
>      "pending": 0
>    },

What's this?

>    "prefs": {
>      "browser.search.defaultenginename": "chrome://browser-region/locale/region.properties",
>      "privacy.donottrackheader.enabled": false,
>      "privacy.donottrackheader.value": 1,
>      "places.history.enabled": true,
>      "browser.tabs.remote.autostart": false,
>      "gecko.buildID": "20141125004001"
>    },

You should not send any pref values with this payload unless I've specifically reviewed them for privacy. For now, I think you should exclude them altogether.

>    "engage": [
>      [
>        1417018120586,
>        "support-firefox-slow",
>        "neutral"
>      ]
>    ]

What's this?

Comment 7

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

Personid:  for this experiment.  Needed to track how many 'attempts' it takes to actually get to 'popping up' a question.  Needed to guide the 'wait until showing' interval.
Flowid:  primary key of the interaction.  For the experiment, the id of the actual 'question experience'.  One at time but a user could have multiple.  At most they will see one 'popup'.
Flow times:  timestamp for when each state of the experience was reached.  Timezones are whatever Date.now() reports at the client.
Profile_age:  (in days) useful (predictive) covariate to explain satisfication.  (I didn't realize there are concerns over this).  From FHR.
Profile_usage:  fhr days and ticks.
extra:
- crashes:  number of reported, total crashes (from FHR)
- engage: clicks on the post-voting engagement page.  Instrumented here for building predictive models.  Also instrumented through GA (untested.  Once known to work, GA will be sufficient for tabulation, but not modeling)
- prefs:  buildID, distributionID from FHR.  We are trying to find markers that predict and explain scoring, response, and differences from General Release audience.

Comment 8

[Benjamin Smedberg]

(In reply to Gregg Lind (User Advocacy - Test Pilot) from comment #7)
> Personid:  for this experiment.  Needed to track how many 'attempts' it
> takes to actually get to 'popping up' a question.  Needed to guide the 'wait
> until showing' interval.

I need more detail. What generates the personid? Is it stored on the client? If so does it expire? Why is it necessary to have a personid in the payload to answer those questions? 

> Flowid:  primary key of the interaction.  For the experiment, the id of the
> actual 'question experience'.  One at time but a user could have multiple. 
> At most they will see one 'popup'.

Is this stored on the client? If so does it expire?

> Profile_usage:  fhr days and ticks.

The data schema doesn't list these. Days/ticks over what time frame?

> extra:
> - crashes:  number of reported, total crashes (from FHR)

Over what time frame?

> - prefs:  buildID, distributionID from FHR.  We are trying to find markers
> that predict and explain scoring, response, and differences from General
> Release audience.

please remove all pref recording for this round.

Comment 9

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

Prefs have been removed.  This removes the ability to understand the impact of search settings on score, response, and engagement.

Personid has been removed.

Personid and flowid are uuids.  they are stored on the client in prefs for the duration of the study, and removed after uninstall (or can be made to).

Personid is what allows proper estimation of the response rate.  They can be dropped this round, but will have to be dealt with eventually. Without this, we will underestimate the response rate, and be unable to measure the number of attempts it takes to trigger the UI element.  In previous studies, fewer attempts was criteria for deciding which UI to use.

Ticks (removed):  sum(FHR clean active ticks) over last 30 days.  (or age of profile).  Removing this removes the ability of using profile age to predict or model score, uptake and engagement.  This variable was significant in previous studies.

Comment 10

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

If you have some suggestions for good possible additional covariates that
- are likely to correspond with differences in user perceived satisfaction
- are consistent with our data collection policy

please suggest them.  

Even this previously collected data performed poorly (<10% of variability explained).

Comment 11

[Benjamin Smedberg]

> Personid and flowid are uuids.  they are stored on the client in prefs for
> the duration of the study, and removed after uninstall (or can be made to).

Yes, please make sure the flow IDs are removed on uninstall.

> Personid is what allows proper estimation of the response rate.  They can be
> dropped this round, but will have to be dealt with eventually. Without this,
> we will underestimate the response rate, and be unable to measure the number
> of attempts it takes to trigger the UI element.  In previous studies, fewer
> attempts was criteria for deciding which UI to use.

I don't understand this. Why don't you just measure the actual thing you care about, which is the number of times the UI is shown, and the # of times people respond to it? This shouldn't require a user identifier.

> 
> Ticks (removed):  sum(FHR clean active ticks) over last 30 days.  (or age of
> profile).  Removing this removes the ability of using profile age to predict
> or model score, uptake and engagement.  This variable was significant in
> previous studies.

I don't mind keeping ticks and active days as long as they are well defined. 30 days sounds reasonable.

Profile age is also ok as long as it's relative to the ping date and isn't an absolute value. I recommend truncating to an integer rather than dealing with floats.

Comment 12

[Benjamin Smedberg]

The reason I suggested recording the heartbeat data directly within telemetry is that we can correlate it against everything else in the FHR/telemetry payload without data duplication. Obviously that's not ready right now, so we'll use the separate collection system, but that means we need to avoid user identifiers.

Comment 13

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

All implemented.

I don't quite grok "Profile age is also ok as long as it's relative to the ping date and isn't an absolute value".  Can you explain that (algorithm) a bit further?  What about profile age / usage would be acceptable?

A.  {ndays_used_in_last_30: x,  total_tick_in_last_30: x, possible: min(profile_age_days,30)}
B.  profileAge: days, capped at 365  (or some other figure?)

The idea here is to use Newness and/or Usage for prediction / explanation.  It's not solved what the Right transform of that info is, thus my looseness in capturing it.

Comment 14

[Benjamin Smedberg]

An absolute profile age can be used to identify a particular user, especially if it has high float precision. A relative age, especially if it is rounded to an integer, provides no way to go back from a ping to identify a user.

Comment 15

[Benjamin Smedberg]

Can you update the JSON attachment so I can mark the feedback+ on it?

Comment 16

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

(sorry, still don't really understand your relative vs absolute profile stuff.  Literally, how do those two measures differ, if one has the ping time?)

Comment 17

[Benjamin Smedberg]

It's the rounding that improves things. The other question was about the data format.

Comment 18

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

Attachment: heartbeat.packet.r2.txt

Sample collected packet, accounting for data collection concerns.

Comment 19

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

Benjamin:  new question..

so, e10s.  Can we exclude them?  If so, manifest, or in addon?  (i will do both).  Would be nice to have a count there, but willing to take it as a wontfix.

Comment 20

[Benjamin Smedberg]

Comment on attachment 8529230 [details]
heartbeat.packet.r2.txt

FWIW, I didn't ask you to omit buildid/partnerid: just the prefs tree. r=me in either case

Comment 21

[Benjamin Smedberg]

I don't understand the e10s question. We're shipping this experiment to beta where e10s isn't even an option.

If it were an issue, we can't exclude them by manifest, so you'd have to do it within the addon.

Comment 22

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

Newest code diff: https://github.com/gregglind/firefox-pulse/pull/90

I claim experiment is ready, but I can't figure out how to test it (in experiment form)

- I build it as a 128.
- but installing it as a normal xpi (by 'file') shows it in the epxerient list, but shows it as completed.

```
cfx xpi \
	  --static-args='{"phonehome": true, "testing": false}' \
	  --templatedir ext-template/ \
	  --output-file addons/heartbeat-telemetry-experiment-1-type-128.xpi
```

Manifest being prepared.

Comment 23

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

Attachment: Result of uploading '128' experiment as xpi from File (and via cfx run)

Unclear how to test a build as 128 experiment

Comment 24

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

Attachment: manifest.json

(manifest file)

Comment 25

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

Attachment: experiment.xpi

(built experiment)

Comment 26

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

(hg push on the telex server wasn't working for me.  Attached instead)

Comment 27

[Benjamin Smedberg]

Kamil can you show Gregg how to test an experiment?

Comment 28

[Benjamin Smedberg]

Comment on attachment 8529857 [details]
manifest.json

This must have a sample of 10% or less, so that there are people available to run other experiments.

Maxversion should be 38.0 not *

I'll check the start/end dates later.

Comment 29

[Benjamin Smedberg]

Comment on attachment 8529857 [details]
manifest.json

Ther other dates (startTime/endTime/maxActiveSeconds) are correct.

Comment 30

[Kamil Jozwiak [:kjozwiak]]

Apologies for the delayed response! I was finishing up some testing/verification/automation updates for the 31.3.0esr and 34.0.5 releases.

I managed to successfully install the hearbeat experiment using the following steps:

Prerequisites: (note: I used Ubuntu 14.04 but it's basically the same steps on OSX)

* Install Mozilla build tools (I'm not 100% sure you need this, but I always install the build tools just incase)
** Ubuntu: wget -O bootstrap.py https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py && python bootstrap.py
** OSX: curl https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py > bootstrap.py && python bootstrap.py
* Install Genshi: http://genshi.edgewall.org/wiki/Download (you'll need pip installed)

Step #1: Pull the telemetry experiment code

* hg clone http://hg.mozilla.org/webtools/telemetry-experiment-server/

Step #2: adding manifest.json and .xpi 

* cd into "cd telemetry-experiment-server/"
* cd into "cd experiments/'
* mkdir heartbeat (place both the .xpi and manifest.json into the "heartbeat" directory)

Step #3: Building the experiment using python

* make sure you're in the "telemetry-experiment-server\" directory
* python build.py <your directory name> <"base URL"> Example --> python build.py _out "http://localhost:8080"
* launch a local server (i used my personal website): python -m SimpleHTTPServer 8080

Step #4: Installing experiment via browser

* install the appropriate browser that will match the criteria
* about:config
* devtools.chrome.enabled;true
* experiments.logging.level;0 (so you can see errors etc.. in the Browser Console)
* experiments.manifest.uri (the url to your firefox-manifest.json, example: http://kamiljozwiak.com/firefox-manifest.json
* disable/enable "experiments.enabled" (the experiment should be installed)

Greg, let me know if you run into any issues. I should be available the entire weekend so just shoot over an email or ping me on IRC. (email probably the better route). You can also take a look at https://wiki.mozilla.org/QA/Telemetry.

Comment 31

[Kamil Jozwiak [:kjozwiak]]

Attachment: installed Heartbeat experiment

Comment 32

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

Attachment: experiment.xpi

V2.0.1 with bugfixes

Comment 33

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

Attachment: manifest.json

Fixes to sample size (which I had misunderstood)

Comment 34

[Blake Winton (:bwinton) (:☕️)]

(I think the info has been provided.  If not, please re-needinfo me!)

Comment 35

[Benjamin Smedberg]

Not sure whether this actually got deployed, but it's no longer needed now.

Comment 36

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

(concur.  This is old news.  Not needed.)