Closed Bug 1414900 Opened 7 years ago Closed 5 years ago

[Shield] Pioneer Study: The ESPER experiment (esper-pioneer-shield-study)

Categories

(Shield :: Shield Study, enhancement)

enhancement
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: fwollsen, Assigned: mlopatka)

Details

Attachments

(10 obsolete files)

Evaluating Similarity of Pioneers as Exemplars of Release

Summary: 
Assess the degree and sense in which users in opt-in cohort of Firefox Pioneer differ from the Firefox release channel population. This should be a one time collection of data that focuses on fields for which we have aggregate statistics pertaining to the Firefox release population in Telemetry.

PHD: https://docs.google.com/document/d/1y3MSF_azfrAoKdj00_GEhC_o8sGetEuLLnloYBpVtNo/edit?ts=59ccadd2&pli=1

Add-on: https://github.com/motin/esper-pioneer-shield-study
Flags: needinfo?(rweiss)
Attached: 0.2.0 for signing, based on: b2fbce.
Flags: needinfo?(rdalal)
I cannot sign addons, NI'ing mythmon.
Flags: needinfo?(rdalal) → needinfo?(mcooper)
Flags: needinfo?(mcooper)
I have some followup questions about this study.  I will say that we will likely need to replicate this same study multiple times post 57, since the population might change a lot as we continue to recruit.  I will also say that I suspect there will need to be an extension of this study comparing survey response to self-reported demographic characteristics, since product questions are rarely focused solely on the range of measurements in Telemetry but rather on addressable markets of specific user types (which include demographic characteristics). 

However, these comments are not relevant for data review, and collecting responses from the study owners is not a blocker.  Providing data review by answering the questions listed here: https://github.com/mozilla/data-review/blob/master/review.md

1. Yes, documentation is provided.  The repo has both a testpilot.md and a testplan.md.
2. Yes, in that people can opt out of Pioneer and data collection via preferences.
3. This is not permanent data collection.
4. These all are either Category 1 or 2, and there does not appear to be any new data collection beyond base telemetry anyway.
5. Default off, default on for Pioneer participants.
6. No new identifiers.
7. Yes, this is covered.
8. No check in needed.
Flags: needinfo?(rweiss)
This all looks good to me. The PHD has been reviewed and data steward has signed off. Talking to mlopatka they are shooting for a Nov 21 launch. I would send the intent to ship Thurs or Friday since people will probably be tied up with the launch earlier in the week. CCing Ritu for visibility. This is for the Pioneer cohort only.
Signed XPI (singed by :mythmon) has been tested and works nicely. Data verified in re:dash for text cases.
What is the process for getting a formal R+ from QA other than filing this bug?
--m
Flags: needinfo?(kraj)
Please file a PI request (to pi-request@mozilla.com) for any QA support to ship this study and I'll have someone assigned to work on it. Thanks!
Flags: needinfo?(kraj)
thanks kraj!
PI request filed 09-11-2017
QA Contact: amasresha
Attached: 0.4.0 for signing, based on: 3f571c9.
Flags: needinfo?(mcooper)
Attachment #8925609 - Attachment is obsolete: true
Flags: needinfo?(mcooper)
Attachment #8925603 - Attachment is obsolete: true
Attached: 0.5.0 for signing, based on: 6de3cf5.
Flags: needinfo?(mcooper)
Attachment #8928517 - Attachment is obsolete: true
Attachment #8928616 - Attachment is obsolete: true
Flags: needinfo?(mcooper)
Looks to me like all QA concerns have been resolved (unless I'm missing something?)
Can I send the intent to ship mail for this study now?
Flags: needinfo?(amasresha)
Outside of QA concerns, we still need to resolve this concern:
https://github.com/motin/esper-pioneer-shield-study/issues/10
(In reply to mlopatka from comment #14)
> Looks to me like all QA concerns have been resolved (unless I'm missing
> something?)
> Can I send the intent to ship mail for this study now?

We retested the latest xpi (from comment 13) on Firefox 57.0, Beta 58.0b3 and Nightly 59.0a1.
Our test runs are green with the exception of having null values for the following fields.

1) On Linux OS (Ubuntu 16.04 amd64):
      "system_cpu.speed_mhz": "null",
      "system_gfx.monitors[1].screen_width": "null",
      "system_gfx.monitors[1].screen_width_zero_indexed": "null",
      "places_bookmarks_count_histogram": "null",
      "places_pages_count_histogram": "null",

2) On Windows 10x64 and 7x86: 
      "places_bookmarks_count_histogram": "null",
      "places_pages_count_histogram": "null",

3) On OSX 10.13:
      "system_gfx.monitors[1].screen_width_zero_indexed": "null",      
      "places_bookmarks_count_histogram": "null",
      "places_pages_count_histogram": "null",

If we are missing actions that trigger these or any configuration,  please suggest and we will verify if these fields are getting non-null values. 
The test document is here: https://docs.google.com/a/mozilla.com/document/d/1pZTInBeQe7oH6j6j3X1gUrSKLAPyXZDG0nnFBkPy45k/edit?usp=sharing
Flags: needinfo?(amasresha) → needinfo?(mlopatka)
The places _histogram fields are most likely null, which is why the failsafe attributes are put in place, see point 1 in https://github.com/motin/esper-pioneer-shield-study/blob/master/TELEMETRY.md#peculiarities

The _zero_indexed field is also a failsafe attribute and will be null if there is no external monitor attached, see point 5 in https://github.com/motin/esper-pioneer-shield-study/blob/master/TELEMETRY.md#peculiarities

The add-on has no Linux-specific probing/exceptions, it simply reports the same values that telemetry reports.
(In reply to fwollsen from comment #17)
> The places _histogram fields are most likely null, which is why the failsafe
> attributes are put in place, see point 1 in
> https://github.com/motin/esper-pioneer-shield-study/blob/master/TELEMETRY.
> md#peculiarities
> 
> The _zero_indexed field is also a failsafe attribute and will be null if
> there is no external monitor attached, see point 5 in
> https://github.com/motin/esper-pioneer-shield-study/blob/master/TELEMETRY.
> md#peculiarities
> 
> The add-on has no Linux-specific probing/exceptions, it simply reports the
> same values that telemetry reports.

Based on this information, concerns in comment 16 are not blockers then.
We completed our testing and all test runs are green.
Attached: 1.0.0 for signing, based on: 71a9f49.
Flags: needinfo?(mcooper)
Flags: needinfo?(mcooper)
Attached: 1.0.1 for signing, based on: 60cc145..
Flags: needinfo?(mcooper)
We will retest this when the addon (v1.0.1)is signed.
We should be ready to go with the latest version, just waiting on signing for the latest xpi.
Flags: needinfo?(mlopatka)
Flags: needinfo?(mcooper)
The latest xpi (from comment 24) does not install properly on all Firefox channels.
It says "it has been added to Firefox", but it does not show in about:addons.  

Any recommendations on installing this xpi properly?
Flags: needinfo?(mlopatka)
Installing the xpi as normal extension shows this error "Not eligable for Pioneer study. Will uninstall the study add-on."
 
I also tried to install it as system addon but it shows the following log: 

511898763558	addons.xpi	WARN	Add-on esper-pioneer-shield-study@shield.mozilla.com is not correctly signed.
1511898763559	addons.xpi	WARN	System add-on esper-pioneer-shield-study@shield.mozilla.com isn't compatible with the application.
1511898763559	addons.manager	WARN	Failed to update system addons: Error: Rejecting updated system add-on set that either could not be downloaded or contained unusable add-ons. (resource://gre/modules/addons/XPIProvider.jsm:2720:13) JS Stack trace: updateSystemAddons@XPIProvider.jsm:2720:13


My update.xml file:
<?xml version="1.0"?>
<updates>
  <addons>
    <addon id="esper-pioneer-shield-study@shield.mozilla.com"
     URL="file:///C:/Users/amasresha/Desktop/sysaddon/esper-pioneer-shield-study@shield.mozilla.com-1.0.1-signed.xpi"
     hashFunction="sha512"
     hashValue="7e106268cd3410cdc687fd3d68521c68b823fc39cd451760650fbfe0f67bdf9cbd0c9e51c91222202e349033208337c5b9ab4537bc2dd6723be09b1fc7a9ec67"
     size="41848"
     version="1.0.1"/>
  </addons>
</updates>
> Not eligable for Pioneer study. Will uninstall the study add-on.

This is a restriction of pioneer-utils. You'll need to install the pioneer opt-in extension: https://addons.mozilla.org/en-US/firefox/addon/firefox-pioneer/

System add-ons need signed in a different way, so this XPI isn't won't be usable as a system add-on. You would have run into the same issue as well if it had installed.
Aftre
I am not getting ping from "shield-study-addon"(In reply to Mike Cooper [:mythmon] from comment #27)
> > Not eligable for Pioneer study. Will uninstall the study add-on.
> 
> This is a restriction of pioneer-utils. You'll need to install the pioneer
> opt-in extension:
> https://addons.mozilla.org/en-US/firefox/addon/firefox-pioneer/

This fixes xpi installation issue. Thanks
But I am not getting any ping from the shield-study-addon even after few minutes of wait.

Here is screen capture: https://testing-1.tinytake.com/sf/MjE1MjEwN182NzA5Nzgy
Flags: needinfo?(mcooper)
I can see the payload data now. 
I found updated test steps here:https://github.com/motin/esper-pioneer-shield-study/blob/use-pioneer-utils/TESTPLAN.md
We will test and update this bug accordingly soon.
Flags: needinfo?(mlopatka)
Flags: needinfo?(mcooper)
We have retested this with the latest addon from comment 24. Our test shows green with the exception of having null values for the following fields in beta (58.0b7) nightly (59.0a1)channels:
“spbeNavigationAboutNewtab": "null",
"spbeNavigationContextmenu": "null",
"spbeNavigationSearchbar": "null",
"spbeNavigationUrlbar": "null",
“spbeWindowOpenEventCount": "null",
"systemGfxMonitors1ScreenWidthZeroIndexed": "null",

Release (57.0) shows non-null values for these fields.

Test cases and runs are here: https://docs.google.com/a/mozilla.com/document/d/1KblMrZYzhUwVisNc3Ubixkvs45LZPsTQVSqKITJUqnQ/edit?usp=sharing
This looks fine to me (expected behavior, mostly covered in comment #17).

Can we ship this ASAP to get as big a data collection window as possible?
Flags: needinfo?(mgrimes)
Please go ahead, looks like this was QA signed off and hits a small segment of release audience.
Flags: needinfo?(mgrimes)
@mlopatka - Now that this study has ended an you recap the results here and close this bug?
Flags: needinfo?(mlopatka)
@Matt, I'm holding off on the analysis until the weirdness encountered in https://bugzilla.mozilla.org/show_bug.cgi?id=1415393 is diagnosed. I'd like to know if the sample we got suffers from the same issues and needs to be rerun. Discussions I've seen seem to indicate that they are close to a confident assessment.
Flags: needinfo?(mlopatka)
Works for me. Thanks for the update!
Attachment #8928668 - Attachment is obsolete: true
Attachment #8928694 - Attachment is obsolete: true
Attachment #8931332 - Attachment is obsolete: true
Attachment #8931575 - Attachment is obsolete: true
Attachment #8931933 - Attachment is obsolete: true
Attachment #8932564 - Attachment is obsolete: true

experiment has completed.

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: