Closed
Bug 1414900
Opened 7 years ago
Closed 5 years ago
[Shield] Pioneer Study: The ESPER experiment (esper-pioneer-shield-study)
Categories
(Shield :: Shield Study, enhancement)
Shield
Shield Study
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: fwollsen, Assigned: mlopatka)
Details
Attachments
(10 obsolete files)
Evaluating Similarity of Pioneers as Exemplars of Release Summary: Assess the degree and sense in which users in opt-in cohort of Firefox Pioneer differ from the Firefox release channel population. This should be a one time collection of data that focuses on fields for which we have aggregate statistics pertaining to the Firefox release population in Telemetry. PHD: https://docs.google.com/document/d/1y3MSF_azfrAoKdj00_GEhC_o8sGetEuLLnloYBpVtNo/edit?ts=59ccadd2&pli=1 Add-on: https://github.com/motin/esper-pioneer-shield-study
Flags: needinfo?(rweiss)
Attached: 0.2.0 for signing, based on: b2fbce.
Flags: needinfo?(rdalal)
Comment 2•7 years ago
|
||
I cannot sign addons, NI'ing mythmon.
Flags: needinfo?(rdalal) → needinfo?(mcooper)
UPDATED PHD Document link: https://docs.google.com/document/d/1AhPGfCUs8lafrs9EznhL80NmiL0z4tUKb7HCMXlPxH8
Comment 4•7 years ago
|
||
Flags: needinfo?(mcooper)
Comment 5•7 years ago
|
||
I have some followup questions about this study. I will say that we will likely need to replicate this same study multiple times post 57, since the population might change a lot as we continue to recruit. I will also say that I suspect there will need to be an extension of this study comparing survey response to self-reported demographic characteristics, since product questions are rarely focused solely on the range of measurements in Telemetry but rather on addressable markets of specific user types (which include demographic characteristics). However, these comments are not relevant for data review, and collecting responses from the study owners is not a blocker. Providing data review by answering the questions listed here: https://github.com/mozilla/data-review/blob/master/review.md 1. Yes, documentation is provided. The repo has both a testpilot.md and a testplan.md. 2. Yes, in that people can opt out of Pioneer and data collection via preferences. 3. This is not permanent data collection. 4. These all are either Category 1 or 2, and there does not appear to be any new data collection beyond base telemetry anyway. 5. Default off, default on for Pioneer participants. 6. No new identifiers. 7. Yes, this is covered. 8. No check in needed.
Flags: needinfo?(rweiss)
Comment 6•7 years ago
|
||
This all looks good to me. The PHD has been reviewed and data steward has signed off. Talking to mlopatka they are shooting for a Nov 21 launch. I would send the intent to ship Thurs or Friday since people will probably be tied up with the launch earlier in the week. CCing Ritu for visibility. This is for the Pioneer cohort only.
Signed XPI (singed by :mythmon) has been tested and works nicely. Data verified in re:dash for text cases. What is the process for getting a formal R+ from QA other than filing this bug? --m
Flags: needinfo?(kraj)
Comment 8•7 years ago
|
||
Please file a PI request (to pi-request@mozilla.com) for any QA support to ship this study and I'll have someone assigned to work on it. Thanks!
Flags: needinfo?(kraj)
Updated•7 years ago
|
QA Contact: amasresha
Reporter | ||
Comment 10•7 years ago
|
||
Attached: 0.4.0 for signing, based on: 3f571c9.
Flags: needinfo?(mcooper)
Comment 11•7 years ago
|
||
Attachment #8925609 -
Attachment is obsolete: true
Flags: needinfo?(mcooper)
Updated•7 years ago
|
Attachment #8925603 -
Attachment is obsolete: true
Reporter | ||
Comment 12•7 years ago
|
||
Attached: 0.5.0 for signing, based on: 6de3cf5.
Flags: needinfo?(mcooper)
Comment 13•7 years ago
|
||
Attachment #8928517 -
Attachment is obsolete: true
Attachment #8928616 -
Attachment is obsolete: true
Flags: needinfo?(mcooper)
Assignee | ||
Comment 14•7 years ago
|
||
Looks to me like all QA concerns have been resolved (unless I'm missing something?) Can I send the intent to ship mail for this study now?
Flags: needinfo?(amasresha)
Comment 15•7 years ago
|
||
Outside of QA concerns, we still need to resolve this concern: https://github.com/motin/esper-pioneer-shield-study/issues/10
Comment 16•7 years ago
|
||
(In reply to mlopatka from comment #14) > Looks to me like all QA concerns have been resolved (unless I'm missing > something?) > Can I send the intent to ship mail for this study now? We retested the latest xpi (from comment 13) on Firefox 57.0, Beta 58.0b3 and Nightly 59.0a1. Our test runs are green with the exception of having null values for the following fields. 1) On Linux OS (Ubuntu 16.04 amd64): "system_cpu.speed_mhz": "null", "system_gfx.monitors[1].screen_width": "null", "system_gfx.monitors[1].screen_width_zero_indexed": "null", "places_bookmarks_count_histogram": "null", "places_pages_count_histogram": "null", 2) On Windows 10x64 and 7x86: "places_bookmarks_count_histogram": "null", "places_pages_count_histogram": "null", 3) On OSX 10.13: "system_gfx.monitors[1].screen_width_zero_indexed": "null", "places_bookmarks_count_histogram": "null", "places_pages_count_histogram": "null", If we are missing actions that trigger these or any configuration, please suggest and we will verify if these fields are getting non-null values. The test document is here: https://docs.google.com/a/mozilla.com/document/d/1pZTInBeQe7oH6j6j3X1gUrSKLAPyXZDG0nnFBkPy45k/edit?usp=sharing
Flags: needinfo?(amasresha) → needinfo?(mlopatka)
Reporter | ||
Comment 17•6 years ago
|
||
The places _histogram fields are most likely null, which is why the failsafe attributes are put in place, see point 1 in https://github.com/motin/esper-pioneer-shield-study/blob/master/TELEMETRY.md#peculiarities The _zero_indexed field is also a failsafe attribute and will be null if there is no external monitor attached, see point 5 in https://github.com/motin/esper-pioneer-shield-study/blob/master/TELEMETRY.md#peculiarities The add-on has no Linux-specific probing/exceptions, it simply reports the same values that telemetry reports.
Comment 18•6 years ago
|
||
(In reply to fwollsen from comment #17) > The places _histogram fields are most likely null, which is why the failsafe > attributes are put in place, see point 1 in > https://github.com/motin/esper-pioneer-shield-study/blob/master/TELEMETRY. > md#peculiarities > > The _zero_indexed field is also a failsafe attribute and will be null if > there is no external monitor attached, see point 5 in > https://github.com/motin/esper-pioneer-shield-study/blob/master/TELEMETRY. > md#peculiarities > > The add-on has no Linux-specific probing/exceptions, it simply reports the > same values that telemetry reports. Based on this information, concerns in comment 16 are not blockers then. We completed our testing and all test runs are green.
Reporter | ||
Comment 19•6 years ago
|
||
Attached: 1.0.0 for signing, based on: 71a9f49.
Flags: needinfo?(mcooper)
Comment 20•6 years ago
|
||
Flags: needinfo?(mcooper)
Reporter | ||
Comment 21•6 years ago
|
||
Attached: 1.0.1 for signing, based on: 60cc145..
Flags: needinfo?(mcooper)
Comment 22•6 years ago
|
||
We will retest this when the addon (v1.0.1)is signed.
Assignee | ||
Comment 23•6 years ago
|
||
We should be ready to go with the latest version, just waiting on signing for the latest xpi.
Flags: needinfo?(mlopatka)
Comment 24•6 years ago
|
||
Flags: needinfo?(mcooper)
Comment 25•6 years ago
|
||
The latest xpi (from comment 24) does not install properly on all Firefox channels. It says "it has been added to Firefox", but it does not show in about:addons. Any recommendations on installing this xpi properly?
Flags: needinfo?(mlopatka)
Comment 26•6 years ago
|
||
Installing the xpi as normal extension shows this error "Not eligable for Pioneer study. Will uninstall the study add-on." I also tried to install it as system addon but it shows the following log: 511898763558 addons.xpi WARN Add-on esper-pioneer-shield-study@shield.mozilla.com is not correctly signed. 1511898763559 addons.xpi WARN System add-on esper-pioneer-shield-study@shield.mozilla.com isn't compatible with the application. 1511898763559 addons.manager WARN Failed to update system addons: Error: Rejecting updated system add-on set that either could not be downloaded or contained unusable add-ons. (resource://gre/modules/addons/XPIProvider.jsm:2720:13) JS Stack trace: updateSystemAddons@XPIProvider.jsm:2720:13 My update.xml file: <?xml version="1.0"?> <updates> <addons> <addon id="esper-pioneer-shield-study@shield.mozilla.com" URL="file:///C:/Users/amasresha/Desktop/sysaddon/esper-pioneer-shield-study@shield.mozilla.com-1.0.1-signed.xpi" hashFunction="sha512" hashValue="7e106268cd3410cdc687fd3d68521c68b823fc39cd451760650fbfe0f67bdf9cbd0c9e51c91222202e349033208337c5b9ab4537bc2dd6723be09b1fc7a9ec67" size="41848" version="1.0.1"/> </addons> </updates>
Comment 27•6 years ago
|
||
> Not eligable for Pioneer study. Will uninstall the study add-on. This is a restriction of pioneer-utils. You'll need to install the pioneer opt-in extension: https://addons.mozilla.org/en-US/firefox/addon/firefox-pioneer/ System add-ons need signed in a different way, so this XPI isn't won't be usable as a system add-on. You would have run into the same issue as well if it had installed.
Comment 28•6 years ago
|
||
Aftre I am not getting ping from "shield-study-addon"(In reply to Mike Cooper [:mythmon] from comment #27) > > Not eligable for Pioneer study. Will uninstall the study add-on. > > This is a restriction of pioneer-utils. You'll need to install the pioneer > opt-in extension: > https://addons.mozilla.org/en-US/firefox/addon/firefox-pioneer/ This fixes xpi installation issue. Thanks But I am not getting any ping from the shield-study-addon even after few minutes of wait. Here is screen capture: https://testing-1.tinytake.com/sf/MjE1MjEwN182NzA5Nzgy
Flags: needinfo?(mcooper)
Comment 29•6 years ago
|
||
I can see the payload data now. I found updated test steps here:https://github.com/motin/esper-pioneer-shield-study/blob/use-pioneer-utils/TESTPLAN.md We will test and update this bug accordingly soon.
Flags: needinfo?(mlopatka)
Flags: needinfo?(mcooper)
Comment 30•6 years ago
|
||
We have retested this with the latest addon from comment 24. Our test shows green with the exception of having null values for the following fields in beta (58.0b7) nightly (59.0a1)channels: “spbeNavigationAboutNewtab": "null", "spbeNavigationContextmenu": "null", "spbeNavigationSearchbar": "null", "spbeNavigationUrlbar": "null", “spbeWindowOpenEventCount": "null", "systemGfxMonitors1ScreenWidthZeroIndexed": "null", Release (57.0) shows non-null values for these fields. Test cases and runs are here: https://docs.google.com/a/mozilla.com/document/d/1KblMrZYzhUwVisNc3Ubixkvs45LZPsTQVSqKITJUqnQ/edit?usp=sharing
Assignee | ||
Comment 31•6 years ago
|
||
This looks fine to me (expected behavior, mostly covered in comment #17). Can we ship this ASAP to get as big a data collection window as possible?
Flags: needinfo?(mgrimes)
Please go ahead, looks like this was QA signed off and hits a small segment of release audience.
Comment 33•6 years ago
|
||
DAU CEP counter deployed prior to launch: https://pipeline-cep.prod.mozaws.net/dashboard_output/analysis.jgaunt_mozilla_com.esper_pioneer_shield_study_estimates.esper_pioneer_shield_study_count.json
Updated•6 years ago
|
Flags: needinfo?(mgrimes)
Comment 34•6 years ago
|
||
@mlopatka - Now that this study has ended an you recap the results here and close this bug?
Flags: needinfo?(mlopatka)
Assignee | ||
Comment 35•6 years ago
|
||
@Matt, I'm holding off on the analysis until the weirdness encountered in https://bugzilla.mozilla.org/show_bug.cgi?id=1415393 is diagnosed. I'd like to know if the sample we got suffers from the same issues and needs to be rerun. Discussions I've seen seem to indicate that they are close to a confident assessment.
Flags: needinfo?(mlopatka)
Comment 36•6 years ago
|
||
Works for me. Thanks for the update!
Attachment #8928668 -
Attachment is obsolete: true
Attachment #8928694 -
Attachment is obsolete: true
Attachment #8931332 -
Attachment is obsolete: true
Attachment #8931575 -
Attachment is obsolete: true
Attachment #8931933 -
Attachment is obsolete: true
Attachment #8932564 -
Attachment is obsolete: true
Comment 37•5 years ago
|
||
experiment has completed.
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•