Closed Bug 1499851 Opened Last year Closed 10 months ago

Expectation of privacy study design and analysis

Categories

(Data Science :: Experiment Collaboration, task, P2)

task
Points:
2

Tracking

(data-science-status Resolved)

RESOLVED FIXED
Tracking Status
data-science-status --- Resolved

People

(Reporter: tdsmith, Assigned: tdsmith)

References

Details

This study is a partnership with the advocacy team to determine user expectations around location privacy. The study lead is Naomi Shiffman.

The study is expected to launch in November.
The study is currently blocked on IRB approval.
We received IRB approval on the 30th; Rob is working with Naomi on running the survey through usertesting and cleaning up the language indicating that telemetry will be linked to survey responses.
data-science-status: --- → Data Acquisition
The survey was deployed yesterday.

The recipe is:

    {
        "recipe": {
            "id": 654,
            "name": "Heartbeat: Location Policy (UCB)",
            "revision_id": "2106",
            "action": "show-heartbeat",
            "arguments": {
                "repeatOption": "once",
                "surveyId": "hb-location-policy-v0",
                "message": "Mozilla & UC Berkeley are interested in your opinion on location sharing on the web",
                "learnMoreMessage": "Learn More",
                "learnMoreUrl": "https://support.mozilla.org/kb/rate-your-firefox-experience-heartbeat",
                "engagementButtonLabel": "Take Survey",
                "thanksMessage": "Thanks",
                "postAnswerUrl": "https://qsurvey.mozilla.com/s3/Location-Policy-Research",
                "includeTelemetryUUID": false
            },
            "filter_expression": "(normandy.locale in [\"en-US\"]) && (normandy.country in [\"US\"]) && (normandy.channel in [\"release\"]) && ([normandy.recipe.id,normandy.userId]|stableSample(0.01)) && (!normandy.isFirstRun)"
        }

The survey was undeployed the next day; we want to both a) target to people who have recently seen geolocation prompts to improve response efficiency and b) link to telemetry. We're currently waiting on those changes. Setting P3 until results are in.

Points: --- → 2
Priority: -- → P3
Depends on: 1519170
Priority: P3 → P2

Results came in; thanks, Rob, for deploying and summarizing the results. Naomi is handling analysis. I spoke with Naomi this morning and she doesn't think that we'll want to link to telemetry yet, so I'll close this out for now.

The final recipes were:

    {
        "id": 666,
        "last_updated": "2019-01-24T19:10:33.149112Z",
        "name": "Heartbeat: Location Policy (UCB) [allowed]",
        "enabled": false,
        "is_approved": true,
        "revision_id": 2117,
        "action": "show-heartbeat",
        "arguments": {
            "repeatOption": "once",
            "surveyId": "hb-location-policy-v2b3",
            "message": "Mozilla & UC Berkeley are interested in your opinion on location sharing on the web",
            "learnMoreMessage": "Learn More",
            "learnMoreUrl": "https://support.mozilla.org/kb/rate-your-firefox-experience-heartbeat",
            "engagementButtonLabel": "Take Survey",
            "thanksMessage": "Thanks",
            "postAnswerUrl": "https://qsurvey.mozilla.com/s3/Location-Policy-Research?v=2&b=3",
            "includeTelemetryUUID": true
        },
        "extra_filter_expression": "!normandy.isFirstRun\n&& [normandy.userId, 'lp']|bucketSample(500, 100, 1000)\n&& !([654,normandy.userId]|stableSample(0.01))\n&& (\n  normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[10]>0\n  || normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[11]>0\n  || normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[12]>0\n)",
        "filter_expression": "(normandy.locale in [\"en-US\"]) && (normandy.country in [\"US\"]) && (normandy.channel in [\"release\"]) && ((normandy.version>=\"63\"&&normandy.version<\"64\")||(normandy.version>=\"64\"&&normandy.version<\"65\")) && (!normandy.isFirstRun\n&& [normandy.userId, 'lp']|bucketSample(500, 100, 1000)\n&& !([654,normandy.userId]|stableSample(0.01))\n&& (\n  normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[10]>0\n  || normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[11]>0\n  || normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[12]>0\n))",
        "latest_revision_id": 2117,
        "approved_revision_id": 2117,
        "approval_request": {
            "id": 1346,
            "created": "2019-01-11T23:30:59.394285Z",
            "creator": {
                "id": 4,
                "first_name": "Robert",
                "last_name": "Rayborn",
                "email": "rrayborn@mozilla.com"
            },
            "approved": true,
            "approver": {
                "id": 6,
                "first_name": "Josh",
                "last_name": "Gaunt",
                "email": "jgaunt@mozilla.com"
            },
            "comment": "approved revision of filtering telemetry"
        },
        "identicon_seed": "v1:yf06kuflo0j"
    },
    {
        "id": 665,
        "last_updated": "2019-01-24T19:10:29.927183Z",
        "name": "Heartbeat: Location Policy (UCB) [not-allowed]",
        "enabled": false,
        "is_approved": true,
        "revision_id": 2120,
        "action": "show-heartbeat",
        "arguments": {
            "repeatOption": "once",
            "surveyId": "hb-location-policy-v2b2",
            "message": "Mozilla & UC Berkeley are interested in your opinion on location sharing on the web",
            "learnMoreMessage": "Learn More",
            "learnMoreUrl": "https://support.mozilla.org/kb/rate-your-firefox-experience-heartbeat",
            "engagementButtonLabel": "Take Survey",
            "thanksMessage": "Thanks",
            "postAnswerUrl": "https://qsurvey.mozilla.com/s3/Location-Policy-Research?v=2&b=2",
            "includeTelemetryUUID": true
        },
        "extra_filter_expression": "!normandy.isFirstRun\n&& [normandy.userId, 'lp']|bucketSample(250, 100, 1000)\n&& !([654,normandy.userId]|stableSample(0.01))\n&& (\n  normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[0]>0\n  || normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[1]>0\n  || normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[2]>0\n) && !(\n  normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[10]>0\n  || normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[11]>0\n  || normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[12]>0\n)",
        "filter_expression": "(normandy.locale in [\"en-US\"]) && (normandy.country in [\"US\"]) && (normandy.channel in [\"release\"]) && ((normandy.version>=\"63\"&&normandy.version<\"64\")||(normandy.version>=\"64\"&&normandy.version<\"65\")) && (!normandy.isFirstRun\n&& [normandy.userId, 'lp']|bucketSample(250, 100, 1000)\n&& !([654,normandy.userId]|stableSample(0.01))\n&& (\n  normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[0]>0\n  || normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[1]>0\n  || normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[2]>0\n) && !(\n  normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[10]>0\n  || normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[11]>0\n  || normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[12]>0\n))",
        "latest_revision_id": 2120,
        "approved_revision_id": 2120,
        "approval_request": {
            "id": 1349,
            "created": "2019-01-12T01:21:02.521785Z",
            "creator": {
                "id": 4,
                "first_name": "Robert",
                "last_name": "Rayborn",
                "email": "rrayborn@mozilla.com"
            },
            "approved": true,
            "approver": {
                "id": 6,
                "first_name": "Josh",
                "last_name": "Gaunt",
                "email": "jgaunt@mozilla.com"
            },
            "comment": "approved revision of filtering telemetry"
        },
        "identicon_seed": "v1:o2fglqagcb"
    },
    {
        "id": 664,
        "last_updated": "2019-01-24T19:10:27.081676Z",
        "name": "Heartbeat: Location Policy (UCB) [not-seen]",
        "enabled": false,
        "is_approved": true,
        "revision_id": 2119,
        "action": "show-heartbeat",
        "arguments": {
            "repeatOption": "once",
            "surveyId": "hb-location-policy-v2b1",
            "message": "Mozilla & UC Berkeley are interested in your opinion on location sharing on the web",
            "learnMoreMessage": "Learn More",
            "learnMoreUrl": "https://support.mozilla.org/kb/rate-your-firefox-experience-heartbeat",
            "engagementButtonLabel": "Take Survey",
            "thanksMessage": "Thanks",
            "postAnswerUrl": "https://qsurvey.mozilla.com/s3/Location-Policy-Research?v=2&b=1",
            "includeTelemetryUUID": true
        },
        "extra_filter_expression": "!normandy.isFirstRun\n&& [normandy.userId, 'lp']|bucketSample(0, 5, 1000)\n&& !([654,normandy.userId]|stableSample(0.01))\n&& !(\n  normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[0]>0\n  || normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[1]>0\n  || normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[2]>0\n  || normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[10]>0\n  || normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[11]>0\n  || normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[12]>0\n)",
        "filter_expression": "(normandy.locale in [\"en-US\"]) && (normandy.country in [\"US\"]) && (normandy.channel in [\"release\"]) && ((normandy.version>=\"63\"&&normandy.version<\"64\")||(normandy.version>=\"64\"&&normandy.version<\"65\")) && (!normandy.isFirstRun\n&& [normandy.userId, 'lp']|bucketSample(0, 5, 1000)\n&& !([654,normandy.userId]|stableSample(0.01))\n&& !(\n  normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[0]>0\n  || normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[1]>0\n  || normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[2]>0\n  || normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[10]>0\n  || normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[11]>0\n  || normandy.telemetry.main.payload.processes.content.histograms.GEOLOCATION_REQUEST_GRANTED.values[12]>0\n))",
        "latest_revision_id": 2119,
        "approved_revision_id": 2119,
        "approval_request": {
            "id": 1348,
            "created": "2019-01-11T23:32:37.548552Z",
            "creator": {
                "id": 4,
                "first_name": "Robert",
                "last_name": "Rayborn",
                "email": "rrayborn@mozilla.com"
            },
            "approved": true,
            "approver": {
                "id": 6,
                "first_name": "Josh",
                "last_name": "Gaunt",
                "email": "jgaunt@mozilla.com"
            },
            "comment": "approved revision of filtering telemetry"
        },
        "identicon_seed": "v1:n3lxplr3csh"
    },
Status: ASSIGNED → RESOLVED
data-science-status: Data Acquisition → Resolved
Closed: 10 months ago
Resolution: --- → FIXED
Depends on: 1544539
You need to log in before you can comment on or make changes to this bug.