1181876 - Make Hello pings easily accessible in Spark

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Katie Parlante]

Hello telemetry probes here: https://bugzilla.mozilla.org/show_bug.cgi?id=1177137

Create Hello specific data set to load redshift table, output as csv, etc.

Comment 1

[Katie Parlante]

Saptarshi and Gareth, it would be great if you could give us a list of the fields that you want from telemetry for hello analysis, and so that we can create a stream for you that is easier to process/analyze.

Comment 2

["Saptarshi Guha[:joy]"]

Hi Katie,
I'm going to say that 
-clientId
-enviroment
-application
-creationdate

-meta#geoCountry
-meta#os
-meta#Timestamp

-payload[ 'subsessionCounter','profileSubsessionCounter','subsessionStartDate','sessionStartDate','sessionLength','timezoneOffset']

- simpleMeasurements#['activeTicks','uptime','totalTime']

Gareth can provide the Histograms for Loop
-

Comment 3

[Gareth Cull [:garethc]]

Hi Katie,

Thanks for looking into this for us. I need the following data for release channel based on this json file:
https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/Histograms.json

Can I get these pieces of data broken down by what Saptarshi requested. Could I also get this rolled up by week and by release window or month if possible?

  "LOOP_CLIENT_CALL_URL_REQUESTS_SUCCESS": {
    "expires_in_version": "never",
    "kind": "boolean",
    "description": "Stores 1 if generating a call URL succeeded, and 0 if it failed."
  },
  "LOOP_CLIENT_CALL_URL_SHARED": {
    "expires_in_version": "never",
    "kind": "boolean",
    "description": "Stores 1 every time the URL is copied or shared."
  },
  "LOOP_TWO_WAY_MEDIA_CONN_LENGTH_1": {
    "alert_emails": ["[e-mail]", "[e-mail]"],
    "expires_in_version": "45",
    "kind": "enumerated",
    "n_values": 8,
    "releaseChannelCollection": "opt-out",
    "description": "Connection length for bi-directionally connected media (0=SHORTER_THAN_10S, 1=BETWEEN_10S_AND_30S, 2=BETWEEN_30S_AND_5M, 3=MORE_THAN_5M)"
  },
  "LOOP_SHARING_STATE_CHANGE_1": {
    "alert_emails": ["[e-mail]", "[e-mail]"],
    "expires_in_version": "45",
    "kind": "enumerated",
    "n_values": 8,
    "releaseChannelCollection": "opt-out",
    "description": "Number of times the sharing feature has been enabled and disabled (0=WINDOW_ENABLED, 1=WINDOW_DISABLED, 2=BROWSER_ENABLED, 3=BROWSER_DISABLED)"
  },
  "LOOP_SHARING_ROOM_URL": {
    "alert_emails": ["[e-mail]", "[e-mail]"],
    "expires_in_version": "45",
    "kind": "enumerated",
    "n_values": 8,
    "releaseChannelCollection": "opt-out",
    "description": "Number of times a room URL is shared (0=COPY_FROM_PANEL, 1=COPY_FROM_CONVERSATION, 2=EMAIL_FROM_CALLFAILED, 3=EMAIL_FROM_CONVERSATION)"
  },
  "LOOP_ROOM_CREATE": {
    "alert_emails": ["[e-mail]", "[e-mail]"],
    "expires_in_version": "45",
    "kind": "enumerated",
    "n_values": 4,
    "releaseChannelCollection": "opt-out",
    "description": "Number of times a room create action is performed (0=CREATE_SUCCESS, 1=CREATE_FAIL)"
  },
  "LOOP_ROOM_DELETE": {
    "alert_emails": ["[e-mail]", "[e-mail]"],
    "expires_in_version": "45",
    "kind": "enumerated",
    "n_values": 4,
    "releaseChannelCollection": "opt-out",
    "description": "Number of times a room delete action is performed (0=DELETE_SUCCESS, 2=DELETE_FAIL)"
  },
  "LOOP_ROOM_CONTEXT_ADD": {
    "alert_emails": ["[e-mail]", "[e-mail]"],
    "expires_in_version": "45",
    "kind": "enumerated",
    "n_values": 8,
    "releaseChannelCollection": "opt-out",
    "description": "Number of times a room context action is performed (0=ADD_FROM_PANEL, 1=ADD_FROM_CONVERSATION)"
  },
  "LOOP_ROOM_CONTEXT_CLICK": {
    "alert_emails": ["[e-mail]", "[e-mail]"],
    "expires_in_version": "45",
    "kind": "count",
    "releaseChannelCollection": "opt-out",
    "description": "Number times room context is clicked to visit the attached URL"
  },
  "LOOP_ROOM_SESSION_WITHCHAT": {
    "alert_emails": ["[e-mail]", "[e-mail]"],
    "expires_in_version": "45",
    "kind": "count",
    "releaseChannelCollection": "opt-out",
    "description": "Number of sessions where at least one chat message was exchanged"

Gareth

Comment 4

[Katie Parlante]

whd is going to create a filter to drop loop data in an s3 bucket

Comment 5

[Wesley Dawson [:whd]]

We now have a derived loop stream, accessible from spark with get_records:

pings = get_records(sc, "telemetry-loop", fraction=1)

The match is for any histograms beginning with "LOOP_". Notably, no LOOP_CLIENT_* histograms appeared to exist in the wild.

The derived stream has data from 2015-12-10 onward and not been backfilled (but can be).