987232 - Ingest FxOS telemetry submissions

Status: RESOLVED FIXED

(Webtools Graveyard :: Telemetry Server, defect)

Description

[Mark Reid [:mreid]]

According to :marshall_law, FirefoxOS pings will be submitted to a URL of the form:
https://fxos.telemetry.mozilla.org/submit/telemetry/ftu-ping/FirefoxOS/30.0a1/default/20140212140455

The 2 easiest ways to get this up and running are:
a) Make "fxos.t.m.o" an alias for "incoming.t.m.o", which would send all the data to the same S3 bucket as all the rest of telemetry. We can modify the server config to treat the fxos reason for "first time use" (ftu-ping in the example above) as a separate dimension, not bucketed in with OTHER, for easy filtering.

b) Deploy a separate fxos-specific instance of the telemetry-server stack, which would send the FxOS data to an S3 bucket of your choice.

If the payloads will be compatible, I recommend option a).  If specialized handling will be required for the FxOS telemetry, we should probably go with option b).

Comment 1

[(dormant account)]

We should go with option a)

Comment 2

[Mark Reid [:mreid]]

I added the new "reason" and "appName" values per option a):
https://github.com/mozilla/telemetry-server/commit/9b3a98dd95f3de1d86f1fd380e852a67a09d3a64

I also verified that we don't have any existing "ftu-ping" values for reason, nor any existing "FirefoxOS" values for appName.  As new submissions come in, they will be easily filtered from other Telemetry data.

Once the DNS alias is added per bug 987922, we'll be good to go.

Comment 3

[Marshall Culpepper [:marshall_law]]

Quick update..

My client side patch just landed in gaia master, and uses the following format:
https://fxos.telemetry.mozilla.org/submit/telemetry/ftu/FirefoxOS/31.0a1/default/20140325104133

So "reason" changes to just "ftu".. this was actually done at Taras' suggestion :)

Some other notes:
- If for some strange reason there is no platform version / update channel / platform build ID, these URI components will have the default value of "unknown"

- We also need to make sure that the ping ID from the payload is removed once we process the data. I assume this can happen somehow as a fork or copy of the custom python processing code?

Comment 4

[Mark Reid [:mreid]]

(In reply to Marshall Culpepper [:marshall_law] from comment #3)
> Quick update..
> 
> My client side patch just landed in gaia master, and uses the following
> format:
> https://fxos.telemetry.mozilla.org/submit/telemetry/ftu/FirefoxOS/31.0a1/
> default/20140325104133
> 
> So "reason" changes to just "ftu".. this was actually done at Taras'
> suggestion :)

OK, I'll update this in the schema
 
> Some other notes:
> - If for some strange reason there is no platform version / update channel /
> platform build ID, these URI components will have the default value of
> "unknown"

Perfect.
 
> - We also need to make sure that the ping ID from the payload is removed
> once we process the data. I assume this can happen somehow as a fork or copy
> of the custom python processing code?

Normally the ping ID is part of the URL (/submit/telemetry/<doc_id>/reason/appName/appVersion/channel/buildid).  I was planning to add a server-assigned id for these FxOS documents.  Is there an additional ping ID inside the payload that needs to be removed?

Comment 5

[Marshall Culpepper [:marshall_law]]

(In reply to Mark Reid [:mreid] from comment #4)
> Normally the ping ID is part of the URL
> (/submit/telemetry/<doc_id>/reason/appName/appVersion/channel/buildid).  I
> was planning to add a server-assigned id for these FxOS documents.  Is there
> an additional ping ID inside the payload that needs to be removed?

Yeah, it is under the key "pingID" inside the JSON payload

Comment 6

[Mark Reid [:mreid]]

Looking at the example ping in comment 2 on bug 969101, it looks like the FxOS pings will definitely require custom processing in their current form.

At minimum, the following changes would be required to make the example ping compatible with the telemetry document format:
- Add a top-level "ver" key to identify the payload version (Firefox and friends submit "ver": 1)
- Add a top-level "info" key with the basic dimensions used to partition the data: reason, appName, appVersion, appUpdateChannel, appBuildID

The "info" object would not add any new information, it would just reorganize how things are structured in the ftu payload. Ideally, other pieces of information could be moved into the "info" object using the same key names as other v1 payloads (to simplify consuming the data).

So a compatible version of the example payload from bug 969101 would look like:
{
    "ver": 1,
    "info": {
      "appUpdateChannel": "default",
      "appBuildID": "20140212140455",
      "appName": "FirefoxOS",
      "appVersion": "30.0a1",
      "reason": "ftu",
      "OS":"Boot2Gecko",
      "version":"1.4.0.0-prerelease"
    },
    "devicePixelRatio":1,
    "firmwareRevision":"",
    "hardware":"qcom",
    "icc":{
        "mcc":"310",
        "mnc":"410",
        "spn":null
    },
    "network":{
        "operator":null
    },
    "pingID":"dd314283-2d30-4a01-8bb4-4d7b494f43ca",
    "productModel":"ALCATEL ONE TOUCH FIRE",
    "screenHeight":480,
    "screenWidth":320
}

All that being said, I don't want to hold up development on the FxOS end, so if this is going to be a major inconvenience I can either do this rearranging on the server side, or detect the FxOS pings and do some custom processing to store them according to your requirements.

Comment 7

[Mark Reid [:mreid]]

Another easier possibility would be to just add a top-level key for "ver": 3 for this payload format.

Then I can easily detect it and translate it accordingly.

Comment 8

[Mark Reid [:mreid]]

Required changes for handling the FxOS pings merged in PR: https://github.com/mozilla/telemetry-server/pull/42

Next I need to deploy the changes to prod.

Comment 9

[Mark Reid [:mreid]]

The changes have been deployed.