1520794 - Investigate attribution value incoherence on DSMO-RS

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect, P1)

Description

[Romain Testard [:RT]]

Per https://sql.telemetry.mozilla.org/queries/60892/source#157103 only 1% of stub installs have attribution (DSMO-RS represents stub ping data) whereas Su's analysis on telemetry shows 18.8% of new profiles with attribution on https://metrics.mozilla.com/protected/shong/shong-reports/firefox-acquisition/darkfunnel/tiger-team-profile-categorization/profile-categorizations.html

This means to me that attribution is working as expected but either the stub ping does not report it (very long strings or strange characters may break it?) or the data pipeline processing the stub ping filters out some of it and interprets as "0" or "null".

This bug is about investigating where the attribution data may break in the data pipeline processing.

Comment 1

[Mark Reid [:mreid]]

Blake, is this something you'd look into?

Comment 2

[Blake Imsland [:robotblake]]

I think this'll be me, I'll get it triaged.

Comment 3

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

A couple drive-by questions:

1. Is https://github.com/whd/dsmo_load the right place to look?
2. Is https://github.com/whd/dsmo_load/blob/master/heka/usr/share/heka/lua_filters/nginx_redshift.lua#L173-L176 doing the right thing -- should that conditional be "v7 or greater" rather than "exactly v7"?

Comment 4

[Mark Reid [:mreid]]

ni :whd re: Tim's questions

Comment 5

[Wesley Dawson [:whd]]

(In reply to Tim Smith 👨‍🔬 [:tdsmith] from comment #3)

A couple drive-by questions:

1. Is https://github.com/whd/dsmo_load the right place to look?
2. Is https://github.com/whd/dsmo_load/blob/master/heka/usr/share/heka/lua_filters/nginx_redshift.lua#L173-L176 doing the right thing -- should that conditional be "v7 or greater" rather than "exactly v7"?

I think these are both correct (that looks like a code bug), but :robotblake would know better.

Comment 6

[Wesley Dawson [:whd]]

This has been fixed by https://github.com/whd/dsmo_load/pull/4 and :robotblake is looking into backfilling the data.

Comment 7

[Blake Imsland [:robotblake]]

Hey all, as :whd said there was a bug where only v7 pings were getting attribution pushed into the db properly.

A fix has been deployed to the DSMO edge servers so data 20190130 forward should be good but I'll need to run a relatively large backfill to fix the data, potentially all the way back to mid-2017. Will update this bug as I get that all running.

Comment 8

[Blake Imsland [:robotblake]]

As a followup, I'm still processing the data, it's taking quite a bit of time since there is a lot of data, hoping to actually initiate the backfill sometime Friday.

Comment 9

[Romain Testard [:RT]]

For info I just re-ran the query and back-filled data does not show-up:
https://sql.telemetry.mozilla.org/queries/60892#157103

Comment 10

[Blake Imsland [:robotblake]]

The backfill had a failure over the weekend and never completed, I've been working on trying to figure out why which is still in progress. Hoping to start a run over night to see if the changes I've made address the issues.

Comment 11

[Blake Imsland [:robotblake]]

Small update, data processing is still chugging along and appears to have gotten past the failure point. Looking like an ETA of sometime this afternoon or early evening and then I can start loading the data into redshift.

Comment 12

[Blake Imsland [:robotblake]]

Alright, so the data processing finally completed after several more fits and starts, I'm doing some validation / smoke-testing that the data looks okay and then I'll be cutting the tables over to the backfilled versions later tonight.

Comment 13

[Blake Imsland [:robotblake]]

Data has been backfilled back to the beginning of June 2017, one side-effect is that any ping that had empty or null attribution (normally v6 is null, and >= v7 is an empty string) all loaded as null. I don't think this is an issue in practice since versioning should be taken into account, and for all intents and purposes null and blank should be in the same "no attribution" category.

If there are any other questions about this or issues you see with the data, let me know!