1548420 - Firefox for Fire TV: verify updated home tile clicked telemetry matches old probe

Status: RESOLVED FIXED

(Data Science :: General, task)

Description

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Brief Description of the request (required):

We used to have two probes:

• User clicked home tile (bundled/custom)
• User clicked YouTube home tile (i.e. this is redundant)

In https://github.com/mozilla-mobile/firefox-tv/issues/2014 we merged the second probe into the first one by including an extra for, "which home tile was clicked."

In the follow-up https://github.com/mozilla-mobile/firefox-tv/issues/2091 we want to 1) use this data science request to verify the new probe gets the same data as the old probe (i.e. it was implemented correctly) and 2) remove the old probe.

Business purpose for this request (required):

We want to remove unnecessary probe to avoid collecting more data than we need to and clarify the existing code.

Requested timelines for the request or how this fits into roadmaps or critical decisions (required):

Ultimately, not urgent but ideally we can do it this sprint (i.e. over the next two weeks) to get it off of our minds and out of our roadmaps so we can focus on more important things.

Links to any assets (e.g Start of a PHD, BRD; any document that helps describe the project):

N/A, I think.

Name of Data Scientist (If Applicable):

Su has some context on these changes.

Please note if it is found that not enough information has been given this will delay the triage of this request.

Comment 1

[Su-Young Hong]

Update:

I've checked the updated behavior of the probe(s) and here are the findings:

• Counting Youtube tileclicks using the updated Bundled Hometile Probe produces results very, very close (within 0.1% for client counts) to counting using the Youtube Hometile Probe.
• Counting non-Youtube tileclicks explicitly using the updated Bundled Hometile Probe produces results that are non-trivially different (undercounts between 1%-2% for client counts) from counting implicitly using the Youtube Hometile Probe and the old Bundled Hometile Probe.
  • Part of the reason for this is there seems to be a bug where extra.tile_id field does not get filled properly (left as a null value) in a small number of cases.

Source: [Adhoc] FFTV-Youtube-Tileclicks-Change

Recommendation:

The error rate seems small enough to ignore going forward with the understanding that non-Youtube usage counts from before and after this change have this bias (the counts from after this change undercount about by about 1-2% relative to before this change), which would affect comparisons over time. If we're fine with having some degree of bias in our comparisons over time, we can turn off the Youtube Hometile Probe and proceed to count non-Youtube usage explicitly using the updated Bundled Hometile Probe.

However, If having very exact comparisons between times before and after this change is important, then we would need to continue sending and using the Youtube Hometile Probe and counting non-Youtube usage implicitly.

Comment 2

[Su-Young Hong]

Let me know how you guys want to proceed, and I'll update the dashboard accordingly (https://bugzilla.mozilla.org/show_bug.cgi?id=1525148)

Comment 3

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

(In reply to Su-Young Hong from comment #1)

- Part of the reason for this is there seems to be a bug where `extra.tile_id` field does not get filled properly (left as a null value) in a small number of cases. 

Looking at the source, we don't fill the extra.tile_id when the clicked tile is of type Pocket or Custom (i.e. user-created). You can distinguish these clicks based on the Value field.

Counting non-Youtube tileclicks explicitly using the updated Bundled Hometile Probe produces results that are non-trivially different (undercounts between 1%-2% for client counts)

Looking at the telemetry source before we made these telemetry changes, it looks like the code should produce the same results, just without the extra. However, it's possible there's a bug in the code that calls out to this telemetry implementation. Also, since the telemetry change, we've added a new tile type – Pocket – but that's never shipped to production I'd think it'd cause us to over count, not under count.

Su, I'm confused that this is happening, given that the code looks the same. Are you using the same methodology to determine what is considered to be a bundled tile before and after? What is it (has a tile_id extra or the tile type value)? Do you have any ideas what could be causing this? Perhaps we should jump on Zoom to quickly discuss – let me know.

fwiw, if we're missing 1-2% of tiles clicks, it's believable to me that the old methodology might include custom tile clicks while the new one may not.

Comment 4

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

I followed up with Su IRL: Su brought me to a shared understanding of his analysis. It looks like there may be a bug in our current implementation [1] where extras is sometimes null for both YouTube tiles & non-YouTube bundled tiles. Our action items as the FFTV team are to decide between:

• Trying to fix the bug
• Accept the discrepancy in data; this would cause the data/dashboards to change when we ship the new implementation (despite no change in user behavior) so we might want to document this discrepancy for future analyses

Personally, I will briefly look into the code to see if I can identify the bug and, if not, consult Ashley about what we want to do.

Su & I decided to leave this bug open until my team fixes the bug or decides to accept the data discrepency.

[1]: Assuming no bugs in the server or telemetry client library, which we probably would have heard of by now

Comment 5

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

The Fire TV has deprioritized this issue in favor of urgent feature work: we'll come back and make a decision later.

Comment 6

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Su, I dug in a little bit more but I still can't find the bug. I notice the queries:

• Don't explicitly filter out non-release builds
• Will also include our beta builds (only tested by Mozillians), i.e. I think the version check substring(metadata.app_version, 1, 3) >= '3.8' will also include beta version 3.9-LAT1)

I'm concerned there may have been a bug in a local debug build or in a beta build that could be causing the data skew. Do you think this is possible?

Comment 7

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Another long shot possibility: could someone have forked the app without changing the app name and changed the code or bundled tile data?

Comment 8

[Su-Young Hong]

Hey Mike,

• 1: So the specific example cases that we looked at (where there is a youtube tileclick event, but corresponding bundled tileclick event isn't tagged in extra, and non-youtube bundled tileclicks are showing null in extra) are on version 3.9 (non-LATX), so I don't think beta leakage is responsible.
  • bundled tileclick, no extra
  • youtube tileclick, no extra
• 2: It's possible it's coming from testing scenarios, but I doubt it given the consistency over time and scale of the discrepancy.
• 3: Not sure about the forking, but it does seem unlikely.

It sounds like ID'ing / fixing the cause of the discrepancy is going to be a significant effort. Maybe we should skip to deciding if accepting the discrepancy is worth living with (and if not, then we can continue digging into the cause).