1282968 - Redash android_events_v1.experiments missing onboarding3-a

Status: RESOLVED WONTFIX

(Firefox for Android Graveyard :: General, defect)

Description

[Chenxia Liu [:liuche]]

Attachment: Core ping from gzipserver

From the discussion in bug 1268641, Barbara was running some Redash queries and wasn't getting any onboarding3-a events so I was investigating:
https://bugzilla.mozilla.org/show_bug.cgi?id=1268641#c2

Locally I'm seeing onboarding3-a in core pings uploaded to my gzipserver (see attachment), but they don't seem to be present in Redash. mcomella mentions that the telemetry payloads are processed before being dumped into Redash, so I'm wondering if the 'experiments' json array is being truncated or mangled somehow.

For what I tried:
Running some queries myself, the 'experiments' json array seemed suspiciously short [1] because I expected every json array to at least include an onboarding event [2], so I set up the local gzip server [3] and was able to see the onboarding3-a experiment. This is code that merged to m-c 5/17 and there haven't been changes to the telemetry code, so this should be present in the Redash database.

[1] https://sql.telemetry.mozilla.org/queries/603/source
[2] https://dxr.mozilla.org/mozilla-central/source/mobile/android/base/java/org/mozilla/gecko/telemetry/pingbuilders/TelemetryCorePingBuilder.java#91
[3] https://wiki.mozilla.org/Mobile/Fennec/Android/Java_telemetry

Comment 1

[Chenxia Liu [:liuche]]

Georg, I'm seeing some inconsistencies between a local core ping payload and queries from in Redash - do you know about how the core ping data is processed and dumped into Redash, and if that might explain the discrepancies I'm seeing in the 'experiments' json array?

Comment 2

[Chenxia Liu [:liuche]]

Talked to mfinkle about this.

Core ping includes all possible experiments, not just the active ones, and the json array is pulled directly from the ping [1]. However, the active experiments shown in android_events_v1 are the active experiments (e.g., ones where 1) a session is active and 2) an event occurs that is tagged with that session).

Looks like all we need to do for this bug is send an event instead of just starting and stopping the session.

[1] https://github.com/mozilla-services/switchboard-experiments

Comment 3

[Chenxia Liu [:liuche]]

Oops, [1] should be https://github.com/mfinkle/user-data-analytics/blob/master/mobile-clients.ipynb

Comment 4

[Chenxia Liu [:liuche]]

Attachment: Bug 1282968 - Redash android_events_v1.experiments missing onboarding3-a.

Review commit: https://reviewboard.mozilla.org/r/61702/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/61702/

Comment 5

[Barbara Bermes [:barbara] - NI please!]

(In reply to Chenxia Liu [:liuche] from comment #2)
> Talked to mfinkle about this.
> 
> Core ping includes all possible experiments, not just the active ones, and
> the json array is pulled directly from the ping [1]. However, the active
> experiments shown in android_events_v1 are the active experiments (e.g.,
> ones where 1) a session is active and 2) an event occurs that is tagged with
> that session).
> 
> Looks like all we need to do for this bug is send an event instead of just
> starting and stopping the session.
> 
> [1] https://github.com/mozilla-services/switchboard-experiments

Thanks for looking into this Chenxia, is this you who can setup to send the event?

Comment 6

[Barbara Bermes [:barbara] - NI please!]

Oppps I guess you did already looking at comment 4, nevermind

Comment 7

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Comment on attachment 8767320 [details]
Bug 1282968 - Redash android_events_v1.experiments missing onboarding3-a.

https://reviewboard.mozilla.org/r/61702/#review59274

Given your analysis, this looks like it'd do the trick.

That being said:
> However, the active experiments shown in android_events_v1 are the active experiments (e.g., ones where 1) a session is active

Makes sense.
> and 2) an event occurs that is tagged with that session).

This seems unexpected to me – I'd expect that we could identify active experiments by client ID. Should we modify the behavior? Is this an issue with the data itself (e.g. in Spark) or with the translation to re:dash?

At the very least, we should consider:

* Adding a comment to the code to explain why these telemetry events are there so they're not removed later – you can avoid comment duplication by having each experiment call out to a single method, which has the comment instead.
* Updating the [re:dash guide](https://wiki.mozilla.org/Mobile/Metrics/Redash#Data_Tables_.2F_Schemas) to explain this behavior
* Announcing this behavior more broadly (e.g. mailing list, things I learned this week Friday meeting)

Comment 8

[Chenxia Liu [:liuche]]

Barbara: actually, talking to mcomella, I had the realization that the reason the onboarding isn't showing up is because the query is for events, and we don't send any onboarding3-a events. Could you use mobile_clients_v1 instead? That's the opt-out client data. I see 3-a present on the query I'm running below. I think that's a better solution to adding the event probe, and that means we don't need to uplift anything.

https://sql.telemetry.mozilla.org/queries/603/source

Comment 9

[Barbara Bermes [:barbara] - NI please!]

Actually, good catch, this seems to work! Thanks.

Comment 10

[Chenxia Liu [:liuche]]

We don't actually need to add onboarding to android_events_v1 because they are already present in mobile_clients_v1.