Redash android_events_v1.experiments missing onboarding3-a

RESOLVED WONTFIX

Status

()

RESOLVED WONTFIX
2 years ago
2 years ago

People

(Reporter: liuche, Assigned: liuche)

Tracking

(Blocks: 1 bug)

Trunk
ARM
Android
Points:
---

Firefox Tracking Flags

(firefox49 affected, firefox50 affected)

Details

Attachments

(2 attachments)

(Assignee)

Description

2 years ago
Created attachment 8766100 [details]
Core ping from gzipserver

From the discussion in bug 1268641, Barbara was running some Redash queries and wasn't getting any onboarding3-a events so I was investigating:
https://bugzilla.mozilla.org/show_bug.cgi?id=1268641#c2

Locally I'm seeing onboarding3-a in core pings uploaded to my gzipserver (see attachment), but they don't seem to be present in Redash. mcomella mentions that the telemetry payloads are processed before being dumped into Redash, so I'm wondering if the 'experiments' json array is being truncated or mangled somehow.

For what I tried:
Running some queries myself, the 'experiments' json array seemed suspiciously short [1] because I expected every json array to at least include an onboarding event [2], so I set up the local gzip server [3] and was able to see the onboarding3-a experiment. This is code that merged to m-c 5/17 and there haven't been changes to the telemetry code, so this should be present in the Redash database.

[1] https://sql.telemetry.mozilla.org/queries/603/source
[2] https://dxr.mozilla.org/mozilla-central/source/mobile/android/base/java/org/mozilla/gecko/telemetry/﷒0﷓
[3] https://wiki.mozilla.org/Mobile/Fennec/Android/Java_telemetry
(Assignee)

Comment 1

2 years ago
Georg, I'm seeing some inconsistencies between a local core ping payload and queries from in Redash - do you know about how the core ping data is processed and dumped into Redash, and if that might explain the discrepancies I'm seeing in the 'experiments' json array?
Flags: needinfo?(gfritzsche)
(Assignee)

Updated

2 years ago
Flags: needinfo?(kparlante)
(Assignee)

Comment 2

2 years ago
Talked to mfinkle about this.

Core ping includes all possible experiments, not just the active ones, and the json array is pulled directly from the ping [1]. However, the active experiments shown in android_events_v1 are the active experiments (e.g., ones where 1) a session is active and 2) an event occurs that is tagged with that session).

Looks like all we need to do for this bug is send an event instead of just starting and stopping the session.

[1] https://github.com/mozilla-services/switchboard-experiments
Flags: needinfo?(kparlante)
Flags: needinfo?(gfritzsche)
(Assignee)

Comment 4

2 years ago
Created attachment 8767320 [details]
Bug 1282968 - Redash android_events_v1.experiments missing onboarding3-a.

Review commit: https://reviewboard.mozilla.org/r/61702/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/61702/
Attachment #8767320 - Flags: review?(michael.l.comella)
(Assignee)

Updated

2 years ago
status-firefox49: --- → affected
(Assignee)

Updated

2 years ago
Assignee: nobody → liuche
(In reply to Chenxia Liu [:liuche] from comment #2)
> Talked to mfinkle about this.
> 
> Core ping includes all possible experiments, not just the active ones, and
> the json array is pulled directly from the ping [1]. However, the active
> experiments shown in android_events_v1 are the active experiments (e.g.,
> ones where 1) a session is active and 2) an event occurs that is tagged with
> that session).
> 
> Looks like all we need to do for this bug is send an event instead of just
> starting and stopping the session.
> 
> [1] https://github.com/mozilla-services/switchboard-experiments

Thanks for looking into this Chenxia, is this you who can setup to send the event?
Flags: needinfo?(liuche)
Oppps I guess you did already looking at comment 4, nevermind
Comment on attachment 8767320 [details]
Bug 1282968 - Redash android_events_v1.experiments missing onboarding3-a.

https://reviewboard.mozilla.org/r/61702/#review59274

Given your analysis, this looks like it'd do the trick.

That being said:
> However, the active experiments shown in android_events_v1 are the active experiments (e.g., ones where 1) a session is active

Makes sense.
> and 2) an event occurs that is tagged with that session).

This seems unexpected to me – I'd expect that we could identify active experiments by client ID. Should we modify the behavior? Is this an issue with the data itself (e.g. in Spark) or with the translation to re:dash?

At the very least, we should consider:

* Adding a comment to the code to explain why these telemetry events are there so they're not removed later – you can avoid comment duplication by having each experiment call out to a single method, which has the comment instead.
* Updating the [re:dash guide](https://wiki.mozilla.org/Mobile/Metrics/Redash#Data_Tables_.2F_Schemas) to explain this behavior
* Announcing this behavior more broadly (e.g. mailing list, things I learned this week Friday meeting)
Attachment #8767320 - Flags: review?(michael.l.comella) → review+
Flags: needinfo?(liuche)
(Assignee)

Comment 8

2 years ago
Barbara: actually, talking to mcomella, I had the realization that the reason the onboarding isn't showing up is because the query is for events, and we don't send any onboarding3-a events. Could you use mobile_clients_v1 instead? That's the opt-out client data. I see 3-a present on the query I'm running below. I think that's a better solution to adding the event probe, and that means we don't need to uplift anything.

https://sql.telemetry.mozilla.org/queries/603/source
(Assignee)

Updated

2 years ago
Flags: needinfo?(bbermes)
Actually, good catch, this seems to work! Thanks.
Flags: needinfo?(bbermes)
(Assignee)

Comment 10

2 years ago
We don't actually need to add onboarding to android_events_v1 because they are already present in mobile_clients_v1.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.