1457388 - Telemetry QA coverage for the GeckoView consumer application

Status: RESOLVED FIXED

(Toolkit :: Telemetry, enhancement, P3)

Description

[Alessio Placitelli [:Dexter]]

In bug 1454902 we'll validate how Telemetry behaves up to the GV API level. We might need to make sure to run a QA pass on the end application as well (Klar). Klar already seems to have a QA process/testplan in place: https://wiki.mozilla.org/QA/Fennec/Klar%2BGeckoView

This bug could just be about adding a few new test cases to that:

- Making sure that pings are correctly assembled;
- Making sure that pings are sent at the right time.

Comment 1

[Alessio Placitelli [:Dexter]]

Anything else we should call out here?

Comment 2

[Georg Fritzsche [:gfritzsche]]

Maybe not all of the below needs to be in QA testing, but here are some thoughts.

Some light testing around some edge scenarios (or combinations thereof) seem interesting to me:
- app getting killed
- ping is triggered close to app launch (e.g. GeckoView not yet started / GeckoView just started / ...)

We should use (say) 1-5 important performance metrics, for which to confirm their values make sense in standard scenarios.

Comment 3

[Frank Bertsch [:frank]]

Also to consider for QA:
- upload limits on number of pings

Luckily this library has been in use for >1 year now, so most of the semantics around sending of pings are already pretty well vetted. Bug 1452166 could also be useful for QA.

Comment 4

[Chris H-C :chutten]

One thing that I noticed in review of the persistence layer is that we now have a longer period between app start and Telemetry being ready. Possibly testing for probes recorded early would be illuminating?

Comment 5

[Chris Peterson [:cpeterson]]

Adding whiteboard tag [geckoview:klar:p1] because Alessio says this bug should blocker Klar+GeckoView release. This bug is to make sure that Klar correctly sends the data out.

Comment 6

[Chris Peterson [:cpeterson]]

Emily or Alessio, is Klar+GeckoView telemetry now working end-to-end? Should I submit a PI Request asking QA to test Klar+GeckoView telemetry now?

Where is the Klar+GeckoView telemetry data available? Is it mixed in with the "Firefox Mobile" Product on telemetry.mozilla.org?

Comment 7

[Alessio Placitelli [:Dexter]]

(In reply to Chris Peterson [:cpeterson] from comment #6)
> Emily or Alessio, is Klar+GeckoView telemetry now working end-to-end? Should
> I submit a PI Request asking QA to test Klar+GeckoView telemetry now?
> 
> Where is the Klar+GeckoView telemetry data available? Is it mixed in with
> the "Firefox Mobile" Product on telemetry.mozilla.org?

That's more of an Emily question :)

Comment 8

[Emily Kager [:ekager]]

Hey Chris, 
This should be working end-to-end now and we can test it! I believe we do use telemetry.mozilla.org (for example https://sql.telemetry.mozilla.org/dashboard/android-focus-vs-klar), but I don't think we have any GV specific queries up yet.

Comment 9

[David Bolter [:davidb] (NeedInfo me for attention)]

Chris will follow up.

Comment 10

[Chris Peterson [:cpeterson]]

This is an administrative bug that does not need to block Focus+GV.

Comment 11

[Chris Peterson [:cpeterson]]

:dexter shared a Redash query to the mobile metrics dataset, so it appears the telemetry data is being reported as expected.

https://sql.telemetry.mozilla.org/queries/57750/source