1541635 - Make nsITelemetryGeckoViewTesting more useful for Performance Testing in automation

Status: NEW

(Toolkit :: Telemetry, defect, P3)

Description

[Nick Alexander :nalexander [he/him] (PTO August 22-September 12)]

In the Performance Team, we want to be able to extract telemetry, specifically gv_measurements.json, in automation tooling for evaluating our Android products. Right now, forcePersist can be invoked and the measurements written to disk. But that API isn't the best it could be:

• there's no promise telling when the write is complete
• there's a file in the middle that doesn't need to be there

Could we have an additional API that just serializes to a string (or a promise of a string)?

If we get such an API, it's easier to make browsertime capture gv_measurements.json as part of its test measurements. browsertime is a Web Driver thing, so this would be done by evaluating some privileged JS in a chrome context.

(This is similar in spirit to making Raptor capture such telemetry as well. Raptor uses a Web Extension, so that would be done by adding a new Web Extension experimental API just for that Web Extension. Under the hood, it would need exactly this functionality.)

Comment 1

[Nick Alexander :nalexander [he/him] (PTO August 22-September 12)]

If this is OK by the Telemetry team, it's possible that :bdekoz can do the actual work of changing or adding the new interface.

Comment 2

[Chris H-C :chutten]

GV persistence is considered to be an implementation detail, so I wouldn't recommend using that directly. Telemetry has snapshotting APIs for reading measurements which might have better ergonomics.

But before we get too deep into the weeds, could you expand a little on the use case? Telemetry isn't commonly used by performance automation, so there might be some detail work we need to look into.

Comment 3

[Nick Alexander :nalexander [he/him] (PTO August 22-September 12)]

(In reply to Chris H-C :chutten from comment #2)

GV persistence is considered to be an implementation detail, so I wouldn't recommend using that directly. Telemetry has snapshotting APIs for reading measurements which might have better ergonomics.

Mmm, this might be sufficient, although it's not clear that this gives the holistic view that we're looking for. More below.

But before we get too deep into the weeds, could you expand a little on the use case? Telemetry isn't commonly used by performance automation, so there might be some detail work we need to look into.

We're trying to understand whether the probes that are currently exposed to GV consumers (like TIME_TO_DOM_CONTENT_LOADED_END_MS) actually agrees with what we see out of perf tooling kits (like browsertime). Presumably yes, but possibly no: there are some questions about whether the GV process model is actually collecting telemetry correctly.

So I guess we want to get our hands on gv_measurements.json in a non-manual way for a short time while we verify that metrics are getting collected correctly. That's possible right now with the existing APIs, but it's awkward.

bdekoz should be able to give a more detailed response. Thanks, chutten!

Comment 4

[Benjamin De Kosnik [:bdekoz]]

Attachment: 2019-02-12-fire-tv-4k-instagram.json

Sample gv_measurements.json file.

Comment 5

[Benjamin De Kosnik [:bdekoz]]

Just to clarify, we are attempting to work on both desktop and android browsers. In particular, with the sitespeed.io browsertime performance testing harness.

For desktop, I can use the persistent telemetry file archived on disk, aka datareporting/archived/*/*main.jsonlz4.

For android, I'm currently persisting to disk like so:

adb shell "run-as org.mozilla.geckoview_example cat gv_measurements.json" > gv_local_measurements.json

From the comment about the snapshotting APIs, I am presuming there is already a better way to do this?

And then, for the use case: the performance team is investigating using telemetry data for analysis of Raptor/Browsertime, in order to better understand what telemetry can tell about "in the wild" performance. We're just starting with some basic page load markers, like FX_PAGE_LOAD_MS_2, TIME_TO_NON_BLANK_PAINT_MS, and TIME_TO_LOAD_EVENT_END_MS.

I've attached one of the gv_measurements files that I persisted to disk via the (cruder, adb) method above for you to inspect.

Comment 6

[Chris H-C :chutten]

The snapshot APIs work like this:

let snapshot = Telemetry.getSnapshotForHistograms();
let values = snapshot[process]["FX_PAGE_LOAD_MS_2"].values;

You can find them defined in nsITelemetry.idl.

Comment 7

[Chris H-C :chutten]

Will the snapshot APIs serve your purposes?

Comment 8

[Benjamin De Kosnik [:bdekoz]]

Is there a getSnapshotForEnvironment or equivalent for the environment part of the payload? I cannot find it.

Is there a self-contained example for using these APIs? Can you point me at it please?

Comment 9

[Chris H-C :chutten]

TelemetryEnvironment.currentEnvironment contains the current Environment.

I'm not sure what you mean by "self-contained example". getSnapshotForHistograms will snapshot the current state of histograms in memory and give them to you. This structure, as documented in nsITelemetry.idl contains each histogram from each process that it was recorded in.

So to get the values of the content processes' pageloads you would

let snapshot = Telemetry.getSnapshotForHistograms();
let values = snapshot["content"]["FX_PAGE_LOAD_MS_2"].values;

Comment 10

[Benjamin De Kosnik [:bdekoz]]

Sorry for the delay. I can confirm that snapshotting is working on desktop (linux) and android (firetv, soon to test fenix).

For the histograms, I'm getting a lot of null values for CYCLE_COLLECTOR_* fields. Is there an easy way to elide just these bits, while keeping the rest of the prerelease probes?

Thanks!

Comment 11

[Chris H-C :chutten]

I'm not sure what you mean. Should we sync up sometime soon and chat through this over Slack or on Vidyo? Throw something on my calendar?

Comment 12

[Benjamin De Kosnik [:bdekoz]]

Will do, thanks