1147395 - Validation: Compare a few telemetry measurements between "saved-session" and "main" pings.

Status: RESOLVED WORKSFORME

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P2)

Description

[Mark Reid [:mreid]]

We should ensure that we're receiving the same data via the new "main" pings as we are receiving via the "saved-session" pings. One simple way is to select a few common measures and calculate the aggregations using both record types. They should come out the same if everything is working as expected.

I propose we use the following commonly submitted values:
SIMPLE_MEASURES_UPTIME
CYCLE_COLLECTOR
GC_MS

And an uncommon measure like:
SHUMWAY_ERROR

We should filter records by
- Application Name: "Firefox"
- Channel: "nightly"

We should aggregate separately by type="saved-session" vs. type="main", excluding duplicate Document IDs.  The aggregation should be done by the appBuildId field.

Ideally, the resulting aggregations should be comparable to the data on telemetry.mozilla.org in addition to comparing with each other.

Comment 1

[Mark Reid [:mreid]]

Roberto, is this work ongoing? Or are you waiting for the follow-up bugs? Are you blocked on anything?

Comment 2

[Roberto Agostino Vitillo (:rvitillo)]

I was waiting for Brendan to be happy with the data first, as he is doing a great job of validating the v4 dataset.

Comment 3

[Thomas Huelbert]

needs info to Katie to find resources for this.

Comment 4

[Roberto Agostino Vitillo (:rvitillo)]

I will start working on this tomorrow.

Comment 5

[Roberto Agostino Vitillo (:rvitillo)]

A preliminary analysis is available at [1]. I compared only few metrics but there appears already to be a mismatch in about 7% of sessions for one of those (GC_MS).

At the end of the notebook I dumped few mismatching sessions. Note that in the notebook I have used a single build-id but the percentage seems to be stable across recent build-ids as well.

[1] http://nbviewer.ipython.org/gist/vitillo/8aec1f023265c9bf2293

Comment 6

[Roberto Agostino Vitillo (:rvitillo)]

Forgot to mention that 7% applies only to multi-fragment sessions.

Comment 7

[Alessio Placitelli [:Dexter]]

After a quite some testing, I was not able to consistently reproduce the issue locally. This is my test procedure (I've changed the telemetry server pref to point to a local, non existent server, so my pings are kept in the pending pings directory, for simplicity):

- Start Firefox
- Browse to about:telemetry
- Wait for Telemetry to start (1 minute)
- Play a bit with the browser to trigger the GC
- Enable a restartless addon to break the session
- Play a bit more
- Close Firefox

The GC_MS histogram in the environment-changed ping and the one in the shutdown ping sum up nicely and the result equals the value reported by the saved-session ping. What I've noticed though is that there's a mismatch in the GC_MS histogram in the "childPayloads" section of the pings.

I'll keep digging further in.

Comment 8

[Georg Fritzsche [:gfritzsche]]

Checking through a few things here, it turns out that the GC_MS data used in the notebook is a sum of both child & parent histograms.

Roberto, can you please rerun this with only looking at the parent data?

The child-data discrepancy needs to be fixed before we'd switch away from saved-session or e10s ships, but its not blocking 41 now.

Comment 9

[Roberto Agostino Vitillo (:rvitillo)]

Using only parent histograms nearly none of the complete multi-fragment sessions have a mismatch. See http://nbviewer.ipython.org/gist/vitillo/b352ec160ce5c5ee2af6

Comment 10

[Georg Fritzsche [:gfritzsche]]

Thanks for rechecking, that is great to hear.
That seems ok and minor mismatches are expected (for bug 1186871).

So i think we can close this off as WORKSFORME and file a follow-up to investigate the child payload discrepancies?
(I think those come mostly down to not collecting child payloads on each subsession collection)

Comment 11

[Roberto Agostino Vitillo (:rvitillo)]

Can we just rename the Bug?

Comment 12

[Georg Fritzsche [:gfritzsche]]

I'd rather close this one as the original question we tracked here is resolved and the context above isn't all related to the e10s issue.

Comment 13

[Georg Fritzsche [:gfritzsche]]

(In reply to Georg Fritzsche [:gfritzsche] [away until july 22] from comment #12)
> I'd rather close this one as the original question we tracked here is
> resolved and the context above isn't all related to the e10s issue.

Filed bug 1187327.