1200553 - Add v2/v4 session matching to the local Telemetry validation tool

Status: RESOLVED WORKSFORME

(Toolkit :: Telemetry, defect, P1)

Comment 1

[Georg Fritzsche [:gfritzsche]]

I've added this in the latest revision here:
https://gist.github.com/georgf/1b0831a6b81b6c9fe240

Comment 2

[Georg Fritzsche [:gfritzsche]]

I've updated this with accumulating time/hours values from the matchup data, this looks much more stable then the previous approach.

needinfo?me to write-up the results tomorrow to fhr-dev and other tasks:
* v2 aborted session time seems always much less than v4 aborted session time
* its very hard to match v2 & v4 aborted sessions because of this
* totalTime seems to line up pretty well otherwise locally between v2 and v4, with max 5sec discrepancies
* sessionLength/subsessionLength accumulation doesn't look as reliable, need to understand the differences here
* i need to file a follow-up on moving over the other accumulations to the session-based approach

Comment 3

[Georg Fritzsche [:gfritzsche]]

@Sam:
Heads-up for a matching algorithm that seems to work reasonably well on local data, will try a proper mail writeup tomorrow.

The important parts here are:
* v2 historic sessions are from cleanTotalTimes & abortedTotalTimes based on their UTC start date
* construct v4 session start dates from the last fragments of sessions: (creationDateUTC - totalTime)
* this usually matches up with <=5s delta to the v2 sessions
* except aborted sessions, which are hard to match... we could do lax matching via "count of aborted sessions on that day" and roughly match on order and v2 abortedTotalTime < the v4 equivalent

Comment 4

[Georg Fritzsche [:gfritzsche]]

(In reply to Georg Fritzsche [:gfritzsche] from comment #2)
> needinfo?me to write-up the results tomorrow to fhr-dev and other tasks:
> * v2 aborted session time seems always much less than v4 aborted session time
> * its very hard to match v2 & v4 aborted sessions because of this
> * totalTime seems to line up pretty well otherwise locally between v2 and
> v4, with max 5sec discrepancies
> * sessionLength/subsessionLength accumulation doesn't look as reliable, need
> to understand the differences here
> * i need to file a follow-up on moving over the other accumulations to the
> session-based approach

I've moved the time comparisons over to be based on the matchup data in the current revision.
The comparison for matched times seems pretty stable.

I'll add a less strict matching for aborted sessions later, then i'll call this done and we should get data from more people.

Comment 5

[Sam Penrose]

Fantastic, thanks Georg. I'm leaving myself needinfo until I try it. Shall we attach results to this bug?

Comment 6

[Georg Fritzsche [:gfritzsche]]

I've just updated the script with a less strict aborted session matching and much less yellowing.

I'd like to collect links to peoples history in this etherpad:
https://etherpad.mozilla.org/TqPEhaUz9H

I want to take a look at the teams local histories and make sure we're confident there too.