808347 - difference between idle-daily and saved-session in telemetry data

Status: RESOLVED INVALID

(Toolkit :: Telemetry, defect)

Description

[Dave Zeber [:dzeber]]

Attachment: boxplots of uptimes

I'd like to clarify the differences between idle-daily and saved-session telemetry pings and how they are collected. This is essential for the metrics analysts in understanding how to structure our analyses. 

My understanding of how idle-daily and saved-session pings are collected is the following. An idle-daily ping is generated during the first browser session of the day and sent. Telemetry for any other sessions occurring after the idle-daily are collected and stored as saved-session pings, one per session. These are sent the next day at the same time as the next day's idle-daily ping.


Day 1 |--|//*////|---|////|---|///|-----|////////|--|
          ID  SS1      SS2     SS3         SS4      
Day 2 |---|///*///|----|////////|-----|///|---------|
            ID SS1         SS2         SS3

These are timelines where |///| is a browser session, and * is the time idle-daily is sent. Is this the correct interpretation?

1) Does idle-daily only include measurements from a partial session, ie from the beginning of the first session of the day until the first idle period?

2) What happens to the remainder of the first session - does it become a separate saved-session (as I indicated in the diagram)? In this case, wouldn't some saved-session pings actually represent partial sessions?

3) Idle-daily is triggered once per day - when does the new day start for this purpose? Are these calendar days?

4) What happens if a session is still running from the previous day? Does idle-daily send all telemetry since the beginning of the session, or since the last ping?

5) Wouldn't idle-daily pings be missing measurements like shutdown time? Looking at the data, they don't seem to be.

6) Wouldn't we expect idle-daily pings to be shorter on average than saved-sessions, if idle-daily is actually a partial session? This is not the case - see attached plot of side-by-side boxplots for log of uptime. For each of the 2 groups, I took 1000 quantiles of the uptime measurements and generated boxplots from those. The data is from v18 nightly telemetry for Sep 3-9 (44025 total pings, 60% ID/40% SS). The medians are practically the same.

7) What does it mean to have uptime = 0? This is the case for ~10% of saved-session pings in my dataset. Does this have to do with discretization - are simpleMeasurements discretized?

8) The previous day's saved-sessions are sent together with the current idle-daily. Is there any identifier variable that will tell me which pings came in together?

Comment 1

[Justin Lebar (not reading bugmail)]

See also bug 790616, which is blocked on bug 794503.  I hope we can get that issue resolved soon, because it's a major data-quality issue.

> are simpleMeasurements discretized?

IIRC, they are not discretized by Firefox, but I believe they are discretized somewhere in the telemetry back-end into very large (24hr?) buckets.  You may or may not be seeing this bucketed data, of course.  Does the rest of your data look like it's been bucketed?

Comment 2

[Vladan Djeric (:vladan)]

(In reply to dzeber from comment #0)
> My understanding of how idle-daily and saved-session pings are collected is
> the following. An idle-daily ping is generated during the first browser
> session of the day and sent. 

Not quite. On Firefox startup, we schedule an idle-daily ping to occur 24 hours after the previous idle daily ping (time of the last idle-daily ping is kept in the prefs file). If it has been more than 24 hours since the last idle-daily ping or an idle-daily has never been sent, we generate an idle-daily event immediately on startup.

See comment and code in http://mxr.mozilla.org/mozilla-central/source/widget/xpwidgets/nsIdleService.cpp#161

> Telemetry for any other sessions occurring
> after the idle-daily are collected and stored as saved-session pings, one
> per session. These are sent the next day at the same time as the next day's
> idle-daily ping.

Correct, but see other answers for more detail

> These are timelines where |///| is a browser session, and * is the time
> idle-daily is sent. Is this the correct interpretation?

Yes

> 1) Does idle-daily only include measurements from a partial session, ie from
> the beginning of the first session of the day until the first idle period?

idle-daily includes all measurements for the current session (i.e. from Firefox startup)

> 2) What happens to the remainder of the first session - does it become a
> separate saved-session (as I indicated in the diagram)? In this case,
> wouldn't some saved-session pings actually represent partial sessions?

If you have a long running session (multiple days), it will send out multiple idle-daily pings for that session. Each idle-daily ping will have cumulative session data from the beginning of the session. Each ping from the same session has the same document ID, so each new ping will over-write the data submitted by a previous ping from the same session. When Firefox exits, it saves the session's final Telemetry data on disk. When this saved-session ping is submitted in a later session, it overwrites any earlier ping and ensures we have a single record of all the Telemetry data gathered for the session, from startup to shutdown.

> 3) Idle-daily is triggered once per day - when does the new day start for
> this purpose? Are these calendar days?

Idle-daily is triggered every 24 hours, see first answer in comment for details

> 4) What happens if a session is still running from the previous day? Does
> idle-daily send all telemetry since the beginning of the session, or since
> the last ping?

It sends all telemetry since Firefox startup (i.e. entire session) and over-writes any existing Telemetry submission for this session

> 5) Wouldn't idle-daily pings be missing measurements like shutdown time?
> Looking at the data, they don't seem to be.

This is because the final Telemetry submission for any session is the saved-session data, and it overwrites any earlier Telemetry pings

> 6) Wouldn't we expect idle-daily pings to be shorter on average than
> saved-sessions, if idle-daily is actually a partial session? This is not the
> case - see attached plot of side-by-side boxplots for log of uptime. For
> each of the 2 groups, I took 1000 quantiles of the uptime measurements and
> generated boxplots from those. The data is from v18 nightly telemetry for
> Sep 3-9 (44025 total pings, 60% ID/40% SS). The medians are practically the
> same.

Not necessarily, consider the case where someone clicks on a link in an IM message and this leads to a brief 5-minute browser session. Unless the browser session happens to coincide with idle-daily firing, the session's data will only be recorded as a saved-session. I think this explanation lines up with your graph -- saved-session pings cover a wider range of uptimes than idle-daily pings

> 7) What does it mean to have uptime = 0? This is the case for ~10% of
> saved-session pings in my dataset. Does this have to do with discretization
> - are simpleMeasurements discretized?

Uptime is in minutes, so I guess it could represent sessions lasting less than 60 seconds (e.g. example of an IM link session above), but I think Justin's explanation might be more applicable

> 8) The previous day's saved-sessions are sent together with the current
> idle-daily. Is there any identifier variable that will tell me which pings
> came in together?

I don't think so, but we can add this if you like

Comment 3

[Vladan Djeric (:vladan)]

(In reply to dzeber from comment #0)
> 5) Wouldn't idle-daily pings be missing measurements like shutdown time?
> Looking at the data, they don't seem to be.

Sorry, I misinterpreted this question. The shutdown time you see in idle-daily (and saved-session) pings is the shutdown time of the previous session

Comment 4

[Dave Zeber [:dzeber]]

Thanks for clarifying these points. I had one more question: 

What happens if the session terminates in a crash? Is telemetry still recorded until the end of the session, and does it generate a proper saved-session ping?

Comment 5

[Nathan Froyd [:froydnj]]

(In reply to dzeber from comment #4)
> Thanks for clarifying these points. I had one more question: 
> 
> What happens if the session terminates in a crash? Is telemetry still
> recorded until the end of the session, and does it generate a proper
> saved-session ping?

No.  This is bug 719167.

Comment 6

[Alessio Placitelli [:Dexter]]

Closing this as it's no longer relevant, we're going to remove the saved-session ping in bug 1320051.