726375 - Investigate sawtooth patterns in Aurora Telemetry Evolution memory data

Status: RESOLVED INCOMPLETE

(Toolkit :: Telemetry, defect)

Description

[Justin Lebar (not reading bugmail)]

See MEMORY_RESIDENT or CYCLE_COLLECTOR for Windows / Aurora on Telemetry Evolution.

There is a pronounced sawtooth in the ranges labeled "Firefox 8" and "Firefox 9" (Aurora 10 and 11).  RSS / CC time is low in early builds, increases in later builds, then drops as soon as we bump the version number.  I don't see this pattern with reporters unrelated to memory usage.

Also notable is the fact that the initial Aurora 10/11 releases (12/22 and 1/31) have ~4x as many pings as other builds.  These initial releases have high RSS/CC times; the RSS/CC times then drop for the next build.

Nightly builds do not show these patterns.

My best guess as to what's going on here is that the builds later in the cycle are restarted less-frequently than builds earlier in the cycle.  Uptime should correlate with RSS, so if the average uptime is larger, the average RSS should be larger, which should result in higher CC times.

But this does not explain why later builds should be restarted less-frequently.  AIUI we don't release an Aurora build if nothing changed on the Aurora tree that night, but even where I can verify that we released builds for three days in a row, the averages do not drop down to the level of the earlier builds.

Telemetry sends uptime in its pings, but the Telemetry front-ends do not display this data, so I can't test this particular hypothesis without assistance from the Metrics team.

I doubt this effect has anything to do with Firefox itself.  But we need to figure out what's going on here (and hopefully compensate for it), because at best this is compromising our ability to read important trends in Aurora builds.

Comment 1

[Justin Lebar (not reading bugmail)]

Attachment: Screenshot of MEMORY_RESIDENT

Comment 2

[Justin Lebar (not reading bugmail)]

Ah, I didn't know that telemetry charts uptime.  It's under SIMPLE_MEASURES_UPTIME.

The 25-90 percentiles are basically fixed at 12 hours exactly.  But there is a gradual increase in mean uptime during the Aurora 11 cycle, which might correspond to the sawtooth we're seeing.

Comment 3

[Justin Lebar (not reading bugmail)]

Ah, looks like someone may be aggressively bucketing uptime data into 12-hour granularity, or something.

Comment 4

[Justin Lebar (not reading bugmail)]

This doesn't seem to be happening anymore.  Or, if it is still happening, we no longer have the data going back, so I can't investigate any further atm.