1414295 - Whole lotta alerts for Oct 24: Network, Child payloads, thumbnails

Status: RESOLVED WONTFIX

(Toolkit :: Telemetry, enhancement, P1)

Description

[Chris H-C :chutten]

Nightly builds starting Oct 24 tripped a lot of alerts:

 HTTP_PAGE_FIRST_SENT_TO_LAST_RECEIVED,
 TIME_TO_DOM_COMPLETE_MS,
 FX_THUMBNAILS_BG_CAPTURE_DONE_REASON_2,
 HTTP_PAGE_COMPLETE_LOAD_NET_V2,
 FX_THUMBNAILS_BG_QUEUE_SIZE_ON_CAPTURE,
 FX_THUMBNAILS_BG_CAPTURE_QUEUE_TIME_MS,
 TIME_TO_DOM_LOADING_MS,
 HTTP_PAGE_OPEN_TO_FIRST_RECEIVED,
 TIME_TO_LOAD_EVENT_START_MS,
 TIME_TO_DOM_CONTENT_LOADED_START_MS,
 TIME_TO_DOM_CONTENT_LOADED_END_MS,
 TIME_TO_DOM_INTERACTIVE_MS,
 TIME_TO_LOAD_EVENT_END_MS,
 TOTAL_CONTENT_PAGE_LOAD_TIME,
 HTTP_REQUEST_PER_PAGE_FROM_CACHE,
 HTTP_PAGE_OPEN_TO_FIRST_SENT,
 TELEMETRY_DISCARDED_CONTENT_PINGS_COUNT,
 SECURITY_UI,
 HTTP_PAGE_COMPLETE_LOAD_V2

All of these alerts show a sudden change that corresponds to no increase in submissions, but a sudden 2-6x increase in samples being submitted: https://mzl.la/2Ae9LSC

We should look into this holistically.

Comment 1

[Chris H-C :chutten]

As of Nov 4 these are back to normal.

Still not sure what happened here.

Comment 2

[Chris H-C :chutten]

It's content-process-only. It happens on all three major Desktop OSs. Seemingly not on Fennec.

The pushlog for the Oct 24 builds is: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=9056f2ee492fa481aa86146aba236c074628e9fd&tochange=a124f4901430f6db74cfc7fe3b07957a1c691b40

Nothing stands out, though. I would assume, from the presence of TELEMETRY_DISCARDED_CONTENT_PINGS_COUNT there's something starting and stopping a lot of content processes. From all of the net probes, I expect this process was loading content over the network.

For now it appears as though it was a temporary effect, quickly remedied. I wish I could nail it down to a pair of changes, but I have no leads.