1603833 - Intermediate Preloading is "failedToDownloadMisc" a lot.

Status: RESOLVED WORKSFORME

(Core :: Security: PSM, defect, P1)

Description

[J.C. Jones [:jcj] (he/they)]

The telemetry for intermediate preloading errors [0] (which was un-landed for six months) is now showing an awful lot of "failedToObserve" errors, which sadly is something of a catch-all for both database update failures and download-from-Kinto failures.

From historical data, it wasn't like this back at the start of 2019, so I'm marking this as regressed by Bug 1559114 as just a starting point.

Mathieu, do you have any immediate ideas on what could be prompting this, or do we need to clean up the telemetry and try to dig in that way?

[0] https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2019-12-12&include_spill=0&keys=__none__!__none__!__none__&max_channel_version=nightly%252F73&measure=INTERMEDIATE_PRELOADING_ERRORS&min_channel_version=nightly%252F72&processType=*&product=Firefox&sanitize=0&sort_by_value=0&sort_keys=submissions&start_date=2019-12-02&table=0&trim=0&use_submission_date=0

Comment 1

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Attachment: bug 1603833 - re-purpose unused intermediate preloading failure telemetry categories to investigate failures r?kjacobs

Intermediate preloading telemetry is overwhelmingly "failedToObserve", which at
the moment is a catch-all indicating that something in
updatePreloadedIntermediates failed. We need to figure out why, so this patch
re-purposes the categories "emptyAttachment", "failedToFetch", and
"unexpectedLength", which are currently not used, to indicate failures in
specific sub-operations in that function.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d66517bc41f0
re-purpose unused intermediate preloading failure telemetry categories to investigate failures r=kjacobs,leplatrem

Comment 3

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/d66517bc41f0

Comment 4

[Mathieu Leplatre [:leplatrem]]

Mathieu, do you have any immediate ideas on what could be prompting this, or do we need to clean up the telemetry and try to dig in that way?

Note that since Bug 1552446 some download errors will be reported in uptake telemetry too (SELECT * FROM telemetry.events WHERE event_category = 'uptake.remotecontent.result')

Comment 5

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Now nearly 100% of the errors are misc. download errors: https://mzl.la/1PGPwMm

(In reply to Mathieu Leplatre [:leplatrem] from comment #4)

Note that since Bug 1552446 some download errors will be reported in uptake telemetry too (SELECT * FROM telemetry.events WHERE event_category = 'uptake.remotecontent.result')

Where would I run this code to look at that data?

Comment 6

[Mathieu Leplatre [:leplatrem]]

You should be able to access the query results here: https://sql.telemetry.mozilla.org/queries/67015?p_source_67015=intermediates

In short, most are network errors, as reported by this code.

Regarding the parse errors, they are issued by Firefox 68 because of Bug 1558709

Comment 7

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Thanks! I guess this is expected, or at least not something wrong with intermediate preloading specifically.