1615087 - Remove/renew expiring update telemetry probes

Status: RESOLVED FIXED

(Toolkit :: Application Update, task, P1)

Description

[Robin Steuber (she/her) [:bytesized]]

The following telemetry probes are expiring in Firefox version 76:

• update.fix_permissions_attempted
• update.startup.from_app_version
• update.startup.mar_partial_size_bytes
• update.startup.mar_complete_size_bytes
• update.startup.intervals.check
• update.startup.intervals.download_bits_partial
• update.startup.intervals.download_bits_complete
• update.startup.intervals.download_internal_partial
• update.startup.intervals.download_internal_complete
• update.startup.intervals.stage_partial
• update.startup.intervals.stage_complete
• update.startup.intervals.apply_partial
• update.startup.intervals.apply_complete
• update.startup.downloads.bits_partial_bytes
• update.startup.downloads.bits_partial_seconds
• update.startup.downloads.bits_complete_bytes
• update.startup.downloads.bits_complete_seconds
• update.startup.downloads.internal_partial_bytes
• update.startup.downloads.internal_partial_seconds
• update.startup.downloads.internal_complete_bytes
• update.startup.downloads.internal_complete_seconds
• update.session.from_app_version
• update.session.mar_partial_size_bytes
• update.session.mar_complete_size_bytes
• update.session.intervals.check
• update.session.intervals.download_bits_partial
• update.session.intervals.download_bits_complete
• update.session.intervals.download_internal_partial
• update.session.intervals.download_internal_complete
• update.session.intervals.stage_partial
• update.session.intervals.stage_complete
• update.session.intervals.apply_partial
• update.session.intervals.apply_complete
• update.session.downloads.bits_partial_bytes
• update.session.downloads.bits_partial_seconds
• update.session.downloads.bits_complete_bytes
• update.session.downloads.bits_complete_seconds
• update.session.downloads.internal_partial_bytes
• update.session.downloads.internal_partial_seconds
• update.session.downloads.internal_complete_bytes
• update.session.downloads.internal_complete_seconds

Many of these probes were added to ensure that BITS updating was working properly and in a timely manner. Most of those have probably served their purpose and should be removed. However, some of them we may want to extend or make permanent in order to make sure that we don't break BITS update in the future.

Some additional notes:

• Although update.fix_permissions_attempted is expiring, it should not be addressed here. It is being renewed in Bug 1615065.
• Some other telemetry probes exist regarding BITS that are not expiring:
  • UPDATE_CAN_USE_BITS_EXTERNAL - Indicates whether BITS could be used to download updates and, if not, why not. This probe is used when the update was initiated externally (by user action).
  • UPDATE_CAN_USE_BITS_NOTIFY - Indicates whether BITS could be used to download updates and, if not, why not. This probe is used when the update was initiated by the update system's internal timer.
  • UPDATE_BITS_RESULT_COMPLETE - Indicates BITS success or failure using one of these result codes. This probe is used when the update downloaded was a complete MAR.
  • UPDATE_BITS_RESULT_PARTIAL - Same as UPDATE_BITS_RESULT_COMPLETE but for partial MAR downloads.

Comment 1

[Robin Steuber (she/her) [:bytesized]]

Answering nalexander's question, from email:

For the download/BITS stuff, as long as we can answer questions about what proportion of our users are getting the BITS flow, what the relative success rates are, and perhaps something about times to download, that feels sufficient.

There are other telemetry probes that cover what proportion of our users can use BITS and what success rates they experience. Those are the UPDATE_ prefixed histograms I documented above. As for the times to download, that would not be covered if we removed the expiring probes. Many of the expiring probes measure different parts of that metric in different ways. If we want to keep something that tracks this, we would have to decide specifically what information we want.

Personally, I think that it was a good idea to measure the time taken, but that it has served its purpose and isn't really needed going forward. I don't think that it is likely that anything we do will increase BITS download time without resulting in an error.

Comment 2

[Nick Alexander :nalexander [he/him] (PTO until Monday, July 28)]

(In reply to Kirk Steuber (he/him) [:bytesized] from comment #1)

Answering nalexander's question, from email:

For the download/BITS stuff, as long as we can answer questions about what proportion of our users are getting the BITS flow, what the relative success rates are, and perhaps something about times to download, that feels sufficient.

There are other telemetry probes that cover what proportion of our users can use BITS and what success rates they experience. Those are the UPDATE_ prefixed histograms I documented above. As for the times to download, that would not be covered if we removed the expiring probes. Many of the expiring probes measure different parts of that metric in different ways. If we want to keep something that tracks this, we would have to decide specifically what information we want.

Personally, I think that it was a good idea to measure the time taken, but that it has served its purpose and isn't really needed going forward. I don't think that it is likely that anything we do will increase BITS download time without resulting in an error.

That's fine by me. To be clear: we'll keep the 4 UPDATE_*_BITS_* probes listed in #c0 and let the very detailed probes expire. Thanks, :bytesized!

Comment 3

[Molly Howell (she/her) (mostly inactive)]

Bug 1616222 makes this a more immediate concern.

Comment 4

[Robin Steuber (she/her) [:bytesized]]

Attachment: Bug 1615087 - Remove expiring BITS update telemetry, supporting code, and tests specific to it r=mhowell

This patch is effectively very close to backing out Bug 1539154, which added telemetry to monitor download times of BITS vs our internal downloader. This was useful to make sure we weren't worsening update times when we added BITS support. But at this point, these probes have outlived their usefulness to us.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/78d142f0f2f6
Remove expiring BITS update telemetry, supporting code, and tests specific to it r=mhowell

Comment 6

[Daniel Varga [:dvarga]]

Backed out changeset 78d142f0f2f6 (bug 1615087) for linting failure at /builds/worker/checkouts/gecko/browser/components/urlbar/tests/browser/interventions_update/head.j

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=290791616&searchStr=linting%2Copt%2Cjavascript%2Cchecks%2Csource-test-mozlint-eslint%2Cjs%28es%29&revision=78d142f0f2f695a19ab12d8140eb14e5fb9f9732

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=290790551&repo=autoland&lineNumber=117

Backout link: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=linting%2Copt%2Cjavascript%2Cchecks%2Csource-test-mozlint-eslint%2Cjs%28es%29&revision=c86774de3bd3d6a64195d27267036b0bd336bba6

[task 2020-02-27T16:59:22.739Z] /builds/worker/checkouts/gecko/caps
[task 2020-02-27T16:59:22.752Z] 16:59:22.750 eslint (94) | Command: /usr/local/bin/node /builds/worker/checkouts/gecko/node_modules/eslint/bin/eslint.js --ext [js,jsm,jsx,xul,html,xhtml] --format json --no-error-on-unmatched-pattern --quiet --ignore-pattern testing/web-platform/tests/tools/third_party --ignore-pattern gfx/sfntly --ignore-pattern testing/talos/talos/tests/v8_7 --ignore-pattern mfbt/double-conversion/double-conversion --ignore-pattern testing/mozbase/mozproxy/mozproxy/backends/mitm/scripts/catapult --ignore-pattern testing/mochitest/MochiKit --ignore-pattern testing/xpcshell/node-http2 --ignore-pattern testing/modules/ajv-4.1.1.js --ignore-pattern browser/components/pocket/content/panels/js/vendor --ignore-pattern browser/extensions/screenshots/build/raven.js --ignore-pattern gfx/wr --ignore-pattern browser/extensions/formautofill/content/third-party --ignore-pattern gfx/ycbcr --ignore-pattern testing/xpcshell/dns-packet --ignore-pattern remote/test/puppeteer --ignore-pattern gfx/graphite2 --ignore-pattern tools/profiler/core/vtune --ignore-pattern gfx/qcms --ignore-pattern testing/xpcshell/node-ip --ignore-pattern testing/modules/sinon-7.2.7.js --ignore-pattern gfx/harfbuzz --ignore-pattern testing/talos/talos/tests/dromaeo --ignore-pattern browser/components/translation/cld2 --ignore-pattern remote/test/browser/chrome-remote-interface.js --ignore-pattern gfx/cairo --ignore-pattern gfx/vr/service/openvr --ignore-pattern gfx/skia --ignore-pattern browser/components/newtab/vendor --ignore-pattern testing/mochitest/tests/MochiKit-1.4.2 --ignore-pattern tools/fuzzing/libfuzzer --ignore-pattern testing/gtest/gtest --ignore-pattern testing/gtest/gmock --ignore-pattern tools/lint/test/files --ignore-pattern testing/talos/talos/tests/kraken --ignore-pattern testing/mochitest/pywebsocket --ignore-pattern gfx/ots --ignore-pattern gfx/angle/checkout --ignore-pattern testing/web-platform/tests/resources/webidl2 /builds/worker/checkouts/gecko/uriloader /builds/worker/checkouts/gecko/browser /builds/worker/checkouts/gecko/tools /builds/worker/checkouts/gecko/mfbt /builds/worker/checkouts/gecko/gfx /builds/worker/checkouts/gecko/docs /builds/worker/checkouts/gecko/testing /builds/worker/checkouts/gecko/hal /builds/worker/checkouts/gecko/remote /builds/worker/checkouts/gecko/caps
[task 2020-02-27T16:59:22.780Z] 16:59:22.780 eslint (92) | Passing the following paths:
[task 2020-02-27T16:59:22.781Z] /builds/worker/checkouts/gecko/gradle  
[task 2020-02-27T16:59:22.781Z] /builds/worker/checkouts/gecko/devtools  
[task 2020-02-27T16:59:22.781Z] /builds/worker/checkouts/gecko/parser  
[task 2020-02-27T16:59:22.781Z] /builds/worker/checkouts/gecko/dom  
[task 2020-02-27T16:59:22.781Z] /builds/worker/checkouts/gecko/xpcom  
[task 2020-02-27T16:59:22.781Z] /builds/worker/checkouts/gecko/media  
[task 2020-02-27T16:59:22.781Z] /builds/worker/checkouts/gecko/python  
[task 2020-02-27T16:59:22.781Z] /builds/worker/checkouts/gecko/editor  
[task 2020-02-27T16:59:22.781Z] /builds/worker/checkouts/gecko/extensions  
[task 2020-02-27T16:59:22.781Z] /builds/worker/checkouts/gecko/modules  
[task 2020-02-27T16:59:22.781Z] /builds/worker/checkouts/gecko/image  
[task 2020-02-27T16:59:22.781Z] /builds/worker/checkouts/gecko/storage  
[task 2020-02-27T16:59:22.781Z] /builds/worker/checkouts/gecko/xpfe  
[task 2020-02-27T16:59:22.781Z] /builds/worker/checkouts/gecko/build  
[task 2020-02-27T16:59:22.781Z] /builds/worker/checkouts/gecko/startupcache
[task 2020-02-27T16:59:22.795Z] 16:59:22.795 eslint (92) | Command: /usr/local/bin/node /builds/worker/checkouts/gecko/node_modules/eslint/bin/eslint.js --ext [js,jsm,jsx,xul,html,xhtml] --format json --no-error-on-unmatched-pattern --quiet --ignore-pattern dom/media/gmp/widevine-adapter/content_decryption_module_export.h --ignore-pattern dom/media/webspeech/recognition/endpointer.cc --ignore-pattern dom/media/webvtt/vtt.jsm --ignore-pattern xpcom/build/mach_override.h --ignore-pattern media/libwebp --ignore-pattern dom/tests/mochitest/dom-level2-html --ignore-pattern dom/media/webspeech/recognition/energy_endpointer.cc --ignore-pattern xpcom/io/crc32c.c --ignore-pattern dom/canvas/test/webgl-conf/checkout --ignore-pattern media/mtransport/third_party --ignore-pattern media/libtremor --ignore-pattern dom/tests/mochitest/dom-level2-core --ignore-pattern dom/webauthn/tests/pkijs --ignore-pattern dom/media/webspeech/recognition/energy_endpointer_params.cc --ignore-pattern media/libyuv --ignore-pattern devtools/client/shared/demangle.js --ignore-pattern modules/fdlibm --ignore-pattern media/libtheora --ignore-pattern media/libmkv --ignore-pattern devtools/client/shared/vendor --ignore-pattern dom/media/webspeech/recognition/energy_endpointer.h --ignore-pattern media/kiss_fft --ignore-pattern dom/media/platforms/ffmpeg/ffmpeg57 --ignore-pattern dom/u2f/tests/pkijs --ignore-pattern dom/media/platforms/ffmpeg/libav54 --ignore-pattern media/libvpx --ignore-pattern media/libaom --ignore-pattern media/webrtc/signaling/src/sdp/sipcc --ignore-pattern media/libsoundtouch --ignore-pattern devtools/shared/sprintfjs --ignore-pattern dom/media/webaudio/test/blink --ignore-pattern devtools/shared/acorn --ignore-pattern media/libcubeb --ignore-pattern media/libspeex_resampler --ignore-pattern dom/media/webspeech/recognition/endpointer.h --ignore-pattern modules/woff2 --ignore-pattern media/openmax_il --ignore-pattern dom/media/platforms/ffmpeg/ffmpeg58 --ignore-pattern dom/tests/mochitest/ajax --ignore-pattern parser/expat --ignore-pattern dom/media/gmp/widevine-adapter/content_decryption_module.h --ignore-pattern dom/tests/mochitest/dom-level1-core --ignore-pattern dom/media/gmp/rlz --ignore-pattern dom/webauthn/cbor-cpp --ignore-pattern dom/media/platforms/ffmpeg/libav53 --ignore-pattern media/libjpeg --ignore-pattern devtools/shared/heapsnapshot/CoreDump.pb.h --ignore-pattern devtools/client/shared/source-map --ignore-pattern devtools/shared/qrcode/encoder --ignore-pattern devtools/client/jsonview/lib/require.js --ignore-pattern extensions/spellcheck/hunspell/src --ignore-pattern media/mp4parse-rust --ignore-pattern modules/zlib --ignore-pattern devtools/shared/qrcode/decoder --ignore-pattern devtools/client/shared/sourceeditor/test/cm_mode_ruby.js --ignore-pattern media/libogg --ignore-pattern devtools/shared/storage/vendor --ignore-pattern media/openmax_dl --ignore-pattern media/ffvpx --ignore-pattern devtools/client/debugger/flow-typed/npm --ignore-pattern devtools/shared/node-properties --ignore-pattern modules/xz-embedded --ignore-pattern xpcom/build/mach_override.c --ignore-pattern media/libdav1d --ignore-pattern dom/imptests --ignore-pattern dom/media/webspeech/recognition/energy_endpointer_params.h --ignore-pattern media/libnestegg --ignore-pattern media/webrtc/trunk --ignore-pattern devtools/client/shared/build/babel.js --ignore-pattern dom/media/platforms/ffmpeg/libav55 --ignore-pattern media/libopus --ignore-pattern devtools/shared/heapsnapshot/CoreDump.pb.cc --ignore-pattern devtools/client/shared/sourceeditor/codemirror --ignore-pattern modules/freetype2 --ignore-pattern dom/media/gmp/widevine-adapter/content_decryption_module_ext.h --ignore-pattern media/libvorbis --ignore-pattern devtools/shared/jsbeautify --ignore-pattern modules/brotli --ignore-pattern devtools/client/shared/sourceeditor/test/codemirror --ignore-pattern build/pymake --ignore-pattern media/libpng --ignore-pattern editor/libeditor/tests/browserscope/lib --ignore-pattern dom/media/gmp/widevine-adapter/content_decryption_module_proxy.h /builds/worker/checkouts/gecko/gradle /builds/worker/checkouts/gecko/devtools /builds/worker/checkouts/gecko/parser /builds/worker/checkouts/gecko/dom /builds/worker/checkouts/gecko/xpcom /builds/worker/checkouts/gecko/media /builds/worker/checkouts/gecko/python /builds/worker/checkouts/gecko/editor /builds/worker/checkouts/gecko/extensions /builds/worker/checkouts/gecko/modules /builds/worker/checkouts/gecko/image /builds/worker/checkouts/gecko/storage /builds/worker/checkouts/gecko/xpfe /builds/worker/checkouts/gecko/build /builds/worker/checkouts/gecko/startupcache
[task 2020-02-27T17:00:59.837Z] 17:00:59.836 eslint (93) | Finished in 97.32 seconds
[task 2020-02-27T17:06:10.621Z] 17:06:10.620 eslint (94) | Finished in 408.09 seconds
[task 2020-02-27T17:06:34.262Z] 17:06:34.261 eslint (91) | Finished in 431.74 seconds
[task 2020-02-27T17:10:38.818Z] 17:10:38.818 eslint (92) | Finished in 676.30 seconds
[task 2020-02-27T17:10:38.826Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/browser/components/urlbar/tests/browser/interventions_update/head.js:250:19 | 'TelemetryTestUtils' is not defined. (no-undef)
[task 2020-02-27T17:10:38.826Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/browser/components/urlbar/tests/browser/interventions_update/head.js:251:3 | 'TelemetryTestUtils' is not defined. (no-undef)
[task 2020-02-27T17:10:38.826Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/browser/components/urlbar/tests/browser/interventions_update/head.js:257:3 | 'TelemetryTestUtils' is not defined. (no-undef)
[taskcluster 2020-02-27 17:10:39.175Z] === Task Finished ===
[taskcluster 2020-02-27 17:10:39.653Z] Unsuccessful task run with exit code: 1 completed in 756.511 seconds

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/41765897b669
Remove expiring BITS update telemetry, supporting code, and tests specific to it r=mhowell

Comment 8

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/41765897b669