Analyze data returned by zero_byte_load event on 2021-02-18
Categories
(Core :: Networking: JAR, task, P2)
Tracking
()
People
(Reporter: zbraniecki, Assigned: zbraniecki)
References
Details
(Whiteboard: [necko-triaged])
As part of YSOD investigation (bug 1675823) in bug 1693146 we landed a probe to collect all zero byte loads.
That probe returned massive amount of data worth analyzing for most common empty loads. To avoid DDOSing our system, in bug 1693708 I'm going to narrow down what we submit, but the data from 2021-02-18 should be analyzed separately for common types, urls, platforms, locales etc.
|
Assignee | |
Comment 1•3 years ago
|
||
From what we see in the broad spectrum
by type:
others 910,761
ftl 255,271
js 157,444
dtd 111,820
xhtml 82,144
properties 2
by url:
omni.ja!/res/dtd/svg11.dtd 101,993
omni.ja!/chrome/browser/res/activity-stream/prerendered/activity-stream-noscript 89,659
webcompat@mozilla.org.xpi!/shims/google-analytics.js 55,361
uBlock0@raymondhill.net.xpi!/web_accessible_resources/googletagmanager_gtm.js?se 33,502
uBlock0@raymondhill.net.xpi!/web_accessible_resources/google-analytics_analytics 29,900
omni.ja!/chrome/browser/content/browser/browser.xhtml 28,409
omni.ja!/chrome/toolkit/skin/classic/mozapps/places/defaultFavicon.svg 26,748
webcompat@mozilla.org.xpi!/shims/google-publisher-tags.js 23,174
uBlock0@raymondhill.net.xpi!/web_accessible_resources/googletagservices_gpt.js?s 21,124
webcompat@mozilla.org.xpi!/shims/empty-script.js 17,050
omni.ja!/update.locale 16,214
omni.ja!/chrome/browser/content/browser/aboutNetError.xhtml 15,930
omni.ja!/chrome/toolkit/content/extensions/dummy.xhtml 14,919
uBlock0@raymondhill.net.xpi!/popup-fenix.html 13,572
omni.ja!/defaults/settings/main/ms-language-packs.json 12,936
omni.ja!/defaults/settings/main/normandy-recipes.json 12,929
omni.ja!/defaults/settings/main/nimbus-mobile-experiments.json 12,926
omni.ja!/defaults/settings/main/rally-studies-v1.json 12,925
omni.ja!/defaults/settings/main/quicksuggest.json 12,923
omni.ja!/defaults/settings/main/websites-with-shared-credential-backends.json 12,923
omni.ja!/defaults/settings/main/doh-providers.json 12,923
omni.ja!/defaults/settings/main/doh-config.json 12,922
omni.ja!/defaults/settings/main/pioneer-study-addons-v2.json 12,920
omni.ja!/defaults/settings/main/pioneer-content-v1.json 12,920
omni.ja!/defaults/settings/main/mobile-experiments.json 12,920
by status:
NS_ERROR_FILE_NOT_FOUND 707,986
NS_BINDING_ABORTED 599,921
NS_ERROR_FAILURE 125,349
NS_ERROR_PARSED_DATA_CACHED 65,941
NS_OK 10,516
NS_ERROR_CORRUPTED_CONTENT 4,924
NS_ERROR_UNEXPECTED 2,329
NS_IMAGELIB_ERROR_FAILURE 369
by sync:
false 1,405,422
true 112,020
|
|
Assignee | |
Comment 2•3 years ago
|
||
Most popular files per type:
Others:
omni.ja!/chrome/browser/res/activity-stream/prerendered/activity-stream-noscript 89,659
uBlock0@raymondhill.net.xpi!/web_accessible_resources/googletagmanager_gtm.js?se 33,502
uBlock0@raymondhill.net.xpi!/web_accessible_resources/google-analytics_analytics 29,900
omni.ja!/chrome/toolkit/skin/classic/mozapps/places/defaultFavicon.svg 26,748
uBlock0@raymondhill.net.xpi!/web_accessible_resources/googletagservices_gpt.js?s 21,124
omni.ja!/update.locale 16,214
uBlock0@raymondhill.net.xpi!/popup-fenix.html 13,572
omni.ja!/defaults/settings/main/ms-language-packs.json 12,936
omni.ja!/defaults/settings/main/normandy-recipes.json 12,929
omni.ja!/defaults/settings/main/nimbus-mobile-experiments.json 12,926
omni.ja!/defaults/settings/main/rally-studies-v1.json 12,925
omni.ja!/defaults/settings/main/quicksuggest.json 12,923
omni.ja!/defaults/settings/main/websites-with-shared-credential-backends.json 12,923
omni.ja!/defaults/settings/main/doh-providers.json 12,923
omni.ja!/defaults/settings/main/doh-config.json 12,922
omni.ja!/defaults/settings/main/pioneer-study-addons-v2.json 12,920
omni.ja!/defaults/settings/main/pioneer-content-v1.json 12,920
omni.ja!/defaults/settings/main/mobile-experiments.json 12,920
omni.ja!/defaults/settings/main/pioneer-content-v2.json 12,919
omni.ja!/defaults/settings/main/regions.json 12,919
omni.ja!/defaults/settings/main/pioneer-study-addons.json 12,918
omni.ja!/defaults/settings/main/cfr-srg.json 12,918
omni.ja!/defaults/settings/main/windows-default-browser-agent.json 12,918
omni.ja!/defaults/settings/main/fftv-experiments.json 12,918
omni.ja!/defaults/settings/main/tracking-protection-lists.json 12,916
FTL:
omni.ja!/localization/en-US/toolkit/global/textActions.ftl 11,167
omni.ja!/localization/en-US/toolkit/printing/printUI.ftl 11,155
omni.ja!/localization/en-US/toolkit/global/videocontrols.ftl 6,726
omni.ja!/localization/de/branding/brand.ftl 2,666
omni.ja!/localization/de/browser/branding/brandings.ftl 2,662
omni.ja!/localization/de/browser/branding/sync-brand.ftl 2,662
omni.ja!/localization/de/browser/browserSets.ftl 2,659
omni.ja!/localization/de/browser/sidebarMenu.ftl 2,659
omni.ja!/localization/de/browser/browserContext.ftl 2,659
omni.ja!/localization/de/browser/protectionsPanel.ftl 2,659
omni.ja!/localization/de/preview/interventions.ftl 2,659
omni.ja!/localization/de/browser/menubar.ftl 2,659
omni.ja!/localization/de/browser/downloads.ftl 2,659
omni.ja!/localization/de/browser/allTabsMenu.ftl 2,659
omni.ja!/localization/de/browser/places.ftl 2,659
omni.ja!/localization/de/browser/browser.ftl 2,659
omni.ja!/localization/de/browser/appmenu.ftl 2,659
omni.ja!/localization/de/browser/newtab/onboarding.ftl 2,658
omni.ja!/localization/de/browser/sync.ftl 2,653
omni.ja!/localization/de/browser/newtab/asrouter.ftl 2,590
omni.ja!/localization/de/browser/tabContextMenu.ftl 2,335
omni.ja!/localization/fr/branding/brand.ftl 1,757
omni.ja!/localization/fr/browser/branding/brandings.ftl 1,757
omni.ja!/localization/fr/browser/branding/sync-brand.ftl 1,757
omni.ja!/localization/fr/browser/newtab/onboarding.ftl 1,755
DTD:
omni.ja!/res/dtd/svg11.dtd 101,993
omni.ja!/res/dtd/svg10.dtd 8,881
omni.ja!/res/dtd/PropertyList-1.0.dtd 578
omni.ja!/res/dtd/svg11-tiny.dtd 205
omni.ja!/res/dtd/svg11-basic.dtd 45
omni.ja!/chrome/toolkit/content/global/DTD/xhtml1-strict.dtd 35
omni.ja!/res/dtd/xbel.dtd 33
omni.ja!/chrome/devtools/content/performance-new/DTD/xhtml1-strict.dtd 20
omni.ja!/chrome/devtools/content/memory/DTD/xhtml1-strict.dtd 19
omni.ja!/res/dtd/loose.dtd 7
omni.ja!/res/dtd/xhtml1-strict.dtd 2
omni.ja!/res/dtd/recoIndex.dtd 1
omni.ja!/res/dtd/ncx-2005-1.dtd 1
JS:
webcompat@mozilla.org.xpi!/shims/google-analytics.js 55,361
webcompat@mozilla.org.xpi!/shims/google-publisher-tags.js 23,174
webcompat@mozilla.org.xpi!/shims/empty-script.js 17,050
webcompat@mozilla.org.xpi!/shims/facebook-sdk.js 8,268
webcompat@mozilla.org.xpi!/shims/google-analytics-legacy.js 6,560
%7B73a6fe31-595d-460b-a920-fcc0f8843232%7D.xpi!/xss/InjectionCheckWorker.js 1,604
uBlock0@raymondhill.net.xpi!/js/popup-fenix.js 1,096
uBlock0@raymondhill.net.xpi!/js/i18n.js 1,037
uBlock0@raymondhill.net.xpi!/js/udom.js 994
uBlock0@raymondhill.net.xpi!/js/vapi-client.js 950
uBlock0@raymondhill.net.xpi!/js/vapi-common.js 913
uBlock0@raymondhill.net.xpi!/js/vapi.js 826
%7B446900e4-71c2-419f-a6a7-df9c091e268b%7D.xpi!/popup/vendor-angular.js 810
%7B446900e4-71c2-419f-a6a7-df9c091e268b%7D.xpi!/popup/main.js 784
uBlock0@raymondhill.net.xpi!/lib/punycode.js 772
uBlock0@raymondhill.net.xpi!/js/fa-icons.js 731
support@lastpass.com.xpi!/static/js/popover.chunk.js 634
jid1-93CWPmRbVPjRQA@jetpack.xpi!/h1-vendors-main-popover-wallet.js 528
%7B446900e4-71c2-419f-a6a7-df9c091e268b%7D.xpi!/popup/vendor.js 434
%7B531906d3-e22f-4a6c-a102-8057b88a1a63%7D.xpi!/lib/single-file/processors/hooks 385
ductloanphuok@gmail.com.xpi!/js/contentScripts/contentAll.js 367
jid1-BoFifL9Vbdl2zQ@jetpack.xpi!/resources/jquery/1.12.4/jquery.min.jsm 356
jid1-MnnxcxisBPnSXQ@jetpack.xpi!/js/popup.js 342
jid1-MnnxcxisBPnSXQ@jetpack.xpi!/js/firefoxandroid.js 319
jid1-MnnxcxisBPnSXQ@jetpack.xpi!/js/htmlutils.js 312
XHTML:
omni.ja!/chrome/browser/content/browser/browser.xhtml 28,409
omni.ja!/chrome/browser/content/browser/aboutNetError.xhtml 15,930
omni.ja!/chrome/toolkit/content/extensions/dummy.xhtml 14,919
omni.ja!/chrome/toolkit/content/global/alerts/alert.xhtml 5,384
omni.ja!/chrome/toolkit/content/mozapps/downloads/unknownContentType.xhtml 5,019
omni.ja!/chrome/toolkit/content/global/commonDialog.xhtml 3,835
omni.ja!/chrome/toolkit/content/global/backgroundPageThumbs.xhtml 1,299
omni.ja!/chrome/browser/content/browser/sanitize.xhtml 1,244
omni.ja!/chrome/browser/content/browser/places/bookmarksSidebar.xhtml 780
omni.ja!/chrome/devtools/content/inspector/markup/markup.xhtml 596
omni.ja!/chrome/devtools/content/inspector/index.xhtml 584
omni.ja!/chrome/toolkit/content/global/pictureinpicture/player.xhtml 556
omni.ja!/chrome/browser/content/browser/webext-panels.xhtml 505
omni.ja!/chrome/browser/content/browser/hiddenWindowMac.xhtml 479
omni.ja!/chrome/browser/content/browser/webrtcIndicator.xhtml 449
omni.ja!/chrome/browser/content/browser/aboutDialog.xhtml 445
omni.ja!/chrome/toolkit/content/mozapps/handling/appChooser.xhtml 188
omni.ja!/chrome/browser/content/browser/places/historySidebar.xhtml 181
omni.ja!/chrome/devtools/content/framework/toolbox-window.xhtml 179
omni.ja!/chrome/browser/content/browser/places/places.xhtml 176
omni.ja!/chrome/toolkit/content/mozapps/handling/permissionDialog.xhtml 148
newtabtools@darktrojan.net.xpi!/newTab.xhtml 124
omni.ja!/chrome/toolkit/content/global/datepicker.xhtml 105
omni.ja!/chrome/devtools/content/responsive/toolbar.xhtml 54
omni.ja!/chrome/browser/content/browser/preferences/dialogs/containers.xhtml 52
Properties:
%7B1018e4d6-728f-4b20-ad56-37578a4de76b%7D.xpi!/locales/en/tooltip.properties 1
%7B1018e4d6-728f-4b20-ad56-37578a4de76b%7D.xpi!/locales/ru/tooltip.properties 1
|
|
Assignee | |
Comment 3•3 years ago
|
||
most popular status by type:
Others:
NS_BINDING_ABORTED 443,025
NS_ERROR_FILE_NOT_FOUND 340,674
NS_ERROR_FAILURE 109,888
NS_OK 9,445
NS_ERROR_CORRUPTED_CONTENT 4,924
NS_ERROR_UNEXPECTED 2,328
NS_IMAGELIB_ERROR_FAILURE 369
NS_ERROR_ABORT 77
NS_ERROR_FILE_CORRUPTED 19
NS_ERROR_NOT_AVAILABLE 8
NS_ERROR_DOCSHELL_DYING 3
NS_ERROR_PARSED_DATA_CACHED 1
FTL:
NS_ERROR_FILE_NOT_FOUND 255,271
DTD:
NS_ERROR_FILE_NOT_FOUND 111,820
JS:
NS_BINDING_ABORTED 156,154
NS_OK 1,071
NS_ERROR_FILE_NOT_FOUND 218
NS_ERROR_FAILURE 1
XHTML:
NS_ERROR_PARSED_DATA_CACHED 65,940
NS_ERROR_FAILURE 15,460
NS_BINDING_ABORTED 740
NS_ERROR_FILE_NOT_FOUND 3
NS_ERROR_UNEXPECTED 1
Properties:
NS_BINDING_ABORTED 2
|
|
Assignee | |
Comment 4•3 years ago
|
||
After filtering out omni.ja!/res/dtd for DTD type I see only:
omni.ja!/chrome/devtools/content/memory/DTD/xhtml1-strict.dtd 19
omni.ja!/chrome/devtools/content/performance-new/DTD/xhtml1-strict.dtd 20
omni.ja!/chrome/toolkit/content/global/DTD/xhtml1-strict.dtd 35
which is a bit disappointing since we see 3 YSOD events with error_core 11 (UNDEFINED_ENTITY) in browser.xhtml on nightly channel. But maybe those three happened on a version that didn't update to our new probe yet, so I'll wait a bit. The planned narrowing down in bug 1693708 should still collect DTD errors that may cause the YSOD.
|
|
Assignee | |
Comment 5•3 years ago
|
||
for browser.xhtml we see:
NS_ERROR_PARSED_DATA_CACHED 28,403
NS_BINDING_ABORTED 6
We've seen 3 YSODs on browser.xhtml in the same time range, so most of those errors are unrelated. Maybe all are since the YSODs may be coming from builds prior to zero_byte_load landing but it would be great to see if any of the three was preceded by zero_byte_load.
|
|
Assignee | |
Comment 6•3 years ago
|
||
I now have two sessions with confirmed correlation between ysod event and zero_byte_load event!
I'm going to conceal the session ID because I'm not sure how public it can be, but I'll document the correlations:
The query I used was focused on YSODs coming from documents in builds newer or equal than 20210218:
with dataset as (
select
*
from
telemetry.events
where
event_category = 'ysod'
and submission_date >= '2021-02-18'
and normalized_channel in ('nightly')
),
pagedataset as (
select
*
from dataset
where
event_string_value not like 'data:image/svg%'
and build_id LIKE "20210218%"
-- event_string_value = "chrome://browser/content/browser.xhtml"
)
select * from pagedataset
Here's the result:
| submission_date | doc_type | normalized_channel | country | locale | app_name | app_version | os | os_version | timestamp | sample_id | event_timestamp | event_category | event_method | event_object | event_string_value | event_map_values | experiments | event_process | session_start_time | build_id |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2021-02-18 | event | nightly | IN | en-US | Firefox | 87.0a1 | Windows | 10.0 | 2021-02-18 15:09 | 44 | 298408 | ysod | shown | ysod | about:preferences | [{'f': [{'v': 'error_code'}, {'v': '3'}]}, {'f': [{'v': 'location'}, {'v': '1:1'}]}, {'f': [{'v': 'last_line'}, {'v': ''}]}, {'f': [{'v': 'last_line_len'}, {'v': '0'}]}] | [{'f': [{'v': 'rollout-doh-nightly-rollout-to-all-us-desktop-users-nightly-74-80-bug-1613481'}, {'v': {'f': [{'v': 'active'}, {'v': '93087308-efdb-4513-a8e2-42706a4c33d1'}, {'v': 'normandy-prefrollout'}]}}]}, {'f': [{'v': 'bug-1686593-pref-http3-on-nightly-nightly-86-88'}, {'v': {'f': [{'v': 'http3-default-off-control'}, {'v': '267392ab-7c51-4dcb-a0a9-1ec8429e77ac'}, {'v': 'normandy-exp'}]}}]}, {'f': [{'v': 'bug-1690684-pref-unenrollment-normandy-test-with-no-op-pref-in-fir-nightly-87-87'}, {'v': {'f': [{'v': 'control-boolean'}, {'v': 'e081d3b6-8249-46d8-bee7-cef5e6c4f2ee'}, {'v': 'normandy-exp'}]}}]}] | parent | 2021-02-19 04:09 | 20210218092411 |
| 2021-02-18 | event | nightly | EG | en-US | Firefox | 87.0a1 | Windows | 10.0 | 2021-02-18 17:36 | 43 | 2558 | ysod | shown | ysod | about:sessionrestore | [{'f': [{'v': 'error_code'}, {'v': '3'}]}, {'f': [{'v': 'location'}, {'v': '1:1'}]}, {'f': [{'v': 'last_line'}, {'v': ''}]}, {'f': [{'v': 'last_line_len'}, {'v': '0'}]}] | [{'f': [{'v': 'bug-1686593-pref-http3-on-nightly-nightly-86-88'}, {'v': {'f': [{'v': 'http3-enabled-treatment'}, {'v': 'c0f1be6e-d494-4088-a6e9-c0a6e466dd46'}, {'v': 'normandy-exp'}]}}]}, {'f': [{'v': 'bug-1690684-pref-unenrollment-normandy-test-with-no-op-pref-in-fir-nightly-87-87'}, {'v': {'f': [{'v': 'control-boolean'}, {'v': 'f1e7a36c-1b7b-4ce1-8752-5ca41cc28f3f'}, {'v': 'normandy-exp'}]}}]}] | parent | 2021-02-18 17:36 | 20210218092411 |
Case 1
| submission_date | doc_type | normalized_channel | country | locale | app_name | app_version | os | os_version | timestamp | sample_id | event_timestamp | event_category | event_method | event_object | event_string_value | event_map_values | experiments | event_process | session_start_time | build_id |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2021-02-18 | event | nightly | IN | en-US | Firefox | 87.0a1 | Windows | 10.0 | 2021-02-18 15:09 | 44 | 298408 | zero_byte_load | load | xhtml | [{'f': [{'v': 'sync'}, {'v': 'false'}]}, {'f': [{'v': 'file_name'}, {'v': 'omni.ja!/chrome/browser/content/browser/preferences/preferences.xhtml'}]}, {'f': [{'v': 'status'}, {'v': 'NS_BINDING_ABORTED'}]}] | [{'f': [{'v': 'rollout-doh-nightly-rollout-to-all-us-desktop-users-nightly-74-80-bug-1613481'}, {'v': {'f': [{'v': 'active'}, {'v': '93087308-efdb-4513-a8e2-42706a4c33d1'}, {'v': 'normandy-prefrollout'}]}}]}, {'f': [{'v': 'bug-1686593-pref-http3-on-nightly-nightly-86-88'}, {'v': {'f': [{'v': 'http3-default-off-control'}, {'v': '267392ab-7c51-4dcb-a0a9-1ec8429e77ac'}, {'v': 'normandy-exp'}]}}]}, {'f': [{'v': 'bug-1690684-pref-unenrollment-normandy-test-with-no-op-pref-in-fir-nightly-87-87'}, {'v': {'f': [{'v': 'control-boolean'}, {'v': 'e081d3b6-8249-46d8-bee7-cef5e6c4f2ee'}, {'v': 'normandy-exp'}]}}]}] | parent | 2021-02-19 04:09 | 20210218092411 | |
| 2021-02-18 | event | nightly | IN | en-US | Firefox | 87.0a1 | Windows | 10.0 | 2021-02-18 15:09 | 44 | 14080 | zero_byte_load | load | xhtml | [{'f': [{'v': 'sync'}, {'v': 'false'}]}, {'f': [{'v': 'file_name'}, {'v': 'omni.ja!/chrome/toolkit/content/extensions/dummy.xhtml'}]}, {'f': [{'v': 'status'}, {'v': 'NS_ERROR_PARSED_DATA_CACHED'}]}] | [{'f': [{'v': 'rollout-doh-nightly-rollout-to-all-us-desktop-users-nightly-74-80-bug-1613481'}, {'v': {'f': [{'v': 'active'}, {'v': '93087308-efdb-4513-a8e2-42706a4c33d1'}, {'v': 'normandy-prefrollout'}]}}]}, {'f': [{'v': 'bug-1686593-pref-http3-on-nightly-nightly-86-88'}, {'v': {'f': [{'v': 'http3-default-off-control'}, {'v': '267392ab-7c51-4dcb-a0a9-1ec8429e77ac'}, {'v': 'normandy-exp'}]}}]}, {'f': [{'v': 'bug-1690684-pref-unenrollment-normandy-test-with-no-op-pref-in-fir-nightly-87-87'}, {'v': {'f': [{'v': 'control-boolean'}, {'v': 'e081d3b6-8249-46d8-bee7-cef5e6c4f2ee'}, {'v': 'normandy-exp'}]}}]}] | parent | 2021-02-19 04:09 | 20210218092411 | |
| 2021-02-18 | event | nightly | IN | en-US | Firefox | 87.0a1 | Windows | 10.0 | 2021-02-18 15:09 | 44 | 7315 | zero_byte_load | load | xhtml | [{'f': [{'v': 'sync'}, {'v': 'false'}]}, {'f': [{'v': 'file_name'}, {'v': 'omni.ja!/chrome/browser/content/browser/browser.xhtml'}]}, {'f': [{'v': 'status'}, {'v': 'NS_ERROR_PARSED_DATA_CACHED'}]}] | [{'f': [{'v': 'rollout-doh-nightly-rollout-to-all-us-desktop-users-nightly-74-80-bug-1613481'}, {'v': {'f': [{'v': 'active'}, {'v': '93087308-efdb-4513-a8e2-42706a4c33d1'}, {'v': 'normandy-prefrollout'}]}}]}, {'f': [{'v': 'bug-1686593-pref-http3-on-nightly-nightly-86-88'}, {'v': {'f': [{'v': 'http3-default-off-control'}, {'v': '267392ab-7c51-4dcb-a0a9-1ec8429e77ac'}, {'v': 'normandy-exp'}]}}]}, {'f': [{'v': 'bug-1690684-pref-unenrollment-normandy-test-with-no-op-pref-in-fir-nightly-87-87'}, {'v': {'f': [{'v': 'control-boolean'}, {'v': 'e081d3b6-8249-46d8-bee7-cef5e6c4f2ee'}, {'v': 'normandy-exp'}]}}]}] | parent | 2021-02-19 04:09 | 20210218092411 |
Case 2
| submission_date | doc_type | normalized_channel | country | locale | app_name | app_version | os | os_version | timestamp | sample_id | event_timestamp | event_category | event_method | event_object | event_string_value | event_map_values | experiments | event_process | session_start_time | build_id |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2021-02-18 | event | nightly | EG | en-US | Firefox | 87.0a1 | Windows | 10.0 | 2021-02-18 17:36 | 43 | 2558 | zero_byte_load | load | xhtml | [{'f': [{'v': 'sync'}, {'v': 'false'}]}, {'f': [{'v': 'file_name'}, {'v': 'omni.ja!/chrome/browser/content/browser/aboutSessionRestore.xhtml'}]}, {'f': [{'v': 'status'}, {'v': 'NS_BINDING_ABORTED'}]}] | [{'f': [{'v': 'bug-1686593-pref-http3-on-nightly-nightly-86-88'}, {'v': {'f': [{'v': 'http3-enabled-treatment'}, {'v': 'c0f1be6e-d494-4088-a6e9-c0a6e466dd46'}, {'v': 'normandy-exp'}]}}]}, {'f': [{'v': 'bug-1690684-pref-unenrollment-normandy-test-with-no-op-pref-in-fir-nightly-87-87'}, {'v': {'f': [{'v': 'control-boolean'}, {'v': 'f1e7a36c-1b7b-4ce1-8752-5ca41cc28f3f'}, {'v': 'normandy-exp'}]}}]}] | parent | 2021-02-18 17:36 | 20210218092411 |
|
|
Assignee | |
Comment 7•3 years ago
|
||
With those two, I believe we have the initial evidence that NS_BINDING_ABORTED may cause YSOD. We see many NS_BINDING_ABORTED errors that do not result in YSOD, and there yet may come cases where YSOD in XHTML is not correlated with NS_BINDINGS_ABORTED but so far it is in both cases we have data for.
|
|
Assignee | |
Comment 8•3 years ago
|
||
Analyzing all NS_BINDING_ABORTED from xhtml files using the following query:
with dataset as (
select
udf.get_key(event_map_values, 'file_name') as file_name,
count(*)
from
telemetry.events
where
event_category = 'zero_byte_load'
and event_object = "xhtml"
and normalized_channel = 'nightly'
and submission_date >= '2021-02-18'
and udf.get_key(event_map_values, 'file_name') LIKE "omni.ja!/%"
and udf.get_key(event_map_values, 'status') = "NS_BINDING_ABORTED"
group by
file_name
)
select * from dataset
I'm getting:
omni.ja!/chrome/browser/content/browser/aboutNetError.xhtml 504
omni.ja!/chrome/browser/content/browser/webrtcIndicator.xhtml 53
omni.ja!/chrome/browser/content/browser/browser.xhtml 6
omni.ja!/chrome/browser/content/browser/preferences/preferences.xhtml 3
omni.ja!/chrome/toolkit/content/global/commonDialog.xhtml 3
omni.ja!/chrome/browser/content/browser/aboutSessionRestore.xhtml 2
omni.ja!/chrome/browser/content/browser/places/historySidebar.xhtml 1
omni.ja!/chrome/browser/content/browser/places/bookmarksSidebar.xhtml 1
omni.ja!/chrome/devtools/content/inspector/index.xhtml 1
omni.ja!/chrome/devtools/content/storage/index.xhtml 1
omni.ja!/chrome/browser/content/browser/aboutTabCrashed.xhtml 1
omni.ja!/chrome/browser/content/browser/webext-panels.xhtml 1
but only two document YSODs, one on preferences, one on sessionRestore.xhtml.
That means that there are many NS_BINDING_ABORTED which do not cause YSOD.
|
||
Updated•3 years ago
|
|
|
Assignee | |
Comment 9•3 years ago
|
||
2021-02-19 got us 5.3 million events!
I started by comparing the data from 19th to the previous day and it seems to mostly overlap, so I applied the same filters as in bug 1693708 to filter out irrelevant data.
First of all, I was able to find more correlations.
The XHTML YSOD at 1:1 with NO_ELEMENTS is (almost?) always preceded by zero_byte_load on the same file with NS_BINDING_ABORTED.
But XHTML YSOD at line 745:15 with UNDEFINED_ENTITY is preceded by zero_byte_load on brand.dtd with NS_ERROR_FILE_CORRUPTED.
Unfortunately, I don't yet see a strong correlation on preferences.xhtml NO_ELEMENTS 1:1 with zero_byte_load.
|
|
Assignee | |
Comment 10•3 years ago
•
|
||
After filtering we're handling just 284k events on 2021-02-19 out of 5.3m.
Almost all of it is coming from NS_ERROR_PARSED_DATA_CACHED which I'm tempted to rule out since it doesn't show up in YSOD sessions.
If I filter it out the most popular XHTML combinations are:
| File | Error | Count |
|---|---|---|
| omni.ja!/chrome/browser/content/browser/aboutNetError.xhtml | NS_ERROR_FAILURE | 50,981 |
| omni.ja!/chrome/browser/content/browser/aboutNetError.xhtml | NS_BINDING_ABORTED | 2,070 |
| newtabtools@darktrojan.net.xpi!/newTab.xhtml | NS_BINDING_ABORTED | 555 |
| omni.ja!/chrome/browser/content/browser/webrtcIndicator.xhtml | NS_BINDING_ABORTED | 150 |
| omni.ja!/chrome/browser/content/browser/blockedSite.xhtml | NS_ERROR_FAILURE | 106 |
| brief@mozdev.org.xpi!/background.xhtml | NS_BINDING_ABORTED | 64 |
| brief@mozdev.org.xpi!/ui/brief.xhtml | NS_BINDING_ABORTED | 50 |
| omni.ja!/chrome/devtools/content/storage/index.xhtml | NS_BINDING_ABORTED | 17 |
| omni.ja!/chrome/browser/content/browser/webrtcIndicator.xhtml | NS_ERROR_FAILURE | 15 |
| omni.ja!/chrome/browser/content/browser/browser.xhtml | NS_BINDING_ABORTED | 14 |
| omni.ja!/chrome/browser/content/browser/aboutSessionRestore.xhtml | NS_BINDING_ABORTED | 12 |
| omni.ja!/chrome/browser/content/browser/aboutNetError.xhtml | NS_ERROR_UNEXPECTED | 12 |
From this list, the non-omni.ja are good candidates to filter out, and aboutNetError.xhtml seems to be omnipresent with all types of errors, but no YSODs (none on nightly and beta, and 1 per day on some days on release).
This cuts us down from 5.3m to 1,274 events on that day. I think it's a good next step.
|
|
Assignee | |
Comment 11•3 years ago
|
||
After the second round of narrowing down in bug 1693955, using the data from 2021-02-19 I see the following:
types:
js: 641
xhtml: 394
dtd: 234
properties: 5
The good news is that it seems like properties are not seen in zero_byte_loads.
The bad news is that that may not be sufficient to claim they're not contributing to the problem, because we don't yet have a confirmation that every YSOD is predicted by zero_byte_loads. In particular, I see about:preferences NO_ELEMENTS/1:1 YSODs without that correlation, but I want to wait a couple days to see if the correlation strengthens.
For DTD the file names are:
| File | Error | Count |
|---|---|---|
| omni.ja!/chrome/toolkit/content/global/DTD/xhtml1-strict.dtd | NS_ERROR_FILE_NOT_FOUND | 87 |
| omni.ja!/chrome/devtools/content/performance-new/DTD/xhtml1-strict.dtd | NS_ERROR_FILE_NOT_FOUND | 86 |
| omni.ja!/chrome/devtools/content/memory/DTD/xhtml1-strict.dtd | NS_ERROR_FILE_NOT_FOUND | 58 |
| omni.ja!/chrome/en-US/locale/branding/brand.dtd | NS_ERROR_FILE_CORRUPTED | 2 |
| omni.ja!/chrome/en-US/locale/en-US/global/charsetMenu.dtd | NS_ERROR_FILE_CORRUPTED | 1 |
The top 3 are worth investigating. Could they cause YSODs we see?
the brand.dtd starts looking like a predictor of browser.xhtml|UNDEFINED_ENTITY|745:15 YSOD.
charsetMenu.dtd preceeded an YSOD in browser.xhtml|UNDEFINED_ENTITY|970:7.
XHTMLs:
File | Error | Count
-|-|-
omni.ja!/chrome/browser/content/browser/webrtcIndicator.xhtml | NS_BINDING_ABORTED | 150
omni.ja!/chrome/browser/content/browser/blockedSite.xhtml | NS_ERROR_FAILURE | 106
omni.ja!/chrome/devtools/content/storage/index.xhtml | NS_BINDING_ABORTED | 17
omni.ja!/chrome/browser/content/browser/webrtcIndicator.xhtml | NS_ERROR_FAILURE | 15
omni.ja!/chrome/browser/content/browser/browser.xhtml | NS_BINDING_ABORTED | 14
omni.ja!/chrome/browser/content/browser/aboutSessionRestore.xhtml | NS_BINDING_ABORTED | 12
This makes sense, and starts loosely correlate with YSODs. It's stil confusing as to why webrtcIndicator is at the top, but I'm willing to let this subset ride to beta and release and see how strong the correlation is at the higher volume.
And for JS:
| File | Error | Count |
|---|---|---|
| omni.ja!/chrome/browser/content/browser/aboutNetErrorCodes.js | NS_BINDING_ABORTED | 85 |
| omni.ja!/chrome/toolkit/content/global/certviewer/pkijs_bundle.js | NS_BINDING_ABORTED | 59 |
| omni.ja!/chrome/toolkit/content/global/certviewer/asn1js_bundle.js | NS_BINDING_ABORTED | 40 |
| omni.ja!/chrome/toolkit/content/global/certviewer/pvutils_bundle.js | NS_BINDING_ABORTED | 39 |
| omni.ja!/chrome/browser/content/browser/aboutNetError.js | NS_BINDING_ABORTED | 37 |
| omni.ja!/chrome/pdfjs/content/build/pdf.worker.js | NS_BINDING_ABORTED | 26 |
| omni.ja!/chrome/toolkit/content/global/certviewer/utils.js | NS_BINDING_ABORTED | 18 |
| omni.ja!/chrome/toolkit/content/global/certviewer/certDecoder.js | NS_BINDING_ABORTED | 17 |
| omni.ja!/chrome/devtools/modules/devtools/client/shared/source-map/worker.js | NS_BINDING_ABORTED | 17 |
| omni.ja!/chrome/toolkit/content/global/customElements.js | NS_BINDING_ABORTED | 16 |
This brings back the aboutNetError which is suspiciously high both here and in XHTML unless we filter it out.
Maybe there's something about cascading failure - if network errors out and that error causes us to load aboutNetError there's a higher likelihood that we'll error while loading it?
But then, we don't seem to get YSODs from it, neither directly nor via DTDs (it still uses them). So how is it possible that we do zero_byte_load it, and then no YSOD?
On the other hand, if we error-load .js file, does it error out the page?
Same goes for the next bunch - /certviewer/ ones - they're only loaded in browser in aboutNetError.xhtml.
Since aboutNetError.xhtml is a third-level UI, I don't think we should focus on those yet.
So far none of the top files related to browser.xhtml or preferences.xhtml.
Let's see how it looks on beta and release.
|
|
Assignee | |
Comment 12•3 years ago
|
||
Among other category the most popular file types in omni.ja loads are:
- JSON
- SVG
- update.locale
- html
- css
In partocular HTML and CSS seem like they may be contributing to the problem.
For JSON majority are dynamic requests and the error is FILE_NOT_FOUND, so hopefully handled ( I suspect our UI JS to handle missing JSON, but maybe not broken?)
|
|
Assignee | |
Comment 13•3 years ago
|
||
This is now completed. We also collected ~1 week of data from Release.
Description
•