1780403 - Crash investigation for pref synchronization in Nightly

Status: RESOLVED FIXED

(Core :: Preferences: Backend, defect)

Description

[Tom Ritter [:tjr] (OOTO until 7/21)]

This bug is to watch for crashes in Nightly and land patches to address things, if needed.

Comment 1

[Tom Ritter [:tjr] (OOTO until 7/21)]

The first and only crash so far is on 'services.settings.preview_enabled' which is a dynamic boolean pref, which means that user had set it to a string.

Comment 2

[Tom Ritter [:tjr] (OOTO until 7/21)]

We also have someone generating crashes on media.decoder-doctor.MediaPlatformDecoderNotFound.formats which is a dynamic pref that seems to be involved here: https://searchfox.org/mozilla-central/search?q=media.decoder-doctor.&path=&case=false&regexp=false

Comment 3

[Tom Ritter [:tjr] (OOTO until 7/21)]

Attachment: Bug 1780403: Allowlist media.decoder-doctor. r?freddyb

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9ac2887f5514
Allowlist media.decoder-doctor. r=freddyb

Comment 5

[Tom Ritter [:tjr] (OOTO until 7/21)]

I found a userchromeJS pref, which we are not going to support; but I tried to let that community know here. https://github.com/xiaoxiaoflood/firefox-scripts/issues/162

Comment 6

[Tom Ritter [:tjr] (OOTO until 7/21)]

Attachment: Bug 1780403: Allowlist the undocumented pdfjs.defaultZoomValue pref r?calixte

Comment 7

[Tom Ritter [:tjr] (OOTO until 7/21)]

It looks like there is an undocumented pdfjs.defaultZoomValue pref that can only be set via about:config and is a string (although it seems like it could be a number?)

Comment 8

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/9ac2887f5514

Comment 9

[Nick Alexander :nalexander [he/him]]

Locally, I'm also crashing on browser.region.update.updated. In Visual Studio debugger, I can see quite clearly:

DumpJSStack()
0 XPCU_defineLazyPreferenceGetter(aObject = "[object Object]", aName = ""lastUpdated"", aPreference = ""browser.region.update.updated"", aDefaultValue = "0") ["resource://gre/modules/XPCOMUtils.sys.mjs":289:36]
    this = [object Object]
1 <TOP LEVEL> ["resource://gre/modules/Region.jsm":72:11]
2 get DEFAULT_REGION() ["resource://autofill/FormAutofill.jsm":82:4]
    this = [object Object]
3 _isSupportedRegion(available = ""detect"", supportedCountries = ""US,CA"") ["resource://autofill/FormAutofill.jsm":106:0]
    this = [object Object]
4 get isAutofillAddressesAvailable() ["resource://autofill/FormAutofill.jsm":135:16]
    this = [object Object]
5 get isAutofillAddressesEnabled() ["resource://autofill/FormAutofill.jsm":156:0]
    this = [object Object]
6 get isAutofillEnabled() ["resource://autofill/FormAutofill.jsm":116:4]
    this = [object Object]
7 handleEvent(evt = [cross-compartment wrapper]) ["resource://autofill/FormAutofillChild.jsm":119:12]
    this = [object JSWindowActorChild]

<void>

Comment 10

[Tom Ritter [:tjr] (OOTO until 7/21)]

Attachment: Bug 1780403: Remove browser.region.update.updated from the blocklist r?nalexander

Comment 11

[Tom Ritter [:tjr] (OOTO until 7/21)]

It appears that there's a large swath of dynamic UI preferences: https://searchfox.org/mozilla-central/rev/ffb50da3ca89100b6ae5054cfe69c187679515f0/widget/nsXPLookAndFeel.cpp#212

Comment 12

[Tom Ritter [:tjr] (OOTO until 7/21)]

Attachment: Bug 1780403: Allowlist the ui. pref branch r?emilio

There's a large list of dynamic preferences in the ui. tree
but none of them should contain personally identifiable data.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0ae4582b58c5
Remove browser.region.update.updated from the blocklist r=nalexander

Comment 14

[Pascal Chevrel:pascalc]

Tom, this is turning into a top crash for our nightly population this morning, I think that if the patch on autoland doesn't reduce the crash volume, we should back out bug 1772345 while you work on this bug so as that we don't loose users on the channel.

Comment 15

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

agibson hit this with Should not access the preference 'browser.uitour.testingOrigins' in the Content Processes.

Comment 16

[Alex Gibson [:agibson]]

Here's my crash report for Comment 15:

https://crash-stats.mozilla.org/report/index/d15ad557-6ea9-4c41-aace-4c2670220721

Comment 17

[Tom Ritter [:tjr] (OOTO until 7/21)]

Attachment: Bug 1780403: Allowlist browser.uitour.testingOrigins r?agibson

Comment 18

[Randell Jesup [:jesup] (needinfo me)]

I've hit this multiple times in debug builds testing Photoshop on the Web; both the update.updated bug and gfx.blacklist.webrtc.hw.acceleration.h264.failureid (and I would assume all or most of the gfx.blacklist prefs are likewise).

I would also take a look at all the prefs in GetPrefNameForFeature() in GfxInfoBase.cpp in conjunction with a gfx person.

Comment 19

[Tom Ritter [:tjr] (OOTO until 7/21)]

Attachment: Bug 1780403: Allowlist the entire gfx.blacklist pref tree r?#gfx-reviewers

Comment 20

[Tom Ritter [:tjr] (OOTO until 7/21)]

Attachment: Bug 1780403: Disable pref sanitization in Nightly temporarily r?pascalc

Comment 22

[Tom Ritter [:tjr] (OOTO until 7/21)]

(In reply to Pascal Chevrel:pascalc from comment #14)

Tom, this is turning into a top crash for our nightly population this morning, I think that if the patch on autoland doesn't reduce the crash volume, we should back out bug 1772345 while you work on this bug so as that we don't loose users on the channel.

I've submitted a patch to disable it - we've gotten a lot of reports, and while I'm landing fixes as fast as I can, people only get the update so fast. Especially since I'll be away for the weekend, we should turn this off and let the reports die down and land all the fixes we see coming in from the tail of reports from un-updated users.

Unfortunately, we ran the full test test suite and ran an opt-in trial via dev-platform that failed to yield the results we're seeing here. I will take another look at some code patterns these reports surfaced, but when we re-enable it there will probably be another stream of reports. I may need to re-architecture this to use Telemetry Events or possibly some code I had written that was designed to limit a crash to a single instance. (Unfortunately it was parent-process only, so it may not work.)

Comment 23

[BugBot [:suhaib / :marco/ :calixte]]

:tjr, since this bug is a regression, could you fill (if possible) the regressed_by field?
For more information, please visit auto_nag documentation.

Comment 24

[BugBot [:suhaib / :marco/ :calixte]]

Copying crash signatures from duplicate bugs.

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6a54d8487859
Allowlist the ui. pref branch r=emilio

Comment 26

[Tom Ritter [:tjr] (OOTO until 7/21)]

Attachment: Bug 1780403: Allowlist some more preferences to avoid crashes r?ckerschb

Comment 27

[Tom Ritter [:tjr] (OOTO until 7/21)]

I've pressed the lando button several times in a row on un-attached-but-dependent revisions, but always in the correct landing order, so hopefully it works...

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/70bce5a0540b
Disable pref sanitization in Nightly temporarily r=Gijs

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a67cf746c61f
Allowlist browser.uitour.testingOrigins r=agibson
https://hg.mozilla.org/integration/autoland/rev/fe97b39c2a97
Allowlist the entire gfx.blacklist pref tree r=gfx-reviewers,jrmuizel

Comment 30

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ba45504aa0d6
Allowlist some more preferences to avoid crashes r=ckerschb

Comment 31

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/0ae4582b58c5

Comment 32

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cbebd35de4a1
Allowlist the undocumented pdfjs.defaultZoomValue pref r=calixte

Comment 33

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/6a54d8487859
https://hg.mozilla.org/mozilla-central/rev/70bce5a0540b
https://hg.mozilla.org/mozilla-central/rev/a67cf746c61f
https://hg.mozilla.org/mozilla-central/rev/fe97b39c2a97
https://hg.mozilla.org/mozilla-central/rev/ba45504aa0d6

Comment 34

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/cbebd35de4a1

Comment 35

[Intermittent Failures Robot]

23 failures in 664 pushes (0.035 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 23

Platform and build breakdown:

• linux1804-32-shippable-qr: 23
  • opt: 23

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1780403&startday=2022-07-21&endday=2022-07-21&tree=all

Comment 37

[Tom Ritter [:tjr] (OOTO until 7/21)]

Attachment: Bug 1780403: Allowlist capability.policy.policynames r?mkaply

Comment 38

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/24872f212643
Allowlist capability.policy.policynames r=mkaply

Comment 39

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/24872f212643

Comment 40

[Intermittent Failures Robot]

71 failures in 3497 pushes (0.02 failures/push) were associated with this bug in the last 7 days.

This is the #11 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 60
• autoland: 11

Platform and build breakdown:

• linux1804-32-shippable-qr: 71
  • opt: 71

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1780403&startday=2022-07-18&endday=2022-07-24&tree=all

Comment 41

[Tom Ritter [:tjr] (OOTO until 7/21)]

Attachment: Bug 1780403: Allowlist a few browser preferences r?ckerschb

Comment 42

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/78c46fea5a7d
Allowlist a few browser preferences r=ckerschb

Comment 43

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1772345

Comment 44

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/78c46fea5a7d

Comment 45

[Tom Ritter [:tjr] (OOTO until 7/21)]

We're not going to do any more work in this bug.