1447604 - Crash in PrefValue::Deserialize

Status: RESOLVED WORKSFORME

(Core :: Preferences: Backend, defect)

Description

[Calixte Denizet (:calixte)]

This bug was filed from the Socorro interface and is
report bp-14aafa25-1577-48cc-9697-369e00180321.
=============================================================

Top 7 frames of crashing thread:

0 XUL PrefValue::Deserialize modules/libpref/Preferences.cpp:303
1 XUL Pref::Deserialize modules/libpref/Preferences.cpp:910
2 XUL mozilla::Preferences::DeserializePreferences modules/libpref/Preferences.cpp:3416
3 XUL mozilla::dom::ContentProcess::Init dom/ipc/ContentProcess.cpp:228
4 XUL XRE_InitChildProcess toolkit/xre/nsEmbedFunctions.cpp:695
5 plugin-container main ipc/contentproc/plugin-container.cpp:50
6 plugin-container start 

=============================================================

There are 16 crashes (from 3 installations) in nightly 61 starting with buildid 20180320100122. In analyzing the backtrace, the regression may have been introduced by patch [1] to fix bug 1436911.

[1] https://hg.mozilla.org/mozilla-central/rev?node=6cffa8738ca5

Comment 1

[Nicholas Nethercote [inactive]]

I suspect this is one of those cases described in bug 1419432 comment 30, where the parent process and the content process end up with different builds. If so, things should work fine when the user restarts after the crash.

I have the following evidence for this.

- The first build that shipped with bug 1436911's change was 2018_03_20_220122. But some of these crashes have a build ID of 2018_03_19_220116 or 2018_03_19_100039 or 2018_03_18_100058, which is *before* the new code shipped. Which doesn't make any sense unless we have mismatched build ids for different processes. I assume the build id comes from the parent process; I know that these crashes are always happening in the content process.

- Most of the crashes (72%) have a build id of 2018_03_20_100122, which is the build immediately before the one that shipped the change. It makes sense that this mismatched build problem occurs most often in adjacent builds.

- I debugged a minidump and saw that the `prefsLen` value in ContentProcess::Init() was 77. This used to be a reasonable value, but since the change landed we are passing much more data, and so it should be much larger, something like 20,000 or more. This observation is consistent with the parent process running a pre-change build and the content processes running a post-change build.

- Likewise, the location of the crash makes sense if the parent is running a pre-change build and the content processes is running a post-change build.

If I'm right about this, the crash rate should drop to zero over the next few days. At the moment there is no sign of it in the March 21 builds. I will keep an eye on it.

Comment 2

[Nicholas Nethercote [inactive]]

The analysis from comment 1 is looking good. The crash rate has dropped off dramatically, and the few crash reports still coming in are from older builds.

Comment 3

[Nicholas Nethercote [inactive]]

We did get a handful of crashes from two installations that had a build id of 2018_03_20_220122, which is the first build that had bug 1436911's change, and so doesn't fit the hypothesis from comment one. I'm not sure how that could happen.

But the trend is clearly headed toward zero, so I think we can close this.

Comment 4

[Nicholas Nethercote [inactive]]

Note: RyanVM was wondering if this would show up again when we move to Beta and then Release. I believe that *won't* happen. Bug 1438678 introduced a new serialization format for prefs, and bug 1436911 changed that serialization format. It was the change from format 1 to format 2 that caused this crash. Because both of those bugs landed in 61, Beta and Release won't experience the transition from format 1 to format 2, and so should avoid the problem.