1342858 - WARNING: some msg dbs left open: '!m_dbCache.Length()' - for last folder that was open in the window and a message was viewed in it?

Status: NEW

(MailNews Core :: Database, defect)

Description

[Jorg K (CEST = GMT+2)]

While working on bug 1334874, bug 1337865 and bug 1342745 which tried to stop leaking nsMsgDBFolder (and nsMsgNewsFolder) objects since their leakage causes the "dynamic atoms leak" assertion, we noticed - what was known before - that various databases also leak, for example running mozilla/mach xpcshell-test mailnews/base/test/unit/test_nsMsgDBView.js gives

"[4932] WARNING: some msg dbs left open: '!m_dbCache.Length()', file c:/mozilla-source/comm-central/mailnews/db/msgdb/src/nsMsgDatabase.cpp, line 87"
"db left open c:\\users\\jorgk\\appdata\\local\\temp\\firefox\\xpcshellprofile\\Mail\\Local Folders\\Trash.msf"
"db left open c:\\users\\jorgk\\appdata\\local\\temp\\firefox\\xpcshellprofile\\Mail\\Local Folders\\gabba0.msf"
(and more)

In fact, looking for this warning in a recent Xpcshell test log I found this warning eight times (which not as bad as I thought).

It's about time we look into this problem.

Chiaki, I know you've been alerting us to this problem for a long time, so finally the penny has dropped ;-)

Please don't paste long logs into comments here, if you have logs that you think might help us, please attach them.

Comment 1

[ISHIKAWA, Chiaki]

Great I will look into this issue finally this week.

Comment 2

[:aceman]

From my experiments, the database left open seems to always be the one last one that was open in the window and a message was viewed in it.

Comment 3

[Jorg K (CEST = GMT+2)]

(In reply to :aceman from comment #2)
> From my experiments, the database left open seems to always be the one last
> one that was open in the window and a message was viewed in it.
Hmm, I looked at warnings in Xpcshell tests and they don't view a message in a window ;-)
BTW, we're down to six of these warnings, all from failing tests from bug 1342745.
I've just looked at a Mozmill run and there are also six of these warnings.

Comment 4

[Jorg K (CEST = GMT+2)]

Hmm, that stuff is a mystery:

Always looking at Windows debug X and Z:

First number: Matches on: m_dbCache.Length, second number: Matches on: "left open"

Fri Mar 03, 2017 18:05:57 - X:6 - Z:18 - 168
Fri Mar 03, 2017 19:56:17 - X:6 - Z:19 - 171
Mon Mar 06, 2017 11:16:18 - X:6 - Z:17 - 144
Tue Mar 07, 2017 18:44:22 - X:0 - Z:0 - Zero!!
Wed Mar 08, 2017 09:00:10 - X:0 - Z:18 - 170
Skipping to the present:
Sat Apr 22, 2017  0:53:25 - X:0 - Z:18 - 82

After landing bug 853881:
Sat Apr 22, 2017 08:49:35 - X:? - Z:16 - 58

Some try results:
Note that with patch (v4) of bug 853881 I get Z: 16 - 59, with patch (v3) it's Z: 18 - 80, but maybe that's because with (v4) the application crashes and some tests aren't run.

Comment 5

[Jorg K (CEST = GMT+2)]

Some figures on the 1st of July 2017:
Matches on: m_dbCache.Length: 18
Matches on: left open: 187!!!

So something has changed in cycle collection. Also, debug builds now crash after reporting the left-open databases, see bug 1377692.

Comment 6

[ISHIKAWA, Chiaki]

I think I have finally figured it out in bug 1677202.

Comment 7

[ISHIKAWA, Chiaki]

Hi, I think I have a patch that solves the problem albeit some verbosity issue.
Can you take a look at
https://bugzilla.mozilla.org/show_bug.cgi?id=1677202#c11
and offer comments there?

Comment 8

[:aceman]

Done, thanks for looking into it.

Comment 10

[ISHIKAWA, Chiaki]

Please see https://bugzilla.mozilla.org/show_bug.cgi?id=1677202#c19

The gist of the issue seems to be m_dbCache references an object, but it is reclaimed by a garbage collector called during shutdown.
So, ..., quoating from the URL above,

How do we make sure an object of which pointer is in m_dbCache[...] (m_dbCache.ElementAt(i)) is not reclaimed by GC during shutdown?

Is there a mechanism to explicitly reference such an object from a system-wide array of pointers that contain a reference to objects so that GC won't reclaim the object since there IS valid reference to the object?

I have seen such mechanisms in Lisp Interpreters to make sure premature GC does not occur when a system function/primitive is operating.

For example, such a mechanism is in GNU Emacs, for example. When a C source code is written to reference an object used inside Emacs-Lisp interpreter, such C source code uses a macro to store the pointer to an object into a static array inside Emacs-Lisp so that a GC triggered by a function call chain from the C source code will not, by chance, eliminate the object. : the reference to the object is in the static array in Emacs-Lisp, and so the object is not eliminated by GC.
After the processing in C source code, the pointer should removed from the static array to avoid the memory leak. There is a macro to do this. The object will be properly eliminated during the subsequent GC if there is no other reference to the object.

TIA