520115 - If compact is executed while standalone message window(or Tab) is open, mail window/tab is closed. Exception is reported to Error Console. Exception/Module may depend on "offset of mail is changed or not by Compact".

Status: REOPENED

(MailNews Core :: Backend, defect)

Description

[WADA:World Anti-bad-Duping Agency]

[Build ID]
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.4pre) Gecko/20090930 Shredder/3.0pre 

This is spin-off of problem of Bug 518920 Comment #4 in Shredder/3.0pre.

(1) If compact is executed while mail is displayed in standalone window, standalone window is closed and next exception is reported to Error Console.
> Error: [Exception... "Component returned failure code: 0x80004003 (NS_ERROR_INVALID_POINTER) [nsIMsgDBHdr.messageId]"  nsresult: "0x80004003 (NS_ERROR_INVALID_POINTER)"
> location: "JS frame :: chrome://messenger/content/folderDisplay.js :: FolderDisplayWidget_saveSelection :: line 258"  data: no]
> Source File: chrome://messenger/content/folderDisplay.js Line: 258
Same error is reported when mail is opened in tab, and tab is closed when compact is executed.
Bug 224825 can't happen with Shredder/3.0pre, because window/tab disappears.

(2) When Tb window(compact is executed) is closed after that, next exception is reported to Error Console.
> Error: uncaught exception: [Exception... "Could not convert JavaScript argument arg 0 [nsIObserverService.removeObserver]"  nsresult: "0x80570009 (NS_ERROR_XPC_BAD_CONVERT_JS)"
> location: "JS frame :: chrome://messenger/content/search.xml ::  :: line 293"  data: no]

[Step to reproduce]
(0) Disable auto-compact. Show "Order Received" column of local mail folder.
    Open Error Console.
(1) Move mail of largest "Order Received" value(offset in mail folder file)
    to other mail folder, and move it back.
(2) Open a mail(not largest offset, offset value is not changed by compact)
    in standalone window(or tab).
(3) Compact
(4) Standalone window/tab is closed. Exception occurs.
(5) Close Tb window. Exception occurs.

Comment 1

[Jonathan Protzenko [:protz]]

Dan, this is the issue I told you about a few days before. messageId holds an invalid reference when, for instance, the original message has been moved in the meanwhile, or the folder containing the original message has been compacted.

STR (from the JS point of view):
- get a reference to a msgHdr
- dump(msgHdr.messageId) //works fine
- compact the folder containing msgHdr
- dump(msgHdr.messageId)

The last line triggers Error: uncaught exception: [Exception... "Component returned failure code: 0x80004003 (NS_ERROR_INVALID_POINTER) [nsIMsgDBHdr.messageId]"  nsresult: "0x80004003 (NS_ERROR_INVALID_POINTER)"  location: "JS frame :: chrome://gconversation/content/selectionsummaries.js :: isNewConversation :: line 1766"  data: no]

Comment 2

[Dan Mosedale (:dmosedale, :dmose)]

This reaches down into message db, where I don't have any real experience or expertise.  Adding bienvenu and jcranmer, who might have some thoughts...

Comment 3

[David :Bienvenu]

it's the msgHdr that's not valid any more - if you plan on holding onto msg hdrs for an extended period of time, then you need to register and react to notifications that the db has gone away, or the folder has been compacted. Or you could remember the messageId instead of the msgHdr.

Comment 4

[Dan Mosedale (:dmosedale, :dmose)]

This seems like a somewhat surprising API, in the sense that it doesn't provide as much encapsulation as one might like.  

If it's not a significant win to change the API (which I'm guessing it isn't?), perhaps this bug wants to morph into at least documenting the object lifecycle model in nsIMsgHdr.idl?

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

In terms of the reported bug per comment 0,

Part 1 got fixed at some point for 3.1; onDestroyingView now invokes _saveSelection inside a try/catch.

Part 2 may still exist but needs its own bug and more recent info on what is going wrong.  (There could be platform issues involved so we'd want to see it with 1.9.2.)

Comment 6

[Andrew Sutherland [:asuth] (he/him)]

(bienvenu fixed in bug 536676)

Comment 7

[WADA:World Anti-bad-Duping Agency]

When offset of viewed mail is not changed by compact(comment #0), problem of comment #0 was not observed with Tb 3.1.1.
However, when offset of viewed mail is changed by compact, following error was shown at Error Console by Tb 3.1.1, and stand-alone mail window/Tab for mail are closed.
> Error: this.folderDisplay.view.dbView is null
> Source File: chrome://messenger/content/messageWindow.js
> Line: 302
When repeated test again(offset of viewed mail is changed by compact), following error was shown at Error Console, and stand-alone mail window/Tab for mail are closed.
> Error: this.folderDisplay.view.dbView is null
> Source File: chrome://messenger/content/messageWindow.js
> Line: 302
> Error: pref is not defined
> Source File: chrome://messenger/content/msgHdrViewOverlay.js
> Line: 399
> Error: OnMsgLoaded is not defined
> Source File: chrome://messenger/content/msgHdrViewOverlay.js
> Line: 614
> Error: displayAttachmentsForExpandedView is not defined
> Source File: chrome://messenger/content/msgHdrViewOverlay.js
> Line: 589

When I reported comment #0, I carefully excluded "offset is changed" case in order to show and focus on result only by "execution of compact", to show that problem occurs even if offset is not changed by compact.
Should we report detailed phenomenon of each slightly different condition and provide test cases for all of slightly different conditions?

Comment 8

[WADA:World Anti-bad-Duping Agency]

Re-opening to check about problem of comment #7.

Comment 9

[WADA:World Anti-bad-Duping Agency]

[Build ID]
> Mozilla/5.0 (Windows NT 5.1; rv:9.0a1) Gecko/20110911 Thunderbird/9.0a1
Following error still occurs in both offset-not-changed case and offset-changed case by; open a mail at stand alone window, compact folder. Stand alone window is closed when the error occurs.
> Error: this.folderDisplay.view.dbView is null
> Source File: chrome://messenger/content/messageWindow.js Line: 283

Comment 10

[WADA:World Anti-bad-Duping Agency]

Error of comment #9 was observed with Tb 6.0 too.

Comment 14

[mgoldey]

Confirming that this behavior still exists in TB 38.0b

Most often, compacting will change the contents of open message tabs, while leaving their title and heading information untouched.  My hunch is that the open tab then displays whichever message has, after compacting, the same internal ID # as the originally-opened message did before compacting (i.e., compacting re-orders the internal ID #, causing a new message to appear where the old one was.)  If, for some reason, the internal ID # is invalid, then the message tab closes, instead. 

(I know that I've described this behavior in a different bug report, but can't locate the report anymore.)

Comment 15

[mgoldey]

This is an update, originally posted here:  https://bugzilla.mozilla.org/show_bug.cgi?id=650267#c9

Additional information has been added.

I am using 40.0 and I just did a compaction of my inbox, with four open message tabs.  All of those message tabs closed themselves after the compaction completed.  This has happened three or four times in the past week.

The error console contains about 100 messages like these:

2015-09-10 09:57:03	gloda.index_msg	WARN	Observed header that claims to be gloda indexed but that gloda has never heard of during compaction. In folder: mailbox://[e-mail]/Inbox sketchy key: 1042354902 subject: Condolences - Ward

([e-mail] is a substitution, of course)

Normally, whenever I compact, I experience the behavior described in the first post:  each open tab displays the same content, typically the content of one of the tabs prior to compacting.  All of the open tabs are borked and unusable.

In the past year or so, I have occasionally noticed that one of the tabs will still contain the correct message and remain a viable, uncorrupted message tab.  Since I compact regularly, my hypothesis is that the compaction and any re-ordering of keys, etc., occurs within the most recent messages, leaving the structure of older messages alone.  Therefore, a sufficiently-old message in an open tab will keep its message key (or whatever the mechanism is) because there is nothing older than it or "above" it to compact.  Such a message in an open tab will remain usable after compaction.  But anything newer or "below" the location of the oldest deleted message -- the message whose former space is being compacted -- will be corrupted.

I have not seen auto-closing tabs lately. They may have closed automatically I opened and then closed the Help > About window right after compacting stopped.  I have a feeling that the "clean up unwanted windows" routine also picks up defunct message tabs.  Just a guess.

ADDITIONAL INFORMATION:  Tabs opened by doing a quick search (CTRL-K) are not auto-closed when compacting completes.  But, any conversation opened from the search results show the same content corruption as described above and elsewhere:  the contents of a seemingly random message are displayed.  To be clear, when results are returned from a quick search, clicking on one of the results opens a new tab that contains the entire conversation in a threaded view.  That's all fine.  After compaction, the body, etc., of the messages shown in the threaded view become corrupted:  when opened the message is no longer the message returned by the search results, but is instead a new, seemingly random message.  Presumably, this is because the message index or key has changed as a result of the corruption, and the search result now points to a different message than before compaction occurred.

Comment 16

[Ludovic Hirlimann [:Usul]]

Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.

Comment 17

[mgoldey]

Bug still present in TB 44.  I compacted with 6 open tabs.  5 of them were closed by Thunderbird when compacting completed.  The sixth (which was the one I had opened first) remained opened, but its contents had been replaced by the contents of the most recent message in my inbox.

Comment 19

[WADA:World Anti-bad-Duping Agency]

(In reply to mgoldey from comment #17)
> Bug still present in TB 44.  I compacted with 6 open tabs.

Can you check with Thunderbird 45?

A biggest reason why problem occurred :
  When mail at Offset=XXX(messageKey=XXX) is shown at Tab/Window,
  if compact is executed when deleted mail exist,
  Offset of the message is changed to PPP where PPP<XXX.
  And, messageKey was also changed to messageKey=PPP,
  so mail of messageKey=XXX doesn't exist any more.
  This happens because messageKey==Offset value was used.

In Tb 45, following change was made.
  When compact is executed, Offset of mail is changed from XXX to PPP as before,
  but messageKey=MMM for the mail is not changed and kept same messageKey=MMM.
  This is because messageKey!=Offset in Tb 45 or newer.

If problem of this bug occurs in Tb 45 or newer, it's following case.
  When mail at Offset=XXX(messageKey=MMM) is shown at Tab/Window,
  if compact is executed when deleted mail exists,
  Offset of the message is changed to PPP where PPP<XXX.
  But messageKey=MMM is kept, so problem doesn't occur.
  However, if Repair Folder is executed after Compact when deleted mails exist, messageKey is re-assigned.
  So messageKey of the mail is changed from MMM to LLL where LLL<MMM.
  Then, mail of messageKey=MMM doesn't exist any more.

Comment 20

[mgoldey]

Behavior still occurs in TB 50.0b3.  

Open a few messages in tabs, then go to older messages and delete a few and/or move them to subfolders, then compact.  The open message tabs are closed.

Error console reports:

2016-12-20 07:23:04	gloda.index_msg	WARN	Observed header that claims to be gloda indexed but that gloda has never heard of during compaction. In folder: mailbox://[user]@[domain]/Inbox sketchy key: 878411893 subject: Typical village home

There are at least 60 such WARNing messages, even though I only opened/moved 6 or 7 e-mails.  There may actually be a warning for every e-mail in the inbox received after the oldest message I deleted prior to compacting.

Comment 21

[WADA:World Anti-bad-Duping Agency]

(In reply to mgoldey from comment #20)
> Behavior still occurs in TB 50.0b3.

It looks same as "not moved case" in former releases.

Compact rebuilds msgDB using "mail folder data after compact". It's perhaps similar to "create new msgDB of nstmp/nstmp.msf, delete xxx/xxx.msf, change nstmp to xxx".
If so, it may be "msgDBHdr held in Tab/Window for messageKey=A before Compact != msgDBHdr for messageKey=A after Compact" or "msgDB for folder XXX held in Tab/Window before Compact != msgDB for folder XXX after Compact".

Warning message by Gloda is different issue from "close tab/window" also both issues ocuur when Compact occurs.

Comment 22

[WADA:World Anti-bad-Duping Agency]

Another guess.
 Message window/tab still expects messageKey=messageOffset in some codes, or expects non-changed messageOffset.
What is changed in msgDBHdr content by Compact is messageOffset.

I'll try to check "events passed to message window or tab when compact is executed" using eventListener.

Comment 23

[WADA:World Anti-bad-Duping Agency]

Following error message was observed in quick test with trunk nightly.
  Open a message in stand alone message window, Compact(because deleted mail exist, Compact is invoked)
> ReferenceError: reference to undefined property "QueryInterface"[Learn More]  msgMail3PaneWindow.js:1124:7
> 
> [Exception... "Component returned failure code: 0x80070057 (NS_ERROR_ILLEGAL_VALUE) [nsIMsgDBHdr.messageId]"  nsresult: "0x80070057 (NS_ERROR_ILLEGAL_VALUE)"  location: "JS frame :: chrome://messenger/content/folderDisplay.js :: FolderDisplayWidget_saveSelection/this._savedSelection.messages< :: line 316"  data: no]  (unknown)
> TypeError: this.folderDisplay.view.dbView is null[Learn More]  messageWindow.js:254:1

Above error itself can be called a kind coding error : doesn't care about "a property is not always initialized".
"if(this.folderDisplay.view && this.folderDisplay.view.dbView) use the dbView property;" like code is a simple bypass of exception.
Problem is: Why such state can occur. (in this bug, it's "Compact happened", but why such state generated by Compact?)
Issue like following?
 Compact -> change in msgDB/msgDBHdr etc. -> try to refresh message window content
 Because "refresh message window content" takes a while, some properties of object is not initialized yet(dbView in above message).

Comment 24

[WADA:World Anti-bad-Duping Agency]

FYI.

As I wrote in bug 1083994, similar phenomenon can be easily observed by "folder rename while message in it is viewed".
  There is no need of "deleted" mails.
  Folder and mails in viewed folder surely doesn't exit any more after rename, so no confusion.
  "rename folder" is better steps to reproduce than compact in many cases of this bug. 
Compact can be called "create nstmp folder from FolderX, delete FolderX, rename nstmp to FolderX" from perspective of this bug.

Comment 25

[WADA:World Anti-bad-Duping Agency]

FYI.

Following is already used scope(and key/value) in Table=webappstore2 of webappstore.sqlite on Tb 45 and Daily.
> INSERT INTO "someTable" VALUES ("","tnetnoc.:chrome","tnetnoc.:chrome","name","King Yatter");
> 
> INSERT INTO "someTable" VALUES ("","gro.allizom.snodda.secivres.:https:443",
>   "gro.allizom.snodda.secivres.:https:443","discopane-url",
>   "https://services.addons.mozilla.org/en-US/thunderbird/discovery/pane/53.0a1/WINNT#
>     {[e-mail]%22:{%22name%22:%22DOM%20Inspector%22,
>     %22version%22:%222.0.16.1-signed%22,%22type%22:%22extension%22,
>     %22userDisabled%22:false,%22isCompatible%22:true,%22isBlocklisted%22:false},
>     [e-mail]%22:
>     {%22name%22:%22Custom%20Buttons%22,%22version%22:%220.0.5.8.9%22,
>     %22type%22:%22extension%22,%22userDisabled%22:false,%22isCompatible%22:true,
>     %22isBlocklisted%22:false},[e-mail]%22:
>     {%22name%22:%22SQLite%20Manager%22,%22version%22:%220.8.3.1-signed.1-signed%22,
>     %22type%22:%22extension%22,%22userDisabled%22:false,%22isCompatible%22:true,
>     %22isBlocklisted%22:false},%22{f69e22c7-bc50-414a-9269-0f5c344cd94c}%22:
>     {%22name%22:%22Theme%20Font%20&%20Size%20Changer%22,
>     %22version%22:%2250.0%22,%22type%22:%22extension%22,%22userDisabled%22:false,
>     %22isCompatible%22:true,%22isBlocklisted%22:false},[e-mail]%22:
>     {%22name%22:%22WinBackMyTrash%22,%22version%22:%220.1.0.020%22,
>     %22type%22:%22extension%22,%22userDisabled%22:false,%22isCompatible%22:true,
>     %22isBlocklisted%22:false},%22{972ce4c6-7e08-4474-a285-3208198ce6fd}%22:
>     {%22name%22:%22Default%22,%22version%22:%2253.0a1%22,%22type%22:%22theme%22,
>     %22userDisabled%22:false,%22isCompatible%22:true,%22isBlocklisted%22:false},
>     %22{e2fda1a4-762b-4020-b5ad-a41df1933103}%22:
>     {%22name%22:%22Lightning%22,%22version%22:%225.5a1%22,%22type%22:%22extension%22,
>     %22userDisabled%22:false,%22isCompatible%22:true,%22isBlocklisted%22:false},
>     %22{fdf1cc26-48c9-18af-3146-d16bf63810bd}%22:
>     {%22name%22:%22Shockwave%20Flash%22,%22version%22:%2224.0.0.186%22,
>     %22type%22:%22plugin%22,%22userDisabled%22:false,%22isCompatible%22:true,
>     %22isBlocklisted%22:false},%22custombutton://buttons/Thunderbird/custombuttons-button0%22:
>     {%22name%22:%22checkCachedDBs%22,%22type%22:%22custombuttons%22,
>     %22isCompatible%22:true,%22isBlocklisted%22:false},
>     %22custombutton://buttons/Thunderbird/custombuttons-button1%22:
>     {%22name%22:%22localStorage%22,%22type%22:%22custombuttons%22,
>     %22isCompatible%22:true,%22isBlocklisted%22:false},
>     %22custombutton://buttons/Thunderbird/custombuttons-button2%22:
>     {%22name%22:%22Adhoc%22,%22type%22:%22custombuttons%22,
>     %22isCompatible%22:true,%22isBlocklisted%22:false}}");
>
> scope is reversed site domain due to same origin policy. 
>   uri=chrome://content
>     -> scope=tnetnoc.:chrome
>   uri=https://services.addons.mozilla.org:443
>     ->  scope=gro.allizom.snodda.secivres.:https:443

"Yatter King" is my name for Windows login ID.
A mozilla-central feature looks to already use localStorage for chrome: uri.

Addon Manager looks to already utilize localStorage using HTTP uri for addon and JSON data.

Following is chrome: uri seen in Thunderbird.
> documentURI example in Thunderbird.
>  chrome://global/content/console.xul
>  chrome://inspector/content/inspector.xul
>  chrome://sqlitemanager/content/sqlitemanager.xul
>  chrome://messenger/content/messenger.xul
>  chrome://messenger/content/messengercompose/messengercompose.xul
>  chrome://messenger/content/messageWindow.xul
>  chrome://messenger/content/addressbook/addressbook.xul
>  chrome://messenger/content/activity.xul
>  chrome://messenger/content/SearchDialog.xul

In Thunderbird, site uri of chrome is always chrome://messenger/content.
So, if chrome: uri is used for data such as panacea.dat content, structured Key is needed for simple/easy utilization of localStorage.
To utilize localStorage easily, following are needed.
- sheme/protocol=https: && faked unique domain name for Thunderbird features
- new scheme/protocol support such as mail-folder:, mail-message: in same origin policy.

Comment 26

[WADA:World Anti-bad-Duping Agency]

Sorry, comment #25 is for other bug. Please ignore it.

Comment 27

[WADA:World Anti-bad-Duping Agency]

FYI.
See bug 1083994 comment #66 and bug 1083994 comment #69 for "similarity in rename folder, move folder, compact folder cases".
See bug 1083994 comment #103 for cause of exception of NS_ERROR_ILLEGAL_VALUE at ClearMessagePane in "imap folder rename while a message in the folder is shown at messagepane" case.

Comment 29

[doug2]

Problem active in 60.0b9

Comment 30

[doug2]

I am not seeing the symptoms of this bug (see bug 711204 marked as dup of this bug) in Daily 77.0.. 2020-04-30 and immediately previous. I would like to recommend NOT closing until at least Beta is tested.

Comment 31

[mgoldey]

I have an update on this bug. I have not noticed a problem in some time, but today, with TB 78.10, I was able to create one.

Steps to reproduce:

1. Open 3 or 4 tabs, each containing an e-mails from your inbox.

2. Delete some e-mails from the inbox, or move them into a subfolder. It probably doesn't matter which.

3. Manually start compacting

4. While the compacting is going on, delete another e-mail.

Result:

The e-mails within the open tabs will become corrupted. Specifically, although the subject of each e-mail displayed in each tab did not change, when I clicked on each tab, the body of the e-mail and the displayed headers (From, Subject and To) had changed. Each tab now displayed the headers and body of the same e-mail. The displayed e-mail was the item that was highlighted in my inbox.

Oddly, if I click "reply" on each tab, one tab did not respond, but the others opened a composition window with the correct information: the To, Subject and Body of the e-mail that was in the tab when I first opened in in step 1, above. I believe that the tab that did not open a composition window is for message that I had moved into a subfolder.

I realize that this is a little vague. It seems like the code that displays the message headers and subject on each open tab can't figure out what to reference any more after the compaction. But, after clicking reply, TB re-locates the message body, etc., and displays the correct information.

Comment 34

[mgoldey]

Since it's been a year, I want to add a comment, and also note that the problem I have experienced is not resolved.

Currently using TB 91.8.1 (Remember, this goes back to before TB 30)

I have 2 tabs open, Inbox, and a message received on April 12.

I deleted and moved a couple of messages from April 5. This so leaves highlighted an e-mail from April 6 (the e-mail immediately below the one I deleted).

I then compacted my inbox

RESULT: The Inbox tab is fine.

The April 12 message tab still shows the title of the April 12 message. But the contents of the tab belong to the April 6 message, the one that was left highlighted. That includes the headers. To all appearances, the April 6 e-mail had replaced the April 12 e-mail that I had left open in this tab, other than the tab title itself, which is unchanged.

Here's the worst part: If I reply to the e-mail in the open tab,, TB picks up the contents of the original, April 12 e-mail, the one that's no longer displayed, and NOT the contents of the e-mail that is actually displayed.

Comment 36

[mgoldey]

Revisiting this for December 2022. TB 102.5.1.

I have three tabs open in addition to the main message window. One open tab contains an e-mail that's 7 or 8 years old. The other two are more recent. In the main message window, I have a message highlighted and visible in the message pane, which happens to have the subject 'InnSight' and happened to arrive yesterday. That message is not open in a separate tab.

TB asks me if I want to compact. I say yes. When it's done, all three of the open tabs contain the title "InnSight" and contain the body of the InnSight e-mail. The other three e-mail messages that were open when compacting started are now gone (although they remain in my inbox; they weren't actually deleted or anything like that).

If I reply to any of these InnSight e-mails, the reply is to that e-mail, which is a change from the behavior I reported in Comment 34