700229 - Unpredictable but Frequent: Deleted/Junked messages not removed from pop Inbox

Status: RESOLVED WORKSFORME

(MailNews Core :: Networking: POP, defect)

Description

[Ruth Milner]

User Agent: Mozilla/5.0 (Windows NT 5.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
Build ID: 20110928134238

Steps to reproduce:

Delete a message (in the case of Trash problems). Mark a message as Junk, either manually or automatically (in the case of Junk problems).


Actual results:

Beginning with Thunderbird 3.x (multiple versions beginning over a year ago) and still happening in TB 7.0.1 (installed two days ago):

Anywhere from a few minutes to a few days after startup, Thunderbird gets into a state where one or both of the following starts happening:
1. Deleted messages may or may not appear in Trash folder but are not removed from Inbox. If Trash folder is viewed, cursor usually stays as hourglass and no actions can be taken on messages there.
2. Junked messages do not get moved anywhere despite Move Message To Junk settings (which normally work). If Junk folder is viewed, cursor stays as hourglass and no actions can be taken on messages there. Newly-delivered spam is marked as Junk but stays in Inbox.
3. Inbox folder view becomes frozen until another folder is viewed and then I return to Inbox.
"Empty Trash" works when Trash is stuck this way but does not fix it. Attempting to compact the Junk folder reports that it cannot because "another operation" is in progress.
In both cases, the problem does not fix itself. Once it starts happening, the only cure I have found is to exit and then restart after several minutes. Removing or repairing .msf files - whether specific ones or all of them - does not prevent the problem from recurring later. After restart, the messages look like they did before exiting, except that redoing the Delete or Mark As Junk on them works normally. Sooner or later, however, the problem starts happening again. It happens more often with Trash than with Junk.


Expected results:

Deleted messages should have been moved to Trash and removed from Inbox. Junked messages should have been moved to Junk folder and removed from Inbox. All folders should have remained functional.

Possibly related, or perhaps a clue to the problem: occasionally when Junk or Trash is stuck like this, if I view the folder contents TB displays the IMAP placeholder message that lives at the beginning of every mail "folder" an IMAP server processes. Sometimes it will go away later. Sometimes I don't see this (it may only appear briefly). This never showed up before TB 3.x, and TB should know enough to ignore it. I mention it because the first appearance of this glitch coincided with the problem reported here.

NOTE: All folders involved are fairly small. Inbox is typically 7-10MB. Junk contains 30 days' worth (automatically trimmed by TB) and is about 10MB. I empty Trash and compact folders every evening routinely.

Comment 1

[:aceman]

When that happens, is there anything in Tools->Error console?

Also, is this on IMAP account?

Comment 2

[Ruth Milner]

Yes, this is on an IMAP account, one that I've been using since 2004 with no problems until the upgrade to TB 3.x about a year ago.

When I first brought up the Error Console, virtually everything there was warnings about fonts or HTML parsing in specific messages. The only other one was this:

Warning: Use of getAttributeNodeNS() is deprecated. Use getAttributeNS() instead.
Source File: chrome://messenger/content/messenger.xul
Line: 0

The Trash/Junk problem happened again while I was looking at the console, but when I highlighted a message in my Inbox and tried to delete it (whether with the Delete key or by right-click->Delete), no new errors appeared in the Error Console. Likewise when I marked a message as Junk and it didn't get moved, no changes to the list of errors.

Prior to exiting TB, I did Empty Trash and compact, and as usual it complained that Junk could not be compacted. One new error did show up then:

Error: An error occurred executing the cmd_compactFolder command: TypeError: this._resetThreadPane is not a function
Source File: chrome://global/content/globalOverlay.js
Line: 100

but of course the real problem was already in effect before then. It might be related to the "other operation in progress" pop-up error that I get.

After one restart, Trash was fixed but Junk was still broken so I restarted again after waiting a while. The only error both times was that same deprecated function (which appears to show up at every TB startup).

The problem has occurred about 10 times since the upgrade to 7.0.1 few days ago - far worse than with 3.x. I'll be away from Thunderbird for a while after tomorrow but will be sure to post updates on any other errors/occurrences before I go. Please let me know if there is anything else I can check for you. This is driving me crazy.

Comment 3

[Ruth Milner]

Watching today, I've observed that the errors I listed above all occur during normal operation, along with a few additional deprecated functions and such. So although they may be problems, they don't appear to be part of *this* problem.

Comment 4

[Ruth Milner]

Back on Thunderbird now and the problem has recurred (~36 hours after starting TB). 

I have been monitoring the Error Console and nothing new showed up there at the time it began acting up. The most frequent error is "aFolder is undefined", but that was being logged sporadically since TB started. I can't correlate it with anything in particular.

Perhaps more significantly, the special-purpose IMAP message I described above showed up in the Trash and Junk folders shortly before TB stopped automatically moving spam to Junk or removing deleted messages from Inbox.

FWIW, the Integrity and Coherence Checks in the log claim that the database is both sane and coherent.

Sidebar suggestion: please timestamp the entries in the Error Console. Would be nice to have more precise data, as well as to know if any of these are occurring in clusters or may be associated with each other.

Comment 5

[Ludovic Hirlimann [:Usul]]

Do you have extension installed ?

Comment 6

[Ruth Milner]

The only extensions I have to TB are Lightning (currently at 1.0b7) and Extra Folder Columns (1.1.4). I never used Lightning and am happy to disable it. The folder columns one provides information vital to me (which ought to be built into TB).

The problem has been quieter this past week. I'm waiting for it to have one of those days where it acts up 3 or 4 times. :-} Is there something I can do when it does occur, like forcing a core dump or some such, that would supply more concrete information about the program status at the time?

Comment 7

[Ruth Milner]

Well, I didn't have to wait very long. <sigh>

Shortly after writing the above, I noticed both Junk and Trash claiming one new message but there weren't any when I looked at them. This is what usually happens when that IMAP placeholder message shows up and then "goes away" again. Now TB has stopped handling deleted and Junked messages properly again.

Some other important points:

 1. All the Thunderbird files/folders reside on my own computer.

 2. Thinking about that point and looking more closely, I realized that I have TB set up to use POP, not IMAP, to download new messages. My humble apologies for the previous misinformation. I set that up a long time ago and got confused by the appearance of those IMAP placeholder messages which began with TB 3.x.

 3. A connection that then occurred to me, looking at my POP server settings in TB: I have selected "Leave messages on server" "Until I delete them". This works with messages that are deleted or Junked - the latter are subsequently moved to Trash by TB automatically - but not with messages that are filed into other folders, which still behave normally when this problem happens. (Technically I suppose those have not been "deleted", though that's a tricky distinction.) I wonder if this has any bearing on the problem. I've had that setting in effect for years (don't remember exactly how long, although per Bug 235741 the option existed in TB 1.0 if not before), but a bug could have appeared in 3.x. The mail host for my account runs Postfix, with Dovecot to handle both POP and IMAP; I will ask which version.

I will leave TB as is for a while in hopes that someone will suggest something else I can do to get more information while the problem is happening.

Also, after I next restart, I will keep an eye on the Trash count and see if it has just been given a batch of auto-deleted spam at the time the problem occurs. Perhaps there is some weird interaction when it tells the server to remove them - though if so, it's odd that it doesn't happen every time (a few times a day, every day).

I did a new search on Bugzilla for "until i delete them" and other relevant strings, but still did not find anything like this behavior reported. However, there are similar descriptions on other sites, e.g.

http://getsatisfaction.com/mozilla_messaging/topics/cannot_delete_in_thunderbird_3_0 (especially see Oct 17 2010 posts)

so this problem has been encountered by others, starting around the same time.

Comment 8

[Ruth Milner]

FYI, Dovecot is version 1.0-1.4.rc15 .

I will try to restart TB using a .bat file to enable POP3 logging thus:

  set mydate=%date:~-4,4%%date:~-7,2%%date:~-10,2%
  set mytime=%time:~0,2%%time:~+3,2%
  set NSPR_LOG_MODULES=POP3:5
  set NSPR_LOG_FILE=%USERPROFILE%\thunderbird_%mydate%_%mytime%.log
  start /d "c:\program files\mozilla thunderbird" thunderbird.exe

(based on info found on the Dovecot Wiki)

Comment 9

[Ruth Milner]

Just an update since it has been a week or so.

First, I used the .bat file to run TB with logging. Unfortunately, it produced such large logfiles (over 150MB per 24 hr) that it was not particularly useful. (I tried changing "POP3:5" to "POP3:4" to see if that might affect the amount of detail, but AFAICT it didn't make any difference.) So I had to stop that. It didn't act up at all during that couple of days anyway.

Second, I disabled the Lightning add-on which comes enabled in TB by default.
That was the only configuration change I have made so far. In the week since then, the problem has not recurred. However, since TB 3+ has occasionally gone this long before without the glitch, I will wait another couple of weeks at least before I point the finger at Lightning. (And I'm at a bit of a loss to see why it would have any effect on the Trash and Junk folders since it doesn't even use them, does it?)

Comment 10

[:aceman]

Thunderbird doesn't come with Lightning preinstalled. Where did you get this TB bundle from?

Comment 11

[Ruth Milner]

I got it through the upgrades that TB prompts for, first 2.x-3.x (at least a year ago now) and then 3.1(.16?) to 7.0 about a month ago. If Lightning wasn't preinstalled in 3.0, it may have been suggested by the post-upgrade page; I don't recall exactly, but I know I didn't go out to look for it and then install it by hand. And if it isn't a default part of 7.0, it was probably kept during the 3.x-7.0 upgrade because it was already there.

On the rare occasions in history when I have manually installed any Mozilla-family product, I've always retrieved the install files from the Mozilla Web site.

Comment 12

[Ruth Milner]

So, it finally happened again.

I noticed the Trash folder showed a new message, and it proved to be one of those bogus messages. Despite showing up in the TB Trash folder, when I took a look at the corresponding file directly, there was nothing that corresponded to it. The message had been automatically marked as Junk, and then apparently moved to Trash automatically because of its apparent 12/31/1969 date.

After this I could no longer delete messages from the Inbox.

Nothing shows up in the Error Console that hasn't been showing up regularly all along, e.g.

  Error: aFolder is undefined
  Source File: chrome://messenger/content/folderWidgets.xml
  Line: 691

which is very frequent. Also stuff like this:

  Warning: Use of attributes' specified attribute is deprecated. It always returns true.
  Source File: about:blank
  Line: 0

is common.

Evidently disabling Lightning did not actually fix it. It isn't clear whether that was responsible for the greatly-reduced frequency of the problem, though.

Comment 13

[Ruth Milner]

OK, I know what operation is causing it: it's something to do with the automatic move of messages from Junk to Trash when they hit the expiry time (in my case, set to 30 days). In particular, I think it's related to those bogus messages with the "zero" date (12/31/1969).

Recall that I had a long period when the problem did not show up except when a couple of messages with that bogus old date were marked as spam and deleted (see comments #7 and #12). Most of this period is 30 days after the time I was out of town (Nov 8-18). When I'm away from home, I access my email through the Web and clean up new spam by hand so it doesn't clutter what I'm looking at. As a result, when I get back to Thunderbird and download messages with POP, there's a time gap in the Junk contents.

Today is the 19th, i.e. 30 days after I got back to Thunderbird. This morning I deleted a bunch of overnight messages with no problem. Shortly afterward, I deleted another and it didn't go away. Trash was hung for actually doing anything, but I can look at what's in it, and three newly-expired spam messages (dated early morning Nov 19) are now there.

Looking at the Error Console, I see the one that always shows up when I compact folders ("An error occurred executing the cmd_compactFolder command: TypeError: this._resetThreadPane is not a function"), which I did as usual late last night, followed by several complaints about an email I was viewing this morning which had a bunch of image attachments. This is the one I attempted to delete and couldn't. So the glitch occurred in the time between.

Those three spam emails were expired and moved from Junk to Trash by TB while I was viewing the image email. However, Trash also shows 1 unread message in the folder pane, even though there isn't one displayed in the message list. This is what happens when one of those bogus messages appears and then goes away before I can look at it. (If I see the unread-message count increment and look at it immediately, as I did in Comment #12, it will be there; otherwise, it will be gone by the time I look, as in #7.)

I had noticed with TB 3.x that Junk expiry was sporadic, and that 7.0 does it much more often. This would explain why the problem I reported suddenly got much worse after that upgrade. Note that the problem doesn't happen every time Junk messages are expired, so I'd bet it's because the more frequent clean-up in 7.0 catches these bogus messages more often.

Of course, I can't explain *why* this is a problem. Doing a repair on the index for Trash or Junk when this happens is not enough to make them useable again; I have to restart TB. But I don't know why these messages appear in the first place. If I search for Thunderbird bugs (all status) with the string "1969", several reports of these exist. Most of them are marked as duplicates of bug #209501, but the most recent report, bug #664645 from June 15 this year, is not. My problem is not simply that they exist, but that TB moving them from Junk to Trash makes Bad Things happen.

Comment 14

[:aceman]

(In reply to Ruth Milner from comment #13)
> Looking at the Error Console, I see the one that always shows up when I
> compact folders ("An error occurred executing the cmd_compactFolder command:
> TypeError: this._resetThreadPane is not a function"), which I did as usual

This seems to be bug 696347.

Comment 15

[Ruth Milner]

(In reply to :aceman from comment #14)

> This seems to be bug 696347.

Not at all.

That bug is simply about the error that appears during the compact. Yes, I see that error too, but it does not cause any problems for me at all. I only mentioned it because - lacking an error timestamp since I don't run the nightly builds - it establishes a time frame for when the bug I reported occurred today.

The bug I am reporting is the freezing of the Trash folder and sometimes Junk as well. This is not at all related to the error during compact. As I reported just a few minutes ago, I did a compact last night and deleted a whole bunch of messages this morning - hours later. The freeze did not occur until messages were auto-expired from Junk and moved into Trash.

Please do not mark this as a duplicate. They are not the same thing at all.

Comment 16

[Ruth Milner]

(In reply to :aceman from comment #14)

> This seems to be bug 696347.

Sorry, I may have overinterpreted your comment. If all you meant is that the routine error I see during compact is bug 696347, then yes, that is true.

I was alarmed that you might be thinking the problem I am actually concerned about/reporting on is that bug as well - which it isn't.

Comment 17

[:aceman]

Yes, don't worry. I only quoted the part of your comment with error message. Only that part is already covered in other bug.

The rest of your comment is still valid and seems unique. Good analysis!

Comment 18

[Ruth Milner]

This problem is still occurring in TB 11.0.1 .

It is definitely related to that phantom mail message with a default timestamp. (Based on other reports about these messages, e.g. bug #209501, the displayed time depends on the user's time zone.) I have found that on the very rare occasions when I spot it before it disappears, if I manually delete it - including out of Trash - that will prevent the Junk and Trash folders from hanging. However, because it doesn't stay visible for long, I don't often catch it this way.

As previously reported, the problem shows up anywhere from once every few days to several times per day. A couple of days ago I had to restart TB half a dozen times in one afternoon. It is driving me nuts.

If someone can find the source of the bogus message and fix that, it would also fix this problem.

Comment 19

[Wayne Mery (:wsmwk)]

perhaps junk processing fails in some way if part of header is bogus?

Comment 20

[Ludovic Hirlimann [:Usul]]

Can we get an example message so we can try to figure out what's going on ?

Comment 21

[Ruth Milner]

(In reply to Ludovic Hirlimann [:Usul] from comment #20)
> Can we get an example message so we can try to figure out what's going on ?

Do you mean the phantom messages? If so, as far as I can tell, there is no message. All the header fields are shown as empty except the date, which is interpreted as the zero date by TB - almost certainly because it's empty (null?) too. I have never been able to find anything in the raw file that corresponds to these messages. They may be caused just by an extra blank line somewhere, or possibly a special character that doesn't appear in text viewers/editors (I normally use "less" and "vim" but perhaps there's an option that shows non-printable characters). Remember that TB only thinks they're there for a very brief period; it's really rare that I am able to even spot them in the message list. Typically TB will indicate in the folder pane that Junk and/or Trash have a new message, but when I look at the message list, nothing there is shown as being unread. (The phantom messages always show up as unread.)

You might get more info asking on bug #209501, which has many followers. It deals simply with the existence of the phantom messages; what I reported here is the impact I am seeing from them, which appears to be much less common - unique, if you go by the number of people reporting it. <sigh> Some combination of TB settings, I'd guess, but I can't tell which ones might matter or why. At any rate, this bug almost certainly depends on that one.

Comment 22

[Wayne Mery (:wsmwk)]

(In reply to Ruth Milner from comment #21)
> (In reply to Ludovic Hirlimann [:Usul] from comment #20)
> > Can we get an example message so we can try to figure out what's going on ?
> 
> Do you mean the phantom messages? If so, as far as I can tell, there is no
> message. All the header fields are shown as empty except the date, which is
> interpreted as the zero date by TB - almost certainly because it's empty
> (null?) too. I have never been able to find anything in the raw file that
> corresponds to these messages. They may be caused just by an extra blank

it's not clearto me how all issues of comment 0 (#1, #2, #3) are related.

what antivirus SW do you use?
and does problem reduce if you disable compact in tools | options | advanced | nework & disk ?

Comment 23

[Ruth Milner]

(In reply to Wayne Mery (:wsmwk) from comment #22)
> it's not clear to me how all issues of comment 0 (#1, #2, #3) are related.

They're related in that they are all behaviors that happen together when the problem occurs. You can add to that list the appearance (and incomplete disappearance unless I catch them in time to delete them manually) of the phantom messages, which I had not yet realized were associated with the problem at the time of that first report.

> what antivirus SW do you use?

Norton. This problem has been happening to me for about two years, on two different computers running different versions of Windows (XP SP3 and Win7 SP1) as well as different versions of Norton Anti-Virus. It did not start until TB 3.x, but I had been running Norton AV for years before then.

> does problem reduce if you disable compact in tools | options | advanced |
> network & disk ?

I don't do automatic compacting at all; I do it by hand - every night under normal circumstances, and sometimes at other times e.g. if I have just deleted a large message. Compacting neither triggers nor fixes the problem.

The only automatic things I have TB do (besides downloading new messages) are
  1) move junk-identified messages into the Junk folder from Inbox, and
  2) expire junk messages "older" than 30 days from Junk into Trash - where
     "older" means messages with a "Date:" earlier than 30 days before, although
     that is not an accurate way to determine age.

Conceivably if I didn't use those features at all, the problem wouldn't happen, but then I'd be left with an Inbox full of junk to manage manually, which is really not an acceptable alternative. Junk email management is one of the features that (normally) makes TB an attractive mail client. Also, the reports under bug #209501 suggest that operating that way would cause the phantom messages to show up in Inbox instead, and if my Inbox were to hang the way my Junk and Trash do now, that would be even worse.

Comment 24

[Wayne Mery (:wsmwk)]

Ruth, does this reproduce when windows is started in safe mode per http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/boot_failsafe.mspx ??

Comment 25

[Ruth Milner]

I can't really test it with Safe Mode since it takes time to show up and I need to do things regularly that won't work in that environment, even with networking enabled.

However, over the past couple of months, the hanging problem seems to have gone away. I still see occasional phantom messages, but they go away on their own after a little while without affecting anything else. No correlation to compacting or other cleanup.

I don't think this had anything to do with versions of Thunderbird (I'm now up to 15.0.1). Rather, it seems to be related to the last time my Junk folder just spontaneously disappeared. It had done this a couple of times previously, without any noticeable impact those times. (For a reference, see https://bugzilla.mozilla.org/show_bug.cgi?id=209501#c153) As a kind of test, that last time it happened I exited TB, renamed the newly-created Junk file - the contents of which show this occurred on September 23 - and then restored the Junk folder from the previous evening. (I had set up automatic backups of key folders 4x/day after similar problems a few months earlier.) I thought that if there was something in the Junk folder that had triggered the disappearance, perhaps it would happen again if the file were put back to the way it was right before that. However, since then things have just kept going, and much more smoothly than they have for me in years.

I have not changed any Junk-related settings in TB since trying things out earlier this year to see what might affect the problem. None of those tests made any difference.

Not sure what to make of this. In the past I had several times deliberately removed both Junk and Junk.msf, to no avail, so I don't understand why it would be different when TB did that itself. It could just be a complete coincidence, of course.

Comment 26

[Wayne Mery (:wsmwk)]

are you seeing this in version 24?

Comment 27

[Ruth Milner]

I'm at 24.1 now and still see the problem with the phantom messages appearing in Junk and being autoexpired into Trash because of the "old" date. From there they just quietly disappear, leaving a bogus 1 new message indicator that goes away after I switch to the folder. However, as mentioned in my last post, they no longer hang the Junk and/or Trash folders. As I say above, the hanging behavior seemed to stop when TB zapped the Junk folder for the third time, even though no amount of removing it either by hand or by TB had made a difference before that.

Comment 28

[Ruth Milner]

Also, thank you for continuing to check on this, I appreciate that.

Comment 29

[Wayne Mery (:wsmwk)]

Hi Ruth. Are you still seeing this problem?

Comment 30

[Ruth Milner]

The hanging problem disappeared peculiarly in 2012 as reported above.

Currently at TB 45.4.0. The phantom new-old messages still appear, although much more rarely because I have since switched email services and very little spam gets through to my Thunderbird. Unlike the previous services, I have no option to disable all spam filtering. Because of this, junkable messages pretty much never have a chance to show up in Inbox, only in Junk and thence to Trash. This makes it nearly impossible for me to test anything.

Unlike before, they pretty much never clear up on their own; I have to Empty Trash. I'm not sure whether this is significant. I see some very old discussion of such behavior in Bug 209501.

One potentially important correction: someone has changed the title of this bug to indicate that Yahoo is the POP3 host. This is misleading. Yahoo is the service I switched to about a year ago, including as the email of record for my Bugzilla account. But for >10 years before that I was using an address at nmt.edu, which was initially hosted at NMT but a few years ago moved to go through Postini and then, after Postini's acquisition, migrated onto Google's services in 2014. The problem persisted for all of that time, and does still happen with Yahoo. So that's four separate POP3 services and two different TB email accounts (running in the same TB environment along with six other accounts that are very low-traffic), so it appears to be completely independent of those variables. In case anyone was wondering about that.

I hope this information is useful. Thanks to you and WADA and everyone else who has had eyes on this weirdness.

Comment 31

[Ruth Milner]

A question: why does the "Whiteboard" field on this bug say "[closeme 2017-01-15]"?

A comment: I discovered I could edit the bug title myself, so I removed the yahoo notation. It is decidedly not limited to Yahoo's POP3 service.

Comment 32

[Wayne Mery (:wsmwk)]

(In reply to Ruth Milner from comment #31)
> A question: why does the "Whiteboard" field on this bug say "[closeme
> 2017-01-15]"?

Because it was uncertain whether you would reply

Comment 33

[Phoenix]

Is this still present in latest Thunderbird version?

Comment 34

[Ruth Milner]

Right now I'm at TB 45.8.0 but I will upgrade to 52.0.1 tonight.

Whether the problem still happens depends on which aspect of the original report you're asking about.

I still see phantom messages, although they mostly seem to show up directly in Trash. Genuine junk messages basically never make it through from my ISP any more (which, believe it or not, I actually find very annoying). I'll upload a screenshot of what I'm seeing right now.

However, like the folder hanging problem in the original report, the "not removed from Inbox" problem also stopped happening a while back. So I don't know if you want to close this particular bug to reflect that. The main phantom-tracking discussion seems to be Bug 209501.

One comment: when working on Bug 209501 in the past, WADA had me enable the Order Received column in my Junk folder display. This showed the "position" of the phantom message as a byte offset; you can see an example there in:

     Screenshot per comment #168 upload 1: 201409242204_ForBugzilla_JunkNewest_byReceivedOrder.jpg

However, when I do that for my Trash folder now, it shows a simple sequence number instead of a byte offset. It appears to do this on every folder. I don't know when this change occurred; was it intentional?

The new screenshot shows that even though the phantom messages have zero size, they increment the sequence number, and messages put in the Trash after them continue the sequence as though they were real.

Comment 35

[Ruth Milner]

Attachment: Screenshot for comment #34

Screenshot to go with Comment #34, showing phantoms in Trash, with Order Received sequence.

Comment 36

[:aceman]

Hi, thanks for your persistence. It is a pity the bug still shows for you. We have seen and fixed some occurrences of such phantom messages, but we never saw them appear in the conditions your describe.

To update the bug, please answer these questions:
1. in the past you saw the problem on an IMAP server. Now you use POP and nothing has changed much? (the bug was moved into POP component which does not seem right.
2. you see the bug when manually deleting messages to Trash and when junk is deleted. Is that junk still marked manually and then deleted by TB via the setting "when manually marking messages as junk -> delete them) ?
3. in which way do you delete the messages? Do you press DEL key or use some button/menuitem?
4. are you still on Windows XP as originally reported? What is your filesystem type (FAT/NTFS) ?

(In reply to Ruth Milner from comment #34)
> However, when I do that for my Trash folder now, it shows a simple sequence
> number instead of a byte offset. It appears to do this on every folder. I
> don't know when this change occurred; was it intentional?

Yes, that is correct to observe and it was an intentioanl change. All is fine with that.

Comment 37

[Ruth Milner]

(In reply to :aceman from comment #36)

> 1. in the past you saw the problem on an IMAP server. Now you use POP and
> nothing has changed much? (the bug was moved into POP component which does
> not seem right.

Actually I realized within a couple of weeks of reporting this problem that I had goofed on that; see comment 7 above. My TB was always, and still is, configured to use POP. It's set to leave the messages on the server "until deleted", although not all POP servers handle that last detail.

The IMAP special preamble message was in the folders because years ago I occasionally viewed them in a Webmail client which used IMAP (the server supported both). I think the bit about the IMAP preamble being shown by TB may have been a red herring, because it was a transient glitch.

> 2. you see the bug when manually deleting messages to Trash and when junk is
> deleted. Is that junk still marked manually and then deleted by TB via the
> setting "when manually marking messages as junk -> delete them) ?

They were never deleted directly by manually marking them as Junk. I have TB set to move them to the Junk folder first, but the glitch that makes them appear with the epoch date (empty = zero seconds offset) means they age out into Trash pretty much immediately. I honestly can't say whether it still happens via Junking. It's extremely rare for spam to get through to my TB any more, so whatever I'm seeing now appears simply from moving/deleting non-Junk messages. In the past, I'm pretty sure that they could result from either automated or manual Junking, i.e. TB spam-filtering them or me marking them manually, but I really have no way to test that now.

The six phantoms in Trash shown in tonight's screenshot are all marked as Junk, but nothing came through that way. Perhaps that flag has something to do with the fact that the messages don't actually exist; they're just TB trying to interpret empty fields that it *thinks* are there.

> 3. in which way do you delete the messages? Do you press DEL key or use some
> button/menuitem?

99% of the time I use the <Delete> key. I do sometimes select a few messages at a time and either <Delete> or <right-click>Move To some folder. I have a sense that bulk operations are more likely to produce the phantoms, but will try to be more conscious of it to confirm.

> 4. are you still on Windows XP as originally reported? What is your
> filesystem type (FAT/NTFS) ?

I switched from XP Pro to a new Windows 7 Pro machine in early 2012 and am still using that. I migrated my TB folders and settings. NTFS on both C drives.

> > However, when I do that for my Trash folder now, it shows a simple sequence
> > number instead of a byte offset. It appears to do this on every folder. I
> > don't know when this change occurred; was it intentional?
> 
> Yes, that is correct to observe and it was an intentioanl change. All is
> fine with that.

Poking around (because I can't help myself), I noticed that my folders with older content look hybrid: the earlier messages' "Order Received" values change per message size, but newer ones simply start to increment by 1. The shift is not consistent by date across folders. FWIW.

Let me know if I can provide anything else. Thank you!

Comment 38

[Ruth Milner]

I had updated a couple of weeks ago to TBird 52.0.1. A few days later I did a raft of overdue OS patches, and then my system became very unstable. I wasn't able to get back on my normal account until just now. But when I deleted a message a few minutes ago (in this case one with a very large attachment), a bogus null message appeared in the Junk folder - even though this wasn't Junk and wasn't marked as such. Size 0 bytes. Within a minute it got "expired" into Trash.

So I'm still seeing the phantom problem.

I'll update to 52.1.0 soon, although I'm not optimistic.

Note that since the original freezing problem went away, this is more a nuisance, but weird.

Thank you.

Comment 39

[Ruth Milner]

I updated to 52.1.0 about a week ago. Tonight the phantoms showed up again: two of them at once, with all the usual characteristics.

One of the emails I deleted in the few minutes before noticing the phantoms had two attachments. This is a similar circumstance to the last incident I reported above, and it makes me wonder whether the remaining occurrences are related to attachments. I'll try to make note of that in future when these show up. Along with the fact that true junk mail is almost entirely blocked by my email services, this could be a factor in why they aren't happening as often, since most of the messages I receive with attachments are saved to folders, not deleted.

In addition to the bogus message, the deleted message shows up in Trash and has functional attachments.

Comment 40

[Ruth Milner]

I emptied Trash and compacted last night, for the first time since the report above. Those two phantom emails were the only ones that appeared, even though I did many subsequent moves and deletes in TB.

Note that the phantom messages used to disappear without direct intervention, but that hasn't been true for some time; emptying Trash (or possibly removing them manually, which I don't generally do) is the only way to get rid of the phantoms that show up there. They keep their position in the message list in Trash in the meantime. This may be a result of the Order Received being changed from a byte offset to a simple sequence. See bug #1106225 for a possible explanation of the behavior.

This morning I noticed three new phantoms together in Trash, followed by a couple of messages I had deleted. A few minutes before that, I had moved three messages at once from Inbox to a folder (select all three, then use right-click->Move To). Per WADA in https://bugzilla.mozilla.org/show_bug.cgi?id=209501#c210, moving messages from Search results to a folder is a useful way to reproduce at least one form of the phantom-message bug. However, I moved these directly from Inbox.

I'll leave these be for a few days and see if these three remain the only ones. If so, I'll test what happens if instead I clean the phantoms up when they appear.

Another thing I can do, since I get so few actual spam messages these days, is to stop having TB automatically move spam to the Junk Folder and automatically expire it. That way, maybe phantoms will stay where they originated. But I'll do that later, so I'm only trying one thing at a time.

Comment 41

[Wayne Mery (:wsmwk)]

Is the only remaining issue described by bug 209501?

Comment 42

[Ruth Milner]

I *think* so. Regular daily use moving/deleting messages from my Inbox still produces occasional phantoms which end up in my Trash and don't go away until I "empty" it. However other odd behavior reported here - freezing, deleted messages staying in Inbox, etc. - has not been seen for a few years. At this point I'm on TB 52.5.2.

I recently moved most of my email usage to a brand-new account where I use IMAP, and I have not seen this problem on that - only on the older POP accounts. But there is much less traffic to those now, so I won't be able to do any decent testing any more.

I really appreciate the suggestions and involvement of the Thunderbird team over the years - thank you for all your effort!

Ruth

Comment 43

[Wayne Mery (:wsmwk)]

Ruth, Thanks for the update