Open Bug 444343 Opened 16 years ago Updated 19 days ago

"Compact Folders" results in some messages dated 1970; regular index rebuild needed

Categories

(MailNews Core :: Networking: IMAP, defect)

Product:
MailNews Core
Component:
Networking: IMAP
Type:
defect

Tracking

(Not tracked)

People

(Reporter: gerv, Assigned: Bienvenu)

References

(Depends on 1 open bug)

Details

This has started happening just recently. I use IMAP, with the "mark messages as deleted" delete model and so I regularly run "Compact Folders". When I do so, in the INBOX of my second IMAP account only (not other inboxes, and not other folders), it regularly ends up such that I have one or two "phantom" messages - no sender, no subject, 1970 date. If I rebuild the indexes, these messages disappear and are replaced by real ones that I previously could not see.

In other words, Compact Folders corrupts the index such that it needs rebuilding, multiple times a day. This isn't great.

Please let me know how I can diagnose this further. Why that one specific folder? Is there some other file I should be deleting to kick it back into life?

Gerv
You could try rebuilding the index for that folder, as long as you don't have tags and a server that doesn't support keywords.
Assignee: nobody → bienvenu
Component: General → Networking: IMAP
Product: Thunderbird → Core
QA Contact: general → networking.imap
Version: unspecified → Trunk
time frame? rkent just filed a regression against bug 435153, which landed almost a month ago - so it's possible...
mark as deleted isn't affected by bug 435153 - it's only deletes to trash that should be affected by that bug.
David: I do rebuild the index, regularly - that's the point :-) It fixes it, but compacting folders breaks it again. I would like to not have to rebuild my Inbox index 20 times a day, please :-)

Gerv
oh, sorry, your comment about rebuilding didn't register - what kind of imap server is it that's having the problems?
telnet tuschin.mythic-beasts.com 143
Connected to tuschin.mythic-beasts.com.
* OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION STARTTLS] Courier-IMAP ready. Copyright 1998-2004 Double Precision, Inc.  See COPYING for distribution information.

Gerv

Oops, sorry, wrong machine. Try this:

gerv@otter:mozilla-central$ telnet-ssl -z ssl limo.mythic-beasts.com 993
Trying 193.201.200.39...
Connected to limo.mythic-beasts.com.
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN ACL ACL2=UNION] Courier-IMAP ready. Copyright 1998-2004 Double Precision, Inc.  See COPYING for distribution information.

I also compacted my 30,000 messages' indexes 150+ folders (some POP3, some RSS/Atom feeds). The effect is even worse here: I do not see newly arrived e-mails anymore in many folders and feeds (no bold font indication, especially not in "unread folders" view anymore), RSS feeds are not even updated anymore. I am so angry, because I have no clue how to fix this. Should I rebuild all indexes? But I thought compacting the folders would have issued an auto rebuild anyway. I also read that compacting folders loses tags, which would be even worse (not checked yet).

I receive so many business e-mails everyday, and I missed a few important ones already, i.e. I noticed them late and only by chance when looking into certain folders manually (then th bold indications suddenly appear).

There are so many symptoms, I do not know how and where to best file this bug.
Oh, one more symptom: Trying to rebuild indexes on some RSS feeds can be clicked now, but the "OK" button of the feed's preferences does not work anymore. I have to close it by "Cancel", but nothing seems to have changed afterwards. Even stranger is this: The default character set is now Arabic for some feeds. D-oh!
bienvenu: is there anything more you need from me here?

Gerv
Gerv, maybe a protocol log of a session where this happens might be useful. I don't see this problem here...
Gerv: you don't have saved searches set up under those imap folders by any chance? I know that causes problems (though haven't figured out the details yet, it's something with two mails for a thread coming in, one of which match the search, i think)
Magnus: depends what you mean by "under". In my Local Folders, I have five or six saved searches which relate to the contents of my IMAP inbox. And that's the only one that seems to be getting corrupted. So you are right, this may well be the link.

Gerv
Magnus: is there anything I can do to help you nail this down?

Gerv
Sorry for the late reply, Gerv. It seems my case has been resolved (by the non-rdf folder work, I think). Once tb3 beta1 is out - or with nightlies - you could compare to see if you still see it.
A late amendment to my own case: Many of the symptoms seem to have been caused my Google Desktop Search with Thundebird plug-in. Once that tool was uninstalled, everything worked okay again after the indexes had been rebuilt.
Product: Core → MailNews Core
This still isn't working for me in Thunderbird 3.0b1. I have to regularly rebuild my indexes after compacting folders, to "restore" phantom messages with no content, no body and a date of 1st January 1970 to their full selves.

Gerv
If Bug 414723 or Bug 501392 occurs, duplicate entry for same UID is generated in msf. It may produce phantom mail after compaction.
Garbes Markham(bug opener), display "Order Received" column(UID if IMAP), and if possible (not so many mails in the mail folder), check whether duplicated entry for a UID exists or not before start of compaction. (check mail folder with which problem most frequently occurs)
can this bug be closed / does anyone still see this problem?

both bugs below were fixed in 3.0b3

(In reply to comment #18)
> If Bug 414723 or Bug 501392 occurs, duplicate entry for same UID is generated
> in msf. It may produce phantom mail after compaction.
> Garbes Markham(bug opener), display "Order Received" column(UID if IMAP), and
> if possible (not so many mails in the mail folder), check whether duplicated
> entry for a UID exists or not before start of compaction. (check mail folder
> with which problem most frequently occurs)
:wsmwk: It might be, it's hard to tell from the descriptions in that (old, morphed) bug. I still occasionally get phantom 1970 messages, usually when deleting. If I rebuild my indexes, they go away, to be replaced by the same number of "real" messages which were "hiding" behind them.

Gerv
What I described in my old comment #8 still happens in Thunderbird 12.0.1 (Win XP Pro). It is even worse because now I regularly use the auto-compact feature offered by TB. After compaction the corresponding folder loses its column sorting settings (I always sort newest message first, the default is oldest first, which is stupid anyway, but kinda off-topic here). And the effect described perviously also happens: Sometimes I do not see messages sorted into freshly compacted folders by a filter because the corresponsing folder is not marked as containing new messages (I usually work in the "unread folders only" view).

This way I missed a few really important messages by clients and friends, sometimes only noticing them months later. This is embarrassing because now they think I am unreliable. Sorry for mentioning this, it is not the problem, but an effect caused by it and should underline that this is not just a minor annoyance but a major bug which is way too old and should be eliminated ASAP. Thank you.
Oh BTW, if you think I shopuld rather split this one off and file a new bug or if you know a duplicate of what I describe, just tell me so and I will do so or continue writing there.
(In reply to Alexander Kriegisch from comment #22)
> What I described in my old comment #8 still happens in Thunderbird 12.0.1
> (Win XP Pro). It is even worse because now I regularly use the auto-compact
> feature offered by TB. After compaction the corresponding folder loses its
> column sorting settings (I always sort newest message first, the default is
> oldest first, which is stupid anyway, but kinda off-topic here).

Alexander , your last symptom is a new issue with version 12 and at least partly fixed in recent nightly build. You might give it a try. As for comment 8, if you have a reproducible test case and it doesn't seem to align with gerv's description, yes, please file a new bug report and then cite the new number back here.
Depends on: 209501
(In reply to Alexander Kriegisch from comment #22)
> After compaction the corresponding folder loses its column sorting settings (snip)

It's perhaps bug 710056. If IMAP, see also Bug 550286.

Anyway, please keep this bug for problem of "Compact Folders results in some messages dated 1970" only.
See Also: → 209501, 536447
Severity: normal → S3

Does this also happen with 115.9.0?

Flags: needinfo?(Alexander)

Thanks for asking again after 12 years. The issue as such is 16 years old, BTW. As for messages having a date from 1970, I do not remember when that happened last time. But frequently necessary folder repairs and read/unread marks for messages being reset or simply not displayed until manually opening the compacted/repaired folder again still happens.

Flags: needinfo?(Alexander)
You need to log in before you can comment on or make changes to this bug.