Closed Bug 1175242 Opened 5 years ago Closed 5 years ago
whole Inbox deleted at every start - maildir
1.17 KB, patch
|Details | Diff | Splinter Review|
42.72 KB, image/png
437.90 KB, application/x-zip-compressed
43.54 KB, text/plain
99.32 KB, text/plain
5.54 KB, application/octet-stream
1.09 KB, patch
|Details | Diff | Splinter Review|
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0 Build ID: 20150602115007 Actual results: Every file in every Inbox-folder is deleted at every start of thunderbird. All Mails are lost, but still listed. A click to fetch new mails sometimes brings a messages about missing space, but there is enough (harddisk, tmpfs). The mails in Send, Draft and other folders stay. Thunderbird worked since installed over a year ago with maildir. arch linux 4
xref/dupe bug 1143231
> Every file in every Inbox-folder is deleted at every start of thunderbird. What exactly do you mean by "file"? > All Mails are lost, but still listed. What is happening/what are you seeing that you think they are "lost"?
Component: Untriaged → Backend
Product: Thunderbird → MailNews Core
@ Wayne Mery Example file: /home/emti/.thunderbird/5gxktq03.tbird1/Mail/pop3.arcor-1.de/Inbox/cur/1434528569682539 these files disappear: ls -la /home/user1/.thunderbird/5gxktq03.tbird1/Mail/pop3.arcor-1.de/Inbox/cur/ insgesamt 28 drwxr-xr-x 2 emti users 4096 17. Jun 10:09 . drwxr-xr-x 4 emti users 4096 17. Jun 10:09 .. -rw------- 1 emti users 2195 17. Jun 10:09 1434528569682539 -rw------- 1 emti users 3576 17. Jun 10:09 1434528569741471 -rw------- 1 emti users 3403 17. Jun 10:09 1434528569985245 -rw------- 1 emti users 4203 17. Jun 10:09 1434528570108662 (that are the mails from bugzilla, if i start thunderbird next time they will be in ext4-bit-nirvana ) i see: ! Fehler: Datei nicht gefunden Die Datei mailbox:///home/emti/.thunderbird/5gxktq03.tbird1/Mail/pop3.arcor-1.de/Inbox?number=1574 konnte nicht gefunden werden. Bitte überprüfen Sie die Adresse und versuchen Sie es erneut. Translation of what i see: Error: File not found The file mailbox:///home/emti/.thunderbird/5gxktq03.tbird1/Mail/pop3.arcor-1.de/Inbox?number=1574 couldn't be found. Please confirm path(adress) and try again.
(assuming major until we know otherwise)
Severity: normal → major
See Also: → 1175890
(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #4) > See Also: → bug 1175890 Same as discribed in bug 1175890, set maildir by edit config too. And crashed when manually starting filter.
I have the same problem: the Inbox Folder (on harddisk) is deleted on every start of thunderbird since I use maildir (I tried maildir for the first time now with thunderbird version 38 (on linux)).
Assignee: nobody → rkent
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Although I still cannot reproduce this, there are a number of statements that mention getting the out of disk space error. Maildir does not currently correctly report disk space used, and the test of available state is correspondingly incorrect. I could pretty easily fix the test of disk space to only test available disk space, and not care about the non-working disk space used.
Because I cannot reproduce this myself, I can't be sure of the issue, but I strongly suspect that it is associated with known issue in folderSize. This patch stops checking folder size, and really is the right thing to do anyway. I would appreciate if anyone who can reproduce this bug try one of the builds at: http://firstname.lastname@example.org/ which have this patch added. If this works, and the testing can be done in the next day or two, there is a chance we could add this to the next release which is 38.2.0 (due this week).
(In reply to Kent James (:rkent) from comment #9) > I would appreciate if anyone who can reproduce this bug try one of the > builds at: > > http://ftp.mozilla.org/pub/mozilla.org/thunderbird/try-builds/kent@caspia. > com-fa620d1115b2/ > > which have this patch added. If this works, and the testing can be done in > the next day or two, there is a chance we could add this to the next release > which is 38.2.0 (due this week). I can confirm that this build mostly fixed my problems [Windows] described in detail in the duplicate bug 1175890. 1) If I create a new profile, I can't reproduce the issue with the described steps. (bug 1175890, comment 12) 2) If I use an earlier corrupted profile in the new version I get the error "There is not enough disk space to download new messages. Try deleting old mail, emptying the Trash folder, and compacting your mail folders, and try again" one last time. In further restarts the error is not shown again. Of course the zero byte messages cannot be found anywhere. They were lost in the old version, there should be no way to recover them. Could it be that you could not reproduce the issue, because of the locale? (German OS here) Perhaps it has something to do with number/date formatting etc. @Kent James, Thunderbird Team: Thank you very much, your work is appreciated.
Perhaps there should be a warning / information for [all users / maildir users / maildir+pop3 users / maildir+pop3 users with zero byte messages] at the next Thunderbird start in the bugfixed version: "Unfortunately there was some bug in Thunderbird 38.0 so that messages in the inbox were lost for pop3 accounts using the new maildir format." I don't know about your best practices here.
Comment on attachment 8646085 [details] [diff] [review] Don't use non-working folder size with maildir Neil, could I get you to review this quickly? I'l like to try to land it for the next version of Thunderbird 38 which we'll build soon.
Attachment #8646085 - Flags: review?(neil)
Comment on attachment 8646085 [details] [diff] [review] Don't use non-working folder size with maildir Aceman, any comments on this? Folder limits don't make much sense in maildir.
Attachment #8646085 - Flags: feedback?(acelists)
Comment on attachment 8646085 [details] [diff] [review] Don't use non-working folder size with maildir I've not tried running with maildir but I agree that checking GetSizeOnDisk against 0xFFC00000 makes no sense at all.
Attachment #8646085 - Flags: review?(neil) → review+
Comment on attachment 8646085 [details] [diff] [review] Don't use non-working folder size with maildir https://hg.mozilla.org/comm-central/rev/b93b131e36c1 As part of this bug, we deleted this comment: - // Allow the folder to only reach 0xFFC00000 = 4 GiB - 4 MiB for now. - // This limit can be increased after bug 1078367 is fixed. For future reference, that bug 1078367 and its dependencies all are issues affecting the proper calculation and use of a large folderSize value. For maildir, folderSize is not currently working at all, plus a large folderSize itself is not really an issue for maildir, the issue is more how to calculate it, and how to handle a large value. So that comment was not really relevant to maildir. I'll consider uplifting to comm-esr38 after a nightly cycle.
Attachment #8646085 - Flags: feedback?(acelists) → approval-comm-esr38?
I'm going to mark this as fixed, but it would be good to verify that after the patch has landed.
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
OS: Linux → All
Hardware: x86_64 → All
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 43.0
Attachment #8646085 - Flags: feedback-
Thanks for the input, aceman. Thinking about the more, the existing code is clearly wrong, as the folder size does not work, and there really is not a good reason to limit the size of a maildir folder to 4 GBytes anyway. So I am not concerned about this patch. But you are right that we don't really know what is causing this. I would have to guess that the probability of this patching fixing the issue is less than 50%. So for now I will reopen this bug until we have some confirmation. The best way to investigate this is for someone who can reproduce it to debug at the code level. Without that, all I can do is make guesses myself, as I cannot reproduce.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
I disabled the option to 'compact' the mailbox to save disk space and it seems that the INBOX is not removed. (I use filters to move mail from INBOX to several maiilbox under 'local folders' and it seems that the INBOX disappear when Thunderbird try to 'compact' the INBOX folder).
rkent, have you tried to artifically trigger the return NS_ERROR_FILE_TOO_BIG; or the return NS_ERROR_FILE_DISK_FULL; paths (so that HasSpaceAvailable fails) to see if it has the effect reported in this bug?
(In reply to :aceman from comment #20) > rkent, have you tried to artifically trigger the return > NS_ERROR_FILE_TOO_BIG; or the return NS_ERROR_FILE_DISK_FULL; paths (so that > HasSpaceAvailable fails) to see if it has the effect reported in this bug? Yes I have, and it does not show the symptoms of this bug. I've also poured over the compact code, it all looks correct to me, and I cannot get a compact to cause any problems. So still unable to reproduce this. (I was actually looking at the compact code yesterday, because it is one of the few parts of the code that will delete files).
Yeah, but the compaction trace should have something to it, as we have bug 1174485 for it.
Comment on attachment 8646085 [details] [diff] [review] Don't use non-working folder size with maildir http://hg.mozilla.org/releases/comm-esr38/rev/7fe010559389
Attachment #8646085 - Flags: approval-comm-esr38? → approval-comm-esr38+
Although we landed this patch in esr38 for Thunderbird 38.2.0, because I suspect that did not fully solve the issue I am not marking status-thunderbird_esr38=fixed
I can confirm, that with this build http://email@example.com/try-comm-central-linux64/thunderbird-42.0a1.en-US.linux-x86_64.tar.bz2 I cannot reproduce the deletion of the inbox anymore with these steps: 1. create a new linux-account 2. start-thunderbird 3. don't use the wizzard to create a new account 4. change the global settings to use maildir as default 5. create a new pop3-account 6. now drop some mails into the inbox 7. restart thunderbird I am willing to help to find the real cause of the error. perhaps you could add some debug logs to the code (where deletion could happen), and I could run such a build (without the fix above, but with debug logs)?
I am also willing to help. Failed to build a custom version. (it took ages to build the project) I think providing a debug version would help much.
Thanks for the offers to help. What would be helpful: 1) Confirm or deny that version 38.2.0 still has this problem (the patch here landed in 38.2.0, but I would be surprised if it really solved the problem). 2) Confirm or deny that completely disabling auto compact solves this problem (see comment 19). I'll have to think about how I could created a debug build that would likely trace where this is occurring.
(In reply to Kent James (:rkent) from comment #28) > 1) Confirm or deny that version 38.2.0 still has this problem (the patch > here landed in 38.2.0, but I would be surprised if it really solved the > problem). Not for me. > 2) Confirm or deny that completely disabling auto compact solves this > problem (see comment 19). Not sure if this is the option you refer to (see attached image). With or without this one it deletes my Inbox.
This zip package contains six lists of files and dirs in my profile dir. tb.filelist.before.first.run.38.2.0.txt - Before first run of 38.2.0 tb.filelist.after.first.run.txt - After first run: one Inbox (my second account) deleted tb.filelist.after.second.run.txt - After second run: second maildir Inbox deleted Similar to those are the files with "disabled" - those are with disabled option from previous attachment, but the story is the same. After first run one Inbox deleted, after second run second Inbox deleted. I have three account there. Two of them as maildirs, one as mbox - if that changes anything.
(In reply to Piotr from comment #30) > This zip package contains six lists of files and dirs in my profile dir. > tb.filelist.before.first.run.38.2.0.txt - Before first run of 38.2.0 > tb.filelist.after.first.run.txt - After first run: one Inbox (my second > account) deleted > tb.filelist.after.second.run.txt - After second run: second maildir Inbox > deleted One more thing: after first run, when one Inbox gets wiped out, I can access messages in the "not yet deleted" account. Works as expected. After second run they're also wiped.
(In reply to Kent James (:rkent) from comment #28) > 1) Confirm or deny that version 38.2.0 still has this problem (the patch > here landed in 38.2.0, but I would be surprised if it really solved the > problem). TB 38.2.0 still has this problem. I am confused, because your debug build worked well for me. > http://firstname.lastname@example.org/ > 2) Confirm or deny that completely disabling auto compact solves this > problem (see comment 19). If you mean the setting Piotr mentioned in attachment #3 [details] [diff] [review], I have to deny that this has any effect: > https://bug1175242.bmoattachments.org/attachment.cgi?id=8649798
Same problem here. Please stop shipping of Thunderbird 38.2.0! Cancel it!
I have created a Windows debug build that I would appreciate people who can reproduce this to use. This build is designed to crash on any attempt to remove a file named: INBOX, INBOX/*, INBOX/cur/*, MAILDIRCRASH, MAILDIRCRASH/*, MAILDIRCRASH/cur/* To use this, download the build, install it in a directory somewhere, and then enter in a command console: c:\path\to\install\thunderbird.exe -P -no-remote Then do whatever it is that reproduces the inbox directory being removed. When that happens, thunderbird should crash, and in the command console you should see a dump of the call stack. You can also force a crash to see what it looks like by deleting any file in the INBOX or MAILDIRCRASH folders (or the folder itself). Obviously the goal of this is not to give you a working Thunderbird, but to generate a stack that shows where the delete is occurring. The program captures attempts to move or delete a file in the low-level file representation in Thunderbird, so if Thunderbird is doing the delete, this should catch it. Please report that stack here. I've tried all kinds of things to reproduce this, on multiple Windows and Linux installs, and I so far have been unable to reproduce the symptoms of this bug. Maybe this will give a clue of what is happening. The download url is: http://mesquilla.net/releases/thunderbirdCrashme-38.2.0.en-US.win32.installer.exe
A linux 64 bit debug build would be great.
(In reply to cyberbeat from comment #35) > A linux 64 bit debug build would be great. (I assume you mean Linux) I only need one person to show me the stack. If I don't get a Windows user in the next few days, I'll try to do a Linux version, but it will take significant work.
Thank you for the build from comment 34. Unfortunately the exception was not triggered when starting a faulty profile. I could trigger it manually by deleting mails of the folder "MAILDIRCRASH". However when starting a faulty profile I found some warnings in the command console output. They occur at the exact time the folder "Inbox" is deleted (the folder "Trash" is not deleted). (I attached the full log FYI)  WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80550013: file c:/tb/9-esr38/src/mailnews/local/src/nsPop3IncomingServer.cpp, line 446  WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80550013: file c:/tb/9-esr38/src/mailnews/local/src/nsNoIncomingServer.cpp, line 149 The result 0x80550013 should be NS_MSG_FOLDER_EXISTS. IMO in both methods "nsPop3IncomingServer::CreateDefaultMailboxes()" and "nsNoIncomingServer::CreateDefaultMailboxes()" the result of calling "CreateLocalFolder" is that the folder (already) exists. Is it correct that CreateDefaultMailboxes is called? I also get this line in the command console output: (There are 35 GB free) GetDiskSpaceAvailable returned: 0 bytes I try to investigate further. Anything you suggest I could try?
The exception would not necessarily be triggered when you start a faulty profile, but should occur at the point where a good profile turns into a bad. If you can 1) create a good profile, 2) do whatever it is that makes it bad, then you should trigger the crash. If your profile goes from good to bad while running this version without crashing, that would also be interesting.
After restarting Thunderbird I get the attached full log. I noticed this lines: GetDiskSpaceAvailable returned: 35719761920 bytes  WARNING: NS_ENSURE_SUCCESS(rv, result) failed with result 0x80520012: file c:/tb/9-esr38/src/mailnews/local/src/nsPop3Protocol.cpp, line 166  WARNING: Could not get disk status from nsIDiskSpaceWatcher: file c:\tb\9-esr38\src\mozilla\uriloader\prefetch\nsOfflineCacheUpdateService.cpp, line 321 It looks that "DiskSpaceAvailableInStore" returns 0 if the parameter aFile does not exist. (IMO only the drive part of the file should be used.) The result 0x80520012 should be FILE_NOT_FOUND. So this one is a result of the earlier corrupted profile.
(In reply to Kent James (:rkent) from comment #38) > The exception would not necessarily be triggered when you start a faulty > profile, but should occur at the point where a good profile turns into a bad. The profile from comment 37 is faulty, but still has the folder "Inbox" in it. When the folder is deleted in thunderbird the version is not crashing. I just tried to use the build from comment 34 to generate a faulty profile, but I did not succeed. Perhaps I can only generate faulty profiles with localized versions of Thunderbird. Could you try to reproduce the error with a localized (German here) version of Thunderbird. Perhaps the difference between the folder in the file system ("Inbox") and the folder shown in Thunderbird ("Posteingang") is the problem. (should be both "Inbox" for you, right?) I will try to reproduce the issue with an English Thunderbird tomorrow.
Just counted the reporting countries: 3 German 1 Czech Republic 1 Poland
Very good hint. I also had the "deleting" problem with other standard folders like "drafts" (in german displayed as "entwürfe"). Perhaps in maildir this "translation" method is not used at some place?
I missed one German in the list: Me. ;-) I cannot reproduce the issue (bug 1175890 comment 12 / comment 25) with an English Thunderbird. (TB 38.2.0) I could reproduce the issue every time I used a German Thunderbird. (TB 38.2.0)
I succeeded to reproduce the bug with an English Thunderbird, unfortunately I had to change a file in the file system, because there is no way in Thunderbird itself to rename the inbox. 1) Follow the steps from (bug 1175890 comment 12 / comment 25) and close Thunderbird. 2) Edit the mork file "/profile/panacea.dat" and replace "=Inbox" by "=XXXXX": <(80=C:\\_Portabel\\Thunderbird4\\Mail\\pop.gmail.com\\Inbox.msf)(81=1004) (82=0)(84=ISO-8859-1)(85=XXXXX)(95=1440314639)(96=1)(86 3) Start Thunderbird and the error will show up. I also succeeded to reproduce the bug with the profiles posted bug 1175890 comment 10 and bug 1175890 comment 12 in an English Thunderbird. (attached one of this profiles here)
Thanks Samuel for all of your hard work. I'll try to dive into this in the next few days (I have a few other critical tasks also ongoing at the moment). Are you able to generate a crash stack from the sample build that I presented? I'd like to see the point where the delete occurs.
(In reply to Kent James (:rkent) from comment #45) > Thanks Samuel for all of your hard work. I'll try to dive into this in the > next few days (I have a few other critical tasks also ongoing at the moment). Take your time. > Are you able to generate a crash stack from the sample build that I > presented? I'd like to see the point where the delete occurs. Unfortunately I could only generate one by deleting messages from the folder MAILDIRCRASH. Perhaps you used the internal name ("Posteingang") and not the file name "Inbox" to trigger the exception. Perhaps you could generate another debug build. See my attempts in comment 37 and comment 40.
Could you rename the inbox to MAILDIRCRASH in your tests? That should do it. I could potentially do another version, but that would take several hours. I'd probably rather spend those hours trying to test a German build, as this is starting to look like a problem related to the renaming of the INBOX.
I also thought of this and will try it. I don't know if I could do it, because I have to manipulate msf or mork files. I fear get corrupt when they differ in size. Will give feedback soon.
I also failed to reproduce it this way. Perhaps the msf and mork files got invalid when changing "Posteingang" (11 chars) to "MAILDIRCRASH" (12 chars). Or my assumption was wrong. I observed that "Posteingang" is renamed to "Inbox" each time I start the English Thunderbird. That would explain why your previous debug version from comment 9 seemed to fix the problems (after first restart no bugs occour), but Thunderbird 38.2.0 didn't.
Using a german locale, I can reproduce easily. Thanks for the help in pointing in that direction! For some reason, when maildirstore was created originally, although the create folder code was largely copied from the equivalent mbox code, the code that deals with this case, returning if the folder exists, was not included. I'm just adding here the equivalent code from mbox. I also in reproducing got the "out of disk space" message. Investigating, that message occurs if you ask for the available disk space on a folder that does not exist, so that message is not really correct in context. I'd like to fix that, but at this point the goal is to do the minimum fix needed for maildir in esr38, so I'd rather not expand the scope of this to include that issue.
Attachment #8654545 - Flags: review?(neil)
Thanks for your time, looking forward for the fix in the ESR. So I was correct about the "out of disk space" message in comment 39 ;) a) IMO it should not be included in the ESR. With the fix it should not be triggered anymore [at least at this point in the code]. b) Is it always true that the parent directory has the same amount of memory available? Consider links to other drives in the file system. So "D:\some folder\linked drive E\" should have different space available than "D:\some folder\" or "D:\" itself. So the solution just checking the root drive shouldn't always work. c) With (b) in mind and for transparency reasons it probably would be better to throw a new exception like "a folder that has been expected is missing..." / show a new warning in TB. d) Otherwise the profile / account base folder could be used.
Thanks for the fix. When and in which official thunderbird version will this fix be applied? I am looking forward using maildir without mbox in parallel :-)
Comment on attachment 8654545 [details] [diff] [review] Don't delete existing directory in create request Seems reasonable.
Attachment #8654545 - Flags: review?(neil) → review+
(In reply to cyberbeat from comment #52) > Thanks for the fix. When and in which official thunderbird version will this > fix be applied? I am looking forward using maildir without mbox in parallel > :-) This will probably be included in the next Thunderbird 38 release, which should be in the week of 2015-09-21 The other maildir blocker is bug 1152651. I have a fix for that for the maildir-maildir copy case, but I need to investigate if there is a similar issue in the maildir-mbox case (as I reported earlier in that bug).
Status: REOPENED → RESOLVED
Closed: 5 years ago → 5 years ago
Resolution: --- → FIXED
Comment on attachment 8654545 [details] [diff] [review] Don't delete existing directory in create request http://hg.mozilla.org/comm-central/rev/95e54339b6a5
Comment on attachment 8654545 [details] [diff] [review] Don't delete existing directory in create request https://hg.mozilla.org/releases/comm-aurora/rev/addd9fa53b40 http://hg.mozilla.org/releases/comm-beta/rev/018e557a74f3 https://hg.mozilla.org/releases/comm-esr38/rev/58290f457bc5
So this will be fixed in TB 41? Is this the next stable release or there is 39 and 40 somewhere in the middle? Guys, you are doing great job, but... I don't understand this! On my list this bug is similar to the "famous" Steam for Linux or bumblebee bugs (well, you know, the kind of "ooops, it deleted your /usr, sorry"). You don't want to delete all user messages. Users don't want suprises like that. Application update that deletes your whole inbox is a show stopper bug and imho should be fixed ASAP in all versions.
I have installed thunderbird 41beta from here: https://ftp.mozilla.org/pub/thunderbird/nightly/latest-candidate-comm-beta/linux-x86_64/de/ the problem above is gone :-) but another problem is still there: I can't move/copy multiple mails from a maildir directory. I think there are some open bugs about that, but the descriptions are too technical. Can you please point me to the right bug-report please? So that I can subscribe there..I need to know, when there are builds, where this is also fixed.
The move/copy issue should be fixed if all of your directories are maildir, but will still be present for copies from maildir to mbox. This is bug 1152651. The maildir->mbox changes were backed out because of some test failures that still needed addressing, so there are no current builds with that fixed.
You need to log in before you can comment on or make changes to this bug.