Closed Bug 1580060 Opened 6 years ago Closed 6 years ago

Deleted items if moved back from Trash to IMAP Inbox never appear

Categories

(MailNews Core :: Networking: IMAP, defect)

Product:
MailNews Core
Component:
Networking: IMAP
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 618809

People

(Reporter: mjurgens, Unassigned)

References

Details

Attachments

(2 files)

imap.log.moz_log.withfailure
1.70 MB, text/plain
Details
imap.log.moz_log.nofailure1
1.71 MB, text/plain
Details

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0

Steps to reproduce:

Delete an email from Inbox (mail appears in trash)
Go to Trash
Move mail-just-deleted back to Inbox (mail does not appear in Inbox)

Actual results:

Mail does not appear in Inbox

Expected results:

Mail should appear in Inbox.
On further inspection using a different mail client (Roundcube webmail), you can clearly see that the mail in the Inbox remains marked for deletion in the Inbox. Presumably, Thunderbird should be unmarking the mail for deletion when it moves it back or creating it as a different instance in the Inbox.

Further Information:
This is for IMAP mailboxes. Currently running v69 Beta 4

If I have 2 non-Thunderbird mail clients using the mailbox at the same time and I perform the steps it all works OK. If I fire up Thunderbird in between steps Thunderbird can see the mail in the Trash (now close Thunderbird, move the mail back to Inbox, start Thunderbird) and then it can see it in the Inbox. If, however, I have Thunderbird open during the entire process, even if performing the steps from another mail client, then the fault occurs and no mail clients can see the message back in the Inbox.

The mail can be moved to any other folder and back to Inbox, it does not have to be Trash.

Summary: Deleted items if moved back to Inbox never appear → Deleted items if moved back from Trash to IMAP Inbox never appear

I'm not seeing it on my end with TB 69.0b4. However, out setup is quite different.

A few question.

Have you been using TB for a while? Did this happen in earlier versions?

comment 1 makes me think this might be a preference or server setting issue in TB whether manually set or automatic you might what to give them a once over to see if anything could cause your issue and toggle it.

Component: Untriaged → Networking: IMAP
Product: Thunderbird → MailNews Core

Reporter Matthew, Do you know the type of imap server you are using? Also, attaching an IMAP:5 log that shows the imap commands that occur when you delete the message and try to copy it back to Inbox may help. Google for "thunderbird log" and attach the log using the "Attach file" link above. Thanks.

The only setting that looks relevant is the account setting of "when I delete a message:". I have this set to "Move it to this folder: Trash". I changed it to "Just mark it as deleted". The behaviour then was that it did just mark it as deleted and I could still see it in all clients. However, when I put the setting back to "Move it to this folder: Trash" is disappeared immediately and I could not see it in Trash or Inbox in Thunderbird but I could still see it marked as deleted in other clients.

I have been using TB since version 1. I am now using the 64bit version. I've done a conversion to "one file per message". I did an upgrade from 1.x to 2.x and then to 3.x and many versions following that. I rebuilt the install and configuration maybe a year ago while doing the conversion to "one file per message".

This problem has been happening for me for multiple of the recent versions. I cannot remember exactly when I first saw it. I have just decided to do something about it now.

I'm not sure I can tell what type of IMAP server I am using. This may be revealed when I collect the logs. I will provide logs later. I may also be able to fire up a fresh install of TB in a new virtual machine and see if that is the same issue. I may also be able to try a different IMAP server - I don't know what type of IMAP server that is at this point either but it is run by another hosting company.

IMAP:5 log with failure

IMAP:5 log without failure

I've now uploaded a couple of IMAP:5 log files. I am now running TB 70b1
Now that I am trying to troubleshoot it, it sometimes works.
I can't quite work out what the criteria is for when it works properly

However, when I put the setting back to "Move it to this folder: Trash" is disappeared immediately and I could not see it in Trash or Inbox in Thunderbird but I could still see it marked as deleted in other clients.

This is expected behavior. All you did in tb was mark it with \Deleted flag at the server, but it is still in Inbox. If you go back to "just mark as deleted" you will see the it with the red X and crossed out. You can right click and undelete it then (just removes the \Deleted flag at the server). Then it will appear again in "move to trash" mode.

Thanks for the logs. I haven't looked at them yet but will now...

I recall now that I first noticed this with messages that had been marked as Junk and moved to a Junk folder. When I marked them as not-junk they disappeared from the Junk folder and never reappeared anywhere else. This just seems to match the pattern of "message can't ever reappear in a folder once it has been marked as deleted" described in this bug

I have tried a brand new install of TB v70.0b1 on a machine that has never had it installed before.
The same problem exists - the message deleted from Inbox, when moved back, does not appear.

Matthew, It works for me on just updated daily build 71.0a1 (2019-9-18). I delete the message from inbox and it moves to trash. Then I can drag it back from trash to inbox. Is that what you are doing? There has not been any changes that I know of that would affect this very basic functionality.
Also, your exact server settings would help to know. And I still need to look closer at the log you attached. Haven't gotten to it yet, sorry.

I can give you an account on the mail server I used which appears to be Dovecot.
You'll have to email me directly to get the details
It would be interesting to see if you can then reproduce it.

Hi Matthew, A real connection to the server is usually the best way to debug. Will send you an email. Thanks.

Attachment #9092267 - Attachment mime type: application/octet-stream → text/plain

(In reply to Matthew Jurgens from comment #0)

Expected results:

Mail should appear in Inbox.
On further inspection using a different mail client (Roundcube webmail), you can clearly see that the mail in the Inbox remains marked for deletion in the Inbox.

Looking at the log, in both cases the email is moved to trash and the server reports the moved email as expunged. This is expected behavior from a server that supports imap MOVE command. Your server is dovecot and it shows MOVE in the capability response. So you should not see the deleted email in inbox just marked with \deleted flag since it is expunged from inbox after the move to trash occurs.

Presumably, Thunderbird should be unmarking the mail for deletion when it moves it back or creating it as a different instance in the Inbox.

When you move the message from trash back to inbox, the same thing occurs for trash. The message that was in trash is expunged by the server after the move to inbox. Tb doesn't remove the \deleted flag from the original message in inbox, in fact, it can't, since the original message doesn't exist in inbox anymore. The message moved from trash back to inbox becomes a new UID in inbox.

Both the good and bad logs show the same thing. It doesn't make any sense that you don't see the email back in inbox after it is moved there from trash. I just tested this on a localhost based dovecot and it works as expected. So maybe I can tell more with the test account.

Thanks for the account info and video description. I haven't tried it yet since you say the "gene" account works OK. I will try it myself shortly.
Edit: Don't see the problem with the test account from here.

What I am wondering is about what you mention in comment 1 above. You talk about two tb clients running and it works OK. Is that still true? If so, can you be a bit more specific since I don't quite understand the procedure you describe to cause the problem in comment 1.

Also I looked at your "with failure" log again and I see the message moved from trash back to inbox. I see the flags for the new message back in inbox but there is no \deleted flag set on that message or any other message in inbox per your log. During or right after the "with failure" log was recorded, is it possible there is another client (e.g., a phone app) that might somehow be setting the \deleted flag on the message moved back to inbox? Seem a bit unlikely but who knows, since it appears that the message goes back to inbox and is visible for a short time then goes invisible.

When I mentioned 2 clients, it was like this

  1. Start phone email client
  2. Start web browser email client
  3. Using web browser client, delete message from inbox
  4. Using web browser client, moved deleted message back to inbox
  5. Works ok

It broke like this:

  1. Start phone email client
  2. Start web browser email client
    2.5 Start TB
  3. Using web browser client, delete message from inbox
  4. Using web browser client, moved deleted message back to inbox
  5. Message never appears in inbox

However, something has changed today.
On the new installation of TB it works ok now, even though you saw it breaking on the video. This is with a web browser and phone also connected to the same account at the same time.
On my main installation of TB it is still broken as previously described even if I turn my phone off so that the only thing connected to the mail account is this one installation of TB

Well, nothing makes sense. Neither the good or bad log you attached show the \deleted flag being stored nor does it list any messages having the \deleted flag set. So there is no evidence in the logs that tb is somehow setting (storing) the \deleted flag on the message that gets moved back to inbox. However, the "It broke like this:" scenario in comment 16 sure seems like tb is causing the problem.

Can you record a log (probably using your main installation) that show tb setting the \deleted flag at some point? You may need to grep the log and look for regexp "senddata.*store.*delete" in the content somewhere. No reason to attach the log unless you see a store of \deleted occur in the log. I'm not sure how roundcube talks to your server; I assume it is via imap between the webserver and the imap server. Maybe there is a log at the imap server you can record that would shows all the client imap activity: tb, roundcube, phone mail. Maybe you could use wireshark or even tcpdump to see the network activity of all clients. Other than this, I can't seem to duplicate the problem and don't know what else to suggest.

(In reply to Matthew Jurgens from comment #10)

I have tried a brand new install of TB v70.0b1 on a machine that has never had it installed before.
The same problem exists - the message deleted from Inbox, when moved back, does not appear.

Matthew, does the new install use "one file per email" (a.k.a., maildir offline storage)? I just used the default "one file per folder" storage (mbox) when I tested the "gene" account. Do you see this as a relevant variable?

That setting appears not to be relevant because:
My main install uses file per message
The new install uses the default file per folder
Both have shown the problem

See Also: → 1582981

One other thing I should note is that I am running the 64 bit version (have been for about 1-2 months). I did previously notice the same issue with the 32 bit version, most notably with regards to the Junk folder. It was the same steps as stated when I lodged this bug:

  1. Mail comes to Inbox
  2. Mark as Junk
  3. Mail automatically moved to Junk folder
  4. Go to Junk Folder
  5. Mark as not junk
  6. Mail disappears from Junk folder but cannot be found in Inbox.

I will spend further time trying to reproduce this problem with whatever combination of folders I can.
Will report back

I tried the exact same step earlier today but 6 did: Mail disappears from Junk folder and returns to Inbox. Anyhow, thanks for helping to troubleshoot this.

Matthew, Thanks for the additional information, test account and examples logs sent via email. I do see in the log you sent via email that tb is setting the \deleted flag (and \seen) after the message is successfully moved to Inbox from Spam and Trash. However, I still can't duplicate the problem here using the test account and have never seen this problem on any of my existing accounts.

In the last example, the video shows that you are using "Unified" folder view. When I tested it I just used normal folder view. Have you seen the problem only with unified view by any chance? (I'm pretty sure the answer is "No" since I think the first video you sent showed normal folder view.)

Looking again at the log you sent via email, I don't see a reason why tb sometimes sets the \deleted and \seen flags after the message is moved back to Inbox (and sometimes it doesn't).

I did find one very remote possibility as to the cause. It would require several seemly unlikely thing to be true according to the code:

  1. Preference mail.server.default.dup_action not the default value (0) meaning keep duplicates.
  2. Somehow the message moved back to Inbox is detected as a "duplicate"

In this case the message gets marked as \seen and \deleted at this point in the code:
https://searchfox.org/comm-central/rev/266e9cc242cd0de076e85eb4aa0b8392fcb2ca01/mailnews/imap/src/nsImapMailFolder.cpp#2846

There is also a confirmed but not fixed bug that reports very similar symptoms as you report but due to dup_action setting at value 2:
Bug 618809

I'm able to duplicate what you observe and I see in your log using the test account if I,

  1. Set mail.server.default.dup_action to 0, i.e., reset to default.
  2. Restart tb to ensure change takes effect.
  3. Copy the same message twice from Trash to Inbox (*)
  4. Set mail.server.default.dup_action to 1, i.e, "delete" message when dup new message arrives in Inbox
  5. Restart tb.
  6. Delete one of the duplicate messages in Inbox to Trash.
  7. Go to Trash and move the just deleted message back to Inbox.
  8. Go back to Inbox. Message moved from Trash not visible and seems lost.
  9. Change delete mode to "just mark as deleted" (**)
  10. Select another folder (e.g., Sent) and go back to Inbox to refresh display.
  11. "Lost" message has red X and crossed out. Can un-delete it if wanted.

(*) I wasn't able to find a duplicate message in the test account without doing this step.
(**) Looking for gray message (marked deleted) in roundcube would be the same as these 3 steps.

If you record a log you will find a line with this case-insensitive regex: "senddata.*store.*deleted"
Well, let me know if this maybe corresponds with your tb configuration. My guess is that at some point in the past you set mail.server.default.dup_action to non-default value, probably 1 since I am only seeing the lost message "marked" \deleted which is what the value 1 actually does. The other options are 2) move to trash and 3) mark as read. This feature is new to me!
If you have only had dup_action set to 0, will just have to keep looking...

Thanks Gene. I certainly do have mail.server.default.dup_action set to 1. I have no idea how it got set like this. I assume that I should just set it to 0 and see how it goes.
I find it strange how it was happening intermittently when set to 1 but maybe it does not matter

Well, that's good to hear that it is set to 1. I was pretty much at the end of what to look for and that seemed to be the only thing that would cause what you saw.

No idea how it could have spontaneously gotten set to 1. But maybe it is because you have been using the same profile for quite a while and something got crossed up. The value 1 would apply globally to all accounts, new and old, within the profile. Only a new profile would cause it to go back to default 0 (no action if duplicate detected).

I haven't studied the code in detail but from what I have read, the subject and message-id are hashed to determine if the message is a duplicate. It doesn't hash all the messages in Inbox but only new ones received during a session. Moving from Inbox to Trash or Junk and then back makes the messages moved back look new in Inbox, so there may be a bug there too, maybe related to unfixed Bug 618809.

Anyhow, setting the dup_action to 0 will cause it to skip all the dup detection and your immediate problems should be solved.
I will go ahead and mark this bug as INVALID but take a closer look at Bug 618809. Post back here or in bug 618809 if you have more information. Thanks.

Status: UNCONFIRMED → RESOLVED
Closed: 6 years ago
Resolution: --- → INVALID
See Also: → 618809

I am changing this to a duplicate of Bug 618809. The trick is to delete the message to trash, move back to inbox, delete it again and then try to move back to inbox. With dup_action 1 it will get marked \deleted in inbox and appear lost. See bug 618809 comment 13 for more details.

Resolution: INVALID → DUPLICATE
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: