460085 - snapshots of composed emails save as separate messages in IMAP drafts folder - multiple drafts (server returns nothing to SEARCH HEADER Message-Id: even though server doesn't return UIDPLUS in CAPABILITY response)

Status: NEW

(MailNews Core :: Networking: IMAP, defect)

Description

[Nelson Bolyard (seldom reads bugmail)]

recent SM trunk nightly

I'm not sure whether this bug should be reported against email composition
or against core IMAP.  I'm guessing core IMAP, but please change that if
I guessed wrong.

I recently switched from POP to IMAP for my primary email account.
I've had IMAP accounts for years, but used them very little until now.

As I compose outgoing email messages, the mail client periodically saves a
copy into the drafts folder on my IMAP server.  Strangely, it saves each
draft as a separate message in the drafts folder.  If I take a long time
to compose an email (say, 30 minutes), then there are 6 separate drafts
of the message in my drafts folder.  After I send the message, all (or 
perhaps all but one) of those drafts remain in the drafts folder.  So, now,
once a day, I have to go read the drafts folder and clear out most of the 
drafts.  

Of the two IMAP accounts that I now have, only one seems to have this 
problem of saving each draft snapshot as a separate message.  The other
one seems to behave as I would expect, keeping only one draft of each 
message and deleting the draft when I send the message.  

Could the server somehow be causing this behavior?  If so, how?

Comment 1

[Magnus Melin [:mkmelin]]

Are the accounts on the same server?

Please get an imap log - http://wiki.mozilla.org/MailNews:Logging

Comment 2

[Nelson Bolyard (seldom reads bugmail)]

Attachment: zip file of requested IMAP log

This zip file contains the IMAP log for a period of time during which
I composed an email to Rob Stradling, subject 
"Re: SSL servers sending out multiple cert chains?"
and I typed characters into it every few minutes. 
After 20 minutes, I had 4 copies of it in the drafts folder.
The unzipped log file is ~1MB.  I'd have filtered it down for you if I
could have done while being certain I would not lose anything useful.

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #2)
> zip file of requested IMAP log

> 2216[4bf7200]: 50ac800:imap.secureserver.net:S-Drafts:ProcessCurrentURL:imap://[e-mail]:993/appenddraftfromfile%3E/Drafts%3EUID%3E:  = currentUrl
> 2216[4bf7200]: 50ac800:imap.secureserver.net:S-Drafts:SendData: 27 append "Drafts" (\Seen \Draft) {1468}
> 2216[4bf7200]: 50ac800:imap.secureserver.net:S-Drafts:CreateNewLineFromSocket: + OK
> 2216[4bf7200]: 50ac800:imap.secureserver.net:S-Drafts:SendData: FCC: imap://[e-mail]/Sent Items
> X-Identity-Key: id10
> Message-ID: <[e-mail]>
> (snip)
> 2216[4bf7200]: 50ac800:imap.secureserver.net:S-Drafts:SendData: 
> 2216[4bf7200]: 50ac800:imap.secureserver.net:S-Drafts:CreateNewLineFromSocket: 27 OK [APPENDUID 1 73] APPEND Ok.
> 2216[4bf7200]: 50ac800:imap.secureserver.net:S-Drafts:SendData: 28 noop
> 2216[4bf7200]: 50ac800:imap.secureserver.net:S-Drafts:CreateNewLineFromSocket: * 12 EXISTS
> 2216[4bf7200]: 50ac800:imap.secureserver.net:S-Drafts:CreateNewLineFromSocket: * 12 RECENT
> 2216[4bf7200]: 50ac800:imap.secureserver.net:S-Drafts:CreateNewLineFromSocket: 28 OK NOOP completed
> 2216[4bf7200]: 50ac800:imap.secureserver.net:S-Drafts:SendData: 29 uid SEARCH UNDELETED HEADER Message-ID [e-mail]
> 2216[4bf7200]: 50ac800:imap.secureserver.net:S-Drafts:CreateNewLineFromSocket: * SEARCH
> 2216[4bf7200]: 50ac800:imap.secureserver.net:S-Drafts:CreateNewLineFromSocket: 29 OK SEARCH done.
>(Mail copy to Sent folder follows)

Your IMAP server returns next to CAPABILITY.
> * CAPABILITY IMAP4rev1 STARTTLS CHILDREN NAMESPACE
Then Tb issues "29 uid SEARCH UNDELETED HEADER Message-ID ...", when tries to delete old draft mail.
But your server doen't return search result.
This is the reason why draft is not deleted.
See Bug 402132 for similar situation of Gmail IMAP(Gmail IMAP bug in this case).
 
Your server returns following to APPEND(to Drafts folder), even though "STARTTLS CHILDREN NAMESPACE" only to CAPABILITY.
> 27 OK [APPENDUID 1 73] APPEND Ok.
This indicates UIDPLUS extension is installed and used.
> http://www.faqs.org/rfcs/rfc4315.html
I guess Tb will not use "uid SEACRH ... Message-ID", if UIDPLUS to CAPABILITY and APPENDUID to APPEND is returned.
And, server possibly will do nothing for "uid SEACRH ... Message-ID" when UIDPLUS extension is installed and enabled.

Server is properly set up?

Comment 4

[Nelson Bolyard (seldom reads bugmail)]

M Wada wrote to me some additional information about this bug.  
Here is an excerpt from that email:

> (1) If no UIDPLUS in CAPABILITY response is intentional, server should 
> return search result to UID SEARCH.
> (2) If server returns nothing to "29 uid SEARCH UNDELETED HEADER Message-ID 
> ...", and if it's due to UIDPLUS extension, server should return at least 
> UIDPLUS to CAPABILITY.
> (3) I think UID SEARCH should work always (even when UIDPLUS is enabled),

Comment 5

[Lewis Rosenthal]

Nelson, is this still an issue?

I am *not* seeing this behavior under:

Mozilla/5.0 (OS/2; U; Warp 4.5; en-US; rv:1.9.1.16) Gecko/20101127 Lightning/1.0b2 Mnenhy/0.8.3 SeaMonkey/2.0.11

My IMAP accounts are hosted on CommuniGate Pro 5.1.x and on Novell GroupWise 8.0.2 systems. I do not have access to an Exchange system to test against that, nor have I tested against Cyrus.

Cheers.

Lewis

Comment 6

[Nelson Bolyard (seldom reads bugmail)]

Lewis,
Yes, I've been living with this continuously for 2+ years now. 
 
When I filed this bug, I was a newbie to IMAP and IMAPS.  I had no other 
accounts or servers against which to compare it.  Now I have numerous IMAPS 
email accounts with various mail service providers, and only one of them has 
this problem.  This fact makes me think that the problem is in that particular IMAPS server, which is imap.secureserver.net port 992, GoDaddy's IMAPS server.
Unfortunately, this is my primary mail service provider.

Comment 7

[Wayne Mery (:wsmwk)]

see also https://bugzilla.mozilla.org/buglist.cgi?type1-0-0=nowordssubstr&order=Bug%20Number&short_desc=draft&field0-0-0=short_desc&bug_severity=blocker&bug_severity=critical&bug_severity=major&bug_severity=normal&bug_severity=minor&type0-0-1=substring&field0-0-1=keywords&type1-0-1=allwordssubstr&resolution=---&classification=Client%20Software&classification=Components&query_format=advanced&value1-0-0=open%20alert%20editor&short_desc_type=allwordssubstr&type0-0-0=anywordssubstr&value0-0-0=two%20mult%20many%20more%20&field1-0-0=short_desc&product=MailNews%20Core&product=Thunderbird&field1-0-1=short_desc

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #2)
> zip file of requested IMAP log

As seen in log, Tb doesn't issue CAPABILITY command after authentication.
> After greeting,
> imap.secureserver.net:NA:CreateNewLineFromSocket: 1 OK CAPABILITY completed
> imap.secureserver.net:NA:SendData: Logging suppressed for this command (it probably contained authentication information)
> imap.secureserver.net:NA:CreateNewLineFromSocket: 2 OK LOGIN Ok.
> imap.secureserver.net:A:SendData: 3 namespace
This causes loss of UIDPLUS capability response if IMAP server returns full set of capability responses after authentication.
It was known bug and is already resolved by Tb 3.0.x or Tb 3.1.

(In reply to comment #6)
> This fact makes me think that the problem is in that particular IMAPS server,
> which is imap.secureserver.net port 992, GoDaddy's IMAPS server.

Nelson, get IMAP log with current Tb and check CAPABILITY response and IMAP flow around draft save and delete of previous version of draft, please.
No UIDPLUS response from server to CAPABILITY command after authentication?

Comment 9

[Nelson Bolyard (seldom reads bugmail)]

Wada, Thank you for taking another look at that log file.  Is there anything 
I can do to get TB to generate a shorter one with much less irrelevant chatter?

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #9)

Yes.
Get log with minimum configuration; new profile, relevant account only, with no non-needed/unwanted activities by Tb.  
(1) Create a new Tb profile, define an imap.secureserver.net IMAP account.
(2) Tools/Options/Advanced/General : disable "Global Search and Indexer"
(3) Modify account settings
    Synchronization&Storage : disable "keep messages ..."
                              (offline-use=off of all folders)
    Server Settings : disable automatic new mail check
    Copies&Folders : choose appropriate Drafts folder,
                     create Drafts folder if required, and save a draft in it.
                     enable "dialog after draft save".
(4) Restart Tb with IMAP logging enabled.
(5) Compose a mail, repeat draft save saveral times.

Comment 11

[Wayne Mery (:wsmwk)]

Puts down a dollar to see if Nelson will post a protocol log before me. :)

As noted in another bug, the frequency of this occurring increase at low values of autosave, like 1 minute. (My current setting for testing is 1 minute autosave, and check for new messages is also 1 minute)


xref:
* Bug 382517 - Deleted copy of draft by auto-save(previous version of draft) is not removed from thread pane until folder re-open
* Bug 279366 - "Options > Send a Copy" makes copy on SAVE; multiple saves results in multiple copies in the Drafts folder
* Bug 482836 - Multiple (iterations) copies of draft messages are saved in the DRAFTS folder (local Drafts folder, not IMAP)
* Bug 549274 - Drafts not deleted on IMAP after message has been sent

from  https://bugzilla.mozilla.org/buglist.cgi?type1-0-0=nowordssubstr&short_desc=draft%20&field0-0-0=short_desc&bug_severity=major&bug_severity=normal&bug_severity=minor&type0-0-1=substring&field0-0-1=keywords&type1-0-1=allwordssubstr&resolution=---&classification=Client%20Software&classification=Components&query_format=advanced&value1-0-0=create&short_desc_type=allwordssubstr&type0-0-0=anywordssubstr&value0-0-0=two%20mult%20many%20delet&field1-0-0=short_desc&product=MailNews%20Core&product=Thunderbird&field1-0-1=short_desc

Comment 12

[Nelson Bolyard (seldom reads bugmail)]

Attachment: shorter imap.log file

Here is the log file produced as WADA requested.
Do I get $1 now?  :)

Comment 13

[Nelson Bolyard (seldom reads bugmail)]

Attachment: YA even shorter log, no extraneous folder fetches

It occurred to me that WADA probably wanted a log without any fetches of
messages from any folders, so my checking the contents of the "send_later"
(drafts) folder in the previous log probably defeated the purpose.  This 
log contains to attempts to seer the contents of any folders.

Comment 14

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #13)
> YA even shorter log, no extraneous folder fetches

Thanks for shorter, simpler, easy to read new log.

It was same as previous log.

(1) Your server sends capability list in greeting, but not full list(no UIDPLUS)
> * OK [CAPABILITY IMAP4rev1 UNSELECT STARTTLS ID CHILDREN NAMESPACE IDLE]
>      Courier-IMAP ready. Copyright 1998-2004 Double Precision, Inc.
>      See COPYING for distribution information.

(2) As capability list exists in greeting, Tb doesn't issue CAPABILITY before login.

(3) Your server doesn't return optional capability list(full capability list which includes UIDPLUS) in OK response to login.
> try to log in
> 2 OK LOGIN Ok.

(4) Tb doesn't issue CAPABILITY command after authentication in this case.
    This is Bug 564642, and it was not resolved yet.
> 3 namespace

(5) Because UIDPLUS is enabled at server, server returns UID to APPEND.
    Tb doesn't know about UIDPLUS in capability response, so Tb doesn't use it. 
> :S-Send_Later:SendData: 9 append "Send_Later" (\Seen \Draft) {825}
> :S-Send_Later:CreateNewLineFromSocket: + OK
> :S-Send_Later:SendData: FCC: imap://[e-mail]/Sent Items
> :S-Send_Later:CreateNewLineFromSocket: 9 OK [APPENDUID 1 969] APPEND Ok.
> :S-Send_Later:SendData: 10 noop
> :S-Send_Later:CreateNewLineFromSocket: * 72 EXISTS
> :S-Send_Later:CreateNewLineFromSocket: * 72 RECENT
> :S-Send_Later:CreateNewLineFromSocket: 10 OK NOOP completed

(6) Because UIDPLUS doesn't exist in capability response, Tb uses SEARCH command. But, server doesn't return search result, probably because UIDPLUS is enabled.
> :S-Send_Later:SendData: 11 uid SEARCH UNDELETED HEADER Message-ID [e-mail]
> :S-Send_Later:CreateNewLineFromSocket: * SEARCH
> :S-Send_Later:CreateNewLineFromSocket: 11 OK SEARCH done.

(a) If server sends full capability response in greeting, no problem.
(b) If server returns optional(full) capability list in OK response to login, no problem.
(c) If Tb issues CAPABILITY after authentication, no problem.
(d) If server supports SEARCH well even when UIDPLUS is enabled, no problem.

Unfortunately, RFC violation is (c) of Tb, as written in Bug 564642.
(d) is possibly RFC violation by server, but I don't know.

Nelson, can you check CAPABILITY response to CAPABILITY command after login using tool like telnet?
(If no-SSL, telnet can be used because plain text login is used)

Comment 15

[Nelson Bolyard (seldom reads bugmail)]

Attachment: capture of login and capability responses

As it happens, NSS's test tools include an SSL-telnet client.  
I used it to generate this log.  I've replaced the password in this.

Comment 16

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #15)
> capture of login and capability responses

My guess seems wrong...
> * OK [CAPABILITY IMAP4rev1 UNSELECT STARTTLS ID CHILDREN NAMESPACE IDLE] 
>   Courier-IMAP ready. Copyright 1998-2004 Double Precision, Inc.
>   See COPYING for distribution information.
> a001 CAPABILITY
> * CAPABILITY IMAP4rev1 UNSELECT STARTTLS ID CHILDREN NAMESPACE IDLE
> a001 OK CAPABILITY completed
> a010 LOGIN [e-mail] XXXXXXXXXXXX
> a010 OK LOGIN Ok.
> a011 CAPABILITY
> * CAPABILITY IMAP4rev1 UNSELECT STARTTLS ID CHILDREN NAMESPACE IDLE
> a011 OK CAPABILITY completed

Your server never returns UIDPLUS in any of next;
  - Initial greeting
  - CAPABILITY command before login
  - CAPABILITY response in OK response to login(No optional capaility response)
  - CAPABILITY command after login(authentication)

Nelson, is your server correctly configuired?
Do you use Proxy server to access your IMAP server?

> As it happens, NSS's test tools include an SSL-telnet client.  

Can the NSS's test tools be usable on Win? Where can get the tools?

Comment 17

[Nelson Bolyard (seldom reads bugmail)]

WADA,
The email server in question is the server that the GoDaddy company makes 
available to its customers who want IMAPS and/or SMTPS service.  As far as 
I know, they own it and run it.  

I do not use a proxy.

The ssl telnet client program (named tstclnt) runs on any platform on which 
firefox runs, which also has a command line shell.  This includes all versions
of windows since Win NT4.  You can pull sources from 
ftp://ftp.mozilla.org/pub/security/nss/releases/NSS_3_12_9_RTM/src/nss-3.12.9-with-nspr-4.8.7.tar.gz
Build instructions are on devmo, IIRC.

Comment 18

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #17)
> I do not use a proxy.

As seen in bug 357284, some IMAP proxy software return his own greeting instead of original greeting from real IMAP server for mainly performance reasons.
bug 357284 is for next.
  Mirapoint proxy returns ACL even though IMAP server doesn't support ACL.
Following is for similar case, and an example of bypass by "perdition proxy".
> http://trac.roundcube.net/ticket/1485750
> To work around this, we've temporarily added an additional perdition proxy
> which reports the same CAPABILITY list as the Cyrus servers in question,
> which resolves the issue. 
This technique can be applied to your case.

Even if bug 564642 will be fixed, your problem won't be resolved, because UIDPLUS doesn't exist in capability response after login in your case.
I guess that administrator of GoDaddy's server merely forgot to add UIDPLUS in greeting string of Courier IMAP when he enabled UIDPLUS extention.
Nelson, ask GoDaddy's support to add string of UIDPLUS to greeting.
If it'll take long, consider bypass by "perdition proxy" and so on. If UIDPLUS exists in initial greeting, Tb can behave as we want in your case.

Comment 19

[Lewis Rosenthal]

But WADA, shouldn't we just fall back from using the UIDPLUS features if the server reports (however wrongly) that it does not support UIDPLUS?

RFC 4315 (full text at http://tools.ietf.org/html/rfc4315 ) says:

   The UIDPLUS extension defines an additional command.  In addition,
   this document recommends new status response codes in IMAP that
   SHOULD be returned by all server implementations, regardless of
   whether or not the UIDPLUS extension is implemented.

   The added facilities of the features in UIDPLUS are optimizations;
   clients can provide equivalent functionality, albeit less
   efficiently, by using facilities in the base protocol.

Obviously, older server implementations are becoming more rare (Courier IMAP has supported UIDPLUS since 2003), but as you point out, it is entirely possible that a proxy - though apparently not in this case - may not fully/properly report actual server capability. I would think that we (on the client side) should have a better way of dealing with these situations over which the end user may indeed have no control.

What happened to the pre-UIDPLUS IMAP routines in MailNews, anyway?

PMFJI...

Lewis

Comment 20

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #19)
> But WADA, shouldn't we just fall back from using the UIDPLUS features if the
> server reports (however wrongly) that it does not support UIDPLUS?

I agree with you on that Tb is better to use uid in OK APPENDUID and OK COPYUID response to UID APPEND/COPY command and better to support NO UIDNOTSTICKY ressponse to SELECT command.
If Tb uses uid in OK APPENDUID response, Tb doesn't need to issue SEARCH Message-Id, then Tb won't meet with server's bug of "SEARCH Message-Id oesn't return search bug" upon trying to delete old draft mail. 

However, I believe such optional responses should be defined as enhancement of IMAP itself such as "IMAP 4 rev 1.1 or rev 2". If "IMAP 4 rev 1.1 or rev 2" will be defined, any IMAP client who is compliant with "IMAP 4 rev 1.1 or rev 2" should support the optional status responses when IMAP4rev1 or higher is returned in capability response. 

In this bug's case, cause is next;
1. Server doesn't return UIDPLUS even though server apparently supports UIDPLUS.
   - Server is Courier-IMAP.
   - Server returns "OK [APPENDUID" response.
2. Server doesn't return search result to "SEARCH header Message-Id".
   This command defined by IMAP4 should work at server always, regardless of
   existence/non-existence of UIDPLUS extension at server.
   I couldn't find statement like next in RFC for UIDPLUS.
   - Server doesn't need to implement "SEARCH HEADER" correctly
     when UDPLUS extension is supported. 

I believe "adding UIDPLUS to greeting at server side correctly" is sufficient for this bug's problem.
Lewis Rosenthal, if you think Tb's enhancement to support "optional status responses defined by UIDPLUS extension" is mandatory, open separate bug, please.

Comment 21

[Nelson Bolyard (seldom reads bugmail)]

3.5 years after opening this bug report, the latest TB still has this issue 
with GoDaddy's servers.  :-/

Comment 22

[Lu Wei]

Attachment: imap log of draft message not deleted after close

the IMAP account is a local service created by davmail, which communicate with a exchange server.

Comment 23

[WADA:World Anti-bad-Duping Agency]

(In reply to LU Wei from comment #22)
> imap log of draft message not deleted after close

After append of draft mail(upload of new version of draft mail), Tb tries to delete old version of draft and issues SEARCH to find old version, because UIDPLUS is not supported so appened uid is not returned to append command.  
> 7 uid SEARCH UNDELETED HEADER Message-ID [e-mail]
> * SEARCH
> 7 OK SEARCH completed
However, server doesn't rerurn search result correctly(nothing is returned).
Server should implement SEARCH command(and UID SEARCH) correctly.

Capabilily list in greeting.
> * OK [CAPABILITY IMAP4REV1 AUTH=LOGIN IDLE] IMAP4rev1 DavMail 3.9.6-1843 server ready
DavMail 3.9.6 looks newest release.
> http://davmail.sourceforge.net/roadmap.html
> DavMail RoadMap
>  3.9.7
>  Next minor release 

Does your server correctly configured?

Comment 24

[Lu Wei]

Attachment: another imap log of deleting draft failed

Comment 25

[Lu Wei]

I created another log, using the newest release of Thunderbird 10 and Davmail 3.9.7. Please check whether it is still the same problem. I did not do any special configuration and just do as Davmail's installation page suggest. 

The log is created as following procedure:
1, run log.bat as:

set NSPR_LOG_MODULES=imap:5
set NSPR_LOG_FILE=.\imap.log
"C:\Program Files\Mozilla Thunderbird\thunderbird.exe"

2, write a new mail, wait until status say like "copying to draft..."
3, close composition window, select "don't save", and close TB.

Comment 26

[Lu Wei]

So...Has this bug been identified to TB or Davmail? 
Additional information: The drafts which are undeleted aforesaid could be deleted "manually", that is, go to draft folder, select them, and delete.

Comment 27

[Lu Wei]

Davmail 4.0.0 has resolved the issue, as its release notes said: " - IMAP: Exchange 2010 does not support header search, workaround to avoid duplicate items in Drafts folder with Thunderbird." It sounds like Exchange's problem?