1055445 - [email] Coremail servers automatically save sent messages in the 'Sent' folder, the email app doesn't know this and saves a second copy. The backend should detect coremail servers by X-CM-EXT-1 capability and avoid redundant appending.

Status: VERIFIED FIXED

(Firefox OS Graveyard :: Gaia::E-Mail, defect)

Description

[Mingyan Liu]

DEFECT DESCRIPTION:
  Two same mails get in "已发送" folder after sending a mail with 126 or 163 account 

Steps to reproduce:
----------------------------------------------------
1. Launch Email
2. Configure a 163 email account
3. send a mail successfully，then open the “已发送” folder

Actual result:
---------------------------------------
   I found two same mails in the folder.
 

Expected result:
--------------------------------------- 
  Only one mail.


Additional info:
--------------------------------------
The problem only exists in 126 or 163 imap accounts.
Reproduce rate:  5/5

Comment 1

[Mingyan Liu]

My test account uid/pwd = [e-mail]/a123456

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

I'm guessing these are IMAP accounts and 126.com/163.com's SMTP server automatically appends messages into the Sent folder so we don't have to.  But we don't know that, so we do it anyways.

We do know that gmail does this and so don't do it for gmail, but we assume every other server is going to make us do it ourselves.  We detect gmail on the basis of their documented "X-GM-EXT-1" capability.

The good/easy news is that 126.com/163.com seem to use http://www.coremail.cn/ and identify themselves with an "X-CM-EXT-1" capability.  It seems a little bit like awkward cargo-culting since this capability does not appear to be meaningfully documented on the internet.  They did adopt XLIST so maybe it was part of that.  But the good news is that this capability means it's really easy for us to do the right thing on coremail servers.

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

Ideally we could also lump in detecting coremail servers in our AutoDiscover process so we can ignore them.  Probably also want to add the autoconfig entry for 126.com with this (currently tracked on another bug).

Comment 5

[ffos_st]

If the mail configuration, using pop3 / smtp way,sent box does not show exactly the same two transmit information,the imap / smtp way, sent two boxes will be displayed exactly the same message.
Temporarily did not find a solution, if there is any way to solve, thanks a lot.

Comment 6

[ffos_st]

diff --git a/apps/email/js/ext/imap/account.js b/apps/email/js/ext/imap/account.js
index 4d7561f..af5862a 100644
--- a/apps/email/js/ext/imap/account.js
+++ b/apps/email/js/ext/imap/account.js
@@ -134,6 +134,11 @@ var properties = {
  return this.meta.capability.indexOf('X-GM-EXT-1') !== -1;
 },

+ // SPRD: bug389292, sent box has the same message two. start @{ 
+ get isNetease() {
+ return this.meta.capability.indexOf('X-CM-EXT-1') !== -1;
+ },
+ // SPRD: bug389292 end @} 
  ////////////////////////////////////////////////////////////////////////////// 
  // Connection Pool-ish stuff 

@@ -805,7 +810,8 @@ var properties = {
  saveSentMessage: function(composer) {
  // (gmail automatically copies the message into the sent folder; we don't
  // have to do anything) 
- if (this.isGmail) {
+ // SPRD: bug 389292, sent box has the same message two.
+ if (this.isGmail || this.isNetease) {
    return;
  }
--------------------------------------------------------
Please check,thanks!

Comment 7

[Andrew Sutherland [:asuth] (he/him)]

Attachment: GELAM patch

Thanks for the patch in comment 6!  I've targeted it against the gaia-email-libs-and-more repo where the back-end lives and done some review cleanup/refactoring.

Comment 8

[Andrew Sutherland [:asuth] (he/him)]

Attachment: gaia pull request

The resulting gaia patch/pull request.

Comment 9

[Andrew Sutherland [:asuth] (he/him)]

landed on master:
https://github.com/mozilla-b2g/gaia-email-libs-and-more/pull/361
https://github.com/mozilla-b2g/gaia-email-libs-and-more/commit/d0091099856877106ce66e29ac08303d422c80ed

https://github.com/mozilla-b2g/gaia/pull/27562
https://github.com/mozilla-b2g/gaia/commit/8df71ff44cc9be5d828176620913f9d5a3a12583

Comment 10

[Andrew Sutherland [:asuth] (he/him)]

Comment on attachment 8552376 [details] [review]
gaia pull request

[Approval Request Comment]
[Bug caused by] (feature/regressing bug #): Not a regression.
[User impact] if declined: Users using 163.com, 126.com, or other CoreMail based mail servers (which seem extremely popular in China!), sent mail will be duplicated in the sent folder.  Since we actually go out of our way to place the second copy in the sent folder, this also wastes bandwidth and battery.
[Testing completed]: :asuth tested with patch on b2g-desktop against 163.com for positive change and gmail.com and yahoo.com to verify no regressions, logically equivalent patch by ffos_st tested by ffos_st against 163.com, all unit tests happy.
[Risk to taking this patch] (and alternatives if risky): Very low risk.
[String changes made]: No string changes made.

Comment 11

[Andrew Sutherland [:asuth] (he/him)]

Er, regression clarification; when we used ActiveSync for 163.com/126.com/other coremail servers a while back.  When using ActiveSync, everything was terribly broken, but sending would not result in duplicate messages.

Comment 12

[Ryan VanderMeulen [:RyanVM]]

v2.2: https://github.com/mozilla-b2g/gaia/commit/95237eb264a01d7558a982ff5efd2c71a8565709

Comment 13

[helen（leave from Mozilla）]

Attachment: verified_3.0.mp4

This bug has been verified as pass on latest build of Flame v3.0 by the STR in Comment 0.

Results: Open the “已发送” folder after sending a mail successfully, you can find no two mails in this folder are the same.


See attachment: verified_3.0.mp4
Reproduce rate: 0/5

Device: Flame 3.0 build(Pass)
Build ID               20150506160205
Gaia Revision          426fe6450ab8da92bb473fef12ccb39c6c920dd0
Gaia Date              2015-05-06 08:40:16
Gecko Revision         https://hg.mozilla.org/mozilla-central/rev/5593ac626826
Gecko Version          40.0a1
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.cltbld.20150506.193508
Firmware Date          Wed May  6 19:35:21 EDT 2015
Bootloader             L1TC000118D0

Comment 14

[helen（leave from Mozilla）]

Attachment: logcat1652.txt

Comment 15

[helen（leave from Mozilla）]

Attachment: v2.2.mp4

This bug has been failed verified on latest Nightly Flame v2.2.

Repro STR:
1.Launch Email.
2.Configure a 163/126 email account([e-mail]/a123456 & [e-mail]/).
3.Send a mail successfully，then open the “已发送” folder.
Result:
Open the “已发送” folder after sending a mail successfully, you can find two same mails in this folder.

See attachments: v2.2.mp4 and logcat1652.txt
Reproduce rate: 5/5

Device: Flame 2.2 build(Fail)
Build ID               20150506002501
Gaia Revision          772a9491909abd02dc67278dd453746e2dd358a8
Gaia Date              2015-05-05 02:02:24
Gecko Revision         https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/3af6a0a79227
Gecko Version          37.0
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.cltbld.20150506.040209
Firmware Date          Wed May  6 04:02:20 EDT 2015
Bootloader             L1TC000118D0

----------------------------------------------------------
Note:
But when I use the account to test([e-mail]),you can find no two mails in this folder are the same.

I will submit a following bug tracking it tomorrow.

Comment 16

[Andrew Sutherland [:asuth] (he/him)]

(In reply to helen from comment #14)
> Created attachment 8602658 [details]
> logcat1652.txt

Thank you for the log!  I can confirm this log indicates our fix is not working.  Although it's confusing because the fix is definitely in the given gaia commit, etc.

The log entries of note:
05-07 16:53:37.871 I/Gecko   ( 1504): WLOG: [slog] smtp:sent^M
05-07 16:53:37.921 I/Gecko   ( 1504): WLOG: queueOp 7 append pre-queues: local: 0 server: 1
05-07 16:53:38.211 I/Gecko   ( 1504): WLOG: runOp(do: {"type":"append","longtermId":"7/4","lifecycle":"do","localStatus":"done","serverStatus":"doing","tryCount":0,"humanOp":"append","messages":[{"messageText":{},")
05-07 16:53:40.911 I/Gecko   ( 1504): WLOG: runOp_end(do: {"type":"append","longtermId":"7/4","lifecycle":"do","localStatus":"done","serverStatus":"doing","tryCount":0,"humanOp":"append","messages":[{"messageText":{},")

Since we capture the capability list at account creation time, the most likely explanations would seem to be:
- The IMAP server is not telling us the "X-CM-EXT-1" capability marker.  This could potentially happen if there's a proxy in use.
- Some type of race related to the life-cycle of the IMAP connection and its list of capabilities.
- Some internal inconsistencies in browserbox as it relates to maintenance of the "capability" list.  For example, care seems to be taken inside the login method to capitalize the capabilities, but the untagged capability handler does not do this.  (But maybe imap-handler or something is doing something?)

Next steps are to:
- check to see if there's a browserbox update on v3.0 that doesn't exist on v2.2 that explains this
- Reproduce with protocol debugging on so we can see what the server says.

I'll wait for helen's new tracking bug to be filed for now since I don't think I can spend more time on this until tomorrow.

Comment 17

[helen（leave from Mozilla）]

Hi,Andrew,
I'm sorry that I can't repro the bug again using latest build of Flame v2.2. I'm confused that I can't repro it using comment 15's build,either. Tomorrow will be Saturday in China, so I will check it again next week.

Comment 18

[Andrew Sutherland [:asuth] (he/him)]

Thanks for the update.  The most likely explanation is probably some fluctuation in the 163.com/126.com server responses to the CAPABILITY command we issue (which should include X-CM-EXT1), which is how we identify them as CoreMail servers.  It's possible they started an upgrade and then reverted, or they had a misconfiguration, or only some of their servers are exhibiting this problem, or there is some type of (misconfigured) proxy involved and there's a race as to whether the proxy or a real server responds.

In any event, I'll wait for your attempts to reproduce next week (and hope you have/had a good weekend! :).  If the problem is not reproducing, there's probably no need to investigate other workarounds.

Comment 19

[helen（leave from Mozilla）]

Attachment: verified v2.2.mp4

Hi,Andrew,
 Thanks for your kindly care about weekend. I've been tracking the latest two flame 2.2 builds and it has been verified as pass all the time.

Results: Open the “sent” folder after sending a mail successfully, you can find the correct behavior that there is only one mail in this folder.

See attachment: verified v2.2.mp4
Reproduce rate: 0/5

The latest one build verson as follows:

Device: Flame 2.2 (Pass)
Build ID               20150511162500
Gaia Revision          c4c1bf443f2b01c2ba918780510fd4c639a3c360
Gaia Date              2015-05-11 14:12:24
Gecko Revision         https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/70782f19acbf
Gecko Version          37.0
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.cltbld.20150511.195739
Firmware Date          Mon May 11 19:57:49 EDT 2015
Bootloader             L1TC000118D0

Comment 20

[Andrew Sutherland [:asuth] (he/him)]

That's great news!  Thanks for re-testing!