991489 - [Sora][Browser] Downloading attachments from 163.com mail server that are exposed as links freaks out browser and download manager

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::System, defect, P2)

Description

[sync-1]

Mozilla build ID:20140323004002
 
 DEFECT DESCRIPTION:
 The attachments can't be download from the attachment link.
 Note: Attachment link function only for 163 and 126 account.
  REPRODUCING PROCEDURES:
 1,Add an 163 account -> Receive an email with attachment -> Tap the attachment link to  download-> KO
 Bettle Lite FF is no ok.
  EXPECTED BEHAVIOUR:
 The attachment can be download from the attachment link.
  ASSOCIATE SPECIFICATION:
 
  TEST PLAN REFERENCE:
 
  TOOLS AND PLATFORMS USED:
 
  USER IMPACT:
 
  REPRODUCING RATE:
 
  For FT PR, Please list reference mobile's behavior:
  TEL:7556

Comment 1

[sync-1]

Attachment: refer the attachment

Comment 2

[Jason Smith [:jsmith]]

Can we confirm this on the Moz side?

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

From the screenshot it's clear that a hyperlink is being followed.  In general, this means that it it's not an e-mail problem and probably is a browser problem; IMAP attachments are handled from inside the e-mail app.  However, we already know that 163.com has a wacky ActiveSync implementation which is why we forced it to use IMAP in bug 951076.  So it's possible it also has some type of wacky attachment handling behaviour and it's detaching attachments and serving them up by some type of web URL and then it assumes there's a cookie in place or something like that.

Reporter, please provide the always-desired information from https://wiki.mozilla.org/Gaia/Email/RequiredBugInfo so we can be sure what type of account we're dealing with (IMAP versus ActiveSync)

And please provide a copy of the e-mail:
https://wiki.mozilla.org/Gaia/Email/ProvidingEmailsForDebugging

Comment 4

[Andrew Sutherland [:asuth] (he/him)]

Er, let me amend that.  It's probably not a browser problem either, it's just manifesting in the browser after e-mail hands off the attachment per spec, the actual fault lies with one of:
- the server linked to
- whoever composed the e-mail/their software
- something wacky going on with the mail server the tester is using

Comment 5

[Pi Wei Cheng [:piwei] (inactive)]

I was NOT able to reproduce this issue on Buri with 1.3, 1.4 Aurora, and 2.0 master.

I applied for an 163 email account, used a gmail account to send a png and a jpg file as attachments to the 163 email. Then I used Buri, set up the 163 email, received the emails, downloaded and viewed the attachments without issue.

Device: Buri 1.3 MOZ
BuildID: 20140414004002
Gaia: 8b92c56267fe50772095f1f25d6cc1f9c9966eb4
Gecko: 3e26908fca71
Version: 28.0
Firmware Version: v1.2-device.cfg

Device: Buri 1.4 Aurora MOZ
BuildID: 20140416000203
Gaia: 441c4bcd8ac4f8c01a9bc5a2f8d64eaa87844803
Gecko: a559848a8a00
Version: 30.0a2
Firmware Version: v1.2-device.cfg

Device: Buri 2.0 Master MOZ
BuildID: 20140416040204
Gaia: 3d4b4b06475d2376bad23ac46da185cd48a68d17
Gecko: dd50745d7f35
Version: 31.0a1
Firmware Version: v1.2-device.cfg

Comment 6

[Warren WU]

163 mail will provide an additional attachment download address if it receive a mail with attachments. such as: http://u.163.com/t/QcC2zM, it can not open in our browser, so I think this is the browser problems.

Comment 7

[Andrew Sutherland [:asuth] (he/him)]

Okay, I am turfing this to browser for more investigation.  The http://u.163.com/t/QcC2zM link is really handy as it can completely reproduce this problem if you just type it into the browser.  I'm not really clear what's going on with the link; Firefox on desktop seems okay with the whole thing, Chrome on desktop seems okay with it too, but its network inspector shows the download in red like it got freaked out by something.

On a trunk buri device the first time I tried to download the file, the browser showed the "oh no, there's a problem with the page", but then we get the download manager UI on top of that telling us the download failed.  When I tried to reload the option the download manager claims it's starting the download but only makes 0 of 0 bytes of progress.

A key thing for displaying the browser sadness page is that the u.163.com is a link shortener; it's a 302 bounce to the following URL which is what gets us our download:
http://preview.mail.163.com/xdownload?filename=Gg.vcf&mid=xtbBZhN0UlD-5dwx5gABsT&part=2&sign=400c995c76e24c4964afa006d2a1db78
&time=1398130177&uid=asasqw66%40163.com

=== tcpdump of the link shortener fetch
GET /t/QcC2zM HTTP/1.1
Host: u.163.com
User-Agent: Mozilla/5.0 (Mobile; rv:31.0) Gecko/31.0 Firefox/31.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Cookie: JSESSIONID=4058F34DAB0292CB219300CB91581BC1
Connection: keep-alive

HTTP/1.1 302 Moved Temporarily
Server: nginx
Date: Tue, 22 Apr 2014 15:02:44 GMT
Content-Type: text/html
Content-Length: 0
Connection: keep-alive
Set-Cookie: JSESSIONID=3A5D154E251449429EACB09603F19407; Path=/
Location: http://preview.mail.163.com/xdownload?filename=Gg.vcf&mid=xtbBZhN0UlD-5dwx5gABsT&part=2&sign=400c995c76e24c4964afa006d2a1db78&time=1398130177&uid=asasqw66%40163.com
X-Cache:  from ngx208-96.163.com



=== tcpdump of the actual fetch (will include the pcap as an attachment in a sec too):
GET /xdownload?filename=Gg.vcf&mid=xtbBZhN0UlD-5dwx5gABsT&part=2&sign=400c995c76e24c4964afa006d2a1db78&time=1398130177&uid=asasqw66%40163.com HTTP/1.1
Host: preview.mail.163.com
User-Agent: Mozilla/5.0 (Mobile; rv:31.0) Gecko/31.0 Firefox/31.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Cookie: TOKEN_COOKIE=400c995c76e24c4964afa006d2a1db78|asasqw66%40163.com|1398130177
Connection: keep-alive

HTTP/1.1 200 OK
Server: nginx
Date: Tue, 22 Apr 2014 15:02:46 GMT
Content-Length: 131
Connection: keep-alive
Set-Cookie: TOKEN_COOKIE=400c995c76e24c4964afa006d2a1db78|asasqw66%40163.com|1398130177; Domain=.preview.mail.163.com; Path=/
Content-Disposition: attachment; filename="=?UTF-8?B?R2cudmNm?="
X-Cache:  from ngx208-92.163.com

BEGIN:VCARD
VERSION:3.0
PRODID:-//Apple Inc.//iOS 7.1//EN
N:Gg;;;;
FN:Gg
TEL;type=HOME;type=VOICE;type=pref:11111
END:VCARD

Comment 8

[Andrew Sutherland [:asuth] (he/him)]

Attachment: pcap of a trunk buri device fetching the URL from the browser by hitting the reload option from the failure page

Comment 9

[Andrew Sutherland [:asuth] (he/him)]

Attachment: har file from chrome download (stable Firefox doesn't seem to want to make har files?)

Comment 11

[Josh Cheng [:josh]]

Hi Andrew, 
per https://bugzilla.mozilla.org/show_bug.cgi?id=1094042#c6. It seems the problem only happen in 2.0/2.0M but not 2.1/2.2. 
Since this issue is reported by partner again in bug 1094042, do you think we have chance to fix it in 2.0M? Thanks!

Comment 12

[Gregor Wagner [:gwagner]]

Aus, any idea whats going on here?

Comment 13

[Gregor Wagner [:gwagner]]

We should also get a reverse-regression window to see what patch fixed it.

Comment 14

[Andrew Sutherland [:asuth] (he/him)]

Just so we're all clear, the differences are like so:
v2.0/v2.0m: The mail server embeds a hyperlink in the message that the browser handles oddly.
v2.1/v2.2: The mail server is not embedding the hyperlink.

In all cases the browser probably does the wrong thing.


The most likely explanation is that v2.0/v2.0m are using ActiveSync instead of IMAP.  It would be handy if people reproducing can indicate the account type as reported by the account as requested at https://wiki.mozilla.org/Gaia/Email/RequiredBugInfo.  In theory we should be using IMAP in all cases for 163.com and ActiveSync for 126.com (a bug), but I'm not going to operate on any assumptions here.

Comment 15

[Peipei Cheng (needinfo if you need my action)]

Attachment: email_163_adblog

Andrew, 
I checked the email configuration on Flame 2.0. It turns out that the we use IMAP+SMTP for 163 mail account. 
But I was unable to check the behavior on 2.1/2.1 because I always get a network error problem while setting up 163 mail account. I tried both automatically setup and manual setup.


Attached is a log I get for 2.0. Could you please take a look first? Test time starts: 11-13 11:37:18.825

Test build
-------------------------------------------------------------------------
Gaia-Rev        5e532a84e762b1bb6231756182cf1465671a061e
Gecko-Rev       https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/124f0bed1700
Build-ID        20141027160201
Version         32.0
Device-Name     flame
FW-Release      4.4.2
FW-Incremental  eng.cltbld.20141027.194042
FW-Date         Mon Oct 27 19:40:52 EDT 2014
Bootloader      L1TC00011880

Comment 16

[Peipei Cheng (needinfo if you need my action)]

I checked with Norry who commented on bug 1094042.(https://bugzilla.mozilla.org/show_bug.cgi?id=1094042#c6).

On V2.1/V2.2, he use pop3+smtp to configure 163 email account. On v2.0, he used imtp+smtp to configure. This explains why the email content is different.

SO I configured pop3 on both v2.0 and v2.1, the behavior is the same.

Comment 17

[Josh Cheng [:josh]]

Triage:
Hi Peipei,
We change 163 to use IMAP per bug 951076. However it seems cause this bug. Can you try set 163 to use POP3 and check if there any regression issue? Thanks!

Comment 18

[Peipei Cheng (needinfo if you need my action)]

(In reply to Josh Cheng [:josh] from comment #17)
> Triage:
> Hi Peipei,
> We change 163 to use IMAP per bug 951076. However it seems cause this bug.
> Can you try set 163 to use POP3 and check if there any regression issue?
> Thanks!

Josh,

1. Before we change 163 to use IMAP, we use ActiveSync for 163 which does not work at all(user could not load email use 163 activesync server). 

2. This should be a bug in Browser (not related to email). Because:
   1) Directly opening this shorten link in Browser will also have this problem.
   2) Opening the same link in Firefox for Android has this problem as well.

3. It might not be a good choice to set 163 to pop3. Because:
   1) All operations user perform on client side will not be synced to pop3 server, such as deleting/move/flag/draft emails.
   2) 163 pop3 does not handle junk mails well.
   3) Using IMAP, user can still download files successfully, if they use the download button in the attachment section.

4. I would suggest we leave this bug open. For woodduck, this should not be a blocker. since the user impact is small. 
   1) Most of 163 email users are in china. And China is not the target market for woodduck.
   2) User can download attachment successfully using the download button in the attachment section.

Comment 19

[Josh Cheng [:josh]]

Per comment 18. Since China is not first wave launch region and there is still workaround. We will revisit this bug when launching China market.

Comment 20

[Andrew Sutherland [:asuth] (he/him)]

(In reply to Peipei Cheng from comment #15)
> Andrew, 
> I checked the email configuration on Flame 2.0. It turns out that the we use
> IMAP+SMTP for 163 mail account. 
> But I was unable to check the behavior on 2.1/2.1 because I always get a
> network error problem while setting up 163 mail account. I tried both
> automatically setup and manual setup.

The log shows:
- An attempt to create a gmail account with the wrong credentials
- A successful attempt to create a 163.com account
- syncing (of the 163.com account?)
- A successful attempt to create a 163.com account (the same account?)
- syncing of the same account as before as evidenced by the timestamps
- (Scary SetCookieString errors)
- A connection timeout during the download of an attachment, followed by an apparent successful download.  In 2.0 we're not very chatty about establishing connections, so it's possible the success is legit.  Direct excerpt:
11-13 11:41:44.695 I/Gecko   ( 1350): WLOG: runOp(do: {"type":"download","longtermId":"3/2","lifecycle":"do","localStatus":"done","serverStatus":"doing","tryCount":0,"humanOp":"download","messageSuid":"3/0/2","mess)
11-13 11:41:54.705 I/Gecko   ( 1350): WERR: Connect error: unresponsive-server formal: Error: Connection timed out on imap.163.com 993
11-13 11:41:59.805 I/Gecko   ( 1350): WLOG: Downloaded 26966 bytes of attachment data.
11-13 11:41:59.805 I/GeckoDump( 1350): DeviceStorage: process save
11-13 11:41:59.845 I/Gecko   ( 1350): WWAR: failed to save attachment to sdcard send MMS from SIM1 to SIM2 and download.png type: image/png
11-13 11:41:59.845 I/GeckoDump( 1350): DeviceStorage: process save
11-13 11:41:59.885 I/Gecko   ( 1350): WLOG: saved attachment to sdcard /sdcard/send MMS from SIM1 to SIM2 and download-1415850119855.png type: image/png
11-13 11:41:59.945 I/Gecko   ( 1350): WLOG: runOp_end(do: {"type":"download","longtermId":"3/2","lifecycle":"do","localStatus":"done","serverStatus":"doing","tryCount":0,"humanOp":"download","messageSuid":"3/0/2","mess)


Since this is v2.0 and you said it worked, this all isn't particularly surprising.  The question would be what is going wrong on v2.1/v2.2.  We are tracking that on bug 1093467 and have test credentials to help us do that.  So let's not deal with this further.

Comment 21

[Andrew Sutherland [:asuth] (he/him)]

Summarizing my understanding based on the accumulated facts:

- 163.com/126.com will include download/preview hyperlinks inside the message content when using IMAP.  This does not happen for POP3 but *the email app does not want to use POP3 for these servers or any servers for a wide variety of UX, feature, and scalability reasons*.  I have filed bug 1099155 on understanding what is going on with the link generation and whether we can avoid it.
- The browser in its interaction with the download manager does not do the right thing for download links.  See comment 7.

So browser/download manager need to be fixed and we can stop dealing with the email app proper on this bug.

Comment 22

[Aus Lacroix [:aus]]

This looks like it's an underlying gecko issue following the the correct route through the sea of redirects to the file. I'm happy to take a look at this but unfortunately the shortened link provided to reproduce is no longer active and I'm unable to sign up for an email account with this provider because I only have a North American phone number at my disposal.

If someone would be so kind to get an updated link posted here I'll get right on it.

Comment 23

[Pi Wei Cheng [:piwei] (inactive)]

(In reply to Gregor Wagner [:gwagner] from comment #13)
> We should also get a reverse-regression window to see what patch fixed it.

Per comment 18.5, this issue is affected in all branches therefore a window is not possible.

Comment 24

[Josh Cheng [:josh]]

Hi Peipei,
Is it possible for you to provide an dummy account with a shorten link per comment 22 from Ghislain?

Comment 25

[Peipei Cheng (needinfo if you need my action)]

(In reply to Ghislain Aus Lacroix [:aus] from comment #22)
> This looks like it's an underlying gecko issue following the the correct
> route through the sea of redirects to the file. I'm happy to take a look at
> this but unfortunately the shortened link provided to reproduce is no longer
> active and I'm unable to sign up for an email account with this provider
> because I only have a North American phone number at my disposal.
> 
> If someone would be so kind to get an updated link posted here I'll get
> right on it.

Here is the new link: http://u.163.com/t/rFDusA

Please let me know if there is any problem.

Comment 26

[Aus Lacroix [:aus]]

The new link reproduces the issue, thanks!

Comment 27

[Nancy]

(In reply to Ghislain Aus Lacroix [:aus] from comment #22)
> This looks like it's an underlying gecko issue following the the correct
> route through the sea of redirects to the file. I'm happy to take a look at
> this but unfortunately the shortened link provided to reproduce is no longer
> active and I'm unable to sign up for an email account with this provider
> because I only have a North American phone number at my disposal.
> 
> If someone would be so kind to get an updated link posted here I'll get
> right on it.

Hi,Aus,

This bug can reproduce on v2.0.
If we changed network status after downloading start, the downloading could complete. For example, disable wifi, and then enable wifi.

Comment 28

[Nancy]

(In reply to Ghislain Aus Lacroix [:aus] from comment #26)
> The new link reproduces the issue, thanks!

Hi, Aus,

Do you have any update?

Thanks

Comment 29

[Aus Lacroix [:aus]]

Not much to report here as this particular issue is on hold for now. I should have more cycles to continue in the coming week.

Comment 30

[[e-mail]]

(In reply to Ghislain Aus Lacroix [:aus] from comment #29)
> Not much to report here as this particular issue is on hold for now. I
> should have more cycles to continue in the coming week.

Hi, Aus,

Do you have any update ?

Thanks

Comment 32

[Wesly Huang (EPM)]

link all Fire C (codename: Sora) bugs to a meta one.

Comment 33

[BMO Automation]

Firefox OS is not being worked on