Closed Bug 1159939 Opened 9 years ago Closed 7 years ago

[SMS] text/html email-gatewayed MMS messages display as an untitled attachment in Messages app thread (such as those produced by the email app in reply to an MMS message)

Categories

(Firefox OS Graveyard :: Gaia::SMS, defect, P1)

ARM
Gonk (Firefox OS)
defect

Tracking

(tracking-b2g:backlog, b2g-v2.2 affected, b2g-master affected)

RESOLVED WONTFIX
tracking-b2g backlog
Tracking Status
b2g-v2.2 --- affected
b2g-master --- affected

People

(Reporter: bzumwalt, Unassigned)

References

()

Details

(Whiteboard: [3.0-Daily-Testing][sms-most-wanted])

Attachments

(3 files)

Attached file Logcat
Description:
When user sends a text only MMS from Messages app with the recipient set to an email address, any all text reply from that email address to the device shows up in Messages thread as an untitled file. If user tries to open the file they are told that there is no application to open that file type.

This does not occur if user receives an SMS from an email address consisting only of text that is not in reply to an existing thread. This message shows up as text with no attachment as expected.

Repro Steps:
1) Update a Flame to 20150429010205
2) Launch Messages app and tap to compose new message
3) Type some text in body and put an email address in the recipient field before sending message.
4) Send a text only reply back to the phone from the received email.
5) View reply in messages app.

Actual:
Untitled media file is received in thread which user is unable to open.

Expected:
User receives text reply from email.

Environmental Variables:
Device: Flame 3.0
Build ID: 20150429010205
Gaia: 6e35b0948c42a4398b8a5916015de167121683a1
Gecko: 1ad65cbeb2f4
Gonk: b83fc73de7b64594cd74b33e498bf08332b5d87b
Version: 40.0a1 (3.0)
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:40.0) Gecko/40.0 Firefox/40.0

Repro frequency: 3/3, 100%
See attached: Youtube video clip & logcat
Youtube video link: http://youtu.be/cdFXVTMRo7o
Issue DOES reproduce on Flame 2.2

Untitled media file is received in thread reply to emailed SMS which user is unable to open.

Device: Flame 2.2
Build ID: 20150429002501
Gaia: 1b7aa7e60788668ed09abf76022dfa231dbe88d4
Gecko: d38ff4717f39
Gonk: ebad7da532429a6f5efadc00bf6ad8a41288a429
Version: 37.0 (2.2)
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:37.0) Gecko/37.0 Firefox/37.0

Unable to test 2.1 as it does not allow email addresses to be used in Messages app.
QA Whiteboard: [QAnalyst-Triage?]
Flags: needinfo?(ktucker)
Brogan, could you please attach your SMS database so that we can analyze what's inside ?

This is essentially the files /data/local/storage/permanent/chrome/idb/*ssm* (if you zip all the content of /idb/ it's fine too).

I suspect either a carrier issue, either a mail issue. Still we should be able to do something more useful with this. This is also somewhat linked to bug 1099418.

Thanks !
Flags: needinfo?(bzumwalt)
Attached file SSM
An interesting discovery: This issue only occurs when replying using email app on Firefox OS Flame device. If user responds to MMS from computer, the issue does not reproduce and user receives text as expected.


File 1: MMS sent to computer email
File 2: Reply from computer email (Gmail)
File 3: MMS sent to device email
File 4: Reply from device email (Gmail)
Flags: needinfo?(bzumwalt)
Eric, can you weigh in on the severity of this issue?
QA Whiteboard: [QAnalyst-Triage?] → [QAnalyst-Triage+]
Flags: needinfo?(ktucker) → needinfo?(echang)
I'll have a look to the files on Monday.
Flags: needinfo?(felash)
I am still trying to reproduce this, using a sim from CHT, sending MMS to an email is fine, but replying message is not coming into the Messages app at all, need to check with telecom.  
 
But if reproducible only from our own email app, I suggest to block this.
[Blocking Requested - why for this release]: to complete user experience, broken functionality.
blocking-b2g: --- → 2.2?
Flags: needinfo?(echang)
Adding :asuth in Cc as he might shed some light about how the Email app replies to such emails.
Hey Brogan,

sorry but I also need your 226660312ssm.sqlite file :/
Flags: needinfo?(felash) → needinfo?(bzumwalt)
Brogan, also, it would make sense to know if other devices can receive the same MMS you send from Firefox OS Email app?
Interesting!  I reproduced using my AT&T account.

What seems to be happening here is that when the FxOS email app receives the MMS email, it is HTML only.  (It's multipart/related containing a multipart/alternative with *only* a text/html part, which makes the use of multipart/alternative use weird.)

We respond with an HTML email with only a text/html part with the rationale:
- A mail client that sends HTML is going to be capable of processing and displaying HTML.
- Downconversion from text/html to text/plain tends to be hard to get right and frequently ends up looking like gibberish.  For example, Thunderbird's downconversion is now pretty horrible, although I think some of this might be a regression.
- Pretty much the only mail clients that don't natively render/support HTML are those used by users that intentionally use a text-mode UI (ext: mutt) and because of the aforementioned downconversion issue, they're likely to use use something link lynx and its dump mode to get a better fidelity downconversion.
- It's a waste of the user's data and other resources to send a text/plain part that won't be read since we assume the other client will just opt to display the text/html.


I tested on an Android 5.1 device, and noticed the following:

- If the reply includes a multipart/alternative with both a text/html part and a text/plain part (ex: produced by gmail), the Android messages app showed the text/plain part (in its entirety).
- If the reply only includes a text/html part (ex: from the FxOS mail app), the Android message app shows the text/html part.  It seems to properly display <blockquote> styling, so I think it's actually using something capable of rendering HTML.
- In both cases the Android app seems to include the quoted content of the message, which seems wasteful.


So it seems like there are two potential fixes here:

1) It seems like the MMS app needs to be able to display or downconvert from text/html.  Many mail apps hedge their bets and send both text/html and text/plain by default, but they can also be configured to only send text/html in various cases.

2) The email app could alter its behaviour when it is able to identify that the message originated from an MMS gateway.  In the cases where we have access to the mail headers (IMAP, POP3), we can observe the presence of headers like "X-Mmms-Message-Type" and infer MMS.  In the case of ActiveSync, we could use a heuristic like noticing that given an address like "5555555555@mms.att.net" that the "local-part" of "55555555555" is all digits; not sure we can rely on the "MMS" part.  If it's an MMS we could do something like alter our reply behaviour to not bother quoting and thus allow us to just reply in plaintext.  We could probably also inhibit the signature since it also seems non-helpful in a text message stream.


Unless there's something simple the MMS app can do like ask the server to perform downconversion of the MMS to text/plain, it seems like both fixes are enhancement-level fixes that are too significant for the v2.2 branch at this point.  Although I suppose the email app could have a hack-fix variant where the front-end uses the all-digits heuristic to just drop to text/plain at the cost of making it 100% impossible for users to reply to messages from all-digit email addresses with quoting.

I'll file a bug on the email enhancement now.
Oh, hm, actually, there may be another, easier fix for the MMS app by doing it in platform as part of the Web API.  Specifically, the API Thunderbird uses to convert HTML to plaintext should still be available.  (Well, unless it was moved to comm-central.)

The counter-argument is that using the converter in this way may be a different security context from how Thunderbird uses it and the risk profile may be unacceptable depending on a security audit of the underlying converter.  (Thunderbird only performs the transformation on outgoing messages and on message display for cases where the user explicitly has requested downconversion of all messages to text/plain which only happens on message display.)  The risk profile may or may not be similar to Thunderbird's sanitizer mode which operates under somewhat similar operational constraints.  Also, Thunderbird is actually capable of releasing security updates.
I filed some email bugs:
- Bug 1161118 - the email app specializing for MMS gateways
- Bug 1161126 - the email app producing text/plain down-conversions of HTML representations (which will eventually need to happen for HTML composition which is bug 1161124).
Summary: [SMS] When user sends MMS to email the reply to that email shows up as an untitled attachment in Messages app thread → [SMS] text/html email-gatewayed MMS messages display as an untitled attachment in Messages app thread (such as those produced by the email app in reply to an MMS message)
Attached file idb.zip
Flags: needinfo?(bzumwalt)
Thanks Andrew, this is really interesting.

Maybe we could use the very simple DOM parser we already have in [1].

[1] https://github.com/mozilla-b2g/gaia/blob/70077825aab2c7a79611befb40a5fe7e610d5443/apps/sms/views/conversation/js/compose.js#L70
(In reply to Julien Wajsberg [:julienw] (PTO May 1st) from comment #14)
> Maybe we could use the very simple DOM parser we already have in [1].

Note that there are security and privacy considerations (mainly to defeat web-bug type things), so you don't just want to hack an existing serialization mechanism used for user-composed data to be used on attacker-controlled data.

The email app's on-worker stream-based HTML parser/sanitizer could be of some interest since it probably could be adapted to do something like that.  See:
https://github.com/mozilla-b2g/gaia-email-libs-and-more/blob/master/js/htmlchew.js#L569 and its dependency, https://github.com/mozilla-b2g/bleach.js

(Note that this is a special situation since we do not have access to the DOM on the worker.  The non-worker branch of bleach.js is rather simpler.  For such a use-case I think there may be other open source projects that might be fancier.)

It's probably worth consulting with the security team when thinking through any approaches involving processing or displaying HTML.  (:freddyb has worked with the email team on stuff like this before.)
My idea was mainly to extract the text from a text/html message, so that we can add it using textContent, so there should be no security issue with this, and this should fix the direct issue.

However I wonder what happens if we send an image by mail as MMS. We'll have to test more.

Also we'll need the DOM API, so it's likely not a long-term solution anyway, and we should have a look at email's solution :)
Comms triage: This bug is a missing part of a new feature. As it was discovered after the milestone, it seems to risky to have it in 2.2. We should have this fixed in 3.0, though.
blocking-b2g: 2.2? → 3.0+
(In reply to Julien Wajsberg [:julienw] (PTO May 1st) from comment #16)
> My idea was mainly to extract the text from a text/html message, so that we
> can add it using textContent, so there should be no security issue with
> this, and this should fix the direct issue.

The code you linked to assumes the HTML has been parsed into a DOM object via use of innerHTML or document.write or document.implementation.createHTMLDocument().  There are potential security/privacy issues with all of these.  Not insurmountable, but things that seem safe like use of document.implementation.createHTMLDocument() can result in your custom elements being fired (because custom elements registrations are inherited via that mechanism for some reason) and potentially abused, etc.
Comms triage: Not doable in the 2.5 time frame. Moving back to hi-pri backlog.
blocking-b2g: 2.5+ → ---
Priority: -- → P1
Whiteboard: [3.0-Daily-Testing] → [3.0-Daily-Testing][sms-most-wanted]
Mass closing of Gaia::SMS bugs. End of an era :(
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → WONTFIX
Mass closing of Gaia::SMS bugs. End of an era :(
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: