804909 - [email/activesync] Reply threading broken for ActiveSync in replies we generate; need to fetch message-id header, references headers or rely on server to generate the reply

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::E-Mail, defect)

Description

[Andrew Sutherland [:asuth] (he/him)]

Per https://github.com/mozilla-b2g/gaia-email-libs-and-more/issues/61 (which is more about noticing this problem while solving a different id problem), we don't currently retrieve the "message-id" header for ActiveSync-fetched e-mails or the "references" headers.  This means that when we reply to the message, we are going to produce messages that will fail to thread in e-mail clients that rely on proper observance of internet standards.  Such as Mozilla Thunderbird.

We could also address this problem by just having the ActiveSync server formulate the reply message more directly for us.  That would be more work, but could be much more efficient for the client since it wouldn't have to tell the server things it already knows.

Requesting blocking-basecamp because this is a particularly egregious violation of internet standards on our part and it will make our messages frustrating for recipients that aren't just using gmail.

Comment 1

[Jim Porter (:squib)]

I think we want to use SmartReply (and SmartForward) here, since as far as I know, there's no easy way to get the References header without getting the whole MIME message.

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

We are assuming SmartReply and SmartForward will actually do the right thing with the References header! ;)

Since I think we are ending up doing HTML only, this may work out easier than if we supported text.  Namely, since for HTML we don't let the user edit the source message, it's trivial for us to just not insert the HTML message and let smartreply and smartforward do their context-providing thing.

The only problem I foresee is that to implement a feature like Thunderbird has where you can change the account you are replying from, one would need to fall back to the MIME case.  But that's a feature we don't have now, and arguably the cost/benefit says that the potential efficiency of SmartReply and SmartForward swamp out a non-existent feature that is probably even rarely used in Thunderbird.

Comment 3

[Alex Keybl [:akeybl]]

We're marking this bug with the C1 milestone since it follows the criteria of "unfinished feature work" (see https://etherpad.mozilla.org/b2g-convergence-schedule).

If this work is not finished by Nov19, this bug will need an exception and will be called out at the upcoming Exec Review.

Comment 4

[Jim Porter (:squib)]

The only thing I can think of to make this work would be to try to grab the MIME headers in the background when you hit the Reply button. None of the documentation shows a way to get the References header otherwise, and nothing I've tried works either.

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

Sounds reasonable.  So when we set up the composition context, we trigger a MIMESupport/MIMETruncation request that gets us the first 4k or 8k of headers, cram that into our header parser, and then feed that into the compose context?

Note: feeding it into the compose context is probably a little awkward right now since all the state is maintained on the client side.  But we can enhance the back-end/daemon side to stash the information there too and lookup when it gets the request.

Comment 6

[Jim Porter (:squib)]

Ok, I found some relatively happy news about this: when using ItemOperations Fetch, I can tell it to only send back the body, which gives us only 123 bytes of WBXML overhead. It also looks like we can pass in an offset when fetching the body, so we could fetch 1-2KB at a time until we find the Message-ID header.

Comment 7

[Jim Porter (:squib)]

Hmm... unfortunately, I'm not seeing a way to truncate the MIME response. It only seems to truncate if we use HTML or plain text.

Comment 8

[Jim Porter (:squib)]

I've investigated this some more: replying via the Hotmail web interface inserts the appropriate References header, but there doesn't seem to be any feasible way to do this via ActiveSync. One would expect the SmartReply command to do this for us, since it already modifies the MIME that we send, but it doesn't. Getting the MIME of the original message (and pulling out the Message-ID) is unworkable as well, since this would require us to download the *entire* message, which may be very large, as the attachment data will come along too.

In short, I don't think this is something we can get for C1 (maybe not for v1 final at all), unless someone has a good idea here. This isn't quite as bad as it used to be, since we now use IMAP for Gmail, so the References headers work fine there. It's just Hotmail (and probably Exchange) that has the problem.

For what it's worth, it appears that the Android email app doesn't insert the References header at all, not even in IMAP.

Comment 9

[Andrew Sutherland [:asuth] (he/him)]

Any chance of the Range request header being a viable evil trick?
https://en.wikipedia.org/wiki/List_of_HTTP_header_fields#range-request-header

Comment 10

[Andrew Sutherland [:asuth] (he/him)]

I'm still a little interested in the potential for an evil trick on that, but otherwise I am of the mind that the unfortunate correct trade-off here is to fail to generate the proper threading headers.  Especially for our current target market, data is more precious than threading correctness, and we aren't even conversation aware ourselves.

We should probably modify mailbridge.js in beginCompose so that we leave referencesStr as null so we don't add any references headers and thereby avoid teasing the other e-mail clients.

Comment 11

[Jim Porter (:squib)]

(In reply to Andrew Sutherland (:asuth) from comment #9)
> Any chance of the Range request header being a viable evil trick?
> https://en.wikipedia.org/wiki/List_of_HTTP_header_fields#range-request-header

I tried that already, and it doesn't work.

Comment 12

[Dylan Oliver [:doliver]]

From C1 review team: if we can verify Jim's note in comment #8 that we are offering the same level of functionality as Android, then we will remove this from the blocking list for v1.

Jim, can you post a couple of screenshots to illustrate the user experience here?

Comment 13

[Jim Porter (:squib)]

I'm not sure what kind of screenshot you want. You won't notice the change in the email app itself, as we don't do threading there. However, if you looked in a client like Thunderbird, you'd see replies that came from Gaia as starting a new thread.

Comment 14

[Dylan Oliver [:doliver]]

Screenshots of Thunderbird would do it -- the team is trying to understand the extent of the bug and the experience if we don't fix it.

Comment 15

[Jim Porter (:squib)]

Attachment: What our broken threading looks like in Thunderbird

Here's what it looks like in Thunderbird. The first row is the message sent from Gaia. The others were sent via Hotmail's web interface and have proper threading.

Comment 16

[Dylan Oliver [:doliver]]

And if you send from Android you'll get the same result as the Gaia email? We can live with that.

Comment 17

[Jim Porter (:squib)]

(In reply to Dylan Oliver [:doliver] from comment #16)
> And if you send from Android you'll get the same result as the Gaia email?
> We can live with that.

From looking at the code, this appears to be the case. However, I don't have an android phone, so I can't test it out to be 100% sure.

Comment 18

[Dylan Oliver [:doliver]]

Jim and I ran a test to verify that the stock Android email app also fails to properly thread replies. Based on that and yesterday's discussion in the C1 Review meeting, changing this to blocking-.

Comment 19

[Jim Porter (:squib)]

I'm not working on this, and indeed I'm not sure if this is even solvable on our end.

Comment 20

[Andrew Sutherland [:asuth] (he/him)]

Potentially contradictory update.

Comment 7 indicates that some version of hotmail's activesync seemed to ignore Truncation requests unless applied to HTML/text.  However these examples for TruncationSize under BodyPreference indicate that it's honored in at least some version of ActiveSync for some server.
- request: https://msdn.microsoft.com/en-us/library/ee237389%28v=exchg.80%29.aspx
- response: https://msdn.microsoft.com/en-us/library/ee220018%28v=exchg.80%29.aspx

So in theory MIME fetching with truncation could work using:
- Truncation (https://msdn.microsoft.com/en-us/library/dn735765%28v=exchg.80%29.aspx) on ActiveSync 2.5
- TruncationSize under BodyPreference (https://msdn.microsoft.com/en-us/library/ee202701%28v=exchg.80%29.aspx) on ActiveSync 12 onwards.

Note that 14.1 seems to also have TruncationSize under BodyPartPreference (https://msdn.microsoft.com/en-us/library/ff850065%28v=exchg.80%29.aspx) which implies that 14.1 allows the individual body parts to be fetched instead of having to deal with the body as a whole.  I don't think we'll likely specialize for this ever, since most of what we care about are Legacy ActiveSync implementations.

The caveat of course is that the our nominal test server is hotmail.com/outlook.com/live.com for which we actually want to abandon ActiveSync and just use IMAP.  (We also had some hosted exchange accounts, but those also tend to be kept up-to-date versus the out-of-date legacy implementations we expect to be used against.)


Relatedly, in bug 1182955 :mcav has found that there seem to be cases where hotmail refuses to convert our body part to HTML and gives us MIME.  We may end up needing to support consuming some MIME from ActiveSync in which case the pain we go through there could help net us a fix here.  Or if we don't have to do this, we should probably just WONTFIX this.

Comment 21

[BMO Automation]

Firefox OS is not being worked on