949498 - [User Story] Email - Move Message to Outbox on Send

Status: VERIFIED FIXED

(Firefox OS Graveyard :: Gaia::E-Mail, defect)

Description

[Adam Rogers (:arog)]

User Story:

As a user, I want emails that I 'send' to be moved to the outbox and sent. This will help me manage any queue of emails I am sending and provides visiblity and managment of any emails I choose to send while I am out of coverage.

Acceptance Criteria:

1. After composing an email when the user clicks send, the email will be moved to the oubox
2. If the email was orriginally in the drafts folder it will be removed from that folder when sent
3. If there is a current data connection emails will be sent immedatly from the outbox
4. If there is not a current data connection emails will be stored in the outbox until such time as they can be sent - see sending requirements
5. Once the email is successfully sent it is remove from the outbox
6. Errors should be handled as per UX guidelines

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

From our e-mail UX review at the Taipei work week I have action items to list user-correctable error scenarios so that Juwei can address them.  My notes can be found at https://etherpad.mozilla.org/fxos-productivity and searching on "E-mail Review UX Notes".

Note that we broadly have 3 types of errors:

1) Transient failures where the solution is to wait and try again at some point in the future:
- Wait for the network to come back (we're offline / the user's network is down / etc.)
- Wait for the user to rectify an account-level problem (the user's password is bad / gmail thinks the user is an evil hacker because they are using a new IP address)

2) Fatal errors related to the message that require user action

3) Suspicious transient failures that seem to keep happening when trying to send the message.  If the server appears to keep hanging up on us during the actual transmission of the message or keeps returning transient errors (ex: SMTP 4xx codes), we simply have to give up after some number of errors to avoid permanent breakage or generating ridiculous data usage.  For example, Postfix's "reject_unverified_sender" default configuration could return 450 error codes until the end of time based on default settings when enabled.

For our handling I think we need treat type 3 errors as type 2 when we eventually give up on them, so I'm including them below.

The authoritative list of enhanced status codes, for reference, can be found here:
https://www.iana.org/assignments/smtp-enhanced-status-codes/smtp-enhanced-status-codes.xhtml

== User correctable errors

- Invalid recipients (DSN: X.1.X).  In the SMTP protocol what happens is we tell the server all the e-mail addresses we'd like the message to go to.  For each of those addresses the server can indicate success or failure.  In general, we expect failures to occur if there is an obvious syntactic problem with the address (ex: "bob@" is not a valid e-mail address) or if the domain in question cannot be resolved and the server performs that checking up-front (this could occur if there is a typo in the domain).

A smart server could also reject addresses for a complicated set of reasons.  See postfix's documentation at http://www.postfix.org/postconf.5.html#smtpd_recipient_restrictions for examples.  A server may provide a string explaining why it is rejecting an address, but it is rather likely to be unlocalized.

We currently do not perform any validation of the mail addresses before trying to send.  Bug 914004 tracks detecting egregious syntactic failures at entry time.  We could also potentially validate domains if we had better platform support (we don't.)

One odd thing in our behaviour right now that is an artifact of our SMTP library is that we will send the message if any of the recipient addresses work.  We should probably change this behaviour.  Very explicitly, if you list "foo@, bar@, and [e-mail]", we will still succeed in the send.

The easiest string we could use is "One or more of the e-mail addresses you entered is not valid, please correct them below:" at the top.  Unfortunately, our current UI for interacting with e-mail addresses is not well tailored to having more than one error.  Our textbox only ever expected to have one e-mail address in it at a time; when the user hits space or comma we convert that into an email address.  Clicking on the bubble does not turn it back into an e-mail address.

The best approach I can come up with this would be for us to display a visual indication on the bubbles that the e-mail address has been determined to be invalid, in general adding an 'edit' button to the bubble menu (currently we just have 'delete' and some options for contacts perhaps) which turns it back into a text-box.  By default we'd make clicking on an erroneous e-mail address turn it back into an editable version.  Visually we'd need something more than coloring for colourblind people; either strike-through or adding a warning icon/angry exclaimation point, etc.

- Disallowed Recipient (DSN: X.7.1, X.7.2).  The user isn't allowed to send a message to the given user / mailing list.  This is likely to be a bounce.  I expect we'd want to handle this like invalid recipients in the UI, just using a different string.  A possible string would be: "The server indicated you're not allowed to send messages to the given e-mail address(es).  Please check for typos."

- Message too big (SMTP status 552, DSN: X.2.3, X.3.4, X.7.16).  We currently enforce a rather low limit on total attachment sizes (5MiB), and usually messages that are too large will be reported as a bounce notification e-mail, it is conceivable that the server could tell us this during sending.


- User's mailbox is full / other mailbox problem. (DSN: X.2.0/X.2.1/X.2.2/X.2.4/X.2.*).  The recipient is okay, but there is something wrong with the recipient's mailbox.  This is most likely to happen as a bounce (which our app will not process), but a reasonable string might be something like: "There is a problem with the recipient's mailbox, if you have another e-mail address for them, try that one instead."  We could do the same thing as an invalid recipient for allowing editing.  We could have a specific message for the user's mailbox is full (X.2.2) and one for other more nebulous mailbox problems.


- Weird problem with the message.  This could be weird security reasons (DSN: X.7.0, X.7.3, X.7.4, X.7.7, others.) or maybe other error codes we don't handle.  This might cover cases where the server thinks the user is a spammer, but those would usually manifest as bounces or account-level problems.  A cop-out message like "The server reported an unknown problem with your message.  Please contact your server administrator and tell them it reported status X.X.X."


== We gave up fatal errors

- We tried to send a number of times but kept getting transient errors reported by the server.  Something like: "We tried to send this message several times, but the server kept reporting transient problems.  There may be a problem with your server.  Your options are to manually try and re-send this later or contact your server support to see if they have any ideas.  Status codes reported: X.X.X, X.X.X"

- We tried to send a number of times but connection loss/weird stuff kept happening AND the message was fairly big (let's say over 500 KiB).  Something like: "We tried to send this large message several times, but encountered network issues.  Manually try to re-send later when you are on a more stable network connection, or if you believe the network is okay, contact your server operator for support."

- We tried to send a number of times but connection loss/weird stuff kept happening and the message was small.  Something like: "We tried to send this message several times but encountered difficulties.  You should manually try again later if you think you are experiencing network troubles or discard this message."

Comment 2

[Juwei Huang[:juwei]]

Thank you Andrew for such profoundly explanation :)
I've consolidated it as 5 different toasts. For those errors which are possible to fix by user themselves, I categorise them as separated toast. For other errors that nothing users can do, I categorise as one toast.

Here's my suggestion:
1. Message send successfully: "Email sent"

2. Network disconnect: "Network unavailable. Email saved to outbox"

3. Message disallowed to sent to recipients / recipient mail box problem: " Recipient service error. Email saved to outbox"

4. Message problem/ all the fatal error: "Error occurred. Email saved to outbox"

5. As for invalid email address, my proposal would be putting the error message on the compose page:
* Turn all the invalid email address to red hint (currently we only highlight those are not with "@", could we also highlight the invalid one?)
* If users do send the message with invalid email address, prompt a confirm window "The message could not be delivered to one or more recipients. Send it anyway?" "Cancel / Send" 

6. Another case is the message too large. I am thinking that block the oversize file before send the mail out, which mean when the added attachment exceeds the maximum size of the total file , an error UI will pop out to forbid users to upload file.
The UI could be "The attachments exceed total file size limit" "Cancel"
**Could we display exactly the number of size limitation on error UI? such as 20MB for gmail or 44MB for zimbra

I put each string mapping to the error cases you provide blow, please check my comment inline:


> - Invalid recipients (DSN: X.1.X).  In the SMTP protocol what happens is we
> tell the server all the e-mail addresses we'd like the message to go to. 
> For each of those addresses the server can indicate success or failure.  In
> general, we expect failures to occur if there is an obvious syntactic
> problem with the address (ex: "bob@" is not a valid e-mail address) or if
> the domain in question cannot be resolved and the server performs that
> checking up-front (this could occur if there is a typo in the domain).

Show an error UI before send message out, please refer to 5.


> - Disallowed Recipient (DSN: X.7.1, X.7.2).  The user isn't allowed to send
> a message to the given user / mailing list.  This is likely to be a bounce. 
> I expect we'd want to handle this like invalid recipients in the UI, just
> using a different string.  A possible string would be: "The server indicated
> you're not allowed to send messages to the given e-mail address(es).  Please
> check for typos."

Toast "Recipient service error. Email saved to outbox"


> - Message too big (SMTP status 552, DSN: X.2.3, X.3.4, X.7.16).  We
> currently enforce a rather low limit on total attachment sizes (5MiB), and
> usually messages that are too large will be reported as a bounce
> notification e-mail, it is conceivable that the server could tell us this
> during sending.

Show an error UI before send message out, please refer to 6.

 
> - User's mailbox is full / other mailbox problem. (DSN:
> X.2.0/X.2.1/X.2.2/X.2.4/X.2.*).  The recipient is okay, but there is
> something wrong with the recipient's mailbox.  This is most likely to happen
> as a bounce (which our app will not process), but a reasonable string might
> be something like: "There is a problem with the recipient's mailbox, if you
> have another e-mail address for them, try that one instead."  We could do
> the same thing as an invalid recipient for allowing editing.  We could have
> a specific message for the user's mailbox is full (X.2.2) and one for other
> more nebulous mailbox problems.

Toast "Recipient service error. Email saved to outbox"
 

> - Weird problem with the message.  This could be weird security reasons
> (DSN: X.7.0, X.7.3, X.7.4, X.7.7, others.) or maybe other error codes we
> don't handle.  This might cover cases where the server thinks the user is a
> spammer, but those would usually manifest as bounces or account-level
> problems.  A cop-out message like "The server reported an unknown problem
> with your message.  Please contact your server administrator and tell them
> it reported status X.X.X."
> 
Toast "Error occurred. Email saved to outbox"

 
> == We gave up fatal errors
> 
Toast "Error occurred. Email saved to outbox"


I think that's all error cases we have right now. Let me know if there's any concern:)

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

(In reply to Juwei Huang from comment #2)
> Thank you Andrew for such profoundly explanation :)
> I've consolidated it as 5 different toasts. For those errors which are
> possible to fix by user themselves, I categorise them as separated toast.
> For other errors that nothing users can do, I categorise as one toast.

The toasts work for me!  Also, I did a lot of that write-up for my own notes and anyone who has to/wants to review the patch.  If anyone thinks what I write is dry and long, they should try the RFC's! :)

From our UX review, we also decided that we would show an error message at the top of the compose page when the user views the draft.  This message would remind the user why the send failed.  We can probably use longer strings for that, although we don't have to.  We can keep things at the same granularity for the toast, but in that case for support reasons we probably want to list the error code either in the message or in a similar fashion to what we do when account setup fails.  (We put an internal error code in the lower right of the message area in brackets.  Unfortunately our codes have some English words in them and at least QA seems to interpret this as a failure to translate something.  For mail sending, we could keep the codes numeric.  Like the send fail what we want to display is "5.2.3".  So in brackets we'd have "[5.2.3]" somewhere, or we could write it out like "Error code: 5.2.3"

If you're okay with my initial strings, we can also start from there.  We can map the errors however we like.  In practice, we may never see a lot of these error codes because the way e-mail transmission works has 2 stages:
1) Our app talks to our user's e-mail server.
2) The user's e-mail server (ex: gmail.com) talks to the recipient's e-mail server.

Most of these failures will occur in the second stage, in which case a bounce e-mail will be generated and which we currently do not try to understand.  (In fact, I don't know of any clients that try to understand them.)  However, they could still occur if both users are using the same server; like if you're sending email from mozilla.com to mozilla.com.

 
> 6. Another case is the message too large. I am thinking that block the
> oversize file before send the mail out, which mean when the added attachment
> exceeds the maximum size of the total file , an error UI will pop out to
> forbid users to upload file.
> The UI could be "The attachments exceed total file size limit" "Cancel"
> **Could we display exactly the number of size limitation on error UI? such
> as 20MB for gmail or 44MB for zimbra

We currently already do this in the compose UI.  Our current limit is 5120000 bytes which is a smidge under 5 MiB.  This was a stop-gap measure from before bug 871897 landed to stop us from crashing on 256MiB RAM devices.  We haven't loosened it yet because we/I never got to the wake-lock issues so it seemed better to not up the limit.  There are a set of bugs on that right now; start from bug 907028 if interested.

Our strings look like this right now:
composer-attachment-large=Attachment too large
composer-attachments-large=Attachments too large
compose-attchment-size-exceeded=The selected attachment is too large to send with this message.
compose-attchments-size-exceeded=The selected attachments are too large to send with this message. Try selecting fewer files.


In general, we can't know what the limits are on the recipient's server.  We could try and build a database and pre-populate it.  We could also try and learn from failures if the server tells us a size limit in its error message.  There is no standard for a size to be reported, but we could look for numbers in the error string that look like they're in the right ballpark, like in the megabytes and roundish so it's  not a phone number or something.  However, that's a lot of work.

For this I'd probably suggest:
- We include the string for now.
- We decide on our new send-size limit that we will increase to once we fix those other problems.  I'll put it and our rationale in a comment in the source with these changes.  It looks like gmail has upped their limit to 25 MiB.  Actually, from http://www.yetesoft.com/free-email-marketing-resources/email-sending-limit/ it looks like all of gmail/hotmail/yahoo/AOL are at 25 MiB, so maybe that would be the appropriate limit.  (Or maybe a little below that if they are including the body size in there.  Or if we need to include the bloat caused by encoding the attachment in an email message.  In that case 20 MiB would be better.)

If you want us to update our strings to explain what the actual limit is and how many bytes we're at without the new attachments, we could probably do that too.

Comment 4

[Juwei Huang[:juwei]]

> We currently already do this in the compose UI.  Our current limit is
> 5120000 bytes which is a smidge under 5 MiB.  This was a stop-gap measure
> from before bug 871897 landed to stop us from crashing on 256MiB RAM
> devices.  We haven't loosened it yet because we/I never got to the wake-lock
> issues so it seemed better to not up the limit.  There are a set of bugs on
> that right now; start from bug 907028 if interested.
> 
> Our strings look like this right now:
> composer-attachment-large=Attachment too large
> composer-attachments-large=Attachments too large
> compose-attchment-size-exceeded=The selected attachment is too large to send
> with this message.
> compose-attchments-size-exceeded=The selected attachments are too large to
> send with this message. Try selecting fewer files.

Is it mean that our mozilla message is limit to 5MB?  And the string is pop out on compose page?

> 
> For this I'd probably suggest:
> - We include the string for now.
> - We decide on our new send-size limit that we will increase to once we fix
> those other problems.  I'll put it and our rationale in a comment in the
> source with these changes.  It looks like gmail has upped their limit to 25
> MiB.  Actually, from
> http://www.yetesoft.com/free-email-marketing-resources/email-sending-limit/
> it looks like all of gmail/hotmail/yahoo/AOL are at 25 MiB, so maybe that
> would be the appropriate limit.  (Or maybe a little below that if they are
> including the body size in there.  Or if we need to include the bloat caused
> by encoding the attachment in an email message.  In that case 20 MiB would
> be better.)

Pre-populate limit size seems like a good idea. How about that's go for this:

(1) In terms of Gmail/Hotmail/Yahoo/AOL, pre-defined limits at 25 MB and forbid users to add oversize file. A toast can be shown on the compose interface.

(2) For other services,  let users send out the mail (to the outbox) and pop out a toast to indicate that the mail has not been sent due to oversize attachments.

Although it seems like we got 2 different flows to achieve the goal, however I believe (1) will cover most of the scenarios. 
The only thing I concern about is  what if gmail/hotmail/yahoo/AOL change their size limit? could we know about that immediately?

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

(In reply to Juwei Huang from comment #4)
> Is it mean that our mozilla message is limit to 5MB?  And the string is pop
> out on compose page?

Yes.  If you try and add 1 or more attachments and it pushes over the limit, we will refuse to add it and show those strings as a popup.
 
> (2) For other services,  let users send out the mail (to the outbox) and pop
> out a toast to indicate that the mail has not been sent due to oversize
> attachments.
> 
> Although it seems like we got 2 different flows to achieve the goal, however
> I believe (1) will cover most of the scenarios. 

As I read this, it sounds like you are suggesting:
- Set our limit to 25 MiB.
- If we have a problem sending, do our standard error handling flow, but do have an appropriate error string.

And we don't actually have any database/mapping based on domain.

This sounds very reasonable to me, especially since I believe that gmail/hotmail/yahoo/AOL have a sufficient market share that it effectively compels all other providers to match those limits.  The exception would be horrible providers whose limits are probably ridiculously low and in which case it's just up to the user to learn from the bounce messages (which are out of scope for us to process for a loooong time.)


> The only thing I concern about is  what if gmail/hotmail/yahoo/AOL change
> their size limit? could we know about that immediately?

It's unlikely they'll lower the values; they might raise them.  We could periodically check a Mozilla-controlled server to see if the limit went up, but that is a lot of work on our part and has potential privacy and security implications we don't need/want to deal with.

Comment 6

[Juwei Huang[:juwei]]

Agree, let's go for the workflow above.
As for limit value, I think we could just stay on 25MB as a general value right now. 
If there's any future planing for syncing message size limit, then we could have more discuss for it at that time.

Comment 7

[Juwei Huang[:juwei]]

Spec attached to the link: https://bug921050.bugzilla.mozilla.org/attachment.cgi?id=8392059

Comment 8

[Marcus Cavanaugh [:mcav]]

Resolved in bug 921050.

Comment 9

[Andrew Sutherland [:asuth] (he/him)]

We didn't address the attachment size issue in this fix, so I've spun off bug 1040861 for that.

Comment 10

[Jordan de Geus(JordanD)]

Verified Fixed - When users send an email, the message switches from the drafts folder to the outbox until it is sent. If users lack a Internet connection, emails will be in a pending state until the device detects a connection which then they start to send.

Flame 2.1

Environmental Variables:
Device: Flame 2.1 (319mb)
Build ID: 20140905000202
Gaia: 95e9b099aa89ded133e44014dd40b19dc0193c01
Gecko: 92a6bbdfd945
Version: 34.0a2
Firmware Version: v123
User Agent: Mozilla/5.0 (Mobile; rv:34.0) Gecko/34.0 Firefox/34.0

Comment 11

[KTucker [:KTucker][Inactive 3/4/2016]]

Please ignore comment 10. We cannot verify this issue at this time. There is a known issue that fails the sending requirements mentioned in step 4. Please see bug 1058443

Comment 12

[KTucker [:KTucker][Inactive 3/4/2016]]

Removing bug 1058443 since I was informed that this is not a blocker. Cannot verify this as fixed though because emails that are in the process of being sent do not show in the Outbox. They only show in the Outbox if an error occurs. This has been written up here:

bug 1065167

This is stated in this user story and shown in the design document on pages 9 and 10: 

https://bug921050.bugzilla.mozilla.org/attachment.cgi?id=8392059