805834 - [email/IMAP][l10n] localize special folder names like INBOX, Sent, Trash, like Thunderbird does

Status: VERIFIED FIXED

(Firefox OS Graveyard :: Gaia::E-Mail, defect, P1)

Description

[Andrew Sutherland [:asuth] (he/him)]

Some mail servers use unlocalized names for all folders, even when not specifically required by the standard (ex: INBOX must be INBOX).  For example, a yahoo.fr account I created has the following mapping (accents elided):
- Inbox: Boite de reception
- Drafts: Brouillons
- Sent: Envoyes
- Spam: Spams
- Trash: Corbeille

Yahoo's IMAP server does not support the somewhat recent (proto-)standards XLIST or SPECIAL-USE, so this all makes sense since otherwise it would be a train wreck of e-mail clients having no idea what's going on.

Thunderbird handles this by maintaining a "prettyName" that's independent of the actual folder name for server communication purposes.  We just maintain the name independent of the folder path.

Its logic and set of l10n mappings look like so:
https://mxr.mozilla.org/comm-central/source/mailnews/base/util/nsMsgDBFolder.cpp#3386

There is no way to get these names from the server via IMAP, so we will need to maintain the mappings locally.  In the case of ActiveSync, this is not a problem, although we have bug 804977 on an encoding issue in that case.

Comment 1

[David Scravaglieri [:scravag]]

This is a feature request

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

(In reply to dscravaglieri from comment #1)
> This is a feature request

Are you saying we shouldn't implement this, or just that we wouldn't block on it if we failed to implement it on time?

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

To elaborate for the blocking request, the problem that will be seen by IMAP users is that their folder names will be "INBOX", "Sent", "Trash", etc. *regardless of their local language*.  So as a French user of yahoo.fr, although my webmail will show "Boite de reception", my Firefox OS phone will just show "INBOX".  This will not happen on ActiveSync accounts like hotmail and gmail.

Comment 4

[Mike Connor [:mconnor]]

Blocking+ on the principle that this will break hotmail, which is a major provider in Brazil.  P1 because this has l10n and feature implications, so it needs to happen now.

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

Jim, localization strategy question for you.  Backend vs. front-end, and where in the back-end if in the back-end.

Right now we cram all the localized strings into the back-end asynchronously from the front-end after we have received indication that the strings have loaded ('localized' event) and that the mail universe has been setup and a MailAPI instance is ready ('mailapi' event).  This works okay because they are all composition strings and the sequence of things is such that the back-end will see the strings before it sees any composition requests.

For localizing the folder names, there are two main options:
1) Do it in the back-end when we first learn about the folder.  This is akin to how SetPrettyName works in Thunderbird.

2) Do it in the front-end quasi-on-demand; probably when we instantiate the MailFolder from the wire protocol.


For #1, the asynchronous thing will no longer work (cleanly) because it's possible for syncFolderList to run before the front-end tells the back-end the strings.  We could perform a fix-up pass when the strings arrive in the back-end, but that seems extra hacky.  So I would propose we change the setup so that the MailUniverse learns the localized strings as part of its startup process and does not complete said startup process until it has them.  (We can still issue the database lookups without blocking on the strings.)

For simplicity from the perspective of the localizers, I would propose leaving everything in the same properties file and just putting the back-end strings in a clearly delimited area.  The loading of the strings would happen in gaia code (not gelam code).  So when we move to have the back-end in its own process/window-context, it's not like gelam would need to learn any new tricks.  That window would just load the same set of strings and still pick out the ones it cares about.  We might try and get mozl10n to dump its string table after initialization for memory usage purposes.


My plan thus far has been #1, back-end since we already have a bunch of strings.  #2 is neat in that it lets us change languages on the fly, but I don't see that actually being a useful feature considering that ActiveSync, for example, will only ever announce things in a single language as far as we are concerned.

Comment 6

[Jim Porter (:squib)]

(In reply to Mike Connor [:mconnor] from comment #4)
> Blocking+ on the principle that this will break hotmail, which is a major
> provider in Brazil.  P1 because this has l10n and feature implications, so
> it needs to happen now.

Note that this *won't* break Hotmail (as far as I know), since Hotmail uses ActiveSync. I believe ActiveSync always sends localized folder names (at least, it does for Gmail; I haven't tested Hotmail).

(In reply to Andrew Sutherland (:asuth) from comment #5)
> Jim, localization strategy question for you.  Backend vs. front-end, and
> where in the back-end if in the back-end.

I don't have any strong opinions either way. Front-end seems a little saner, but then we already pass strings to the back-end anyway...

Comment 7

[Alex Keybl [:akeybl]]

We're marking this bug with the C1 milestone since it follows the criteria of "unfinished feature work" (see https://etherpad.mozilla.org/b2g-convergence-schedule).

If this work is not finished by Nov19, this bug will need an exception and will be called out at the upcoming Exec Review.

Comment 8

[Cody Brocious [:Daeken]]

Attachment: MailAPI patch for folder localization

This patch to MailAPI -- in conjunction with https://github.com/asutherland/gaia/commit/8812f4628ea2a38d4dbbb0efc631d0a9f6bf40d5 -- enables folder name localization based on type.  Outside of the lack of a test, it seems to be pretty solid.  The two questions I have are:
1) Should this actually be done in MailAPI or in the email app?
2) Should this do the matching based solely on folder type, or should it match only folders named a specific way as well, as Thunderbird seems to do?  Not sure how specific that route will be to a given service.

Comment 9

[Andrew Sutherland [:asuth] (he/him)]

(In reply to Cody Brocious [:Daeken] from comment #8)
> The two questions I have are:
> 1) Should this actually be done in MailAPI or in the email app?

MailAPI is the right place for this because all e-mail UIs that consume the backend will want properly localized names.

> 2) Should this do the matching based solely on folder type, or should it
> match only folders named a specific way as well, as Thunderbird seems to do?
> Not sure how specific that route will be to a given service.

It needs to also check that the name is (case-insensitively) equivalent to a specific hard-coded string, just like Thunderbird does in https://mxr.mozilla.org/comm-central/source/mailnews/base/util/nsMsgDBFolder.cpp#3386

The rationale behind this is that the IMAP LIST command is unable to convey fancy meta-data so names per convention must be used.  But XLIST is able to provide the meta-data so localized strings can be provided, and in that case there is no need for us to override already localized names.  This is especially true because fancy server implementations could have more than one folder with a given type, and it would be wrong of us to name them all the same thing.

Like for the similar (but different) bug 804977, we can't really do a unit test for this right now without an IMAP fake-server, so no unit test is required.  I can test with my yahoo.fr account, but if you could also get a yahoo.fr account too and test it out before putting up the revised patch, I'd appreciate it!  (Make sure not to let them divert you to a yahoo.com e-mail.  I found signing up with Chrome and its automatic translation support to be the easiest way.)

Comment 10

[Andrew Sutherland [:asuth] (he/him)]

Comment on attachment 680655 [details] [diff] [review]
MailAPI patch for folder localization

(clearing review request, please request with the new patch)

Comment 11

[Cody Brocious [:Daeken]]

Attachment: Finalized localization patch

This patch checks both the folder type and name, which conveniently is just a lower case version of the type.

Comment 12

[Andrew Sutherland [:asuth] (he/him)]

r=asuth with some tweaks to round out the yahoo.fr cases (landed).  IMAP exposes spam as 'Bulk Mail' for unclear reasons, and their drafts folder is called 'Draft', so it's not a 1:1.  (My mapping in comment 1 appears to have been of the web UI; their strings for their English web interface are more sane than their IMAP folder names, sadly.)

For those attempting to duplicate, beware that currently our account creation process fares poorly on yahoo.fr entries because of bad behaviour by https://yahoo.fr/.  I filed bug 812969 for that.  I was then seeing subsequent ISPDB flakiness, but there could be some local network issues involved in that, as XHR onerror was firing which is an unusual one.

gelam pull request (merged):
https://github.com/mozilla-b2g/gaia-email-libs-and-more/pull/81

gaia pull request (merged):
https://github.com/mozilla-b2g/gaia/pull/6482

Comment 13

[Misty Byrd]

This issue is fixed for device Unagi; build #20130113070202v.1
Yahoo IMAP, outlook activeSync,  special folders are changed with localization on device. (change language on device)

Comment 14

[Tracy Walker [:tracy]]

we'll need to add a test case to moztrap for special chars in folder names to generalize this.