1880043 - Incorrect status messages when downloading messages from Gmail

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Silvia Bonavista i Serrat]

Attachment: gmail-download.png

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36

Steps to reproduce:

I've set up my Gmail account with about 32.000 messages in total.

Actual results:

While downloading mail on a set up localised in Spanish (both TB client and Gmail web UI), I get confusing status updates, see attached screenshots:
Gmail: Descargando mensage 2428 de 5308 en Enviados...
Descargando mensaje 1 de 2 en [Gmail]/Enviados...

Obviously, the "1 of 2" is wrong. While downloading the inbox, I even got a non-localised name:
Gmail: Descargando mensage 1 de 5 en INBOX...
The count was also wrong since the inbox has about 18.000 messages.

This happens for all folders, also for the "All Mail" ("Todos") folder.

Expected results:

Correct and fully localised status updates.

Maybe this is related to the localised Gmail setup.

Comment 1

[gene smith]

I create my gmail account (having about 16k messages) in a test profile that never had my gmail account in it.
I never saw anything like "1 of 5" when downloading folders having thousands of messages.

I even got a non-localised name:
Gmail: Descargando mensage 1 de 5 en INBOX...

I don't think this can be fixed easily (could be wrong). "INBOX" is the "selected folder name" that IMAP returns and IMAP (gmail) is returning "INBOX" and not the translated string for INBOX. So I think all languages will see INBOX here.
Others like "Enviados" and "Todos" are the actual names of the folders returned by IMAP (actually gmail's names they give for "Send" and "All Mail" folders). But imap RFC requires that Inbox be called INBOX (or InboX, case is not important) so that gets encoded in the status string. So all but Inbox can have whatever name the user or the server provider gives them.

I did see one issue and I've seen it before over many years. Sometime I see something like "Downloading 500 of 100". But I think I only saw this today when moving between gmail folders while downloads are in progress and it seem to be that the previous folder's progress is used when moving to another folders. I noticed this especially when "repairing" folders (just to trigger a re-download) and then repairing another folder while the first one is in progress. I've been meaning to look at this for a long time but never have.

Comment 2

[Silvia Bonavista i Serrat]

Attachment: gmail-bandeja-de-entrada.png

Thanks for checking this issue. The localized version of INBOX is "Bandeja de entrada" and that's what shows in the status message when downloading the 18.000+ messages. Only the interspersed incorrect status updates show INBOX. As you can see in the other screenshot, the incorrect status updates use a different folder name with [Gmail]/ at the front, whereas the correct messages don't have that, so there are likely different code paths involved. It's likely that to reproduce the issue you need to a) use a localized version of TB and b) switch your Gmail UI to a different language.

Comment 3

[gene smith]

Ok, so you only see INBOX when you see the the wrong ones like "1 of 5" appear and these are interspersed during message downloads with the "correct" number and the Spanish inbox name?
The [Gmail]/ in front of "Sent" and "All Mail" is expected in status string since they are under the "noselect" folder [Gmail]. Gmail for 115 has just been artificially "flatted" so you don't see [Gmail]/ in the folder list.
Are you seeing [Gmail]/<spanish inbox> or [Gmail]/INBOX ? If so, that would be wrong since gmail inbox is not under [Gmail]/

I'll try a spanish version of TB asap. Do I have to do something at the gmail site too for all the folder to be in Spanish? I.e., you said "switch your Gmail UI to a different language" so I suppose this is at gmail site?

In the email, it mentioned "recreated" the account. Not sure what that means. Do you mean the account was deleted a while back but all the folder files (mbox, etc) were not deleted? If that's what it means, do you have the same problem(s) if you create the account in a fresh profile or in a profile that never had that same gmail account? I don't think the "flattening" affected the file structures, just the TB/gmail folder list presentation.

Comment 4

[Silvia Bonavista i Serrat]

Attachment: gmail-settings.png

(In reply to gene smith from comment #3)

Ok, so you only see INBOX when you see the the wrong ones like "1 of 5" appear and these are interspersed during message downloads with the "correct" number and the Spanish inbox name?

No, the incorrect interspersing happens for all folders, see first image and comment #0, for example:
Gmail: Descargando mensage 2428 de 5308 en Enviados...
Descargando mensaje 1 de 2 en [Gmail]/Enviados... (interspersed)

For the inbox that would be:
Gmail: Descargando mensage xxx de 18763 en Bandeja de entrada...
Descargando mensaje 1 de 2 en INBOX... (interspersed)

Note that the folder name/path is differently formatted, so that would likely come from two different places in the code, the second one giving wrong information.

The [Gmail]/ in front of "Sent" and "All Mail" is expected in status string since they are under the "noselect" folder [Gmail]. Gmail for 115 has just been artificially "flatted" so you don't see [Gmail]/ in the folder list.

Yes, but as was said, sometimes the [Gmail]/ appears, other times, it doesn't.

Are you seeing [Gmail]/<spanish inbox> or [Gmail]/INBOX ? If so, that would be wrong since gmail inbox is not under [Gmail]/

See above and see first screenshot. The messages for the inbox don't have [Gmail]/ prepended.

I'll try a spanish version of TB asap. Do I have to do something at the gmail site too for all the folder to be in Spanish? I.e., you said "switch your Gmail UI to a different language" so I suppose this is at gmail site?

Yes, see screenshot, here shown for a different account in English.

In the email, it mentioned "recreated" the account.

Yes, initially after moving from 102 to 115, the wrong interspersed message appeared constantly, although there was nothing to download, slowing the system down considerably. That's why I completely deleted the account including all the files and set it up again. After the "recreation" the interspersed messages have only appeared during the initial download phase, now the account seems to work as it should. However, that's not 100% certain since status message pass quickly, so the fault is likely only seen during mass download.

I haven't tried a new profile.

Comment 5

[gene smith]

Ok, thanks. I understand the problem now.

Comment 6

[gene smith]

I was also informed that the same problem occurs for German localized TB. I downloaded the DE version of 124.0b3 (latest beta) and set the gmail server to DE and started a new profile. My gmail has about 16k messages in All Mail so it's fairly big.
The problems I see are sometimes I see "1 of X" (where X is small) interspersed with normal ranges. I really didn't see just "INBOX" while it downloaded everything (I could have just missed seeing it while looking away).
However, when I set my Inbox to only download headers (turned "sync" off in properties) I ONLY saw INBOX and not the localized name while downloading full messages.
When I re-downloaded inbox in the same session, the starting count never got reset so it would start with something like "2000 of 1000".
Otherwise, after it all downloaded I never saw any re-downloading. But I didn't start with an existing profile previously running 102 (don't have right now) but used a new profile to run 124.
So, for now I'll try to see:
Why there are small count ranges interspersed with normal ranges,
Why just "INBOX" appears instead of the localized inbox name (maybe only when downloading headers?)
Why re-downloads of headers and/or full messages use a final count from previous download as the new starting point.

Comment 7

[Silvia Bonavista i Serrat]

Attachment: 1880043.png

Just for the record, here some screenshots provided by a colleague using a German localized TB on a non-Gmail server. One can see the missing localization and the interspersed "4 of 4".

Comment 8

[gene smith]

Attachment: imap-progress-localization-fix-v0.diff

Here's a first version of a fix.
The progress produced directly by imap protocol uses the standard imap name for inbox, "INBOX". So when headers are downloaded I saw only INBOX in the status line even though I had my inbox name localized to "ZZZZz". However, when full messages are downloaded I did see ZZZZz in the status line. That's because the download of full messages are driven by "autosync" and it produces it's own status and uses the localized (called pretty) name for the folder. However, imap protocol code also was doing writes to status (doing autosync requests) causing the interspersed "1 of X" to INBOX quite often where X is 1 to maybe 5, depending on how many messages are requested at a time by autosync.

I changed the imap protocol code to also use the "pretty" name instead of the name selected by imap, INBOX. So now download of headers also shows the localized name (in my case, ZZZZz).

Also, changed protocol code so that it doesn't write status lines while autosync is in progress, which prevents the interspersed 1 of X status.

There is also another status triggered by the imap parser that indicates % complete on larger messages downloaded, this was also interspersing with the normal progress messages, causing short messages saying just "Downloading message...". I also inhibit this while the autosync status are occurring so you get a mostly constant, non-flickering status line and no longer see "INBOX" in the status.

The first file in the diff just shows how I set inbox to the localized name, ZZZZz. So I did the work with my usual daily build but only with that one change to the inbox name (which now appears on all my test accounts, even POP3).

Comment 9

[gene smith]

Attachment: imap-progress-localization-fix-v1.diff

Since the progress message now just uses the "pretty" folder/mailbox name for the folder currently "sinked" by imap protocol, there seems to be no need to pass around a mailbox name in several functions dealing with progress. This just removes the essentially unused mailbox name parameter from them.
Otherwise, there is no change in functionality from the v0 diff.

Comment 10

[gene smith]

Attachment: Bug 1880043 - IMAP download progress status and localization improvement. r=mkmelin

Comment 12

[[e-mail]]

Two issues here:
The comment "Message Downloading..." is not correct since what is suppressed is:
imapDownloadingMessage=Downloading message…

We believe the entire hunk in nsImapProtocol::ProgressEventFunctionUsingName() is not correct: When opening a profile that hasn't been opened for a while, without the hunk, we see the "Downloading message…" status display. With the hunk, that status display is suppressed, so the status bar remains empty. There is no overall "Downloading X of Y..." which would be interrupted. So we believe the condition on which to suppress the "Downloading message…" doesn't cater for the "just open profile after a while" case.

Comment 13

[gene smith]

In the patch, I changed the comment with the example local folder name as requested by Magnus.

Also, changed the comment containing Message Downloading... to Downloading message... as pointed out in comment 12.

But I was unable to duplicate the status bar issue pointed out in comment 12. When updating an out-of-date profile, even when new messages need to be downloaded to several folders, on startup I always see Downloading message X of Y for each folder updated. I never see a completely empty status line due to Downloading message... being suppressed.

Downloading message... does appear correctly when when downloading the message to disk cache when the folder has no offline store and it appears correctly when downloading to offline store when autosync is turned off.

I think Downloading message... should only appear in response to a user action on a single message causing it to be downloaded. It should not occur when messages are automatically downloaded when the message Downloading message X of Y appears (even when Y is one).

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/26b35edb35ff
IMAP download progress status and localization improvement. r=mkmelin

Comment 15

[Wayne Mery (:wsmwk)]

Comment on attachment 9389478 [details]
Bug 1880043 - IMAP download progress status and localization improvement. r=mkmelin

[Triage Comment]
Approved for beta

Comment 16

[Daniel Darnell [:dandarnell]]

Thunderbird 125.0b4:
https://hg.mozilla.org/releases/comm-beta/rev/dee6a9466928