Closed Bug 1373161 Opened 7 years ago Closed 7 years ago

check_all_folders_for_new=true resets gmail imap folders on startup in 52.2.0

Categories

(MailNews Core :: Networking: IMAP, defect)

Product:
MailNews Core
Component:
Networking: IMAP
Platform:
x86_64
Windows 7
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
Thunderbird 54.0

People

(Reporter: ephbase-moz, Unassigned)

References

Details

(Keywords: regression) 

this is new in 52.2.0

after upgrade from 52.1.1 (on win7 64) all gmail account folders are reset (deleted, recreated, messages re-downloaded)

the activity manager is full of "deleted folder ..." entries

this seems to continue on every restart

I narrowed it down to mail.server.default.check_all_folders_for_new=true
(Likely won't change but...) Does it also happen with Thunderbird in safe mode?

And if you revert to https://releases.mozilla.org/pub/thunderbird/releases/52.1.1/win32/en-US/ ?

https://bugzilla.mozilla.org/buglist.cgi?o1=equals&v1=54%2B&f1=cf_tracking_thunderbird_esr52&query_format=advanced&list_id=13634735 is our list of "fixed" bugs for 52.2.0
Happens in safe mode.  The first run in safe mode ignores user.js and resets check_all_folders_for_new to default so it works, but if you set it to true manually and restart then it manifests.

52.1.1 is ok
We didn't change anything in IMAP between 52.1.1 and TB 52.2.
(In reply to Jorg K (GMT+2) from comment #3)
> We didn't change anything in IMAP between 52.1.1 and TB 52.2.

This should be easy to repro, then you can bisect

1. win7 64, TB 52.2.0
2. start with new profile
3. add gmail account (let it sync at least headers)
4. restart once and check activity manager to make sure everything is normal
5. set mail.server.default.check_all_folders_for_new=true
6. restart
7. you should see account folders disappear (they may reappear right away or not) and "deleted folder" messages in the activity manager
I can confirm the same thing happening for me when updating to TB 52.2.0.

Sometimes TB will even crash when I click on the accounts fast enough. (Submitted two bug reports so far...)
I can also confirm that resetting mail.server.default.check_all_folders_for_new to false fixes it for me, too.
Same here. Everything was running fine in 52.1.1, updated to 52.2.0 and Gmail IMAP was not happy!

On one of my Gmail accounts, all folders except Inbox were not showing up, and a new folder named [Gmail]^Junk was created for most of my Gmail accounts.

It also made a real mess of the AppData ImapMail folder as well; it was creating numerous iterations of each folder and file, such as [Gmail].sbd [Gmail-1].sbd [Gmail-2].sbd etc.

Reverted to 52.1.1 and all was well again.
(In reply to Jorg K (GMT+2) from comment #3)
> We didn't change anything in IMAP between 52.1.1 and TB 52.2.

In 52.2 I see the window telling about wrong IMAP command.
After that TB shows empty messages list, broke all folders excepting Inbox and tends to crash very often.
See the Bug 1373493 for details.
Looking at bug 1373493, perhaps this "I have few Gmail accs with OAuth2 auth" gives a hint.

As I said, there were no changes in IMAP between TB 52.1.1 and TB 52.2. However, we did include bug 1176399 in TB 52.2, so maybe that's causing it: https://hg.mozilla.org/releases/comm-esr52/rev/fdb3acad393d

Philipp and Magnus, can you see whether your changes broke it.

Can someone bisect this on a Daily, please. Bug 1176399 and bug 682474 both landed on M-C on 2017-04-11 so a Daily of April 10th should work and one of April 11th should not work:
Looks like builds were broken on those days, so try

8th:
http://ftp.mozilla.org/pub/thunderbird/nightly/2017/04/2017-04-08-03-02-05-comm-central/

and 13th:
http://ftp.mozilla.org/pub/thunderbird/nightly/2017/04/2017-04-13-03-03-00-comm-central/

So Wayne, here's your but that might make a TB 52.2.1 necessary, since by the looks of it, we broke Gmail in the middle.
Flags: needinfo?(philipp)
Flags: needinfo?(mkmelin+mozilla)
I can confirm problems with my GMail (IMAP) account after the update to TB 52.2.0.
Anyway I setup TB with more IMAP accounts, ie: Yahoo.com, Outlook.com, Libero.it beside to my university account (University of Padova, Italy).
These accounts are all in IMAP mode with SSL security and I have no problems until now for these 4 different accounts.
So, in my case the problem seems specific for GMail. Please note that tha for GMail (but also for Outlook.com) account I have a 2 steps security (OAuth).
Even if I tried to delete and recreate the GMail account in TB, after the first correct upload of all the existing mail to Thunderbird, after closing -> restarting TB session, by ispecting GMial account I did not find any email message (inbox or sent etc..) and I found only the junk folder inside TB GMail accoun.
My 2 cents tip
Leopoldo Saggin aka Topoldo
Hi,

I have this issue too, with TB 52.2.0.

My PC is a Win10, I have 3 IMAP accounts and the only one I have problems with is the GMail account.
I even tried to synchronize the account to work off-line, but every time I restart TB my GMail folder is empty (or not read properly) and TB starts Downloading it again.

I don't know if the issue is with TB or GMail itself (since the problem involves only GMail) but with TB 52.1.0 nothing was wrong.

Roberto.

P.S.: Sorry for my bad english.
Please no more "Me too" comments. Can someone affected try the two Dailies as per comment #8.
(In reply to Jorg K (GMT+2) from comment #11)
> Please no more "Me too" comments. Can someone affected try the two Dailies
> as per comment #8.

Reporters there are 6 of you. We need this info ASAP - one or all of you.

Also, a few of you have mentioned crashes. We need your crash IDs posted here in the bug. see https://support.mozilla.org/en-US/kb/mozilla-crash-reporter#w_viewing-crash-reports
Flags: needinfo?(rcombet)
Flags: needinfo?(leopoldo.saggin)
Keywords: regression, regressionwindow-wanted
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #12)
> Also, a few of you have mentioned crashes. We need your crash IDs posted
> here in the bug. see
> https://support.mozilla.org/en-US/kb/mozilla-crash-reporter#w_viewing-crash-
> reports

bp-439c28f3-462d-4ef1-b157-096c30170616
	6/16/17	3:48 AM
bp-a133d661-0cfa-4583-b39f-26c860170616
	6/16/17	3:29 AM
bp-5c6cd8a0-7658-401a-a69b-6d8a50170616
	6/16/17	3:28 AM
bp-8317c4d0-c1d4-4253-a59f-352dd0170616
	6/16/17	3:14 AM
bp-9590ef01-de55-404f-856c-e595c0170615
	6/16/17	2:52 AM
bp-16e04b45-f2eb-47bc-94fe-58a290170615
	6/16/17	2:52 AM
Just to confirm rolling back to 52.1.1 resolves the issue and all mail is instantly available again, with no further action.
Actually I may have spoken too soon, its now download mail from one of my big folders 1575 of 34k...
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #12)
> (In reply to Jorg K (GMT+2) from comment #11)
> > Please no more "Me too" comments. Can someone affected try the two Dailies
> > as per comment #8.
> 
> Reporters there are 6 of you. We need this info ASAP - one or all of you.

http://ftp.mozilla.org/pub/thunderbird/nightly/2017/04/2017-04-08-03-02-05-comm-central/ just tested and don't break Gmail folders.
mail.server.default.check_all_folders_for_new default is false as in 52.1.1 and I don't touch it.

But... I notice strange window about one of non-Gmail IMAP accounts, SSL + normal password, saying that it is not IMAPv4 account. It can be vestiges of 52.2 crashes, because it reads messages normally.
(In reply to Jorg K (GMT+2) from comment #19)
> Please also test
> http://ftp.mozilla.org/pub/thunderbird/nightly/2017/04/2017-04-13-03-03-00-
> comm-central/
> Then we're pretty certain that bug 1176399 broke it.

Yes, this version broke folders as described many times here and in Bug 1373493 with one difference - I don't see a window about invalid IMAP command.
It looks like even with the downgrade it is downloading all mail again.

In the profile directory I see global-messages-db.sqlite is nearly 4GB here and ImapMail subfolder was small and its not 1.2GB after a hour or so of running.

So it appears whatever happened with 52.2.0 deleted ALL of the download Imap mail :(
Thanks!!

All we can do is backout bug 1176399 and bug 682474 from the TB 52 ESR and ship a new version TB 52.2.1 real soon since breaking Gmail is not so much fun.
(In reply to Steven Hartland from comment #17)
> Crash report:
> bp-4a6a3175-ac60-4d4b-81e2-825440170616	16/06/2017

This is nsACString_internal::Assign | nsACString_internal::Assign | nsImapProtocol::CreatePossibleTrashName
Which has signficantly increased activity in 52.2.0 and is in fact a topcrash. And also what Ache is seeing.
https://crash-stats.mozilla.com/topcrashers/?product=Thunderbird&version=52.2.0

I can say with certainty that not all gmail users are affected, so I think we can take a couple days to determine what is causing the problem and perhaps develop a fix.
I rolled back to TB 52.1.1 and everything went back to work.
Then I installed this "daily", as requested:
http://ftp.mozilla.org/pub/thunderbird/nightly/2017/04/2017-04-08-03-02-05-comm-central/thunderbird-55.0a1.en-US.win32.installer.exe
Apparently I did not have problems anymore with my GMail account.
Just an info more: now I have 3 folders for "Junk" emails in my GMail on TB
I'll stress this because an error at this level was the first signal of the problem with GMail & TB 52.2.0, at least in my case.
The tree of folders for my GMail account in TB appears this way:
GMAIL
- Inbox
- Junk
  [GMail] (folder)
  - Bozze   (in italian. The original should be "Drafts")
  - Posta inviata  (in italian. The original should be "Sent")
  - Tutti i messaggi (in italian. The original should be "All the messages")
  - Spam (in italian. The original should be "Junk")
  - Cestino (in italian. The original should be "Trash")
  - Chat  (in italian. The original should be "Chat" too)
  - Conferme di recapito  (in italian. The original should be "Return Receipts")
  - Importanti (in italian. The original should be "Important")
  - Junk (I suppose it did not bmatched the name with "Spam")
  - Lavoro (in italian. The original should be "Work")
  - Modelli  (in italian. The original should be "Templates")
  - Personale (in italian. The original should be "Personal" or "Private")
  - Speciali (in italian. The original should be "Special")

Leopoldo Saggin aka Topoldo
Flags: needinfo?(leopoldo.saggin)
See Also: → 1373493
Fallen on IRC writes [in some of his original testing of bug 1176399] that the folders went away, only a restart helping
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #23)
> 
> I can say with certainty that not all gmail users are affected, so I think
> we can take a couple days to determine what is causing the problem and
> perhaps develop a fix.

Along these lines I have many gmail accounts, 4 of which are oauth, several more which are not. But the auth is saved/stored. I'm not seeing any problems with or without check_all_folders_for_new with 52.2.0 and betas. So this isn't a univeral problem and there is some factor which causes this to happen for some users and not others. 

Several of our release testers are gmail users and did not report issues. And of course there are many beta testers with gmail. And I did find Bug 1362881 - Thunderbird does not always load emails (and folders) from gmail account - reported by a beta 54 user)

Are all of you prompted to authenticate the gmail account at startup or when getting new mail?
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #26)
> Are all of you prompted to authenticate the gmail account at startup or when
> getting new mail?

For what I'm concerned, I saved all my password (not only the Gmail one - in fact I use 8 different accounts) into Tools -> Options -> Security -> Password.
They are all protected by the extension "Startup Master v.1.6.3").
Please note that I also save my Google Calendar password because I sync Lightning with several Google Calendars I have, using the WebDAV technique.
Besides, I also use two other extensions Google-related, ie "GoogleTasksSync v.0.5.3" and "gContactSync v. 2.0.12" both of which require to save my GMail account password.
Anyway I did NOT experienced any problems even using these extensions until TB 52.2.0.

Regards,
Leopoldo Saggin aka Topoldo
One more info.
I have 3 different PC using TB.
I had the problem with the PC at work starting this morning (CET-DST)about 4-5 hours ago, after a couple of days in which I did not have any problem.
I'm curious to see what's about the other 2 computers I have. Until yesterday night they where ok.

Regards,
Leopoldo Saggin aka Topoldo
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #26)
> Are all of you prompted to authenticate the gmail account at startup or when
> getting new mail?

I also use "StartupMaster 1.6.2", but have tried without and the issue persists (and Thunderbird behaves weirdly) One of my crashes is in fact from a run without StartupMaster enabled.
2017-04-08-03-02-05: ok
2017-04-13-03-03-00: bug

> Are all of you prompted to authenticate the gmail account at startup or when getting new mail?

No, why would that happen? In my repro directions for a new profile (Comment 4), after you add an account you are prompted for oauth login, after which the token is saved and there are no further prompts.
FWIW

I had 1 gmail account which threw an imap error 'unknown command' took quick to read on first startup after update, and I restarted so it is not in the log now. 

Two of my gmail accounts with subfolders were fine, the other did not display any subfolders, and gave me messages that folders for which I had messages redirected by rules couldn't be found. The only setup difference I could see between two of the accounts was that the one that was working had just xxxx for the username, and the one that didn't had zzzz@gmail.com. When I removed the @gmail.com, all the folders loaded for that account. However, none of my Favorite folders in that account are marked Favorite any longer.

A third account has also has @gmail.com in the username, and is working. I can only imagine that changing the username caused Tbird to recheck for folders, but I'm at a loss to explain why it didn't hit the same error that time.
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #26)
> Are all of you prompted to authenticate the gmail account at startup or when
> getting new mail?

Not for me. Token was saved long time ago.
An additional side effect of issue is that has caused is several filters where automatically disabled and needed to be manually re-enabled
Please read Gijs' analysis in bug 1374244 comment #4.
(In reply to Jorg K (GMT+2) from comment #35)
> Please read Gijs' analysis in bug 1374244 comment #4.

What is the plan for this? Why don't you revert bug 1176399 and release 52.2.1? Then you can take whatever time you need to redo it.
I do not mean to add to the chorus of "me too" -- I am unable to check my university email (IMAP) MS Exchange account using TB. I want to help out to correct the IMAP connection problem. Please let me know what information is needed from the Error Console, etc to facilitate a solution. 

Thank you,
R.V.
Will a new version be pushed out real soon to fix this?  I'd hate to have to tell people to reinstall 51.1.1, just to have the new version come out right away.
(In reply to R.V. from comment #37)
> I do not mean to add to the chorus of "me too" -- I am unable to check my
> university email (IMAP) MS Exchange account using TB. I want to help out to
> correct the IMAP connection problem. Please let me know what information is
> needed from the Error Console, etc to facilitate a solution. 
> 
> Thank you,
> R.V.

Your issue is not related to this bug.  This bug is specific to Gmail.

You should get support for the add-on "ExQuilla for Microsoft Exchange" by looking at that add-on's webpage:
https://exquilla.zendesk.com/hc/en-us
With all due respect. I had no problems using TB to check my Exchange email account by setting up the account as an IMAP connection until the update to 52.2. I have used TB for many years and I really do not appreciate being told that the problem I'm encountering is unrelated to the IMAP server connection.

Thank you for not being of assistance.
R.V.
(In reply to R.V. from comment #40)
> With all due respect. I had no problems using TB to check my Exchange email
> account by setting up the account as an IMAP connection until the update to
> 52.2. I have used TB for many years and I really do not appreciate being
> told that the problem I'm encountering is unrelated to the IMAP server
> connection.
> 
> Thank you for not being of assistance.
> R.V.

Bruce didn't say that your problem was not related to the IMAP sevrer connection, only that it was not related to this bug, which is specific for Gmail. Aside from that, there are problems with Exquila in relation to Thunderbird 52.2.0, which do need to be solved in the add-on.
My symptoms are the same:

 * Lots of deleted folder messages ("Deleted folder Inbox" etc) in Activity Manager
 * Re-downloads and re-indexes everything upon start-up
 * Reverting to 52.1 fixes the issue

I would like to note two things:

 * Out of 5 Gmail accounts, only 2 had the issue.
 * Globally setting check_all_folders_for_new=false did not fix the issue, it was already false!

However, the 2 Gmail accounts that had the issue both had "Check for new messages on startup" ticked on individual account settings. I'm now investigating if I can successfully download all messages, restart TB and see that the symptoms do not show up again.

Thanks for looking into this so promptly, seems to be a major issue. Downloading and re-indexing all messages takes hours during which re-indexing can make a system slow-responsive.
See Also: → 1374244
Thanks Jorg!

I switched on "Check for new messages on startup" for one account, which would previously produce the issue in question.

I can confirm that the behaviour is gone for 52.2.1-rc-build1.

There were some other activities that I didn't recall usually happening out of the blue on an otherwise fully sync'ed system (my previous version was 52.2.0). It's both indexing an Inbox (unspecified which account) of over 6,000 messages, and downloading messages for another Sent Mail (1,000) and then downloading ~200 messages from another Inbox. Before the upgrade, I had done all the indexing, restarted a number of times, and the activity manager was all chilled out.
(In reply to Jorg K (GMT+2) from comment #43)
> All people affected, please try TB 52.2.1:
> http://ftp.mozilla.org/pub/thunderbird/candidates/52.2.1-candidates/build1/

appears to be fixed (3 restarts, no folder deletion)
Also in my hands (I installed TB 52.2.1-rc-build1 onto 2 different PCs) at the moment seems to work fine.
Let's hope it could be really ok for all of us!
Topoldo
Same issue, on CentOS 7.
(In reply to Martin S. from comment #47)
> Same issue, on CentOS 7.

And it seems (so far) that switching away from OAuth2 fixes the issue.
52.2.1 is released and available
Same problem on TB 54.0 betas and 55.0beta1-rc-build1 with mail.server.default.check_all_folders_for_new=true.
Resetting mail.server.default.check_all_folders_for_new to false fixed the problem.
(In reply to Kosuke Kaizuka from comment #50)
> Same problem on TB 54.0 betas and 55.0beta1-rc-build1 with
> mail.server.default.check_all_folders_for_new=true.
> Resetting mail.server.default.check_all_folders_for_new to false fixed the
> problem.

Yes. These will be broken at least until we can build another beta. We have no ETA when that might be possible
Just updated to 52.2.1 and it didn't fix the issue, its reset everything again so will now spend most the next week re-downloading all my mail which is very annoying to say the least :(
Even worse mail in folders which is years old (26 Nov 2004) is now showing as current for today due to this.
(In reply to Steven Hartland from comment #52)
> Just updated to 52.2.1 and it didn't fix the issue, its reset everything
> again so will now spend most the next week re-downloading all my mail which
> is very annoying to say the least :(
52.2.1 readownload everything due to previous version was installed, but not damage the folders structure at least.
Fixed by backouts from TB 52 and later. Open report remains: Bug 1379475.
Flags: needinfo?(rcombet)
Flags: needinfo?(philipp)
Flags: needinfo?(mkmelin+mozilla)
Target Milestone: --- → Thunderbird 54.0
Status: UNCONFIRMED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.