494784 - GMail IMAP SPAM folders no longer allow automatic moving of junk mail to local junk folder

Status: RESOLVED INVALID

(Thunderbird :: Folder and Message Lists, defect)

Description

[John L. Galt]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1pre) Gecko/20090525 Firefox/3.5b4pre (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1pre) Gecko/20090525 Shredder/3.0b3pre

I cannot be sure of the date when this started occurring, but...

I set up my GMail accounts originally as POP, but earlier this year copied all my mail back to the servers after establishing the IMAP interface in Thunderbird 3.0b2 - everything went well, except for a few glitches, and once having done that I set up my account to automatically mark msgs that I mark as Junk to be marked Read and moved to the Junk folder on my local machine.  This worked for a while, but all of a sudden, for the Spam folders on GMail, it has stopped working.  Any other folder (from any server, GMail, Yahoo, Hotmail, POP, etc) I currently use will automatically move the mail to the junk folder locally - however, only the Spam folder in GMail accounts via IMAP have stopped doing this, an I have to manually move the msgs to the Junk folder every time.

Stats:

Windows 7 64bit RC
Core2Quad 6600 OC'd to 3.21 GHz
4 GB PC8000 RAM
Tb 3.0b3
3 GMail accounts (IMAP)
several POP accounts
4 Hotmail Accounts (Webmail)
1 Yahoo Account (Webmail)

Reproducible: Always

Steps to Reproduce:
1.  Open Tb, check GMail account
2.  Open Spam folder, mark all as Junk
3.  Msgs stay in folder instead of moving to local Junk folder as set in account settings
Actual Results:  
Msgs manually marked as Junk in Spam folders do not get moved to Junk folder on local machine.  Msgs manually marked as Junk on *any* other folder do get moved automatically.

Expected Results:  
All messages marked as Junk should be moved to the local Junk folder.

I have tried rebuilding the index on both Spam folders, as well as changing various other settings as well, including handling options (marking Server default as well as removing that and setting my own options) and nothing seems to help.

Comment 1

[Magnus Melin [:mkmelin]]

Fallout from the xlist support (bug 476260) perhaps...?

Comment 2

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> Gecko/20090525 Shredder/3.0b3pre
>(snip)
> Steps to Reproduce:
> 1.  Open Tb, check GMail account
> 2.  Open Spam folder, mark all as Junk
> 3.  Msgs stay in folder instead of moving to local Junk folder as set in
> account settings
> Actual Results:  
> Msgs manually marked as Junk in Spam folders do not get moved to Junk folder on
> local machine. Msgs manually marked as Junk on *any* other folder do get moved
> automatically.

Sounds that your case is not Bug 487610.
"Mark as Junk at IMAP folder + Automatic move to local Junk folder" roughtly consists of; (1) uid xxx FETCH for the mail, (2) Copy mail data to local Junk folder with Junk-score, (3) uid xxx store flag \junk like operation, (4) uid xxx store flag \Deleted.
But, [Gmail]/Spam(Spam *FOLDER* at Web Interface of Gmail) is very special folder. In addition to it, Tb 3 knows that [Gmail]/Spam is special folder.

To John L. Galt(bug opener):

(Q1) Get IMAP log. Is there any Tb's fault in IMAP log?
    (See Bug 402793 Comment #1, Bug 402793 Comment #16, Bug 402793 Comment # 17)

(Q2) What is your purpose to move spam mails in Gmail's Spam to local folder?
I think following is practical way;
  (a) If local mail or IMAP mail should be marked as Junk,
      move(or copy) it to [Gmail]/Spam. (If copy from Gmail's folder to
      [Gmail]/Spam, it's same as "move to [Gmail]/Spam)".)
      It's similar to "report of new Junk/Spam to Gmail".
      Bayesian Filter of Tb also learns about new Junk/Spam.  
  (b) If mail in [Gmail]/Spam should be markd as Junk in order to pass Junk mail
      information to Tb's Bayesian Filter, "Mark As Junk" is sufficient.
      There is no need to move spam mail in [Gmail]/Spam to local Junk folder.
  (c) If you want to collect Junk mails in local Junk folder,
      I think "copy from [Gmail]/Spam to local Junk after (a)&(b)" is sufficient.

(Q3) Did you define the Gmail IMAP account by "Account Wizard for Gmail IMAP"?
    (new feature of Tb 3.) Or you defined Gmail IMAP account by ordinal account
    wizard of Tb 2 or Tb 3?

Comment 3

[John L. Galt]

@WADA:

re:  Q1 - so, if I understand the documentation correctly from https://wiki.mozilla.org/MailNews:Logging I need to set the environment variables

NSPR_LOG_MODULES
NSPR_LOG_FILE

And then it should start logging the information?  Or should this already be occurring automatically in these nightly builds?  If the latter, I see more than 1 file with the word log in it, an I even see each IMAP account having the filterlog.html file (which must be is there from creation, b/c AFAICT I have filter logging turned off)....

Q2 - I have 12 different email accounts that I have accumulated over the various years (dating back to 1996 when I created my first HoTMaiL account) that I keep consolidated in one single application.  I have always used POP previously, with Webmail extension for the Webmail accounts, so that I could easily keep track of my different accounts in one interface.  (BTW, these 12 are the remaining accounts - I have dropped several due to server abandonment (a la Mailandnews.com / shanjemail) as well a servers where I am getting entirely too much spam as the service has removed SPAM scanning from free accounts (a la Softhome.net).

My main idea here is to move *all* mail marked junk to a *single* repository from all the accounts - unmarking it as Junk used to send the mail back to its respective account inbox, but I have noticed that now with he IMAP accounts, it also has ceased to function correctly, moving it back to the local Inbox instead...

I keep each account separated (in terms o fits own inbox, outbox, sent folder,  etc) but like to keep the junk on my local machine b/c my machine is speedy enough, and with the proliferation of junk and spam, and *especially* as I have a single 'honeypot' account via HotPOP that I get nothing but junk in and that I use to test for 0-day exploits in the wild with various anti-virus and anti-malware programs, My junk folders will start getting full quickly (in a typical well, my local junk folder has ~2000 msgs in it - but I also like to peruse the folder periodically to make sure something I am expecting has not moved to junk b/c of an overzealous filter setting, or else b/c of poor choice by the sender in terms of subject line or attachments - thus, keeping it localized to a single repository makes much more sense.

re (a) I may just allow the IMAP accounts to use their own SPAM folders, but unfortunately, then I really don't want to be looking at the SPAM folders every time there is a new mail entered into the folder - and every time *GMail* moves msgs into those folders it does *not* mark the msgs read, and thus Tb will start synchronizing those folders for me.  *In addition*, as stated above, I am never 100% trusting of *any* filtering method as being foolproof - one of the 3 GMail accounts I have is one associate with my domain, and while the amount of spam in that particular account is relatively small (1-2 per week - lucky me) I'd still rather not have all mail automatically filtered without having a final say so in whether the mail is actually junk or not.

Just as a case in point - this particular email notification of your reply to me was in my GMail SPAM folder - not sure why, as there is another reply already in my account, but the times are different by over an hour - but, at any rate, had this been the only notification I had received, and if I had not paid attention to the Spam folder, I would not have read replied until some time further down the road....

re: (b) - already doing that

re: (c) - I understand that copy to is 'sufficient' - however, still having those mails in the folder is a PITA b/c I have (already) close to 100 folders that are synchronizing, and having to sync yet another folder, and one that gets as much activity as it does (roughly 20-30 spam msgs per day) gets taxing on resources, particularly when you miss checking mail for a day or two.  

I also know that I can manually move the msgs, as noted in my original post, however, my point in opening this bug is that, *previously*, on 3.0b1(?) or else 3.0alpha1/2 this was automatic.  Something changed somewhere, and I am not 100% sure *when* it changed, but the automatic moving of msgs stopped somewhere along the way.

re: (Q3) - Account wizard  in Tb 3.0a1/2, I believe - or whenever that feature was first available.

What had actually happened was that I experience a crash and lost all my filter settings and accounts settings earlier this year - I already knew that GMail had started allowing IMAP, so since I had to create everything by scratch, I crated the new profile with all IMAP for GMail, as well as IMAP for GMX.com accounts, and used POP for the remaining POP accounts an WebMail for the remaining WebMail accounts, *primarily* in the interest of always having my mail with me without having to keep my profile on a UFD in my possession all the time - I can just log in to my GMail account online and see the same thing I see here at home....

I then proceeded to access my old accounts (which had all the POP mails on the local HD) manually (by creating a dummy account and linking it to the existing folder) and then manually copied, one by one, each folder I had in the account locally to the IMAP account.  Everything finally was accomplished successfully, but of course, since I had lost my filters I have to create them from scratch - those are only partially working.  Regarding filters, I have toyed with the idea of creating the filters in the GMail web interface itself, as opposed to in Tb, but that is currently too much of a hassle - I previously had well over 100 filters to move mail to various folders - and that is in just one of the two main GMail accounts....

Comment 4

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #3)
> And then it should start logging the information?
> Or should this already be occurring automatically in these nightly builds?

It depends on what you did.
If you set the environment variables as system level setting, all thunderbird.exe instances start logging and tries to write log data to specified file.
If you set the environment variables as "Command Prompt" level setting, only thunderbird.exe which is started at the "Command Prompt" window starts logging.
Difference between Tb 3 latest-trunk builds and some Tb 3 beta builds/Tb 2 is "timestamp parameter is supported or not" only.

Comment 5

[Wayne Mery (:wsmwk)]

(In reply to comment #1)
> Fallout from the xlist support (bug 476260) perhaps...?

magnus, still think this might be the cause?  If so, John would want to test a nightly prior to   2009-05-14.

Comment 6

[John L. Galt]

It would be interesting to know - I never made any headway with the testing mentioned above, but when I installed Windows 7 RTM, I went to set up GMail IMAP and found this page:  http://mail.google.com/support/bin/answer.py?answer=78892 which specifically states not to use the native Junk filters in TB for GMail IMAP accounts.  Following those rules, I can now mark items spam, and the GMail servers will then move them to the appropriate folder (but do not remove them from the current folder either, somewhat problematic for me but I can deal with it).

Anyhow, though, the original report still stands, and I am beginning to think it may not be all Tb's fault, seeing as those instructions from GMail specifically ask the user to not move msgs to the Spam folder via filtering.

Comment 7

[John Andersen]

(In reply to comment #6)
 
> IMAP and found this page: 
> http://mail.google.com/support/bin/answer.py?answer=78892 which specifically
> states not to use the native Junk filters in TB for GMail IMAP accounts. 
> Following those rules, I can now mark items spam, and the GMail servers will
> then move them to the appropriate folder (but do not remove them from the
> current folder either, somewhat problematic for me but I can deal with it). 

There is another bug report here https://bugzilla.mozilla.org/show_bug.cgi?id=494784 that addresses the issue of having to move the mail manually to Spam when adaptive filters are turned off as Gmail advises. 

Since Gmail's filters are way better, it is sound turn off Tbird's filters.  In that case Move junk mail check box should be honored regardless of the setting for using adaptive filters.

ANYONE trying to train a server side spamassassin would want the same thing: Movement by the Junk button to the designated folder and deletion from what ever folder it was found in.  So perhaps weighing in on that bug would be in order.

My testing was with Tbird 3.0b4.

Comment 8

[John L. Galt]

Thanks for the info, John Anderson.  I'd look into that bug, however, you linked to *this* bug....

Comment 9

[John L. Galt]

5 months later and nothing new on this bug....

It is not so bad, except for the fact that the occasional message that *does* slip through the filters now can be marked as spam, but still stays in the same folder where it was delivered to.  The user is forced to manually move the message to the SPAM folder in order o get rid of it.

Does anyone know if this is a Google issue or if it is a TB issue?  Would be nice to have *someone* confirm this officially, as this is still marked as unconfirmed.

Comment 10

[John L. Galt]

This might be now solved.  Testing further to make sure (Tb 3.1.x and Shredder 3.1a1pre

Initial testing indicates that this functionality has been restored, but as of when I cannot say.  I plan to test with lower versions of TB as well to make sure all is well.

Comment 11

[John L. Galt]

OK, Thunderbird 3.1.x will not automatically move messages when they are marked as junk, but 3.2a1pre *does*.

Comment 12

[John L. Galt]

Marking this Resolved and Invalid as this is not a bug, as this is by design how GMail SPAM is set up.