827897 - Manual get messages can cause process collision: "This folder is being processed. Please wait until processing is complete to get messages." (15 pop accounts)

Status: REOPENED

(MailNews Core :: Networking: POP, defect)

Description

[Dan Pernokis]

User Agent: Mozilla/5.0 (Windows NT 6.1; rv:17.0) Gecko/20100101 Firefox/17.0
Build ID: 20121128204232

Steps to reproduce:

Usually concurrent GET MAILs -- manually initiated getmail for a list of accounts while a regular timed getmail is already in progress, or getmail while compact is in progress.  But this time the problem happened spontaneously.


Actual results:

Usually while progressing through list of accounts, getmail trips over an account which is already doing its own getmail and gives popup message "This folder is being processed.  Please wait until processing is complete to get messages."  Does not go any further until OK is clicked.

This time, the message appeared by itself while I was away -- I had not manually initiated anything as far as I know -- and was waiting for OK to proceed.


Expected results:

Procedural lockout should either pre-check before continuing (probably it does), but in any case should retry occasionally and not wait for user to click OK.  In this latest incident, had I been out of the office, TB would have waited forever and nothing would have come down all day!

Comment 1

[Dan Pernokis]

This issue has existed since early TB and continues to today's (version 17.0).

Also note this bug at its essence may be the cause or triggering factor in several listed bugs which also display the "folder is being processed" message.

To reproduce, simply do a getmail while getmail is active.  (With only one or two accounts, you may have to repeat getmails a to time it right.  On a long list of accounts, it will happen fairly quickly.)

Comment 2

[:aceman]

This will probably be solved universally by the migration to maildir(lite) where each message is a separate file so concurrent processes will not trip over themselves (they will probably not work on the same file, as they do now). You can already try it out manually, see bug 402392.

Comment 3

[Wayne Mery (:wsmwk)]

(In reply to Dan Pernokis from comment #1)
> To reproduce, simply do a getmail while getmail is active.  (With only one
> or two accounts, you may have to repeat getmails a to time it right.  On a
> long list of accounts, it will happen fairly quickly.)

I remember now that at least one example of this shouldn't happen. A bug was fixed a few years ago where repeated getmail of pop account would easily trigger this.

Dan, are your accounts imap or pop?

Comment 4

[Dan Pernokis]

Hi Wayne...

I have one IMAP and fifteen POP3 accounts, and all except the IMAP have multiple addresses associated with them.  Important accounts trigger a getmail every ten minutes, others are maybe once per hour.

The problem still persists -- almost daily.  I often get the warning popup when I manually do a getmail (which gets all), but I've never seen it happen spontaneously when getmails are triggered automatically on their own.

(I also get the message -- understandably -- when getmail is attempted while compact is in progress.  But that is usually because I can't see any progress of the compact session -- hint, hint, would be nice to see that "Such-and-such mailbox is being compacted" -- and the status bar is often erased so the "Compacting Complete" message can easily be missed.)

-dan-

Comment 5

[Wayne Mery (:wsmwk)]

Dan, did you use global inbox when you created these pop mailboxes?

current list of bugs with " This folder is being processed"
https://bugzilla.mozilla.org/buglist.cgi?f1=OP&o3=anywordssubstr&list_id=7361341&v3=folder%20backend%20pop&resolution=---&classification=Client%20Software&classification=Components&o2=anywordssubstr&f4=CP&query_format=advanced&j1=OR&f3=component&f2=short_desc&longdesc=This%20folder%20is%20being%20processed&v2=folder&product=MailNews%20Core&product=Thunderbird&longdesc_type=substring

Comment 6

[Dan Pernokis]

I don't honestly recall but I don't think so.  I do use one global inbox ("INBOX") to capture all incoming mail from all accounts, but I'm pretty sure most or all accounts were created with whatever default, then switched to global.

And I can't say which (if any) particular mailbox is being polled when the msg comes up -- I think they all have been at some time or other.

Comment 7

[WADA:World Anti-bad-Duping Agency]

"This folder is being processed. Please wait until processing is complete to get messages."　itself is evidence that locking of mail folder is successfully done.
- When "Repair Folder" is in progress, 
  similar message is shown when Compact is requested.
- When Compact is in progress,
  similar message is shown when "Repair Folder" is requested.

What is your auto-compact setting? Is threshold value appropriate?
Check following via Config Editor(Tools/Options/Advanced/General).
> mail.prompt_purge_threshhold = true / false (true==auto-compact is enabled)
> mail.purge_threshhold_mb     = 20 (total deleted mail size when auto-compact is invoked)
> mail.purge.ask = true / false ("confirmation before auto-compact" is shown, or not shown)
>      (if default or reset, this may not be shown)
> note: Auto-compact is invoked "once per one hour",
>       So, if auto-compact is invoked, next auto-compact happens after
>       one hour or later. 

Set mail.purge.ask = true, and restart Tb(note: restart is mandatory).
How frequently is "confirmation dialog before auto-compact" shown in your environment?
(never check "do it automatically..." at dialog. otherwise, you need to set mail.purge.ask=true and restart Tb again.)

When dialog is shown, reply Cancel, check file size of Inbox of Local Folders(file named Inbox, not file named Inbox.msf), and Do "Compact" of Inbox of Local Folders(Global Ibox for your many POP3 accounts).
How long does "Compact of Inbox" take usually?
What is file size difference after end of Compact?

Comment 8

[Dan Pernokis]

I agree that the locking process works.  My concern is that GETMAIL does not look to see if certain processes like COMPACT are in already in process before fetching anything.  In any case, GETMAIL should not wait forever when a lock is detected -- it should time out and check again, then proceed when the coast is clear.

ANSWERS for WADA:
mail.prompt_purge_threshhold = TRUE (default boolean)
mail.purge_threshhold_mb = 60 (user set, integer)
mail.purge.ask = TRUE (default boolean)

Confirmation dialogue???  Environment???
Nothing listed in DOS/WINDOWS environment (MSCONFIG), nothing listed in TB Config Editor.  And I've never seen any such pop-up (dialogue box), nor have I ever seen "do it automatically..." checkbox.

Since mail.purge.ask was already true, I never changed it and just proceeded with compact.  I didn't get a dialogue -- the compact just ran when requested. 
INBOX (Before) = 466,541 kb
Compact = 59 seconds (typical)
INBOX (After) = 447,863 kb (2810 messages)
Difference is 18.6 mb decrease in size

Comment 9

[Dan Pernokis]

I had another spontaneous collision a few days ago.  I was just sitting at my desk -- hadn't touched the computer for some time -- when suddenly the "folder being processed" message popped up all by itself.  So two automatic getmails must have been in progress.  Strange, because although each timed getmail can trigger at any time according to its clock, they each poll their own independent mailbox.  But I suppose if they're both trying to write to INBOX simultaneously -- and almost everything goes there -- that would be a conflict.

(FYI, have been running TB 17.0.8 for quite some time, new version 24.0 just now installed.)

Comment 10

[:Hb]

(In reply to Dan Pernokis from comment #6)
> ... I do use one global inbox ("INBOX") to capture all incoming mail from 
> all accounts, but I'm pretty sure most or all accounts were created with 
> whatever default, then switched to global.

Please try to deactivate the Junk Mail handling in all sub accounts while having it activated for th e main inbox.

Comment 11

[WADA:World Anti-bad-Duping Agency]

(In reply to Dan Pernokis from comment #4)
> I have one IMAP and fifteen POP3 accounts, and all except the IMAP have
> multiple addresses associated with them.  Important accounts trigger a
> getmail every ten minutes, others are maybe once per hour.

If 15 POP3 accounts uses Global Inbox, and if each new mail check of each POP3 account takes 1 minutes, and if new mail check interval is 1 minutes for all POP3 accounts, there is no room to use Global Inbox for download of mails by manual new message check. 

Get POP3 log and check "required time" for an new mail check of each POP3 account.
If many mails are held in Mbox of POP3 server by "Leave Messages on sever", it takes long, because LIST command and UIDL command is needed and server returns message number/size of all mails and UIDL of all mails.

Another possible case : internal rebuild-index(Repair folder)
Get NSPR log with MSGDB:5. Is log like next seen?
> nsMsgDatabase::Open(C:\DOCUME~1\mjurgens\LOCALS~1\Temp\MozillaMailnews\<FolderName>.msf ...
(see bug 905576 for this log. this log is written when rebuild-index is invoked by Tb)

Comment 12

[Dan Pernokis]

Most of those 15 inboxes are empty by now -- they're inbound from various businesses we do and they're not very busy, and their polling is once an hour (or something long like every 45 minutes).  The few that are regular (about four or five), have at most a hundred or two emails "left on the server" and only a few emails occasionally to bring down, and they're the ones polled every ten minutes.

When all systems (me and my ISP) are running well, polling of all 15 takes but a few seconds.  (I can see the connection/polling/etc messages whizzing by.)  If an email comes down, of course the message lingers while downloading, but otherwise the polling is very quick.  If my ISP is slow, the connect messages linger, but otherwise the other messages still whiz by.  Only if my own system is slow do the messages crawl by, and that's rare.  In ALL cases, the "folder is being processed" popup message may occur.

And just to reiterate, it isn't the occurrence of the message that bothers me -- the message obviously is preventing a damaging collision.  My beef is that the message never times out to try again -- so if I'm away when it happens, nothing else moves along and no further emails come down.

@HB:  This problem happens on our laptop too -- and it has little or no filtering active, and polls only half of the accounts.

Comment 13

[Dan Pernokis]

One more thing, HB:  Most of the time, as happened just now, the polling is happening but not fetching anything -- and therefore not filtering -- the polling messages zip right by, and bang! there's a popup midway through.  No emails come down (nothing there), nothing filtered, just "This folder is being processed".

@WADA:  Total time right now for GETMAIL to poll & check all accounts -- 15 POP3 and one IMAP -- about 8 seconds.

Comment 14

[:aceman]

Interesting idea about closing the alert message after a while (if you are away).

Neil, are there any alert dialogs with a timeout available in the widgets?

Comment 15

[WADA:World Anti-bad-Duping Agency]

IIRC, "error response from server to an IMAP command" is reported;
  1. "elevator error message panel" at right bottom corner of Desktop.
     After a while, this message goes down, and disappears.
  2. Error is logged in Activity Manager panel.
"New mail alert" also uses style like 1.

Comment 16

[[e-mail]]

(In reply to aceman from comment #14)
> Interesting idea about closing the alert message after a while (if you are away).
> 
> Neil, are there any alert dialogs with a timeout available in the widgets?

I don't know of any dialogs with an auto-close timeout. (There were some dialogs with a focus timeout to ensure that you read them before you click OK but that's the reverse of what you want.)

A spontaneous biff should of course never alert in the first place.

Comment 17

[[e-mail]]

Bah, midairs prevent needinfo clearing :-(

Comment 18

[:aceman]

Actually the biff idea is interesting too. The dialog could be converted to the sliding alert in the corner (like the "new messages" notification or IMAP errors notification). Those do disappear after a while.

Comment 19

[Dan Pernokis]

OK, but it isn't just the disappearing -- the process has to loop back and re-try.  (I presume that happens now when I click OK, but I don't know that for a fact.)

And I think a good time-out would be about 60 seconds -- short enough to have no impact on automated processes, but long enough to avoid constant thrashing & checking every few moments.

Comment 20

[Wayne Mery (:wsmwk)]

is this not bug 707933?

Comment 21

[Dan Pernokis]

Bug 707933 is similar, even probably the same cause.  But it specifically emphasizes that the collisions *never* happen during manual GETMAILs, whereas this bug came about when an irritant in manual mode started to occur spontaneously in automatic polls.  And for the record, it still happens regularly on manual GETMAILs and occasionally on automatic on TB 24.6.0.

In both bugs, I think we all agree that some kind of lockout is happening -- as it should -- but I think a timeout would alleviate most or all of the concerns.

Comment 22

[Wayne Mery (:wsmwk)]

Unless wada thinks otherwise, I think we should focus on bug 707933. And if bug 707933 doesn't end up fixing your problem then we reopen this bug.

Note I believe this got worse in version 24.

Comment 23

[Dan Pernokis]

Cross-posted from Bug 707933...

Today I updated to TB 102.0.3 Release and re-enabled my GET MESSAGES button configuration. This bug still occurs.

I'm getting the occasional popup warning that certain accounts are "busy". Happened when I clicked GET MESSAGES first time and the timed fetches started too. In fact, for the first time I had two popups simultaneously -- one immediately on top of the other -- as the two accounts competed for attention, and a total of three popups altogether (out of ten accounts actively polled on this machine).

Comment 24

[Wayne Mery (:wsmwk)]

Dan do you still see this problem when using version 128?
If it is gone then this will have been solved by bug 1847137

Comment 25

[Dan Pernokis]

Sorry, I'm not running v128. However we are running the latest v115 and it STILL occurs there very regularly.

As for the fix in Bug 1847137:
That bug is a little different than this one, because the errors repeatedly occur on their own ("spontaneously", I've said in my reports) when two timed GETMAIL operations collide since they happen to run at the same time. In my case I have rarely (but not never) had it happen spontaneously. But every day, manual collisions still occur if an automatic (timed) GETMAIL or GET ALL MAIL happens to be silently in progress when I manually trigger a GET ALL MAIL request and one of the fetches then collides with the automatic fetch already in progress. (I see the guys have stated that v115esr is affected but its status is "wontfix".)

Comment 26

[Wayne Mery (:wsmwk)]

Thank you for reminding that this can still happen with manual operations.