1074055 - IMAP: after "not junk" of about 30-50 individual messages, slow, stops working (yahoo account)

Status: RESOLVED INCOMPLETE

(Thunderbird :: Folder and Message Lists, defect)

Description

[Jim Michaels]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0
Build ID: 20140923175406

Steps to reproduce:

1.IMAP (yahoo mail): set filter to match all messages, mark as junk, move to junk folder. this will zap all emails in the account (not the folder).
2.click junk folder
3.click on individual messages that are not junk and click "not junk" button.
4.repeat #3 about 30-50 times or more




Actual results:

button stops working. "not-junk"ing process takes about 1 minute per email, making the process really slow. I have 1500 messages to process, so at about 5 seconds per email that's 50sec*1500=7500/3600=20 hours of clicking.

I noticed that there's a large download on every "not junk", just watching the progress bar "Downloading message...". it's only 1k, so it's not THAT big on a 59Mbps connection that it should take a minute to download - is it downloading the entire folder or the entire account over and over?


Expected results:

button should continue to work.
downloading 1 IMAP message should not take 1 minute on a 59Mbps connection, should it?

Comment 1

[Jim Michaels]

also, when I start massively clicking the fire icon (about 100-200 items), it doesn't always react by moving the messages out of junk folder. also, sometimes, messages I range-select and drag into Inbox after this come right back, sometimes with possibly duplicates (I have them sorted by From:).

Comment 2

[WADA:World Anti-bad-Duping Agency]

(In reply to Jim Michaels from comment #0)
> Steps to reproduce:
> 1.IMAP (yahoo mail): set filter to match all messages, mark as junk, move to junk folder. 
>    this will zap all emails in the account (not the folder).
> 2.click junk folder
> 3.click on individual messages that are not junk and click "not junk" button.
> 4.repeat #3 about 30-50 times or more
> Actual results:
> button stops working. "not-junk"ing process takes about 1 minute per email,
> making the process really slow. I have 1500 messages to process, so at about
> 5 seconds per email that's 50sec*1500=7500/3600=20 hours of clicking.

"Marking many many mails as Junk or as Not Junk at same time" takes O(n^2) like time and causes high memory consumption. (never O(n) order) This is known phenomenon and is observed even on local mail folder.

> 4.repeat #3 about 30-50 times or more

IIUC, "mark as Non-Junk at Junk Folder" invokes "move mail back to Inbox" currently.
If so, "repeat marking as Non-Junk of same mails at Junk Folder" is impossible.
(Q1) Does the "repeat #3" mean next?
   At Junk Folder,
      (i)    Select a mail(or several mails) at thread pane,
                  (a) via context menu, Mark -> As Not Junk, 
              or (b) at each thread pane row of selected mail, click Junk mark(toggle Junk/Not Junk),
              or (c) at header pane box, click "Junk" button or at above message pane, click "Not Junk" button,
      (ii)   Select another mail(or otther several mails) at thread pane,do (a) or (b) or (c),
      (iii)  Repeat step (ii).
   If yes, "several mails" in this context = 30 to 50 mails in your case? (bug summary sounds this case)
   Or "30 to 50" is "repeat 30 to 50 times"? (comment #0 say so)

Note: if multiple mils are selected at thread pane, (c) is possible only when mail.operate_on_msgs_in_collapsed_threads=false.

(Q2) What is your mail.operate_on_msgs_in_collapsed_threads setting? (Tools/Options/Advanced, Config Editor)
         Do you see difference in performance
         between mail.operate_on_msgs_in_collapsed_threads=false and mail.operate_on_msgs_in_collapsed_threads=true?

"Mark as Not Junk" approximately consists of;
    (d) "Bayesian Filter process" + (e) "Move mail back to Inbox" process
"Tb's behavior in  (e) " depends on Offline-Use=On or Offline-Use=Off of folder, if IMAP and if auto-sync is enabled.
"Process after (e)" depends on "Global Search and Indexer is enabled or not" if Offline-Use=On folder.

(Q3) Is auto-sync enabled? (default = enabled)
        If enabled, is Junk folder and Inbox folder Offline-Use=On folder? (Folder Properties/Synchronization, default=On)
        If Junk folder and Inbox folder is Offline-Use=On folder, is "Global Search and Indexer" enabled?

(Q4) Do you see your problem with -safe-mode of Thunderbird? (force "all add-ons is disabled" status)

(Q5) "Part of slowness in your case" is measured by "time to take for process of (e)".
        "time to take for process of (e)" is measured by "IMAP log with timestamp". (see bug 402793 comment #28 for getting log.)
        Sorry but I don't know whether "bulk move" is used upon (e) by Junk Move or "move individual mail" is used.
        "move mails from Junk to Inbox" is done by Thunderbird via following IMAP command ;
            Move extension is not supported by server  : at Junk, "uid copy xx:yy  Inbox" + "uid store xx:yy +Flags(\Deleted)"
            Move extension is supported by server        : at Junk, "uid move xx:yy  Inbox"
        Check log file content using Text Editor by yourself, please.
How long does it take to execute above IMAP command for "each mail "or "30 to 50 mails" when mark as Not Junk is requested?

Comment 3

[Wayne Mery (:wsmwk)]

Please reply to comment 2

FWIW, using current nightly in 400 messages in my inbox I did mark not junk and it was barely a blip. But I know that's not the same as marking messages in Junk folder as not junk.

Comment 4

[Jim Michaels]

I had such a huge volume of messages (thousands) that wound up in junk folder (some were junk, some were not) that I had to do #3 ovber and over and over. so #4 just says to repeat #3 50x as necessary.

Comment 5

[Jim Michaels]

a "sure you want to do this?" confirmation dialog would be good for moving all to junk folder, just so the user doesn't stub their big toe :-)

Comment 6

[Wayne Mery (:wsmwk)]

(In reply to Jim Michaels from comment #5)
> a "sure you want to do this?" confirmation dialog would be good for moving
> all to junk folder, just so the user doesn't stub their big toe :-)

this is not relevant to this bug report. If you wish to report a new issue please file a new bug report.
(but I doubt it would be implemented/worth the work)

Comment 7

[Wayne Mery (:wsmwk)]

(In reply to Jim Michaels from comment #4)
> I had such a huge volume of messages (thousands) that wound up in junk
> folder (some were junk, some were not) that I had to do #3 ovber and over
> and over. so #4 just says to repeat #3 50x as necessary.

There are more than 5 questions in comment 1.  You haven't touched Q2, Q3, Q4.

Comment 8

[Jim Michaels]

sorry, I have gotten on others for the same thing.
Q2: no about:config like in ff, so how would I go about this? it's in an sqlite db, so I can't just open a .js file or .json file to grab the text. opening the db as binary doesn't work, can't find the text.

Q3:where exactly is auto-sync so I can find out? there are s huge number of settings to look at.

Q4: when I went through this ordeal, no I didn't want to try it in --safe-mode. I wanted my data back to semi-normal like anybody else and was working hard at that since email is one of my lifelines.

Q5: I have not looked at the IMAP protocol, but I suppose about >1/2 HTML emails have attachments of some sort, but not that significant in size. 1min*50Mbps/8bitsPerByte=60*50e6/8=375,000,000 BYTES. I seriously doubt there's that much data transfer going on, unless it's transferring the whole catalog of email headers, that I could understand might take some time, since I have about 20k emails I think in yahoo mail (rolls over). it DID seem like when it was first going that it took an exeeding amount of time just trying to get the message headers and download messages. about a day or two in fact, with one problem I remember: mail auth/login errors that should not have been happening unless the mail server was saying "too much all at once, get off my server". the IMAP download process just looked generally looked broken, but I wondered if it was really TB's fault, so I didn't mention it. many auth retries seemed to encourage tb to finish the job.

the emails in this yahoo mail account range from about 2002-2014 at the time.
I know this is probably not the answer you wanted, but I am unsure how to answer the question. I have not studied up on IMAP logs. I am reasonably familiar with POP3 and SMTP protocols, I don't remember IMAP being in TCP/IP illustrated. I could really use a more up-to-date and nicely-detailed protocols book, since mine is really dated (any suggestions?), I know they are out there.
I know a POP3/SMTP email is not anywhere near the 375MB size, probably more like a few KB usually if it has no pictures (and 90% of those are usually on the web now anyway).

Comment 9

[Wayne Mery (:wsmwk)]

Without rereading the whole bug I don't know if this is feasible - to workaround the issue of "I didn't want to try it in --safe-mode"...just wanted to be working again - a possibility for users of imap accounts is fire up a new profile https://support.mozilla.org/en-US/kb/using-multiple-profiles

Getting beyond that, yahoo accounts seem to have issues related to junk - something I've recently been researching.  Do you see any of the issues listed in http://mzl.la/1TkmCd2 ??

Comment 10

[Jim Michaels]

have you ever thought of automated testing? and what kind of file tree structures should that take the form of? you know you can ask a window for its size, class, etc and there are tools for doing this sort of testing and make the whole process simpler like wintest - is that still around or has the company gone belly-up?
of course safe-mode testing would have to be done differently than non-safe-mode since dialogs and such are somewhat different, like fewer tabs maybe or...
what other windows/mac/linux test tools are out there for sale or free? sf.net, github for example.

Comment 11

[Wayne Mery (:wsmwk)]

what in the world are you talking about? 
what does this have to do with yahoo being a pain the butt?

Comment 12

[Wayne Mery (:wsmwk)]

(repeating)
Do you see any of the issues listed in http://mzl.la/1TkmCd2 ??

Comment 13

[Jim Michaels]

umm, is that a safe web site to visit? just revisiting my bug reports.
tb is failing thunderously, failing at nearly every IMAP try. what I really need now is a converter from IMAP to POP3 as a nice little button to bail out. as a result, I keep losing sent emails and drafts which never make it to being fully saved in teh drafts and sent folders. ugh. please someone fix. I was wondering if it was server trouble, but not so sure now.

Comment 14

[Jim Michaels]

ok, visited it, yeah, it's probably what you pointed out. 

was having mail trouble once with yahoo pop3 or something, and yahoo support once actually tried to get my IP behind my router and ask my router kind (which was dd-wrt at the time "because they could not get through").
I wonder sometimes. and suddenly it started working. unusual.

cancel comment 13.

Comment 15

[Matt]

Do you still see this occurring in Thunderbird 68?

Comment 16

[Phoenix]

Resolved per whiteboard