1943854 - Improve total time to update all email accounts - updating pop accounts serially is slow when not fully parallelized.

Status: NEW

(MailNews Core :: Networking: POP, defect)

Description

[Pepi Huber]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:134.0) Gecko/20100101 Firefox/134.0

Steps to reproduce:

I have 20 eMail accounts and use pop3 to get the messages every 15 minutes. I use a unified inbox and get all mails into this inbox. This works without problem, however, there are two issues:

1. the pop3 access seems not to use parallel connections, but goes in sequence, e.g. first account, then second, .... then 20th. This takes a long time to complete - several minutes. Is there a way to parallelize the pop3 access of different accounts?
2. most of the 20 accounts are not really used for eMail and 99% of the time do not have any incoming mail at all, but 4 accounts are important. I would like to be able to define in the pop3 access which accounts shall be processed first. Currently my most important accounts are polled at the end which means i have to wait for minutes until the major accounts are being processed. So how can the sequence of accounts be defined? In the settings / account dialogue i have already ordered the accounts as desired, e.g. the important ones on top, but this does not apply as i use a unified inbox and the sorting order defined here has no influence on the sequence of pop3 accesses. A way to configure the sequence of pop3 accesses shall be possible.

Actual results:

See above,

1. pop3 access for all 20 accounts is slow, and
2. sequence of accounts for pop3 access cannot be configured.

Expected results:

1. parallel pop3 attempts for all accounts rather than the current sequential mode.
2. sequence of accounts shall be configurable for pop3, e.g. which is the first account to be pop3ing, which is the second, etc.

Comment 1

[Magnus Melin [:mkmelin] (away until Aug 10)]

We used to do it serially in parallel, but that did have some issues as delivery may be to the same file.
If there are not messages, checking mail should be very fast (a sec or so) unless the server is super slow.
For the order, did you try re-ordering the accounts in the account manager?

Comment 2

[Pepi Huber]

You seem confused ... "we used to do it serially" - that's the very issue of speed, did you mean parallel?

How can "delivery be to the same file"? This does not make sense.

Embarrassing answer - someone asks about a speed issue and the answer is "should be very fast". No, it is not, it is minutes for 20 accounts.

It seems you overlooked the post which states "In the settings / account dialogue i have already ordered the accounts as desired, e.g. the important ones on top, but this does not apply as i use a unified inbox and the sorting order defined here has no influence on the sequence of pop3 accesses. ". Otherwise I do not understand why you ask exact this question.

I disagree with your sloppy and disrepectful response! You have not properly read and not understood the issue, have no competence to assess it, otherwise you would not have asked wheter i played around with the sorting order. So you do not know the underlying issue. But still you modify it's status as RESOLVED, while you have not resolved anything. It shall be clear to everybody that this is a waste of time, and not a bit the way to improve things.

The topic remains open, please refrain from further comments as you are not competent to engage here.

Comment 3

[Magnus Melin [:mkmelin] (away until Aug 10)]

Oh yes, we used to do it in parallel, sorry.
Pop can be configured to deliver all accounts to the same inbox (unfortunately).

Comment 4

[Wayne Mery (:wsmwk)]

I'm going to modify this bug report, because it is effectively a regression from Bug 1847137 - 115.1 hangs randomly "not responding" when automatically checking multiple accounts for new messages. Workaround stagger mail check interval.

We aren't going to change back to parallel checking of accounts. But we should try to determine why the access of some individual accounts is slow.

1. For now, are your four most important accounts at the top of the accounts list?
2. Are you using Global Inbox for these pop accounts? (where multiple pop accounts store their messages in a single Inbox, also known as deferred accounts)
3. Do you have a long(er) interval for checking for new mail for the less important accounts, for example 30 minutes?

Comment 5

[Pepi Huber]

Thanks Wayne, your question #1. indicates that you did not grasp my issue #2 in the opening post. I would be happy to be able to "put the four most important accounts on the top of the list". But there is no such list. My whole question for issue #2 relates to how accounts can be sorted. And no, arranging them in the accounts setup window does not change anything on the sort order of how accounts are polled by the thunderbird implementation.

To answer your
question 1: Yes. they are. (But this does not change anything on the sequence thunderbird polls accounts).
question 2: yes, i use global inbox for all but one account, i.e. two "inboxes" - a global one, and a specific one in a different folder.
question 3: no, all accounts are checked every 10 minutes.

What is really annoying, when one gets an email (e.g. registration on a website which send a confirmation) and then refreshing the polling and waiting for a minute or so until thunderbird has crawled through all of them to finally get the message i am waiting for. And no, selecting the specific account and updating it is too many clicks away, i go for shift F5.

How are the accounts arranged for polling? There must be some sort order which is NOT the sort order of the accounts in the setup. I have not found any correspondance with the sequence of adding the accounts to thunderbird. However, there are indications its reversed, i.e. the oldest account (the one added first) gets polled last.

Comment 6

[David]

I also have this same issue, though not as many accounts. It would be nice to check in parallel, but I understand that was causing problems.

So the issue is that there seems to be no way to change the order the accounts are checked. The first account created always gets checked first, and the last account created always gets checked last. You can drag the order of the accounts in the account settings, but it has no effect on the order the accounts are checked.

Comment 7

[David]

I'll add, if somebody wanted to fix the parallel checking issue, you could do something like this:

1. When an account has to write to a file, set a lock on the file. When it's done writing, release the lock.
2. If another account needs to write to the file, check for the lock, and if there is a lock, wait until it is released, then it can lock/write/release.

This would be a good improvement for those of us checking multiple pop accounts.

Comment 8

[Wayne Mery (:wsmwk)]

Regarding comment 7, parallel checking was fraught with issues such that simplistic checks would not avoid the main problems. And I think bug 1847137 not only changed to serial but also improved some performance areas where parallel was deficient?

But perhaps there are some areas for possible improvement. I have linked some. Gene, have any thoughts?

Comment 9

[Wayne Mery (:wsmwk)]

Isaac, how has your performance been when using current versions?

Comment 10

[Pepi Huber]

(In reply to Wayne Mery (:wsmwk) from comment #9)

Isaac, how has your performance been when using current versions?

The question is valid, but seems not to account for the "unknown" - thunderbird might be fast as hell, which will not help if the pop/smtp/imap on the other end is slow.
Having 20 accounts and 3 sec each makes one wait for a minute when the last account is the one you are wanting to check.

A relieve of this behavior can be that the sequence of the account queries can be configured. The sequence now is not the same sequence as shown in the account settings list, and there is no way to configure or change the sequence. Currently it seems the sequence is the reverse order of creating/setting up the accoutns, i.e. FILO - First In, Last out. My first and main email account is the topmost in the account settings list, but is querying the last when updating all accounts.

A solution would be the parallelization - which it seems has made problems before and therefore is serial now.

Yes, its possible to query single accounts, but this is going to several menus and going through lists ....

Comment 11

[Wayne Mery (:wsmwk)]

(In reply to Pepi Huber from comment #10)

(In reply to Wayne Mery (:wsmwk) from comment #9)

Isaac, how has your performance been when using current versions?

The question is valid, but seems not to account for the "unknown" - thunderbird might be fast as hell, which will not help if the pop/smtp/imap on the other end is slow.

It is understood and known that server side delay is a significant factor - even more so if there are many accounts. And by unknown I assume you mean not fully quantifiable, and not under Thunderbird's control. Again, understood.

Regardless we are still going to look for ways to improve.

Comment 12

[gene smith]

When we went from doing pop3 polling in parallel to serial in bug 1847137 it was never really determined why parallel seemed to cause complete TB lockup. The lock-up occurred for some users but definitely not all.
However, a few months ago Hartmut fixed this bug which seems like it could be a major contributor to whatever was causing the app lockups for pop3 users: Bug 2020627.
From the reports in bug 2020627, this bug didn't cause a complete application lockup but stopped pop3 polling until TB is restarted.

Anyhow, I have made a "try" build based on current daily and wondering if reporter Pepi would be willing to run this and see if it helps. It includes the fix from bug 2020627 and reverts back to using parallel pop3 polling.
Here's the link to the try build: https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=0f7c46987a4b1056c410835c27cf17d4bd0c0f94

I don't know which platform reporter Pepi is using so the build includes them all. To find the appropriate installer, click the green "Ba" next to the platform. Then down below click on "Artifacts and Debugging" column. A list of files appears and you can pick the appropriate installer to download:
build/target.tar.xz (for linux, just expand and run the thunderbird executable inside, not really an installer)
build/target.dmg (for mac)
build/target.installer.exe (for win32==Windows x86 Opt or win64==Windows Opt)

Comment 13

[Magnus Melin [:mkmelin] (away until Aug 10)]

I think use of deferred inbox (where multiple pop accounts deliver to same inbox file) was also causing issues. I'd think that's a complication we do not want to have, so we should always force such accounts to check serially.

Comment 14

[gene smith]

(In reply to Magnus Melin [:mkmelin] from comment #13)

I think use of deferred inbox (where multiple pop accounts deliver to same inbox file) was also causing issues. I'd think that's a complication we do not want to have, so we should always force such accounts to check serially.

I appears there has always been an attempt to serialize for deferred accounts as indicated here:
https://searchfox.org/comm-central/rev/c522a5f0f3336d4d45dff99fa5dd2b7ca30767bf/mailnews/local/src/Pop3IncomingServer.sys.mjs#270
It also serializes checks for normal pop3 accounts when checking all pop3 servers.
I put the "Note" there since it still allows simultaneous connections but this appears to force serial POP3 activity (based on observed testing with 3 POP3 test accounts).

The serialization change at bug 1847137 really only affects checks for a single POP3 server and several queued up quickly, and during timed biff checks of multiple server and does not appear to affect manual checks for ALL servers, e.g., shift-F5.

So my try build in comment 12 reverting bug 1847137 probably won't make a difference for reporter Pepi since he's only noticing slowness during shift-F5 checks.

Comment 15

[gene smith]

Here's a WIP change which allows you to set the pop3 poll order using a simple pref called server.serverX.pop3_poll_order. This new pref can be created and set for each configured pop3 server in "Config Editor" to an integer value. If not set for a pop3 server the value defaults to 0. This controls the order that each pop3 server is checked with servers with lower values occurring first and higher values last on each shift-F5 or when All servers or accounts are checked via menus. This also controls the order in the same way when checks are made at startup.

With my 3 pop3 accounts I created and set the values to this:
server.server54.pop3_poll_order 0 <--- first
server.server55.pop3_poll_order 2 <--- last
server.server48.pop3_poll_order 1 <--- middle

This doesn't affect the poll order that occurs during biff (timed checks).
If this is helpful and solves the problems, it will probably need to be added to the pop3 server settings screen to make the settings easier.

Here's the try build based on current c-c/daily. Just to be clear, this keeps the serial checking as it currently is unlike the try build of comment 12.
https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=d89696b198a92dc7e2a4583ae7ec5ccd8f1d71e9

Comment 16

[gene smith]

Attachment: t.diff

Here's some more changes. This keeps the user specified poll order for all cases better (still except for timed biff).
Note: There is some overlap with my patch to disable accounts so there are "disableAcct" vars visible. Probably not a useful patch to anyone except me.

Comment 17

[Pepi Huber]

Thanks @gene smith for the work and patches!! This sounds great!
I am far too unfamiliar with thunderbird build and code etc. to be able to make more from your post than understanding there is a patch which hopefully will make it into a published solution?

Comment 18

[gene smith]

@Pepi, before this is published for release it needs to be tested and verified that it actually fixes or at least improves your issue described in your comment 0 and in comment 10. I've tested my proposed fix and now see that the order of pop3 server polling is occurring in the order that can now be specified using preference set in Config Editor. But I've not tested with as many servers as you access (20 vs. my 3) and my servers responses are fairly quick, so, with or without my proposed patch, I don't see a big difference in Inbox folder update speed.

So I'm hoping you are willing to test the new "try" build I have here using your current profile with 20 pop3 accounts: https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=0fdf1e042a731ebb3a3d4dced20237773f33f63e&selectedTaskRun=HCGNlgdjRIW6iv18IbZJ8w.1
I don't know which platform you are running so the build includes them all. To find the appropriate installer, click the green "B" next to the platform. Then down below click on "Artifacts and Debugging" column. A list of files appears and you can pick the appropriate installer to download:
build/target.tar.xz (for linux, just expand and run the thunderbird executable inside (it's not really an installer):
build/target.dmg (for mac or mac AArch64)
build/target.installer.exe (for win32==Windows x86 Opt or win64==Windows Opt)

Based on "User Agent" in comment 0, it appears you are running linux. Here's the direct link to the "installer":
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/HCGNlgdjRIW6iv18IbZJ8w/runs/1/artifacts/public/build/target.tar.xz
To run this, I just unzip this into my ~/Downloads which creates a ~/Downloads/thunderbird directory. Then I just run the thunderbird executable like this in a terminal:
~/Downloads/thunderbird/thunderbird
Running this will use your existing tb profile and won't affect your currently installed TB release version. When done testing this you can deleted the ~/Downloads/thunderbird directory.

Of course, you may want to backup your current profile if something goes wrong.

My description for how to set the account "polling order" in comment 15 is still valid for this build.

I'm also wanting to know exactly how you have your 20 pop3 accounts configured. You said this above in comment 5:

question 2: yes, i use global inbox for all but one account, i.e. two "inboxes" - a global one, and a specific one in a different folder.

Are you saying 19 pop3 account are directed into a "global" inbox? Is this global inbox actually Local Folder's Inbox or possible the inbox of another pop3 account?
Also, how is the other (the 20th) account setup exactly? Is it just a default setup where the pop3 account Inbox is just its own Inbox?

Comment 19

[gene smith]

Ok, I am able to introduce (simulate) an extreme network delay to my pop3 servers that are not on localhost using tc command
sudo tc qdisc add dev wlp0s20f3 root netem delay 2000ms
I have one server on localhost which is not affected by this command and remains fast.
I set the poll order like this:
1 for localhost server
2 for server on internet
3 for server on lan
All servers are set up to have a messages ready to be downloaded by TB and all 3 pop3 accounts are "deferred" to Local Folders/Inbox (global inbox).
I go to Local Folder and select Inbox and click the "get mail" icon. (Note: shift-F5 does the same thing with this setup.)
The messages from localhost server come in immediately and are readable.
After several seconds the messages from server on internet come in and are readable.
After several more seconds the messages from server on lan come in a are readable.

So I conclude from this (and other testing not described) that the ordering has the desired effect and it should fix the comment 0 issue described.
Hopefully, reporter Pepi can test this too in his "real world" setup.

Comment 20

[Wayne Mery (:wsmwk)]

Pepi, are you able to test https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/HCGNlgdjRIW6iv18IbZJ8w/runs/1/artifacts/public/build/target.tar.xz ?

Comment 21

[Wayne Mery (:wsmwk)]

pepi, did you have good results with https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/HCGNlgdjRIW6iv18IbZJ8w/runs/1/artifacts/public/build/target.tar.xz ?

If you didn't get to test, we will need a new test build.