1893307 - POP3 with lots of messages on server gets stuck inside LineReader.sys.jsm doing spinEventLoopUntilEmpty() and times out

Status: RESOLVED FIXED

(MailNews Core :: Networking: POP, defect)

Description

[gene smith]

Bug 1875633 added sending NOOPs to avoid server timeouts when dealing with server responses with lot of "lines" of data. But recently I've noticed that this doesn't always help and NOOPs are never sent because code goes into this and may never come out:
https://searchfox.org/comm-central/rev/769985a1e178315c97985c1374c0e45f1bdb5ee9/mailnews/base/src/LineReader.sys.mjs#67-70

I've noticed this a lot while testing Bug 1847137 when I've shutdown TB while selected on the "root" of a POP3 account with 35k messages on the server. I then start TB and check for new messages (f5) on that POP3 account while seeing in "preview/thread" pane only some account info. Since POP3 logging is enabled and I'm running a debug build started from a terminal, I can see that LIST command is sent and the response comes in but no further activity occurs until a timeout messages pops up (this is after the typical 1 minute timeout allowed by POP3 servers). Also, no NOOP is sent during this time.

This is consistent over many TB startups.

But I notice that while waiting for the LIST processing to finish, if I click inside the "preview/thread" pane or do any other UI activity that a NOOP is sent and the LIST processing quickly completes and then the UIDL processing starts and is received. In a similar way, the UIDL processing hangs and can be "un-hanged" by doing some UI activity. Actually, the UIDL processing, after triggering the LIST processing to finish, does finish without additional UI activity.

When I comment out these lines:
https://searchfox.org/comm-central/rev/769985a1e178315c97985c1374c0e45f1bdb5ee9/mailnews/base/src/LineReader.sys.mjs#67-70
and rebuild and do the same things, there is no hang processing the LIST and UIDL responses.

Magnus suggested changing this to use "setTimeout" here bug 1875633 comment 51 which was tried then but the bug reporter didn't see that it made a difference. However, from what I can tell, completely removing these 4 lines fixes the problem I'm reporting here and I don't see any UI response issues with this gone. This conditional call to spinEventLoopUntilEmpty was put in pre-supernova (toward the end of 102) when apparently UI code had lower "priority" and it was easier to block the UI with mail protocol/network activities. It appears that now, with supernova changes, the UI is more likely to block the mail protocol (e.g., fast scrolling using scroll bar will completely stop network sending).

Comment 1

[gene smith]

Attachment: Bug 1893307 - Avoid POP3 server timeout processing LIST and UIDL lines. r=mkmelin

...

Comment 2

[gene smith]

Attachment: Bug 1893307 - Avoid POP3 server timeout processing LIST and UIDL lines. r=mkmelin

With 38k message on POP3 server, setTimeout occurs 33 times during
LIST and 33 times during UIDL line processing with this change.
No observable UI slowdown is detected with loop count at 1000.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/4a6272605636
Avoid POP3 server timeout processing LIST and UIDL lines. r=mkmelin

Comment 4

[Magnus Melin [:mkmelin]]

This is causing perma oranges for at least
comm/mailnews/news/test/unit/test_nntpGroupPassword.js
comm/mailnews/news/test/unit/test_xover.js
comm/mailnews/news/test/unit/test_server.js

Comment 5

[gene smith]

Attachment: Bug 1893307 - Follow-up to fix some failing tests. r=mkmelin

Note: This change was put into the main comment 2 commit.
Avoid test error "FAIL - A promise chain failed to handle a
rejection" due to LineReader function "read()" now async.
In NntpClient.sys.mjs don't use runningUri when it is null..

Comment 6

[Magnus Melin [:mkmelin]]

Backout: https://hg.mozilla.org/comm-central/rev/400a75fcadc14b418bea7e87dc78b54aaa46ecd7

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/18b7b86cd412
Avoid POP3 server timeout processing LIST and UIDL lines. r=mkmelin,babolivier

Comment 8

[Wayne Mery (:wsmwk)]

Comment on attachment 9402034 [details]
Bug 1893307 - Avoid POP3 server timeout processing LIST and UIDL lines. r=mkmelin

[Triage Comment]
Approved for beta

Comment 9

[Daniel Darnell [:dandarnell]]

Thunderbird 127.0b4:
https://hg.mozilla.org/releases/comm-beta/rev/4ebc65ceddbe