1143519 - "Number of articles per one XOVER command" should be limited with a threshold. Opening big newsgroup results in 100% CPU, high memory consumption, because 100,000 articles requested by single XOVER. then repeated because XOVER times out

Status: NEW

(MailNews Core :: Networking: NNTP, defect)

Description

[WADA:World Anti-bad-Duping Agency]

Opening big newsgroup takes too long in trunk, with 100% CPU, with high memory consumption.

[Build ID]
39.0a1, User Agent 	: Mozilla/5.0 (Windows NT 5.1; rv:39.0) Gecko/20100101 Thunderbird/39.0a1

[Steps to reproduce]
1. Subscribe mozilla.general of news.mozilla.org, download  all headers(93500 articles).
2. Open mozilla.general.
    Tb 31.5.0 : takes long, with , with 100% CPU, with high memory consumption.
    Trunk nightly on 2015/02/22, Tb 38.0a1pre :
         takes critically long, with , with 100% CPU, with high memory consumption.
    Trunk nightly on 2015/03/15, Tb 39.0a1pre :
         takes critically long, with , with 100% CPU, with high memory consumption.
3. If Tb is terminaated with mozilla.general selected at folder pane, it7s miserable.
    Restart of Tb takes too long, so I must kill Tb trunk nightly.
    To recover, "start with Tb 31.5.0, wait for end of open, select small local folder, terminate Tb", is needed.

"Critically long newsgroup open time" occurs even with "Work Offline mode".
So, it&s not network/NNTP relevant issue. It's issue in folder open.
O(N^2) issue in msgDBHdr processing for thread pane display?
Issue in Thread processing? 
(because of newsgroup, many Threads, deep threads with many references, exist)

Comment 1

[Wayne Mery (:wsmwk)]

Are you suggesting regression?
If not, see bug  539194, bug 185634

Comment 2

[WADA:World Anti-bad-Duping Agency]

(In reply to Wayne Mery (:wsmwk) from comment #1)
> Are you suggesting regression?

Yes, of course. I wanted to say : slowness in Tb 31.5.0 is acceptable, but severe slowness in Tb 38, 39 is never acceptaable.

Comment 3

[Alfred Peters [:infofrommozilla]]

I can confirm this Bug.

I tested with a local news-server and a group with 102k messages.

My private trunk build:

Thunderbird (32.0a1) - Daily BuildID=20140602210516
Rev: Comm-Central:9bd77606066e / Mozilla-Central:6a984e21c2ca

needs ~55 seconds to download all headers and display it.
And my next build:

Thunderbird (32.0a1) - Daily BuildID=20140605190150
Rev: Comm-Central:f6c069c7563c / Mozilla-Central:7297dedf2416

needs ~160 seconds for the same group (my actual trunk is a bit faster ~100 seconds).

Dupe of Bug 1055077?

Comment 4

[WADA:World Anti-bad-Duping Agency]

(In reply to Alfred Peters from comment #3)
> Dupe of Bug 1055077?

Problem is not observed with mozilla.test(10,000 articles) and mozilla.dev.apps.thunderbird(14,000 articles). Sufficiently quick if 10,000 articles.
If such kind of problem, I think it should be O(N) issue.
It looks for me O(N^2) problem. N=F(Number of articles,Number of threads,Depth of thread), Time required to open = O( N ^ 2 ).

Comment 5

[WADA:World Anti-bad-Duping Agency]

After some tests with Search Folder(search target=mozilla.general, VF-A: Local Search, match all, VF-B: Online Search, subject doen't contain !!!???!!!), "severe slowness in newsgroup open of mozilla.general" is not observed.
Phenomenon is following?
    After subscribe and initial header download, re-sync with server was continuously done, so open was slow.
    Re-sync completed, then open is quickly comeletes.
If so, due to Bug 1055077?

Comment 6

[Alfred Peters [:infofrommozilla]]

(In reply to WADA from comment #4)
> It looks for me O(N^2) problem. N=F(Number of articles,Number of
> threads,Depth of thread), Time required to open = O( N ^ 2 ).

I don't see O(N^2). With current trunk:

- 50k: 37 seconds
- 102k: 100 seconds

- 289k: 266 seconds

Comment 7

[WADA:World Anti-bad-Duping Agency]

Attachment: nntp:5 log for mozilla.test.multimedia(24 articles) of news.mozilla.org

After sending XOVER for 100,000 articles,
> Sending: XOVER FromArticleNumber-ToArticleNumber
> Receiving: 224 xover information follows
100,000 * NNTP_XOVER_RESPONSE are returned from server at once.
It takes pretty long, but I perhaps terminated Tb before all XOVER_RESPONSE was returned and processed, because open newsgroup didn't complete.
Then, XOVER for not-fetched-yet articles were requested upon each trial of newsgroup open.
Finally, XOVER_RESPONSE of all articles were returned and processed.

This "all articles are requested by single command at once" is already known issue, and it's produces performance issue if big newsgrouup.
This is same in imap "uid fetch aa:bb body.peek[ Headers( ... ) ]" for headers and  "uid fetch aa:bb body.peek[]" for entire mail data.
In Tb trunk, "long XOVER_RESPONSE process time with high CPU usage" was newly added.

"Number of articles per one XOVER command" is better limited to a threshold.

Comment 8

[Alfred Peters [:infofrommozilla]]

(In reply to WADA from comment #7)

> This "all articles are requested by single command at once" is already known
> issue, and it's produces performance issue if big newsgrouup.
> This is same in imap "uid fetch aa:bb body.peek[ Headers( ... ) ]" for
> headers and 

mail.imap.hdr_chunk_size limits this to (by default) 200 messages per chunk.
With this value I notice a significant slowdown on downloading a complete folder.

> "Number of articles per one XOVER command" is better limited to a threshold.

The only benefit would be, to see a result after every received chunk.
But I think, it wont speed up the download in total.

Comment 9

[WADA:World Anti-bad-Duping Agency]

(In reply to Alfred Peters from comment #8)
> > "Number of articles per one XOVER command" is better limited to a threshold.
> The only benefit would be, to see a result after every received chunk.

"Reducing memory consumption" is objective.
If "num of articles per one XOVER command" is reduced to 1/N, memory requirement to hold response is reduced to 1/N.

> But I think, it wont speed up the download in total.

But this is newsgroup, is not mail, and user requested loading of huge number of old news articles.
I believe following is practical if newsgroup.
   If NumArticlesToBeLoaded<10000, nornally open newsgroup, Else Lock the newsgroup,
   In background,  Do nn=0 to NumArticlesToBeLoaded/10000; XOVER  From_nn-To_nn; End; Sleep a while for garbage collwction; End;
   after end of batch job, unlock the newsgroup.
"Locked newsgroup for 10 to 15 minutes upon first access after subscribe" is far better than "complete UI freeze for 5 minutes".
 , 1loa,gis is

Comment 10

[Wayne Mery (:wsmwk)]

(In reply to WADA from comment #2)
> (In reply to Wayne Mery (:wsmwk) from comment #1)
> > Are you suggesting regression?
> 
> Yes, of course. I wanted to say : slowness in Tb 31.5.0 is acceptable, but
> severe slowness in Tb 38, 39 is never acceptaable.

did we never determine the source of this regression?

Comment 11

[Wayne Mery (:wsmwk)]

Walt, do you know of any large newsgroups where this could be tested?

Comment 12

[WaltS48 [:walts48]]

(In reply to Wayne Mery (:wsmwk) from comment #11)

Walt, do you know of any large newsgroups where this could be tested?

I just unsubscribed from my alt.binaries.pictures.scenic group on my Usenet-News account using version 91.4.1 on Fedora.
Selected the account and clicked Manage newsgroup subscriptions.
It takes awhile for that server to respond, but selected the newsgroup and subscribed when the group list appeared.
Downloaded 146362 messages. My System Monitor indicated Thunderbird was using about 38-40% CPU and around 500 MB of memory.
CPU and RAM usage go up when trying to navigate the Folder Pane after leaving the a.b.p.scenic folder, and Thunderbird was slow to respond. Back up around 38% and 500 MB.
Forgot I had "Delete messages more than 30 days old" set for that account.
The a.b.p.scenic folder went down to having 100 messages, and Thunderbird response is quicker.