693668 - For newsgroups marked for offline use, automatically save bodies when downloading new headers.

Status: NEW

(MailNews Core :: Networking: NNTP, enhancement)

Description

[pnixte]

User Agent: Mozilla/5.0 (Windows NT 5.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
Build ID: 20110928134238

Steps to reproduce:

I intend to use Thunderbird as an offline NNTP reader.

I have set the NNTP account settings to "Do not delete any messages" and made sure (to the best of my knowledge and hours spent researching) that there is no hidden option that could somehow circumvent this ("Remove bodies..." is also unchecked).

I have also disabled the junk filter.



Actual results:

In the Activity manager I see that Thunderbird does not obey the "Do not delete" policy and goes on with the deletion of some messages. Version 6.0.2 Windows.


Expected results:

Thunderbird should respect my choice of not deleting ANY messages at all.

Is there an option to see specifically which messages are deleted? I only get the total number for each group.

Comment 1

[:aceman]

From where does it delete the messages? Your local offline copies of them? Can you attach screenshot of your settings. There is Offline & disk space config tab per account, There also may be some global setting in preferences. There are also per folder retention settings (see them in right-click on folder).

Comment 2

[pnixte]

Yes, it says it deletes the local offline copies.

Attached are 4 screenshots. Note that the group file in \Data\profile\News\newsserver-5 is 20MB, the .dat file is 1 Kb and the .msf is 3MB.

Comment 3

[pnixte]

Attachment: account settings

Comment 4

[pnixte]

Attachment: activity manager

Comment 5

[pnixte]

Attachment: disk space properties

Comment 6

[pnixte]

Attachment: folder properties

Comment 7

[Joshua Cranmer [:jcranmer]]

Are the newsgroups set up for offline storage?

Comment 8

[:aceman]

That would be the "select newsgroups for offline use..." button on attachment 566469 [details] ?

Comment 9

[pnixte]

Yes, they are and the offline browsing works as it should be.

Although, a funny thing I just noticed. 
In the attachment for the "item offline settings" (to be uploaded now) there is the number of unread messages next to the newsgroup name. I do not know whether this is the correct behavior or has anything critical to do with this bug.

Comment 10

[pnixte]

Attachment: items for offline use

Comment 11

[:aceman]

Does it still happen in Thunderbird 8?
Also, can you somehow determine which messages are being deleted? Is it possible they are deleted from the server and TB is just syncing to that (I am not sure this applies to newsgroups, just an idea). Do you have any other means to look on the state on the server?

Comment 12

[pnixte]

> Does it still happen in Thunderbird 8?
Unfortunately, yes, it still does. In the last comment there is (finally) the reason why this happens. But bear with me for a minute...

> Also, can you somehow determine which messages are being deleted? 
It happens to *some* (not all) old messages in *some* (not all) groups.

> Is it possible they are deleted from the server and TB is just syncing to that
Most of the messages are long expired on the server but they remain accessible locally (offline) through Thunderbird. Also I "star" all messages before closing the program and I have the option "never delete starred messages" always checked in order to use Thunderbird as an *offline* newsreader. Also, I have switched on/off the variable "mailnews.offline_sync_news" to no avail.


> Do you have any other means to look on the state on the server?
I tried snooping around the network traffic and discovered (possibly) the real reason this happens.


Here is the showdown (according to a meticulous net snooping):

- The program correctly sends a "XOVER" message for the new messages of the group
("XOVER 1112-1129") and the server responds with an Overview (headers of the particular messages).

- Then, the program sends "ARTICLE 1024" (which corresponds to an "ancient" message that should have been *already* stored the last time Thunderbird run (articles are kept on the server for 14 days and Thunderbird runs every 2-3 days) and the server responds with a "No such article". This means that Thunderbird had only the headers of the message (through the Overview response) but did not bother storing the body within the period of 14 days that the article remains on the server.
- Ditto for articles 1027, 1038, 1043 (for a total of 4 "ancient" articles) which is also what the Activity Manager reports as "Messages that were deleted".

- However, the body of article 1067 which is "ancient" as well but has not been deleted from the server is retrieved and stored normally.
- Ditto for the bodies of articles 1069, 1072, 1074 and 1081.

- Thunderbird goes on and retrieves all the bodies from the XOVER command, that is articles 1112-1129.
- However, some bodies are NOT stored in the mbox file that Thunderbird keeps locally (name_of_newsgroup). In this session, while article 1125 was correctly retrieved from the server (network dump testifies for that), its body was *not* stored in the mbox file. Thus, when I tried to read the article through Thunderbird I was presented with the message "The body of this message has not been downloaded from the server for reading offline. To read this message, you must reconnect to the network, choose Offline from the File menu and then uncheck Work Offline..."
- Ditto for articles 1113, 1114, 1117, 1118, 1120 and 1122.

- That means that I have to go "serial accessing" each and every message in the group in order to test for non-stored articles (then connect to the server in order to retrieve and store the body of the article) which is a tedious process and frankly a bit of a deal-breaker for Thunderbird use (I read hundreds of newsgroups and the serial inspection of each and every message retrieved in each and every group I subscribe to is absolutely unrealistic).


Hope to have helped pinpoint the bug.

Comment 13

[pnixte]

> - However, the body of article 1067 which is "ancient" as well but has not
> been deleted from the server is retrieved and stored normally.
> - Ditto for the bodies of articles 1069, 1072, 1074 and 1081.

In this part, I mean that Thunderbird sends to the server the commands "ARTICLE 1067", "ARTICLE 1069", "ARTICLE 1072", "ARTICLE 1074" and "ARTICLE 1081", before it goes and retrieves the bodies of the articles from the "XOVER" command.

Comment 14

[:aceman]

Thanks for the thorough examination. I'll try to move the bug to the appropriate component.

Comment 15

[Wayne Mery (:wsmwk)]

Joshua, thoughts?  confirmed bug?

Comment 16

[pnixte]

Waiting for Joshua's input, I would like to add that this bug still persists in Thunderbird 13.0.1.

Comment 17

[Wayne Mery (:wsmwk)]

(In reply to pnixte from comment #16)
> Waiting for Joshua's input, I would like to add that this bug still persists
> in Thunderbird 13.0.1.

Comment 18

[Joshua Cranmer [:jcranmer]]

It sounds like the real problem is that we don't automatically download bodies for offline retention in newsgroups, which I don't think we've ever supported, but which I think is worthwhile to support.

Comment 19

[Hartmut Welpmann [:welpy-cw]]

(In reply to Joshua Cranmer [:jcranmer] from comment #18)

It sounds like the real problem is that we don't automatically download
bodies for offline retention in newsgroups, which I don't think we've ever
supported, but which I think is worthwhile to support.

Yes, "Select this newsgroup for offline use" might be misleading, as it does not mean that bodies will be saved as well when new headers are fetched.