534539 - RSS feeds configured for offline use duplicate the entire feed whenever the RSS feed is checked

Status: RESOLVED DUPLICATE of bug 258465

(MailNews Core :: Feed Reader, defect)

Description

[Vebyast]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.15) Gecko/2009102815 Ubuntu/9.04 (jaunty) Firefox/3.0.15
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0

Last night I told Thunderbird to download messages for offline use, just to see what would happen. Now whenever Thunderbird checks for new items in RSS feeds, it downloads the entire feed and marks every newly downloaded duplicate as a new post. This happens even when an unread duplicate is present.

Reproducible: Always

Steps to Reproduce:
1. Go to edit->preferences->advanced->network and disk space->offline and set "Download messages for offline use when going offline?"
2. Get RSS feeds.
Actual Results:  
A copy of every item in the RSS feed appears in the feed's folder. Each one is marked as unread.

Expected Results:  
Not downloaded duplicate RSS items.

I am running a dual-boot Windows 7 and Ubuntu linux (9.04, kernel 2.6.28-17-generic), with Thunerbird 3 on both operating systems. They share a folder (created by TB2) that's stored in the standard TB location on the windows partition (c:\Users\$(username)\AppData\Roaming\Thunderbird\Profiles). Windows accesses it normally, and linux accesses it using ntfs-3g (/media/disk/$(...)). I haven't yet tested whether this behaviour is also present in windows.

Comment 1

[Vebyast]

I just restarted my computer to take care of a new install, and now the problem is not dependent on TB being told to download mail for offline use. It is occurring at all times. Simply tell an RSS feed to check for updates and downloads and begins indexing the entire feed. This has made the RSS feeds basically unusable. For example, accidentally checking http://www.giantitp.com/comics/oots.rss grinds my computer to a halt for nearly two minutes while TB downloads and indexes 800+ messages.

Comment 2

[Ludovic Hirlimann [:Usul]]

Attachment: Screenshot

Comment 3

[Ludovic Hirlimann [:Usul]]

(In reply to comment #1)

> http://www.giantitp.com/comics/oots.rss grinds my computer to a halt for nearly
> two minutes while TB downloads and indexes 800+ messages.

I don't have it on this feed but I'm seeing that on another feed of mine -> marking new based on the symptoms.

Error console is empty.

Comment 4

[Ludovic Hirlimann [:Usul]]

Myk what would you need to be able to debug this ?

Comment 6

[Vebyast]

Didn't think to check the error console. I get one of the following whenever Thunderbird downloads a new set of duplicates.

Error: not well-formed
Source File: file:///media/disk/Users/$(username)/AppData/Roaming/Thunderbird/Profiles/77agdh2r.default/Mail/News%20&%20Blogs/feeditems.rdf
Line: 5, Column: 34
Source Code:
                   urn:forumzilla:stored="true"


This error only occurs when checking the feed generates duplicate items. If I clear the feed's folder and check the feed, this error does not appear.

Comment 9

[Myk Melez [:myk] [@mykmelez]]

(In reply to comment #4)
> Myk what would you need to be able to debug this ?

Sorry for the delay replying.  Primarily I just need time and attention. :-(

My Thunderbird-created feeditems.rdf file (at [ProfD]/Mail/News & Blogs-3/feeditems.rdf) uses "fz:stored" rather than "urn:forumzilla:stored" attributes, where the "fz" prefix is declared in this xmlns declaration on the root node in the file: |xmlns:fz="urn:forumzilla:"|.

So perhaps this is some kind of bug in the serialization of the feeditems.rdf file in which the serializer prepends the namespace rather than its prefix to the "stored" attribute.

Axel: have you seen this happen before?

Comment 10

[Axel Hecht]

Switching over to my RDF-me.

No, haven't seen anything like that. Sounds like a good tangent to create some testcases, though. In particular, round-tripping through the xml parser and serializer would be worthwhile.

Or a pointer to the code that serializes, maybe the DS doesn't keep the namespace mappings around.

Comment 11

[Myk Melez [:myk] [@mykmelez]]

The serialization is just nsIRDFRemoteDataSource::Flush:

http://mxr.mozilla.org/comm-central/source/mailnews/extensions/newsblog/content/Feed.js#490

Initial file creation happens through the writing of plain text to the file, though (although I can't think of how this might cause problems):

http://mxr.mozilla.org/comm-central/source/mailnews/extensions/newsblog/content/utils.js#351

Comment 12

[Axel Hecht]

What's the implementation of the nsIRDFRemoteDataSource::Flush? Plain nsRDFXMLDataSource.cpp?

Comment 13

[Axel Hecht]

... and it'd be good to see an actual feeditems.rdf file that exposes the problem.

Comment 14

[Vebyast]

Attachment: Broken $(profile)/Mail/News & Blogs/feeditems.rdf

Comment 15

[Mark Banner (:standard8)]

Vebyast: do you know if this is a regression from Thunderbird 2?

Comment 16

[Axel Hecht]

I could load and FlushTo a sanitized version of feeditems.rdf allright. No idea where we'd loose the prefix.

Comment 17

[Vebyast]

Partly. In TB2, I did have a much more limited form of the problem. Only part of one of the feeds would be duplicated (usually about a third), and it didn't happen very often (at a very rough estimation, once every few hundred get mail operations). I updated to TB3 I believe one day after it came out; this problem came up I think two days later.

Comment 18

[Vebyast]

Also, something I just thought to confirm: the problem is present with brand-new RSS feeds, and with RSS feeds that have been deleted and recreated in the same location without removing already-downloaded items.

Comment 19

[Dan Mosedale (:dmosedale, :dmose)]

After discussion with Standard, we don't believe this bug meets either of our current blocker criteria:

a) make the upgrade experience from TB2 very painful for a large number of users

or 

b) be a new, reproducible, severe quality issue (eg dataloss, frequent crashes)

particularly because it's not the default option setting, and the problem can be made to go away by unsetting that option, so marking blocking- and wanted+.

That said, a patch would be great here, and if there's not time to fix or workaround the core RDF bug, just disabling the "offline" setting for feeds would be fine.

Adding mkmelin in case he's interested in taking a run at it.

Comment 20

[Leho Kraav (:macmaN @lkraav)]

i have just run into this with a profile that has originated from TB2-time. rss feeds had accumulated tens of thousands of items (duplicates) and the user stopped paying attention and also didn't think to report the issue.

mv feeditems.rdf feeditems.rdf.old seems to make TB3 regenerate the file and stop duplicating items on feed checks. can someone confirm is this the user-side solution to the issue?

Comment 21

[alta88]

duping this to bug bug 258465.

see this: 
https://bugzilla.mozilla.org/show_bug.cgi?id=608520#c5