40817 - localstore.rdf grows very big with international characters in mail folders

Status: VERIFIED FIXED

(Core Graveyard :: RDF, defect, P3)

Description

[Robert Kaiser]

I've got a mail folder called "Local Folders/Old Mail/ÖVP" which seems to be the
source of a problem I ran into with localstore.rdf - it grew up to over 16 MB
with multiple calls of mailnews!

The following are lines I extracted from a 2 MB version of localstore.rdf:

<RDF:Description about="chrome://messenger/content/messenger.xul">
<NC:persist resource="mailbox://nobody@Local%20Folders"/>
<NC:persist resource="mailbox://nobody@Local%20Folders/Old Mail"/>
<NC:persist resource="mailbox://nobody@Local%20Folders/Old Mail/Ã?Â?Ã?Â[...]?Â?
Ã?Â?VP"/>

where [...] is to be replaced by about 250K of that "Ã?Â?Ã?Â?Â?Ã?Â" crap.
I'll attach a 31K zip of that 2M version of localstore.rdf

Seen with multiple nightly M16 builds, last of them was 2000052615 (win32).

Note that Mozilla is able to start up and work properly with that localstore.rdf
files, it only takes ages to launch.

Comment 1

[Robert Kaiser]

Attachment: zip of my 2MB localstore.rdf

Comment 2

[nhottanscp]

reassign to putterman

I haven't seen this problem, working fine before 2000052615?

Comment 3

[Robert Kaiser]

That did happen with a few builds for me after M15... could also be that it's 
generated if I want to use the right-click popup menuitem "Move To" to move a 
message from Inbox to that folder, or if I try to do the same via d&d. That's 
the only actions I've done to that folder (or its subfolders) afaik.

Comment 4

[scottputterman]

Can you try starting with a new profile and seeing if this happens on a current 
build?

Things that make a folder appear in localstore.rdf include changing to threaded 
mode, changing the sort, or opening the folder if it has subfolders.

Comment 5

[Robert Kaiser]

putterman, yes, it's still in recent builds. Verified again with 2000-05-29-08.
I now realized that's it's not going from almost zero to 16M in one step...

If it gests into localstore.rdf once (I made a "ÖVP" folder with two subfolders
and viewed them), the crappy code doubles (or in some other form multiplies)
with every call of Mozilla, even browser!
So once it's in, it grows and grows almost "forever"...

Comment 6

[scottputterman]

I created the same folder that you listed below and I can't seem to reproduce
this.  cc'ing waterson in case he has any ideas.  cc'ing momoi to see if he can
reproduce this. If this is easily reproduced on I18N machines then this should
probably be an nsbeta2 bug. But I'd like to find that out before doing that.

Comment 7

[Robert Kaiser]

OK, tried again in 2000060308-win32.

Exact steps to reproduce:
1) start Mozilla -ProfileManager
2) Select "Create Profile" and create a nerw Profile (used the default settings)
3) Launch Mozilla
4) Open up Mozilla Mail
5) create an existing Mail account (used "foo" settings)
6) click the twisty to open up "Local Mail" folders in sidebar
7) right-click on "Local Mail", selcet "New Folder"
8) Enter "ÖVP" as folder name (be sure you enter "Ö" <iso-8559-1: #214>)
9) Click on the new folder in sidebar
10) Quit Mozilla
11) View localstore.rdf and see "<NC:persist 
resource="mailbox://nobody@Local%20Folders/ÖVP"/>" somewhere in the file
12) Launch Mozilla again with that Profile (it's enough to launch Browser)
13) Quit Mozilla again
14) View localstore.rdf again and see the line now telling "<NC:persist 
resource="mailbox://nobody@Local%20Folders/ÖVP"/>"

it's growing everytime you launch Mozilla again.

Comment 8

[Chris Waterson]

this is a problem with nsLocalStore or the RDF/XML serializer. stealing from 
putterman.

Comment 9

[Makoto Kato [:m_kato]]

This issue ocuurs "file://" of IE Import Favorties other than mailbox, too.
Chris, is charset of localstore.rdf UTF-8??

Comment 10

[Chris Waterson]

Hmm. I don't know if there is *any* encoding set for this file. See

http://lxr.mozilla.org/mozilla/source/rdf/datasource/src/nsLocalStore.cpp#355

which is the code that first creates the file. That said, shouldn't the file be
read and written using some sane encoding? I'm suspicious that there may be
other code that's not I18n-friendly along the chain that writes out the file...

Comment 11

[Makoto Kato [:m_kato]]

This problem is the following line.
http://lxr.mozilla.org/seamonkey/source/rdf/base/src/nsRDFXMLDataSource.cpp#1303
http://lxr.mozilla.org/seamonkey/source/rdf/base/src/nsRDFXMLDataSource.cpp#1394

But I don't know good fix code...
And uri of mailbox will change by bug 39757.

Comment 12

[Chris Waterson]

makato: what do you think the problem is?

Comment 13

[Katsuhiko Momoi]

Hi, can we do something about this bug soon?
It's preventing my work. In the last 2-3 days, I had to trash
2 profiles for Local Folder testing because the memomry consumption
gets to be 100% of my 256MB RAM!
I see that my localstroe.rdf file is now 36 MB.
I'm designatingt his as a dogfood bug because it greatly
hamper my local folder testing work. 

Comment 14

[Jay Patel [:jay]]

Putting on [dogfood+] radar. 

Comment 15

[Katsuhiko Momoi]

I am going to nominate this for nsbeta2 because it is happening to
me all too often when I am doing international test involving
Japanese, Latin 1 or UTF-8 messages. 
I didn't realize until the last several days that what was causing
interminable slowdown such as waiting for a window to open
for nearly 5 minutes was due to this bug.

According to Scott P., growth of localstore.rdf could be due to
changing to threaded mode, changing sort order, and opening
sub-folders, etc. These are pretty common activities. 

I would like to know what other activities could be contributing
to this growth because I am not necessarily doing these things but
yet managed to grow localstore.rdf to 15MB or more in less
than 2 hours of normal use. This slows down all other applications on
my 256MB RAM Windows to a point I cannot do anything but terminate
Mozilla before doing anything else.
I don't think average users will ever figure out what is wrong
and probably stop using this product immediately. 

I don't think we can ship PR2 with this bug.
 

Comment 16

[bobj]

RDF (like XML) by default is encoded in UTF-8 (unless there is a Unicode Byte
Order Mark at the begining of the file then it is UTF-16), so the
localestore.rdf should be encoded in UTF-8.  By convention, all the internal
Seamonkey XML, XUL, and RDF should be in UTF-8.

The original string, ÖVP, looks like correct UTF-8 for "ÖVP", if rendered as
Latin1.  The expansion looks suspiciously like the UTF-8 string was mistakenly
run through a Latin1-to-UTF8 converter.  So I'd suggest looking at how we are
 converting the strings.

Comment 17

[Jay Patel [:jay]]

Putting on [dogfood+][nsbeta2+] radar. 

Comment 18

[Chris Waterson]

Attachment: use appropriate string conversion routines on outbound text

Comment 19

[Chris Waterson]

cc'ing rjc for some r= lovin'.

rjc: see the attached patch. I've changed all the stuff that assumed that URLs
were encoded in ASCII to assume that URLs are encoded in UTF-8. The problem was
that we would take a UTF-8 encoded value, widen the bytes "as is" to UCS-2 (or
UTF-16, or whatever) without decoding it, and then write it out. Doing this over
and over would basically double the number of characters in a URL for each
write. And that gets to be big pretty quick!

Comment 20

[Robert John Churchill]

Looks good.  r=rjc

Comment 21

[Chris Waterson]

fix checked in

Comment 22

[Katsuhiko Momoi]

OK. I have been using the same environment which caused a huge
bloat in the past doing the same sorts of things which led to
the slowdwon problem. I have not run into this problem after
10 days or so.
So as imprecise as this would seem, in the absence of precise steps,
I think this is good enough. 
Marking it verified as fixed.