localstore.rdf grows very big with international characters in mail folders

VERIFIED FIXED in M18

Status

()

Core
RDF
P3
normal
VERIFIED FIXED
18 years ago
18 years ago

People

(Reporter: Robert Kaiser, Assigned: Chris Waterson)

Tracking

Trunk
x86
Windows 98
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [dogfood+][nsbeta2+] FIX IN HAND)

Attachments

(2 attachments)

(Reporter)

Description

18 years ago
I've got a mail folder called "Local Folders/Old Mail/ÖVP" which seems to be the
source of a problem I ran into with localstore.rdf - it grew up to over 16 MB
with multiple calls of mailnews!

The following are lines I extracted from a 2 MB version of localstore.rdf:

<RDF:Description about="chrome://messenger/content/messenger.xul">
<NC:persist resource="mailbox://nobody@Local%20Folders"/>
<NC:persist resource="mailbox://nobody@Local%20Folders/Old Mail"/>
<NC:persist resource="mailbox://nobody@Local%20Folders/Old Mail/Ã?Â?Ã?Â[...]?Â?
Ã?Â?VP"/>

where [...] is to be replaced by about 250K of that "Ã?Â?Ã?Â?Â?Ã?Â" crap.
I'll attach a 31K zip of that 2M version of localstore.rdf

Seen with multiple nightly M16 builds, last of them was 2000052615 (win32).

Note that Mozilla is able to start up and work properly with that localstore.rdf
files, it only takes ages to launch.
(Reporter)

Comment 1

18 years ago
Created attachment 9239 [details]
zip of my 2MB localstore.rdf

Comment 2

18 years ago
reassign to putterman

I haven't seen this problem, working fine before 2000052615?
Assignee: nhotta → putterman
(Reporter)

Comment 3

18 years ago
That did happen with a few builds for me after M15... could also be that it's 
generated if I want to use the right-click popup menuitem "Move To" to move a 
message from Inbox to that folder, or if I try to do the same via d&d. That's 
the only actions I've done to that folder (or its subfolders) afaik.

Comment 4

18 years ago
Can you try starting with a new profile and seeing if this happens on a current 
build?

Things that make a folder appear in localstore.rdf include changing to threaded 
mode, changing the sort, or opening the folder if it has subfolders.
(Reporter)

Comment 5

18 years ago
putterman, yes, it's still in recent builds. Verified again with 2000-05-29-08.
I now realized that's it's not going from almost zero to 16M in one step...

If it gests into localstore.rdf once (I made a "ÖVP" folder with two subfolders
and viewed them), the crappy code doubles (or in some other form multiplies)
with every call of Mozilla, even browser!
So once it's in, it grows and grows almost "forever"...

Comment 6

18 years ago
I created the same folder that you listed below and I can't seem to reproduce
this.  cc'ing waterson in case he has any ideas.  cc'ing momoi to see if he can
reproduce this. If this is easily reproduced on I18N machines then this should
probably be an nsbeta2 bug. But I'd like to find that out before doing that.
(Reporter)

Comment 7

18 years ago
OK, tried again in 2000060308-win32.

Exact steps to reproduce:
1) start Mozilla -ProfileManager
2) Select "Create Profile" and create a nerw Profile (used the default settings)
3) Launch Mozilla
4) Open up Mozilla Mail
5) create an existing Mail account (used "foo" settings)
6) click the twisty to open up "Local Mail" folders in sidebar
7) right-click on "Local Mail", selcet "New Folder"
8) Enter "ÖVP" as folder name (be sure you enter "Ö" <iso-8559-1: #214>)
9) Click on the new folder in sidebar
10) Quit Mozilla
11) View localstore.rdf and see "<NC:persist 
resource="mailbox://nobody@Local%20Folders/ÖVP"/>" somewhere in the file
12) Launch Mozilla again with that Profile (it's enough to launch Browser)
13) Quit Mozilla again
14) View localstore.rdf again and see the line now telling "<NC:persist 
resource="mailbox://nobody@Local%20Folders/ÖVP"/>"

it's growing everytime you launch Mozilla again.
(Assignee)

Comment 8

18 years ago
this is a problem with nsLocalStore or the RDF/XML serializer. stealing from 
putterman.
Assignee: putterman → waterson
Product: MailNews → Browser
(Assignee)

Updated

18 years ago
Status: NEW → ASSIGNED
Component: Internationalization → RDF
Target Milestone: --- → M18

Comment 9

18 years ago
This issue ocuurs "file://" of IE Import Favorties other than mailbox, too.
Chris, is charset of localstore.rdf UTF-8??
(Assignee)

Comment 10

18 years ago
Hmm. I don't know if there is *any* encoding set for this file. See

http://lxr.mozilla.org/mozilla/source/rdf/datasource/src/nsLocalStore.cpp#355

which is the code that first creates the file. That said, shouldn't the file be
read and written using some sane encoding? I'm suspicious that there may be
other code that's not I18n-friendly along the chain that writes out the file...
(Assignee)

Comment 12

18 years ago
makato: what do you think the problem is?

Comment 13

18 years ago
Hi, can we do something about this bug soon?
It's preventing my work. In the last 2-3 days, I had to trash
2 profiles for Local Folder testing because the memomry consumption
gets to be 100% of my 256MB RAM!
I see that my localstroe.rdf file is now 36 MB.
I'm designatingt his as a dogfood bug because it greatly
hamper my local folder testing work. 
Keywords: dogfood

Comment 14

18 years ago
Putting on [dogfood+] radar. 
Whiteboard: [dogfood+]

Comment 15

18 years ago
I am going to nominate this for nsbeta2 because it is happening to
me all too often when I am doing international test involving
Japanese, Latin 1 or UTF-8 messages. 
I didn't realize until the last several days that what was causing
interminable slowdown such as waiting for a window to open
for nearly 5 minutes was due to this bug.

According to Scott P., growth of localstore.rdf could be due to
changing to threaded mode, changing sort order, and opening
sub-folders, etc. These are pretty common activities. 

I would like to know what other activities could be contributing
to this growth because I am not necessarily doing these things but
yet managed to grow localstore.rdf to 15MB or more in less
than 2 hours of normal use. This slows down all other applications on
my 256MB RAM Windows to a point I cannot do anything but terminate
Mozilla before doing anything else.
I don't think average users will ever figure out what is wrong
and probably stop using this product immediately. 

I don't think we can ship PR2 with this bug.
 
Keywords: nsbeta2

Comment 16

18 years ago
RDF (like XML) by default is encoded in UTF-8 (unless there is a Unicode Byte
Order Mark at the begining of the file then it is UTF-16), so the
localestore.rdf should be encoded in UTF-8.  By convention, all the internal
Seamonkey XML, XUL, and RDF should be in UTF-8.

The original string, ÖVP, looks like correct UTF-8 for "ÖVP", if rendered as
Latin1.  The expansion looks suspiciously like the UTF-8 string was mistakenly
run through a Latin1-to-UTF8 converter.  So I'd suggest looking at how we are
 converting the strings.

Comment 17

18 years ago
Putting on [dogfood+][nsbeta2+] radar. 
Whiteboard: [dogfood+] → [dogfood+][nsbeta2+]
(Assignee)

Comment 18

18 years ago
Created attachment 11623 [details] [diff] [review]
use appropriate string conversion routines on outbound text
(Assignee)

Comment 19

18 years ago
cc'ing rjc for some r= lovin'.

rjc: see the attached patch. I've changed all the stuff that assumed that URLs
were encoded in ASCII to assume that URLs are encoded in UTF-8. The problem was
that we would take a UTF-8 encoded value, widen the bytes "as is" to UCS-2 (or
UTF-16, or whatever) without decoding it, and then write it out. Doing this over
and over would basically double the number of characters in a URL for each
write. And that gets to be big pretty quick!
(Assignee)

Updated

18 years ago
Whiteboard: [dogfood+][nsbeta2+] → [dogfood+][nsbeta2+] FIX IN HAND
Looks good.  r=rjc
(Assignee)

Comment 21

18 years ago
fix checked in
Status: ASSIGNED → RESOLVED
Last Resolved: 18 years ago
Resolution: --- → FIXED

Comment 22

18 years ago
OK. I have been using the same environment which caused a huge
bloat in the past doing the same sorts of things which led to
the slowdwon problem. I have not run into this problem after
10 days or so.
So as imprecise as this would seem, in the absence of precise steps,
I think this is good enough. 
Marking it verified as fixed.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.