Closed Bug 116369 Opened 23 years ago Closed 16 years ago

Large filter list gets incorrectly migrated; non-functional in Mozilla; and gets corrupt

Categories

(MailNews Core :: Filters, defect, P1)

Tracking

(Not tracked)

RESOLVED WONTFIX
mozilla1.2alpha

People

(Reporter: julien.pierre, Unassigned)

Details

Attachments

(5 files, 1 obsolete file)

I have a very large profile in Netscape Communicator 4.61 for OS/2, with about a
half gigabyte of email and 50 filters, using POP3 mail.

When I migrate it using Mozilla build 2001122019, I can view all my mail folders
the first time I bring up Mozilla, including Inbox. All folder message count is
displayed.

However, if I close mozilla and then restart it, I can't see the messages in
Inbox anymore. I can still access all other folders. However, the header message
list on the top right does not display when the Inbox folder is selected. It is
thus not possible to read the messages in Inbox. Closing and restart mozilla
after that occurs does not help.

I have noticed that in that situation, if new mail is fetched in Mozilla, the
filters are not applied. All the messages end up going to Inbox, but Inbox
cannot be read. If mozilla is closed after the new mail is stored in Inbox, then
restarted, the Inbox gets its messages counted when clicking on the Inbox
folder. Subsequently, as long as mozilla is not closed, the Inbox folder is
properly viewable, and further mail messages retrieved are filtered properly to
their respective folders. But the problem reappears if Mozilla is closed and
restarted - the Inbox can then no longer be viewed and the scenario repeats itself.

This has forced me to once again back out of Mozilla and revert to using
Communicator 4.61 for my home e-mail :(

I haven't done extensive testing yet, but it would appear that new profiles as
opposed to migrated profiles do not exhibit this same problem with Inbox display
& message filtering.
If you look in the filesystem, did your 1/2 gig of email get transferred over
correctly?
Status: NEW → ASSIGNED
The migration did take up some space and create new files, yes. I will have to
try again on a more recent build of mozilla.
I still have that problem on Mozilla build 2002012216. At the moment I can't
open my inbox in Mozilla, and the "File/Get new messages" has no effect whatsoever.
I have access to all my other migrated folders, just not Inbox, and I lost the
ability to receive any new mail. This is a showstopper for me.
Can you check your encoding and see if it is set to Arabic?
Mike,

View / Character-coding is set to Western (ISO-8859-1), not Arabic.

Right now I'm able to view the inbox. I don't know how I got to that state. I
will have to pick up a new build and migrate one more time to reproduce again.
Currently, when I retrieve mail, mozilla complains about missing folders that
apparently weren't migrated properly. So I am getting more convinced that this
problem is related to filter migration. I have 39 filters though, so I'm not
certain where the problem lies. More experimenting with this is needed and I
don't have much time right now to do it.


OK, I retested this. This is an exact step by step summary of what I did :
1) I deleted my old mozilla build
2) I grabbed mozilla build 2002012908 .
3) I migrated my only Netscape 4.61 OS/2 profile on startup
4) I brought up Mozilla. Mail came up
5) I got prompted for password and hit "cancel"
6) I went to the mail preferences to set the POP to "leave messages on server";
also set the left mail window to display folder "unread" and "total" message counts
7) I hit "get msgs" 
8) I got prompted for password again, and entered it
9) mozilla downloaded a couple messages into my Inbox, which I could read
10) I closed  mozilla
11) I started mozilla
12) mail came up, prompting for POP password
13) I entered the POP password
14) there were no new msgs, but the inbox was invisible (nothing displayed)
15) I clicked on other folders, and got a message count and message list
16) I closed mozilla
17) go back to step 11, everything repeats the same after. Inbox is no longer
visible.
In case this matters (and it well might), my 4.61 was set to come up with mail
first, rather than the browser. So was the migrated mozilla browser profile. It
might affect this bug.
FYI, in 0.9.8, I can access the inbox after migration.
However, none of the filters work. Every message ends up in the Inbox, rather
than in the folder where it's supposed to be. I looked at some of the filters in
the dialog, and they appear to be fine. I will attach a copy of my rules.dat
FYI, I copied the OS/2 4.6 profile to a Win2K partition and installed 
Communicator 4.79. The profile worked fine there. Then I installed 0.9.8 on NT. 
It didn't see the communicator profile and didn't perform any migration. I filed 
a bug at 127632. Because of this I'm unable to verify if this is OS/2-only. I 
get stuck even earlier under Windows.
Component: Mail Window Front End → Profile Migration
Summary: Unable to access INBOX after migration, odd filter and display problems in INBOX → Filters don't work after migration
This bug still exists for me in the 2002032616 build for OS/2 . In the migrated
profile, the mail filters just aren't working. Everything goes to the Inbox .
This is very frustrating and I really need this fixed or I will not be able to
migrate off communicator 4.61 for OS/2 ...
I don't have any reason to believe the bug is OS/2 specific. The rest of my
profile migrates fine. Even the filters look fine, except the rules don't work.
It may be a bug in the filter engine. Does anyone care to try the rules file I
attached and see if it works ? I'm using POP, FYI. You could setup a POP account
with "leave messages on server" and see what happens. One of the filters
migrated puts all mail to/from the netscape.com  domain into one folder. So it's
rather easy to verify if it's a migration bug or a filters bug.
OK, so I booted to NT, copied the rules.dat, and did the little pop account
setup with "leave messages on server". I created a netscape subfolder out of
inbox, and then sent a message to my address at netscape.com . When I fetched,
the message ended up in my inbox, not the netscape subfolder. So the set of
filters I have attached simply aren't working in mozilla. Please look into that.
Since it isn't OS/2-specific, I'm setting this to all platforms.
OS: OS/2 → All
Priority: -- → P1
Hardware: PC → All
Target Milestone: --- → mozilla1.0
FYI :

one of the filters at the top is for amazon.com in "to or cc". I tried to send
mail to bogus@amazon.com, bcc'ing myself. The mail did end up in my amazon folder.
But at the point of the netscape one, it fails.
So there is probably a filter somewhere half-way in my 40-filter list that the
Netscape filter engine doesn't understand and chokes on. You should be able to
determine which one.
Please update this bug with an [adt1] - [adt3] impact rating (or take it off the
list if it doesn't even rate adt3.)  Thanks!
Steve, I am the reporter and set the target, because I consider it to be a 
critical mozilla mail bug, especially given that there appears to be no filter 
logging and therefore no way to diagnose what's going on. I don't know about the 
adt[x] priority though, since I don't directly work on the client but on 
NSS, so please set it up.
Summary: Filters don't work after migration → Some filters don't work (discovered after migration)
Target Milestone: mozilla1.0 → mozilla1.2alpha
Julien, 
Based on current comments 11, 12 & 13.  I created a profile in 4.79 on my win98
system with a filter to a folder called "netscape" under my POP account Inbox. 
I sent myself a few messages that fit the filter criteria.  The messages
filtered OK.  
I launched the latest build  with -installer so it would see the 4.79 profile
that needed migrating.  It migrated it then I opened it.  I sent myself a
message that fit the criteria for the filter and it worked OK.  I will look at
your filters to see if I can reproduce the netscape filter and reproduce the
problem.
Esther,

Which filter did you specifically test ?
What I see is that some filters at the top of the list work, but after a certain
point they are ignored. For example the "netscape" filter didn't work.

That filter checks for jpierre@netscape.com in Sender or To or cc.
This used when I send myself e-mail to/from home.
I always Bcc myself on all my email since I use POP rather than use a Sent
folder. So all the messages I send from home to jpierre@netscape.com end up in
my "netscape" folder in Communicator. But in Mozilla, they end up in Inbox.

Actually, I just found the problem. When I look at the filters through "message
filters", some don't have a folder set for moving the messages to.  Only the
very first two filters are valid. "troll" is a delete filter, so it doesn't have
a folder.
"amazon" is showing a valid folder, "amazon on madbrain on pop..." .
All subsequent filters, from Sonicwall on, show the folder as: "madbrain on
pop.tsoft.net" . That's the name of my account and my pop server, but not of a
specific folder. This is why none of these filters match. If I fix the folder by
hand, then it works. I just tried with the Netscape filter by choosing the
netscape folder in the filter, then sent myself a message, and it ended up in
the proper folder. So based on this, I would say the bug is in the migration
code.






Attachment #71281 - Attachment is obsolete: true
I have just added three attachments :
- the first one is the original rules.dat which runs fine in 
Communicator 4.61 (latest communicator for OS/2 ...)

- the second one is the content of the rules.dat file in 
e:\internet\mozilla\bin\mozilla\profiles\default\nnoim9iv.slt\mail\pop.
tsoft.net immediately after migration when mozilla.exe is first started 
. As you can see, it is identical to the rules.dat from Communicator, 
so the migration program didn't touch the rules

- the third one is the content of rules.dat in 
e:\internet\mozilla\bin\mozilla\profiles\default\nnoim9iv.slt\mail\pop.
tsoft.net after first getting messages in the mail program. As you can 
see, the "amazon" filter has been modified by Mozilla to look like this 
:

actionValue="mailbox://madbrain@pop.tsoft.net/Inbox/amazon" 

All the other filters however are still the same as in Communicator, 
eg:

actionValue="INBOX.sbd/Sonicwal"

And those filters don't work.

I hope this prrovides enough information for you to track the problem 
down.
Thanks, the new info helps a lot.
Julien, 
I read your rules.dat and created a profile in Communicator 4.79 added a pop
account and added 4 filters with criteria similiar to yours.  1& 2 were
Sender/Contains, the third was Subject/Contains and the 4th Sender/Contains,
they all moved to folders under the Inbox.  
I confirmed they worked in 4.79 then ran NS6 with -installer so it would list
the newly created profile from 4.79 as needing to be migrated.  I migrated it
and opened the Profile then exited so I could open another profile to send
messages meeting the 4 filter criteria.  I then opened the migrated pop profile,
got messages and all messages filtered correctly.  I then exited and looked at
my rules.dat file and all of them showed the correct action values
(actionValue="mailbox://xyz@xyz.com/Inbox/subtwo"). 
In comments #10, #11 and #12 you mention you manually copied profiles from place
to place.  You mention 0.9.8 didn't see the profile so you couldn't migrate it.
 Later you mention in the migrated profile the filters just don't work.  Moving
profiles and rules.dat files could be part of the problem you are seeing.  I
need to think about this more to see what to try next.
Esther,

In comments 10, 11 and 12, I was trying to reproduce the problem in a Windows 
build. I didn't know if the problem was OS/2-specific or not, so I booted to my 
seldom-used Win2K partition at home. But I was completely unsuccessful at doing 
prrfile migration in Windows. Mozilla refused to recognize the presence of 
Communicator 4.79 on my Win2K partition, and therefore did not migrate any 
profiles or filters. This is why I was forced to copy the rules files manually 
in a new Mozilla profile on windows.

Since I was stuck on Windows, all my other test cases of filters were done on 
OS/2 - which is what I want to use anyway. On OS/2, Mozilla correctly detected 
the presence of Communicator 4.61, and migrated the profile including the filter 
rules.dat . I never copied the rules.dat manually on OS/2. But the filters 
didn't work after the migration. The three files I attached today were all from 
Communicator and Mozilla builds on OS/2.

Since I'm at the office right now, I think I may have more chance of getting the 
migration to work on Windows. I'll create a new Communicator 4.79 profile and 
try one more time with my filters here ...
OK, I was able to do the migration on Windows here at work.
After migration, my filters didn't work either in Mozilla.
The content of the rules.dat in 
C:\Documents and Settings\jpierre\Application 
Data\Mozilla\Profiles\madbrain\oi39lcjj.slt\Mail\pop.tsoft.net>
was corrupt. I have attached the file. There is some binary garbage at the end.

This probably means that there is a static buffer somewhere that's too small for 
my list of filters.
Attachment #77709 - Attachment description: rules.dat after migration in Mozilla 2002040409 → rules.dat after migration in Mozilla 2002040409 on OS/2
Attachment #77711 - Attachment description: rules.dat after first getting e-mail in Mozilla → rules.dat after first getting e-mail in Mozilla 2002040409 on OS/2
Attachment #77788 - Attachment description: rules.dat file after getting mail in Mozilla on Windows → rules.dat file after getting mail in Mozilla 2002040403 on Windows
What is the timeframe for the "mozilla1.2alpha" target that you set for this bug 
? I depend on this bug to be able to use mozilla for mail. Please don't tell me 
I have to wait 2 more years for that to happen ...

Note that this is not a migration issue but a filter issue - the engine is not 
able to handle the large number of filters that I have, even if they aren't 
migrated. So I'm changing the component to filters instead of profile migration.

Component: Profile Migration → Filters
Setting a more accurate description of the problem.
Hopefully this can be resolved easily and quickly. It appears that a static 
memory buffer is used to store filters in memory, and that buffer isn't large 
enough when you have many filters as I do. This is a regression from 
Communicator 4.x .
Summary: Some filters don't work (discovered after migration) → Filters don't work and get corrupt if they are numerous (50)
I finally got to the bottom of this problem.
As you can see in my previous attachments, the migration process leaves
actionValue="Inbox.sbd/folder"

Only a couple of filters end up getting correct
actionValue="mailbox://madbrain@pop.tsoft.net/Inbox/folder"

I did a search and replace in my rules.dat file of Inbox.sbd to
mailbox://madbrain@pop.tsoft.net/Inbox right after migration, but before getting
any e-mail. This made all my filters work, and I did not get any corruption in
the rules.dat file anymore. Therefore, this is primarily a problem with filter
migration.

I'd like to point out that there is still corruption in the filter code if the
search/replace is not done. I was able to reproduce corruption of some entries
in the rules.dat file before I figured out the search/replace. So there are 2
defects associated with this.
Summary: Filters don't work and get corrupt if they are numerous (50) → Large filter list gets incorrectly migrated; non-functional in Mozilla; and gets corrupt
I'd like this bug to get reconsidered for mozilla 1.0 . Since it is a migration 
bug, it will get in the way of people upgrading from 4.x . The fix for the 
migration part of the problem is probably not so involved, and it will make 
transition much smoother.
Has anyone interested in this looked to see if bug 163733 might be related?
Product: MailNews → Core
do you still see this problem?
Felix, Andrew...
dupe or gone?
Assignee: sspitzer → nobody
Status: ASSIGNED → NEW
QA Contact: esther → filters
At this point in time I doubt anyone cares if Netscape 4 migration works. 
(In reply to comment #35)
> At this point in time I doubt anyone cares if Netscape 4 migration works. 

doesn't matter to me. Just want to see the bug resolved in some manner - duped or whatever. OTOH I see there is test data above. 
perhaps ditto bug 47137
as with bug 45777...
->WONTFIX since 4.x migation has been dropped on trunk. 

If one has an overwhelming need to see this fixed on branch, please attach a patch and reopen the bug.
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → WONTFIX
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: