552127 - All message filters disappear permanently from interfaces, become inaccessible, but are still invoked.

Status: RESOLVED WORKSFORME

(MailNews Core :: Filters, defect)

Description

[mike montagne]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.8) Gecko/20100202 Firefox/3.5.8 (.NET CLR 3.5.30729)
Build Identifier: version 2.0.0.23 (20090812)

I have some 700 message filter rules and practically so many mail boxes. All of a sudden one day, none of the existing filters were accessible to regular interfaces. I am able to add filters, and the new filters are still honored along with the previous filters; but not being able to control filter order (because I can't access them), I now have mail going to wrong boxes constantly. It's worse than having no mail filter capability at all.

Reproducible: Always

Steps to Reproduce:
1. (I'm a software engineer.) The only thing I can associate the problem with is (evidently) a Mozilla crash. Several temporary folders were created during a background process; and evidently, after that process crashed, it left these folders, full of some 20,000 emails (it's necessary for me to keep records). I'm constantly working with my filters to streamline my system. The next time I went to work on my filters (a few minutes later), they would not display in the Message filters dialog/window. One filter only may be listed, which belongs to a separate account. This one filter is listed yet as the only filter for the other options, including Local Folders, where my several hundred filters exist.

If I add a filter, it is displayed until the dialog is closed. And it can be applied. But like all the other filters, it disappears in the next Thunderbird session.
2. So, I NEVER HAVE ACCESS TO MAY MAIL RULES ANY MORE.
3. AND THIS MEANS EVER MORE DYSFUNCTIONAL BEHAVIOR, BECAUSE IT IS CRITICAL TO CONTROL THE ORDER OF R4ULES AS WELL... and of course, without access to the rules, there's no way to control where mail is going. 
4. So now, mail is ending up in the wrong boxes by the dozens and hundreds.
Actual Results:  
No access to mail filters: 500-700 filters or so are permanently inaccessible.

Expected Results:  
Regular access to mail filters.

My theory is that either the method of storing the filter data or method of populating the filter dialog/window are at fault -- that when the background process crashed (compacting?), this somehow resulted in a corrupted data file which the dialog population method cannot handle. There must be a way to access and repair this file (instead of failing to populate the dialog).

OR, there must be a fault in the methods which populate the dialog with filter controls.

Comment 1

[mike montagne]

Please note that this issue has more views in the "Reporting Thunderbird Bugs" forum than any other issue:

http://forums.mozillazine.org/viewtopic.php?f=31&t=196958&p=8907255#p8907255

Should developers need further information, screen shots, etc. I will be glad to cooperate as time permits. This is a huge problem for me.

Comment 2

[Kent James (:rkent)]

Mike, an issue similar to this in TB2 was resolved in an earlier bug that now should be fixed in TB3. Could you upgrade to TB3 and tell me if the issue still exists?

Comment 3

[mike montagne]

Kent,

I'm very happy to report that the upgrade from TB2 to TB3 fixed this issue for me. I've had very limited time on the upgrade, so if anything re-surfaces, I'll make a new report should this case be closed, or I'll add further reports, should the case stay open for some while.

For those following this issue, faults of the previous TB2 installation had prevented (somehow disabled) automatic updates. I downloaded TB3 *and custom installed* to a separate c:\Program Files\Mozilla Thunderbird 3 directory -- distinguishing the TB3 install from the previous TB2 installation with the "3" appended to the default destination directory. This of course was to be certain the TB3 install did not overwrite previous files.

There was no more customization or attendance required. The install was very slick. It automatically imported all my mail -- including complex directories; all my formerly dysfunctional mail rules; my address books... and so forth. Even stationery settings.

If you have a lot of former email (as I do), it will take quite a while for it all to re-index, so let it run on its own until it finishes.

Great job.

Comment 4

[mike montagne]

(In reply to comment #2)
> Mike, an issue similar to this in TB2 was resolved in an earlier bug that now
> should be fixed in TB3. Could you upgrade to TB3 and tell me if the issue still
> exists?

Kent,

I'm assuming this will get to you. In any case, looks like the upgrade indeed does fix the problems. I'll try to get back to the forums and report this so others will know.

BIG THANKS!

mike

Comment 5

[mike montagne]

OOOPS. (Ouch.)

I deleted two mail filters; added one; and immediately got strange scrolling behavior in the message filters window. The message filter I added can be moved in the list, albeit very slowly, but after it was scrolled, it would no longer be displayed as if it was selected. Odd thing is, wherever it is listed in the rules, the adjacent rules appear to be selected, but not this rule. I closed the message filter window and re-opened it, and then the filter appears to be selected. Any filter I move has the same problem, and the move down and move up behavior is atrocious. Every row the moved filter transcends, the whole window is re-drawn, and the scroll bar gyrates wildly up and down.

Then I returned to this new rule, because I had to add another condition. The further condition will not be applied.

Just a guess from my end... but are we up against a redundant limitation on objects, a hash, conditions?

Looks like we still have serious problems. I'd be glad to demonstrate on TeamViewer some time.

Hope this helps.

mike

Comment 6

[Kent James (:rkent)]

"but after it was scrolled, it would no longer
be displayed as if it was selected."

Are you sure you are using the latest version of 3.0? This was an issue like this in 3.0.0 that was fixed in the current version.

Also, are you using LOTS of filters?

The filter and filter list UI really needs another major revamp. There was a major rewriting of the filter list editor (which is what you are discussing) for TB 3, but its motivations were to remove the need to use a particular backend feature, not to improve it. I've ended up having to do a lot of rework to fix regressions from that "fix".

Comment 7

[mike montagne]

Sorry to report I've got further, more serious problems with the new installation. I've got 3.0.3. Just checked for updates; TB reports none available, it will check periodically.

Yes, I have LOTS of filters. In fact it doesn't look like any of the filters I added *after* I started having problems came across in the update. Is there a limit on the objects in the list?

Anyway, by rough count, there's about 400 filters in the present list. I'm rough counting about 480 mail boxes. Both counts suggest that the new installation didn't bring everything over. By previous count (according to memory) I had more like 700 mail boxes, 550 or 600 filters -- multiple conditions in many of them.

***Worse however, I can now report that this installation is majorly awry. It won't even send any mail today (I almost never have any problems with our ISP):

When I try to send mail, a progress dialog reports that TB is trying to connect with the outgoing mail server, then that it *has* connected. Then it returns this error message:

Sending of message failed.
An error occurred sending mail: Unable to authenticate to SMTP server out.[ServerURI]. It does not support authentication (SMTP-AUTH) but you have chosen to use authentication. Uncheck 'Use name and password' for that server or contact your service provider.

I checked my account settings (for 2 accounts): both are unchecked already.

I also tried checking them and sending. Same error report. I unchecked them again and tried sending. Same error report. I could be having ISP issues, but I doubt it. I'll be trying again of course in a few hours.

Another thing I've noticed already is it's taking huge spans of time half an hour or longer just to delete a few mails from trash.

Anyway, between the terrible scrolling behavior of the filters list and these further dysfunctionalities, we have serious issues here. My personal bet is it's a memory management issue related to the filters list and/or mail box list, which is breaking a number of processes.

That's the best I can tell you for now. I was able to send mail at least from the earlier installation (which I haven't removed). So I may have to return to that until this issue is resolved.

Thanks for giving this some attention, Kent.

m

Comment 8

[mike montagne]

Rebooted system and double-checked for mail sending functionality. Still cannot send mail. Will post back tomorrow by this time if this changes. 

When I restarted TB, it went into what sounded like (from hard drive) a folder compacting function. I sent it on that yesterday; I don't know why it would resume today; but an odd thing was that it didn't report that it was performing such a process. Anyway...

m

Comment 9

[mike montagne]

BOY, it looks like I'm *really* screwed now. The former executable won't start, even when I double-click directly on the exe? Instead, the new installation starts? Any way I can fix that? I'm completely mail-less now. This is really no good.

Comment 10

[mike montagne]

Would an OSX installation likely have these same issues if we brought my data over to an OSX system?

If not, is it possible to do that?

Comment 11

[Kent James (:rkent)]

Mike - I am very sorry that you are having multiple problems, but for the purposes of bugzilla you really need to pick a single issue per bug.

Trying to sort through what you are saying, I think this bug may be moving toward claiming that the TB 3 filter list editor is usable when you have lots of filters. But I'm not really sure either. It would be helpful if you could define clearly what issue that you would like this bug to discuss. You've mentioned sending problems, migration problems, not being able to see filter after being added at least. If the goal is to help you get your installation working, that would better be covered in a support forum than bugzilla.

Comment 12

[mike montagne]

Kent, 

This may seem confusing, but it really isn't. When an end user reports "a bug," all they can do is report an evident dysfunctionality. That's all we're doing here. As you know, bugs can (and will) result in whatever possible number of ramifications. Memory leaks can affect multiple processes. Sure, one thing which goes awry can be difficult to trace to its real underlying fault; but the best an end user can do is report consequences. 

I was able to send mail until I upgraded. We have to think hard if we're going to resolve this. Users don't corrupt their data files; they don't write the code; they can't account for the faults of the code. When there's a problem, *all* the evidence of "the" problem points, somehow or another to the problem. In reporting "a bug," and end user can only report further ramifications.

So, let's be clear about this -- I'm only trying to help; if I were writing the code, I'd be evaluating the whole train of ramifications. If I recall, the most reported faults are issues related to disappearance of mail filters from the mail filter admin window ("dialog," whatever). I cannot tell you for sure if my problems started there (no end user can "identify *the* (?) bug in such a way). But the first I noticed real problems (beyond issues which were not destructive to application functionality), was with the very important dysfunctionality of my mail filters admin window. *That* (of course) is not "the bug." It's a consequence of *some* bug(s). 

We don't even know how many. So it's a misunderstanding to call this a bug report in the first place. We can't report the bug. We can only report the aberrant behavior (which is obviously a fault). I didn't have dysfunctional outgoing mail capability until I upgraded as you suggested. I could mail just fine. I had lost all control of my mail rules. Why, I can't tell you; but of course, either of us can make some intelligent deductions from which we can proceed to investigate the matter.

I've only written a few million lines of code, myself. But not this code. I don't even know how you guys arrange all this (I'd certainly be interested). From where I'm sitting, I'm wondering about some very serious issues (for such a mail client at least). Would I presume these various issues are "different bugs?" In fact, I wouldn't, because the end user reported *one* issue. The fact this precipitated into further issues with an upgrade is evidence as to what's gone awry.

I haven't made a serious attempt at even finding my data files as yet. Maybe if you wanted to share with me where the related data is kept, and how it is kept, I might be of further service to the interests of resolving this issue. 

What I'm thinking is that there is a data corruption issue. Perhaps for instance mail rules are stored in XML. Perhaps TB can freeze during a file re-write, leaving the data file corrupted permanently in such a way that the existing system is hampered as I have described; while an upgrade, having no capability to resolve the corruption, suffers further destruction from the fact(s) (?) that XML terminators are destroyed in such a way that corrupted data which originally (in TB2x) affected mostly the mail rules, now saddles TB3 with a gargantuan data field it can't handle: because of *the particular* form of corruption, it thinks it's authenticating what it is not; it isn't authenticating as it's supposed to; and it can't even interface all this correctly.

FYI, from the earliest days of Windows, I've never used Windows style ini files, in part to avoid the potential for such flaws. I've always, always, always used databases for initialization.

In any case, my first guess is we have a serious data corruption issue here. However many ramifications manifest from this issue... well, the sky is the limit, isn't it? Especially if the data storage technique is something like XML. Now, are those different bugs an end user can report? Absolutely not. We have further ramifications which may or may not be attributable to the original reported *condition* (versus "bug" -- end users don't report bugs; they report consequences of faults). 

I would say, the best thing for you folks to do would be to provide instructions for sending you somehow (I don't have mail sending capability) the data files for my system. My first step to resolve this would be to study those files, side by side for instance with the method code for displaying the mail filters in the mail filter admin window. That's where your clue is going to be to resolve this. I mean, you might immediately discover that there's no terminator for a list of rules -- that this would undermine a procedure which indeed may not crash as a consequence of the anomalous data, but simply refuses to display it, because it has no power to display infinity perhaps. I don't know; and I can't know from where I'm sitting.

Anyway, I'll help all I can; and I think I'll be able to provide damned good help for resolving this problem. But as to whether it's one bug, fifteen bugs, or whatever... no way to determine that from this end.

Unless you have a better idea, it looks to me like we have a data corruption issue. That may mean manual or automated means of recovering from the corrupted state. Or perhaps there are faults *for instance* in the filter window, where this corruption seems to manifest. One way or another, the evidence exists -- and probably in the data. If TB3 is well proven to send mail successfully (which I presume it is), then hmmm... it would seem to me that we just might have data corruption which undermines that capability. Only way to tell is to get after the data.

I can post it in this form... hopefully I can get something running so I can send... I can FTP... whatever you need for help on this, I'm ready and willing to help you. Just let me know how you want to proceed.

Best regards,

mike

Comment 13

[timeless]

bugzilla is for tracking bugs.

if you want support, please visit http://support.mozilla.com/chat

Comment 14

[mike montagne]

(In reply to comment #13)
> bugzilla is for tracking bugs.
> 
> if you want support, please visit http://support.mozilla.com/chat

Bugs cause dysfunctional application behavior. Users do not. Nor can support resolve dysfunctional application behavior caused by a bug. "Resolved, invalid," my hind quarters.

Comment 15

[Wayne Mery (:wsmwk)]

Mike, to best help Kent understand your issue, comment 12 which was 3 screens of information really needed to be no longer than a paragraph or two. You've provided so much information that no one can digest it. And you have raised a great many issues, which makes it likely that you need some assistance in addition to whatever bugs you think need reporting.

IMO this bug should be closed WORKSFORME because you wrote that the issue you first reported is working per comment 4.  Please seek help for the other issues first in support http://getsatisfaction.com/mozilla_messaging/ where more people are available to help you, and then report any outstanding problems as bugs.  See also https://developer.mozilla.org/en/Bug_writing_guidelines

Comment 16

[mike montagne]

Wayne,

If in a software engineer's opinion, that some 2/3 of former rules are restored only to visibility; and that after removing one rule, adding another, leaves the dialog dysfunctional even in generic scrolling behavior, I have no doubt why *all* these issues exist. Frankly, that you want to sweep them under the rug totally destroys any credibility of this process. There's thousands of reports of this dysfunctionality. It's one of the most popular and most reported Thunderbird threads. The new install won't even send mail. What broke that? The user, or the install process, or this bug?

Here's how I feel about it: If you don't intend to resolve such issues, then take down this forum. If you think your software has been reported fixed by the aforesaid, that's a sad testament to software engineering. If you don't want any details, how do you expect to fix anything?

Maybe if one of you wants to coach me to inspect the data files of this installation, I can find the clues needed. Otherwise, if one after another of you is going to insist you've fixed what you haven't... thanks for the warning. But that's as pathetic as pathetic gets. If this software didn't break itself, then you call this working the way it's supposed to? Won't even send mail!

Great!

Comment 17

[Ludovic Hirlimann [:Usul]]

You are seeking both help and trying to help. Thanks. Seeking help shall be done on our support forums.

Once we figured out what bugs you have we'll file the bugs and address them.

Comment 18

[Wayne Mery (:wsmwk)]

(In reply to comment #16)
> Wayne,
> 
> If in a software engineer's opinion, ... Frankly, that you want to sweep them under the
> rug totally destroys any credibility of this process. 

_clearly_, I was attempting no such thing. and if you felt my judgment was in error, rather than going on a rant all you had to do was say, "comment 4 was inaccurate and comment 5 now better describes this issue."

Granted, it is frustrating when one is faced with multiple issues. However, everyone here has tried to help you, and each issue is more effectively solved separately, and calmly. 


> There's thousands of reports of this dysfunctionality. 

hyperbole isn't helpful in resolving problems. If there are useful references, please point us to them.

Comment 19

[[e-mail]]

I too have this problem.  I have filters which are being applied to incoming messages which do not appear in the listing of filters.  Therefore, it is impossible to edit them.  And, frequently, no filter is applied.  This is simply to inform that this problem still exists, nothing more.