Closed Bug 116443 Opened 23 years ago Closed 19 years ago

Deleted inbox after receiving virus infected mail (McAfee, Norton, AntiVir)

Categories

(MailNews Core :: Backend, defect)

Product:
MailNews Core
Component:
Backend
Platform:
x86
Windows 2000
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: rocky-beach, Assigned: Bienvenu)

References

(Depends on 1 open bug,
URL
)

Details

(Keywords: dataloss, Whiteboard: [asaP1])

Attachments

(2 files)

add (currently) hidden pref "mailnews.downloadToTempFile" which makes pop3 download each message to a file before appending to inbox
24.81 KB, patch
mscott
: superreview+
Details | Diff | Splinter Review
fix multiple message case
1.06 KB, patch
mscott
: superreview+
Details | Diff | Splinter Review 
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.7+)
Gecko/20011220
BuildID:    2001122003

Today for the third time my inbox was emptied by receiving a mail which was
infected by the W32/Badtrans@MM virus.

Reproducible: Always
Steps to Reproduce:
1. Some mails are in the inbox.
2. I receive an infected mail. My NAV software pops up and warns me.
3. 

Actual Results:  The messages in the inbox are still displayed but the Inbox
file has file size 0 (some information of the inbox mails are still stored in
Inbox.msf) so I think they are displayed in the inbox, but it is impossible to
delete these messages. I had this problem (first time) also with the build from
17th of Dezember.

Expected Results:  Mails in the inbox should not be deleted.
Navin, don't you have a dup of this?
Assignee: bienvenu → naving
Status: UNCONFIRMED → NEW
Component: Mail Database → Mail Back End
Ever confirmed: true
I'm pretty sure this is a dup also.  There's another bug out there with the
suggestion that we store our attachments outside of the mail file so that in the
worst case scenario just that attachment would be deleted.
Status: NEW → ASSIGNED
Now I have more information. NAV software moves  - after receiving infected
mails - the now infected Inbox file to quarantine.
I think mozilla should prevent this while running.
Just had this hit a user I had converted over to Mozilla for his E-mail.  Was
running InoculateIT for anti-virus protection.  There were reports of BadTrans
in the AV log files, but unfortunately no dates attached.  The entire inbox got
wiped out.  No real indication of what all happened.

From what the user told me, he had just click on Get Messages while in the
process of dragging older messages to folders outside his inbox to be filed. 
Prior to even seeing what had been sent, the inbox went blank.  I went and
checked his profile folder, and sure enough the inbox.sbd file showed a file
size of zero. Don't know what to tell him at this point.

Personally, I would concur with putterman's opinion that attachments be stripped
off an E-Mail prior to being stored in the inbox.  Prior to my seeing this bug I
would have thought of that as a usability issue.  Now it seems that this
suggestion should be thought of as a security or stability issue.

I've moved the severity of this bug to blocker and added "dataloss" to the
keywords.  I did so as I can't imagine how much worse a bug can get than
actually losing so much critical stored data so quickly.  I would further
recommend that a target milestone be established for this due to the extreme
severity.
Severity: major → blocker
Keywords: dataloss
Same user, got hit again wiping out everything in his inbox.  This may be 
taken as a flame, but I'm moving this user off of Mozilla mail to something 
more stable.  This user has lost significant data twice over now.  The second 
time was doing nothing more than changing which folder was in view.

I wish I had more info to provide, but that's pretty much all I could dig out. 
By the time I got to the box the data was lost beyond recovery.  Based on what 
this user described this was not a virus issue, but a database corruption 
issue.  No reliable way to reproduce that I can see.  It just decides to blow 
up every week or so with unread mail permanently destroyed.

Now I need to figure out how to get the mail out of Mozilla and onto something 
like Eudora or Pegasus.  I just can't allow this user to trust his E-Mail to 
Moz any longer.


I don't think that it's a good idea to store attachments separate from the rest
of the mails. This would violate the mbox file format and would make restoring
the original mail (for export, proof, external processing, whatever) harder or
impossible.

If it's really the anti-virus software deleting the folders, I'd said that it's
a severe bug in *that* AV software, not Mozilla. How could Mozilla possibly
prevent that?
Eudora has been storing attachment files seperate from the messages for at 
least the last 2 major versions now.  The folder for these files is placed by 
default in a folder along with the message body files.  For backing up, you'd 
back up the entirety of the /Mail folder, just like you would with Moz or NS 
4.7x.  All the attachments go with.

If an attachment no longer exists in that folder, Eudora notes the fact the 
file is no longer there, and retains the message body.  Without exception, 
everyone I have ever personally discussed this with would LOVE to be able to 
remove an attachment from an E-Mail while retaining the original body text.  
That of course would be a usability issue, where as this is an integrity bug.

As to how Mozilla should have prevented this from happening, much as I have 
already stated.  By removing the attachment before entering the mail into the 
InBox no virus protection software would feel compelled to muck about with 
mail folders.  The AV software is left scanning only the attachments, thus no 
conflict is possible.

The alternative is to have every AV vendor be fully aware of every mail client 
vendor's method of storing mail.  Conversely, Mozilla would need to be aware 
of every AV vendor's methods as well.  This would simply lead to finger 
pointing on both sides as to who was at fault.  All the while, users are 
suffering data loss.

All I would seriously ask of any developer reading this bug is to give a hard 
look at how Eudora handles attachments.  It's quite a popular feature amongst 
it's users, and insures a safe seperation between the message body and what 
needs to be scanned by AV products.


This is an antivirus issue. The anti virus should not be allowed to touch the
mozmail inbox. Since InnoculateIT has no email protection (only file
protection), that folder should be exclude when scanning (I know that this means
no email scanning, but that is what you get for using an anti virus scanner that
does not scan emails).

OTOH, mork should be smarter about data corruption, though I doubt this can
happen between now and 1.0 (if ever).
> everyone I have ever personally discussed this with would LOVE to be able to 
> remove an attachment from an E-Mail while retaining the original body text.  

That's another bug (already filed) and different from separating *all*
attachments from the bodies.

> The alternative is to have every AV vendor be fully aware of every mail client 
> vendor's method of storing mail.

mbox is a very common storage method. They *have* to know that, if they mess
with emails at l.

Mozilla's (and 4.x's) usage of mbox was very helpful in the past. Removing mbox
just because some AV software is braindead and can't behave is not a good move IMO.
Replying to Micheal Collette's post <a32f9g$9j73@ripley.netscape.com> in
n.p.m.mail-news:

> Eudora strips an attachment from an E-Mail, [...] The anti-virus 
> software never gets involved with the actual mail folders as they contain 
> nothing but plain text.

This is not necessarily true. In fact, to do anything at all with Mozilla's
attachments, the AV software has to MIME decode them, which would be special
email knowledge. If they can MIME-decode, they really should know mbox, too. But
 you said that the website of InnoculateIT's AV software (one of the problematic
ones, as it seems) says that it doesn't scan email.

IIRC, some AV software scans for some typical "signatures" of virii, and that
signature might be in the subject or body of the mail or the filename of the
attachment. The circumstances suggest that this might be what happened here. In
this case, a separation between bodies and attachments wouldn't help at all - in
the worst case, all your mails (without attachments) are gone while the
attachments (incl. the infected one) are still there.
_basic, this bug has nothing to with mork. Did you mean mbox?
heh! yeah typo, (should learn to not try to look at too many discussions at the
same time ;-)
From what I've seen of InnoculateIT in action, I don't believe it takes any
actions until the file is trying to be saved.  Reason I say that is it turned
out this user had a bunch of mail with viruses attached.  As I converted him
over to Eudora and these files were stripped that's when the AV caught the bulk
of these, and removed them.  The inbox was not touched.

The reason BadTrans caused problems with Norton is that the mail included HTML
that attempted to automatically run the attachment.  This is going to trigger
any AV app into some kind of action.  Only Mozilla Mail, so far as I know,
attempts to launch this virus.

Mind you, I have zero evidence to suggest that the second wipe out had anything
to do with AV software or even an attachment.  There was no warning from the AV
software, no was there even a message being viewed at that moment.  The user was
in the process of switching folders.

The main thing I'm trying to get across here is that due to Eudora's handling of
attachments, at least 20 (most likely many more, lost count) infected E-Mails
were cleaned up with zero data loss in the actual E-mail. Where as in Mozilla
mail the inbox was wiped out twice over, each time taking one other folder
(seemingly at random) and wiping it out as well.  How could I possibly sit here
and recommend that he continue to use Mozilla?  Telling him that Moz is
compliant with the mbox format isn't exactly a selling point when a user hadn't
had any AV problems under NS 4.7x or now under Eudora.

One last thing to consider here is that most users aren't going to be able to
figure out what the heck happened.  This bug has tracked two major AV vendors
that very possibly caused the dataloss in question.  The only thing an end user
is going to honestly know is that Mozilla lost their E-mail.  Even if the mail
box is just renamed and moved, how many folks actually know where to look for it?

Just pointing to Norton and/or InnoculateIT and saying they're wrong isn't going
to do any good for Mozilla or it's users.  If McAfee gets to doing the same
thing should I just tell my users that they shouldn't run AV software at all? 
It seems most likely that most users will react by switching to a mail client
that is capable of living with the AV product they have installed rather than
the other way around.

> when a user hadn't had any AV problems under NS 4.7x or now under Eudora.

From what I understood about his bug, 4.x would be affected just as Mozilla,
because they work very similar (both use mbox).

> The only thing an end user
> is going to honestly know is that Mozilla lost their E-mail.

That's what the user *thinks*, yes. We all know that.
If a virus does an |rm -rf ~/.*|, did "Mozilla lose all your email"?

Did anyone contact the AV software vendors (not via the usual support channels)
to make them aware of the problem? They are creating themselves one of the
problems they claim to protect they users from...
Does anyone have any other suggestions?
> They are creating themselves one of the problems they claim to protect they
> users from...

At least, that's what it looks like. Would be a good chance to test the assertion.


In my case I haven't contacted the AV vendor as this particular product no
longer exists.  It was purchased from Computer Associates by e-Secure.  Although
I have run Norton in the past, I haven't got this installed on any machines to
do testing with.  I also don't have a copy of the suspected virus that may have
triggered the AV software.  Worse yet, I don't actually know that the AV
software was at fault here.  The only thing we do know is that something is
causing critical data loss, and we suspect 2 major AV apps might have something
to do with it.

One thing that is different from Netscape's mail is how it handled HTML in the
message body that attempted to save to the users hard drive some attachment.  NS
simply wouldn't do this.  Mozilla does.

The office I admin is very nearly all NS 4.7x E-Mail users running InnoculateIT.
 I have not seen any reports or complaints from users that would suggest that
similar things are happening with this combination.  Been about 6-7 months of
using that as our AV software.  The user I'm referring to in this report is a
friend of mine that had been using this same combination for at least as long,
without corruption of his mail box.

> That's what the user *thinks*, yes. We all know that.
> If a virus does an |rm -rf ~/.*|, did "Mozilla lose all your email"?

Good point, except the virus never launched.  Had the virus actually launched
this wouldn't be a Mozilla bug report.

I feel I should mention that I have been a long time advocate of providing the
ability for a user to delete an attachment from an E-Mail.  I get asked how to
do this by too many people to not believe this is a highly wished for feature. 
Prior to a few days ago I would not have been an advocate of stripping
attachments on the fly as Eudora does.  Seeing this bug in action has definitely
changed my mind about that.  Just thought I should add where I'm coming from on
this.

> One thing that is different from Netscape's mail is how it handled HTML in the
> message body that attempted to save to the users hard drive some attachment.  NS
> simply wouldn't do this.  Mozilla does.

Can you please investigate this and file a bug report (if needed with security
flag on)?
Bug 109249 is already handling this aspect of it.  Not sure if one of these 
should be considered a dependancy on the other.  They seem to be related.  At 
present, 109249 is listed as a MIME bug.

Sorry to interrupt your discussion.

NS 4.78 and receiving a BadTrans virus the inbox get wiped out too.
Definitely not an interruption Titus.  In fact, very much to the point.  Which 
AV app was involved?

It was Norton AV Corporate Edition 7.50.846
Is there no backup copy of the inbox file that could be maintained until Mozilla
could be sure that the updated inbox file was written succesfully?
Adding relnote,
decreasing severity from blocker to critical.
Severity: blocker → critical
Keywords: relnote
*** Bug 136623 has been marked as a duplicate of this bug. ***
What about enhancement request 103487?  I can't help but think that Moz might be
able to work WITH the AV software proactively to prevent data loss.  If an item
were forcefully scanned before even being entered in the inbox, any deletions or
quarantines should be isolated to that particular item.  Any thoughts about this?
Just as additional info, a coworker's inbox was wiped today by McAfee's
VirusScan 6.02 in response to an email labeled "Re: Your Password"
(W32.Frethem.D@mm). The antivirus prompted for user choice (delete, quarantine,
and others). The user selected "delete", and the messages remained in the
listing, but where not available. When new messages arrived, the message list
goes nuts (wrong message, etc)

I ran into this with Norton AV. FYI, for anyone who has this problem, you can 
go into the Norton tools, look in the quarantine and select the Inbox file and 
restore it to recover the data. It appears to strip the 'virus' email in the 
process. It's not an ideal solution, but if you're running Norton it's a way to 
recover lost data.
mass re-assign.
Assignee: naving → sspitzer
Status: ASSIGNED → NEW
Commnet #26: Actually, Mozilla and the AV software both worked correctly. The 
user choosed "delete", the AV did delete the file in which the virus was 
hiding - the Mozilla Inbox. This is correct behaviour IMHO. The user should 
know what he is doing when commanding an AV software.
But there may be some workarounds:
1. The AV probably said which file it is going to delete and the user should 
have spotted that.
2. Mozilla can lock its mail files. Currently it locks its files in the way, 
that they can be read, but not opened for write. And not even copied. Maybe a 
more prohibiting locking should be implemented. If it could deny reading, the 
AV wouldn't spot the virus and there would be no problem. The virus will be 
detected when it is opened and saved. No problem, as I see it.
Sorry, I was a bit wrong at point 2. Mozilla locks only the .msf files of 
folder that were opened. These can't be copied. The mbox files (like Inbox) are 
freely accessible. That's probably why the AV succeeded in deleting the file.
I would argue that this bug should probably be a WONTFIX.

Here's the argument,

The user, in most cases, is in control of the virus scanning software and can
control whether a file is left alone, moved, or deleted during the "realtime"
scanning of the file system.  As such, it is the user's responsibility to decide
what the policy should be.  AV vendors will not be able to constantly write new
code to handle each new mail storage format.  Thus most have gone to a "proxy
server" technique where a local server is inserted in the mail retrieval loop. 
This avoids the problem with mail storage formats because the mail is scanned
prior to storage.  

For local file scanning, the solution is to simply not delete the mailbox.  Here
is an excerpt from Symantec's knowledge base on the subject: (sorry for the long
URL)
http://service1.symantec.com/SUPPORT/ent-security.nsf/9d94c8571a91ba4788256bf3007f62b5/712247a53df336e088256a22002724ad?OpenDocument&prod=Symantec%20AntiVirus%20Corporate%20Edition&ver=8.x&src=ent&pcode=sav_ce&dtype=corp&svy=&prev=&miniver=sav_8_ce

-----------
Netscape or Outlook Express Inbox is quarantined when infected email is detected

Situation:
Symantec AntiVirus Corporate Edition (Symantec AV) or Norton AntiVirus Corporate
Edition (NAVCE) has detected an infected email, and the options are set to have
infected files quarantined. You now find that your Netscape or Outlook Express
5.0 Inbox has been quarantined.

Solution:
Netscape and Outlook Express store email in a single file. Symantec AV or NAVCE
scans the file, and quarantines the file if a virus is detected. Because all
email is stored in a single inbox file, the entire inbox is quarantined in some
circumstances. To avoid quarantining the entire inbox file, we suggest excluding
the inbox file from being scanned.

How to avoid having the entire Inbox placed in Quarantine
Excluding the inbox file from being scanned prevents the inbox file from being
quarantined while still allowing a virus to be detected. When a virus is found
in an opened email message, rather than during a scan or when downloading, the
opened message can be safely quarantined--or deleted--without causing problems
with the Inbox itself.

-----------

As the virus scanning vendor suggests that the user configure the scanner to
leave the file alone, it would seem that it rests upon the user to take the
action, rather than Mozilla.

If not marked WONTFIX, I suggest that "dataloss" keyword be removed and severity
be reduced to normal (I could do this, but I'll leave it to someone with more clout)

A addition to the release notes should be made indicating to the users that the
mail folders should not be scanned by file system scanners.  Scanning of mail by
"proxy servers" is still a viable option for scanning mail. 

Kevin
*** Bug 186930 has been marked as a duplicate of this bug. ***
*** Bug 216948 has been marked as a duplicate of this bug. ***
I strongly recommend looking seriously at this bug.
Here's what happened to me:

Symantec AntiVirus Corporate detected with its heuristic virus auto-detection 
(bloodhound) a virus in one of the attached emails (showing inline as my 
preferences) of a bugtraq email I got.  The attached email had some malicious 
html code in it and it triggered bloodhound.  Wiped out my mail folder.  

Since we are running a big network of computers, we don't let the user chose 
what to do with the infection.  It's desinfect or quarantine.  So this is kind 
of odd since folder deletion is not preventable.  Notice that I've never opened 
an attachment or did anything that could lead to virus infection.  I only had 
enabled inline attachment viewing and am using bloodhound with SAV (enabled by 
default).

IMO, the problem wouldn't be that bad if it was triggered only when clicking on 
an attached virus (because it will actually prevent infection! the opposite of 
if they remove this folder from realtime protection like suggested above!) but 
as my case showed it was triggered automatically in an entreprise scenario (no 
control over AV at all) where we don't trust OE but do trust Thunderbird.

I imagine that this will be complicated to fix.  Goodluck!  Great product!
Please report this to Symantec, because it's clearly *their* program which
deleted your mailbox. The more customers complain about dataloss there, the more
likely they are to fix it.
The thing I don't understand is that if it's *their* program's fault how come 
outlook express handles everything perfectly?  I will report it to Symantec but 
I feel they'll tell me it's Thunderbird's fault.
Outlook Express stores all of your mail in one giant database instead of
individual mailbox files. Symantec probably has a special case for dealing with
the OE file, but not for individual mailbox files like INBOX. Usually this is
done with file extensions (INBOX has no file extension). I forget what the OE db
is called but it has a special file extension.
bienvenu, do you think it would have sense to give mboxes a file extension, at
least on Windows?
No, that would be a nightmare. Virus checkers can be configured to deal with
files w/ no extensions.
If this is a problem that should preferably be solved by symantec, shouldn't we 
call in some people from Evangalism to add some pressure to our case?
doesn't symantec already have the option to deal with files with no extension? I
don't have a copy myself...
(In reply to comment #6)
> If it's really the anti-virus software deleting the folders, I'd said that it's
> a severe bug in *that* AV software, not Mozilla. How could Mozilla possibly
> prevent that?

While I agree that it's the anti-virus software that absolutely sucks
for moving files out from under running problems, Mozilla should be more 
robust (at least since the anti-virus behavior is now a known problem and
the severity of the data loss is so great).

Also see bug 235588 (Inbox zeroed out when filter target folder file 
quarantined).
NAV has something called "exclude at risk files", according to the manual this
option will be available if the could not be deleted. Maybe it could be locked
or something so that NAV cannot delete it?
If someone here could email me (bart at mozilla dot org) the name of ANY contact
person at Symantec so we could discuss an arrangement for getting this problem
solved, I'd be happy to try & help connect the dots.  I just tried making a
blind call over there and ran into a nightmarish experience ("pls call technical
support") that got me nowhere (I think)...
*** Bug 233966 has been marked as a duplicate of this bug. ***
Hi.

I just installed Mozilla mail in order to prevent Netsky virus to cause havoc in
my computer with Internet Explorer (which for unknown reasons I cannot update).

I had the exact same problem : all my emails were "erased" after Panda cleaned
the messages infected with various sorts of viruses. In fact, I could read my
mails in plain text after opening my inbox file with the Mozilla composer.

I tried to find a solution after looking for bugs solutions (and discover that
this bug hasn't been resolved) and found that if you uncheck the "Display
Attachments Inline" in the "View" menu of the Mozilla Mail windows, then, even
if Panda detects viruses and cleans them, then you keep all your messages
without problem.

I hope this aids.

If this can be useful, I run Windows ME OS, and not Windows 2000.
(In reply to comment #47)
> Hi.
> I just installed Mozilla mail in order to prevent Netsky virus to cause havoc 
in
> my computer with Internet Explorer (which for unknown reasons I cannot 
update).
> I had the exact same problem : all my emails were "erased" after Panda cleaned
> the messages infected with various sorts of viruses. In fact, I could read my
> mails in plain text after opening my inbox file with the Mozilla composer.
> I tried to find a solution after looking for bugs solutions (and discover that
> this bug hasn't been resolved) and found that if you uncheck the "Display
> Attachments Inline" in the "View" menu of the Mozilla Mail windows, then, even
> if Panda detects viruses and cleans them, then you keep all your messages
> without problem.
> I hope this aids.
> If this can be useful, I run Windows ME OS, and not Windows 2000.


Hello,

I have today the same problem, with Netsky virus : when arrives, deletes Inbox.
For similar raisons, I have recently move from Opera 7 to Mozilla 1.6 
(Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.6) Gecko/20040113).

Note that (with the same mail), this problem doesn't happend with MS Outlook 
Express 6.

When this problem will be solved ?

Thanks.


(In reply to comment #48)
> (...)
> Note that (with the same mail), this problem doesn't happend with MS Outlook 
> Express 6.
> (...)

Does anyone know what are the reasons for this difference? Are viruses detected
without deleting mail in other mail clients, or is there any different behavior?
Possibly it would be interesting to know the general mechanisms for this.

Perhaps the problem is that some antivirus software is currently developed with
MS mail clients in mind, mainly. (?) In this case, it seems that it would be
important to talk to them for collaboration with Mozilla to fix this bug.
Mailbox files of Microsoft clients have a unique extention and those are added
to the exclusion list of most mailclients. Mozillamail (and thunderbird) use
mbox format mailboxes without an extention. adding the mbx or mbox extention
would make it much easier to exclude the files for the AV companies.

This seems a case where the Maldir format would show its strengths over mbox. In
fact, in this particular case it would even display the expected behaviour: the
infected message would be deleted leaving the rest unchanged. Anyway, I think
changing the mail database format is out of the question. An alternate solution
would be saving the incoming message to a temp file (each message to its file)
on arrival, and only after that, if it was still there, it would be added to the
inbox. This way the AV software would be able to catch *only* the infected
messages. This way it wouldn't be necessary to contact *all* the AV software houses.

OTOH, the mbox is a pretty well known format, so it would be expected that AV
software producers would know that it's this kind of file they're dealing with
(without the need to check the file extension) and delete only the infected
message, levaing the rest in place...


(In reply to comment #51)
> This seems a case where the Maldir format would show its strengths over mbox. In
> fact, in this particular case it would even display the expected behaviour: the
> infected message would be deleted leaving the rest unchanged. Anyway, I think
> changing the mail database format is out of the question. An alternate solution
> would be saving the incoming message to a temp file (each message to its file)
> on arrival, and only after that, if it was still there, it would be added to the
> inbox.

This only works for incoming messages with known virusses. The inbox would still
be deleted if a new virus snuck into your inbox and is detected on a schedulled
scan two days later.

> This way the AV software would be able to catch *only* the infected
> messages. This way it wouldn't be necessary to contact *all* the AV software
houses.

That's true.

Just changing teh extension and announcing this in public with a huge font and a
note to software houses in press releases all over the world should get their
attention don't you think? (probably not). But changing the extension would
probably be the best way to get around this.

I vote for renaming inbox -> inbox.mbx or inbox.mbox. same for all subfolders.
*** Bug 243162 has been marked as a duplicate of this bug. ***
*** Bug 245414 has been marked as a duplicate of this bug. ***
I am having the same problem, and as far as I can tell it is a bug in Symantec
AntiVirus Corporate Edition. My office just upgraded from version 8.1 to 9.0 and
it performed a full system scan (all files). A bunch of my mailboxes were
immediately quarantined.

To recover my mail, I restored mailbox files from the quarantine and parsed the
messages into smaller sub-mailboxes. Then I scanned each sub-mailbox file to
find the infected section. Many iterations later I found and deleted the
infected message and then put the "healthy" messages back where they belong.
Some of the infected mailboxes were large so this was a very long process (not
done yet). Since I manually scanned each mailbox/sub-mailbox file outside of
Mozilla, I don't see how this can be a Mozilla problem. As far as I know,
mailbox files are just palin text files as far as the anti-virus program is
concerned. Having said that, it appears that the anti-virus program does
recognize Outlook pst mailbox files as it extracts and quarantines the infected
message from the pst file and leaves the rest of the mailbox alone. I consider
this a Symantec bug, and I think they should be notified.

Unfortunately, the tedious process described above doesn't always work. I am
still in the process of trying to fix my Inbox (fortunately I don't leave
anything important there). When I parsed the messages into sub-mailboxes and
scan each one they were all fine. When I put all the messages back in a single
mailbox, Symantec found a virus and quanrantined the whole thing again. I'm
already losing my hair, and by the time I'm done with this, I may not have any
left. Of course our sysadmin says I should just switch to Outlook.

The ability to strip attachments from email messages would certainly be helpful
here. On top of that, I often get mail with attachments I don't need to keep or
that need to be saved elsewhere, e.g. in a file management system. These
attachments end up just bloating my mailboxes. It is important to keep the email
messages themselves as they provide a record of when the mail was received and
who sent it. For sent mail, it is often important to know to whom I sent a file
and when, but I don't need to leave the file attached.

So to whoever out there is counting votes for an option to strip attachments off
email messages, please add mine.

Sorry for the length of this posting, but I thought the detail might be helpful.
(In reply to comment #55)
> (...)
> 
> The ability to strip attachments from email messages would certainly be helpful
> here. (...)
> 
> So to whoever out there is counting votes for an option to strip attachments off
> email messages, please add mine.

If you wish, you may add yours to the current 355 votes of Bug 2920 ["Delete
attachment from mail message in folder (remove/strip attached files from email
messages)"].

Some programmers are trying to fix it, a not easy task within the complex
Mozilla code.
Many have posted that this is a user problem, and that simply excluding the
Mozilla inbox folder(s) from the scan fixes this problem. Many of us are on
corporate networks where we don't have the luxury of being able to pick and
choose exclude folder for virus scanning. In particular Norton's corporate
edition tends to lock users out of any such configuration options. As has been
stated many times, this is a problem that could be easily resolved by storing
attachements as separate files. While I'll admit I don't know how much work will
be involved in doing such a thing, it seems like a better solution than simply
blaming the problem on the user. One of the advantages of Open Source is that in
general- developers don't turn a blind eye on user problems. I love
Thunderbird/Mozilla, but I have to seriously look at going back to a M$ product
now that I have lost all of my inboxes for the fourth time in the last month.
(In reply to comment #57)
> (...) As has been
> stated many times, this is a problem that could be easily resolved by storing
> attachements as separate files. (...)

In this case (something like the Eudora automatic functionality as a possible
Mozilla option), it's bug 9309 -"option to have attachments auto saved to chosen
location"- rather than bug 2920. Naturally it would be great to have these two
enhancements soon.
> this is a problem that could be easily resolved by storing
> attachements as separate files.

This is *everything* but easy. Not only would it be a lot of work, it would also
be very dangerous in case of bugs  in the new code.

> better solution than simply blaming the problem on the user.

I'm blaming the anti-virus product, not the user. mbox is a very old, standard
and widely-used format, they wipe it, they have no excuse. Please call them, the
more people the better.
Look at Symantec knowledge base Document ID:2000051809560948 
which describes their solution...
(In reply to comment #60)
> Look at Symantec knowledge base Document ID:2000051809560948 
> which describes their solution...

That's:

Symantec Knowledge Base
Netscape or Outlook Express Inbox is quarantined when infected email is detected
http://service1.symantec.com/SUPPORT/ent-security.nsf/6fffc7260966992188256bf300818635/712247a53df336e088256a22002724ad
Blocks: 247126
Adding dependency on 9309 which would effectively contrinbute to solve this.

Confirming this bug. It also deletes/moves the Junk mail and Trash folders and
whatever folders have emails with viruses in them. These folders should not be
easily modified/moved/deleted by other applications.

Platform: PC
OS: Windows 2000 Pro
Product: Thunderbird
Version: 0.7.2 (20040707)
Component should be changed to Mail Window Front End

This has also been reported on the MozillaZine forums:
http://forums.mozillazine.org/viewtopic.php?t=101013
http://forums.mozillazine.org/viewtopic.php?t=75470
http://forums.mozillazine.org/viewtopic.php?t=79453

Workaround:
http://kb.mozillazine.org/index.phtml?title=Thunderbird_:_FAQs_:_Anti-virus_Software
Depends on: 9309
*** Bug 254453 has been marked as a duplicate of this bug. ***
No longer blocks: 247126
*** Bug 247126 has been marked as a duplicate of this bug. ***
As mentioned once, the whole issue should be resolved by moving to the maildir
format. Although, there is a clear disadvantage of switching to a different
default mail storage format, there are several other advantages to the format,
not the least of which is that new user wouldn't have to know ahead of time of
this risk in order to avoid having their inbox deleted. I believe there's
another bug about supporting the maildir format.
(In reply to comment #63)
> *** Bug 254453 has been marked as a duplicate of this bug. ***

Sorry for filing this bug again, I checked for other bugs but didn't see bug
116443. I read a little of the previous discussion on this bug. I think it is
fine and prudent that a virus checker removes or quarantines an infected file -
email or others - that's the whole point of a virus checker, after all. In this
context, excluding files from the virus checker (comment #60/61) is precisely
what I don't want to do because incoming mail is the number one potential virus
source on my machine.

After thinking I had a great idea involving a temp file before saving into the
inbox, I read comments # 25, 51 and 52 and discovered that others had the same
idea years before me. Now, my  question is: why is not every single e-mail given
some counter number or time stamp as a name and saved individually in text or
mbox format in an inbox FOLDER? I realize that this increases storage space, but
since we're talking GB meanwhile, it would be well worth it for me.
*** Bug 238499 has been marked as a duplicate of this bug. ***
Blocks: 233833
Blocks: 235588
*** Bug 200287 has been marked as a duplicate of this bug. ***
Blocks: 256034
I wanted to note that this is something that Mozilla needs to handle.  The
latest versions of the two biggest (I believe) corporate virus scanning clients,
McAfee and Norton, seem to have problems with this, and these problems do not
occur when used in conjunction with Outlook or Eudora.  Also, in the corporate
environment, anti-virus program settings may be locked down, so the
MailNews/Thunderbird mailbox files cannot be exempted and individuals in these
situations flat out CANNOT use MailNews/Thunderbird as their e-mail client.  As
long as incoming files were written to an individual file first, then appended
to the mailbox after the scanner has a chance to react, many (not all) of these
problems would go away.  Maildir would be the ultimate fix, but in the mean
time, something MUST be done if corporate deployment is a goal.
Blocks: eudora
Summary: Deleted inbox after receiving virus infected mail → Deleted inbox after receiving virus infected mail (McAfee, Norton)
Blocks: 229235
No longer blocks: 256034
I'll mark a number of bugs as dups (currently deps), which are about different
situations (filtering, compacting folders, Eudora import), but essentially the
same cause.
No longer blocks: 229235, 233833, 235588
*** Bug 229235 has been marked as a duplicate of this bug. ***
*** Bug 235588 has been marked as a duplicate of this bug. ***
*** Bug 233833 has been marked as a duplicate of this bug. ***
(In reply to comment #70)
> I'll mark a number of bugs as dups (currently deps), which are about different
> situations (filtering, compacting folders, Eudora import), but essentially the
> same cause.

OK, but does that imply that whoever fixes this needs to test that the fix
actually works in all of these situations? It may turn out that the cause is not
quite as identical as you assume.
Flags: blocking-aviary1.0?
*** Bug 245538 has been marked as a duplicate of this bug. ***
*** Bug 260384 has been marked as a duplicate of this bug. ***
*** Bug 260855 has been marked as a duplicate of this bug. ***
Flags: blocking-aviary1.0? → blocking-aviary1.0-
Anyone interested in having this solved should also check 9309 and vote for it /
contribute to it.
(In reply to comment #78)
> Anyone interested in having this solved should also check 9309 and vote for it /
> contribute to it.

That should be restricted to those who think that the best way to solve this
problem is a complete redesign of Mozilla's data formats, although the experts
seem to say that that is not the best way. To put it another way, I don't agree
that there is real a dependency on bug 9309.

An alternative and probably much simpler way of solving this problem would be to
save and reopen each incoming message (or each one with an attachment) as a
separate temporary file before adding it to the inbox. This would trigger a
virus scan of the temporary file. This is the same kind of strategy as the good
one used with downloaded executable files, that they are always saved and
reopened before being executed, and so any virus scanner has the chance to scan
them. But I'm not sure if the performance hit would be acceptable.

Is anyone seeing this problem with latest versions of anti-virus software? I
haven't seen it since I have been using Norton AV 2003 and now 2004, with e-mail
protection. Maybe that is because they catch the virus before it makes it to the
inbox. (Or maybe it is because my ISP is catching the viruses before they get to
me.) None of the recent duplicates mention exactly what anti-virus software is
in use or whether e-mail protection is enabled.

Saving the file to a temp location and then adding won't help if you're behind
with signiature files (happens to the best, just 10 minutes behind can be 10
minutes too many); the virus creeps into the inbox file; signiatures are
downloaded; inbox get's deleted.

I think you're no longer seeing this with NAV 200[345] because most virusses are
stopped during teh POP3 transfer, but the above situation applies to this aswell.
How is this solved in ms outlook?
It saves all it emails in one file, too, doesn't it?
Or is this file just encrypted, so the anti-virus programms can't scan it?
(In reply to comment #80)
> Saving the file to a temp location and then adding won't help if you're behind
> with signiature files (happens to the best, just 10 minutes behind can be 10
> minutes too many); the virus creeps into the inbox file; signiatures are
> downloaded; inbox get's deleted.

NAV 2004 at least supports automatic update of signature files, if enabled as
recommended. Of course they still come out after the viruses first appear. But
in this scenario the anti-virus program should not delete an infected file, but
should clean it or quarantine it (and quarantined files can be retrieved).

The basic problem here as I observed it is that Mozilla overwrites the inbox
before the AV program gets the chance to quarantine it. I think what is
happening is that Mozilla can't read a file, because the AV program is locking
it, and so instead Mozilla overwrites it, which is not blocked by the AV program
- all before the user has had the chance to respond to the "what do you want to
do about this virus" dialog. The fallacy here is Mozilla's assumption that
because it cannot read a file that file may be safely overwritten; there are
many reasons other than virus problems why this is a bad assumption e.g.
multi-user locks, network glitches. This is certainly happening when an inbox is
deleted while being compacted: Mozilla can't read the inbox because it is
infected, so deletes it - see my report of bug 233833 of which this is
considered a duplicate.

By the way, ;-) would the problem be solved if mailbox files were given the
extenstion .pst? That would automatically exclude them from at least some AV
scans. Or is this extension a Microsoft copyright? ;-)
In response to Christian Fersch's question.  My experience is with Outlook and 
McAfee, where McAfee hooks itself into Outlook and can scan the mail messages 
and delete them using Outlook's COM API.
*** Bug 264172 has been marked as a duplicate of this bug. ***
(In reply to comment #82)
>
> The basic problem here as I observed it is that Mozilla overwrites the inbox
> before the AV program gets the chance to quarantine it. I think what is
> happening is that Mozilla can't read a file, because the AV program is locking
> it, and so instead Mozilla overwrites it, which is not blocked by the AV program

I'm not sure if this is what really happens. When Norton AntiVirus 8.0 found the virus in my inbox, an
alert box popped up that stated:

  Action taken:  Clean failed : Quarantine succeeded : Access denied

  (see my dup 264172 for further infos)

I read into that, that NAV tried to clean my mbox file, but didn't succeed because of a lock. However, it 
succeeded with moving the file to the quarantine.

Furthermore, does anyone know, why I could use Thunderbird / NAV for over half a year now with
hundreds of viruses trickling in my Inbox before this bug occurred twice this week (with the same kind
of virus)? Does this affect only viruses found with bloodhound?

Anyway. It's a nasty situation for anyone that gets into it. Even if it might be the AV-Vendors fault, it
won't be easy to convince all of them to provide a fix, so a solution on the Mozilla side would really be 
appreciated.


This problem also exists with McAfee's VirusScan Enterprise 8.0i; I got my junk
mail folder deleted even though I have only set it to automatically clean files
and move into quarantaine them if the clean failed.
Would be alot better if McAfee's Email Scanner would also do
Thunderbird/Mozilla/Netscape 7 (which is all the same product).
Despite all the finger-pointing at anti-virus applications, this still appears
to be an E-mail client problem.  

I use Norton Anti-Virus 2003.  My E-mail client is Eudora Lite 3.0.6 (yes,
several years old).  I do NOT have this problem.  Why?  Because viruses are not
found in the E-mail messages themselves, only in the attachments to those
messages.  Eudora Lite merges messages into long files, one per folder; but it
saves the attachments separately, each attachment as an individual file.  Thus,
a virus results in the deletion, repair, or quarantine of the afflicted
attachment file, leaving the merged message file untouched.  

I know the process used by my Eudora Lite is somewhat old-fashioned.  But it
works.  The more modern process used by Mozilla and Thunderbird is broken with
respect to anti-virus protection.  

Some questions:  
*  How likely do you think it would be for all anti-virus application vendors to
change their products for compatibility with Mozilla products?  
*  Given the unending problem of computer viruses, which is more likely --
someone disabling their anti-virus application to use a Mozilla E-mail client or
someone enabling their anti-virus application and refusing to use a Mozilla
E-mail client?  

I must agree with comment #85 and suggest that this be fixed in the affected
Mozilla products.   There should be no room for an attitude that says: "We're
right and we don't care how adversely it impacts our users or the future growth
of our organization."  
*** Bug 264916 has been marked as a duplicate of this bug. ***
I Have had the same problem with Norton AV 2004. 

It detected copies of netsky downloaded into the mailbox file and so quite
sensibly blocked access to that file. To get mozilla mail working again I had to
log in as admin, turn off the AV, manually edit the mailbox file to remove the
content of said bad e-mails and then run mail and hope it all worked ok.

As soon as I had it working 2 more Netsky mails came in and I had to do it all
again :(

Possible solutions:

 - Don't download attachments until the user tries to open them (as a pref)
 - Don't store the e-mail attachments in the mailbox as mentioned earlier.
    Prehaps store a ref to the file that contains the attachment?
    (Much better solution although both combined would be nice)

This is getting to be quite an issue as I have had to go and fix quite a few
machines with this issue. It also appears to be getting mozilla mail a bad
reputation as "outlook never did this".

I have to agree with previous comments about the fact that this needs fixing. Is
there a good reason why the attachments are stored in the mailbox or just a
legacy thing?

(See Also: BUG 9309 and 2920)


(In reply to comment #89)
...> Possible solutions:
> 
>  - Don't download attachments until the user tries to open them (as a pref)

Please, no! This would be disastrous for those who download e-mail by sometimes
expensive dial-up and read it later off-line. Anyway, I don't suppose POP3
supports this.

But then I would suggest using an ISP which filters viruses - not 100% of
course. Or save and reopen the file as I suggested in comment 79 - again not
100% but not far short.

...
> This is getting to be quite an issue as I have had to go and fix quite a few
> machines with this issue. It also appears to be getting mozilla mail a bad
> reputation as "outlook never did this".
> 
> I have to agree with previous comments about the fact that this needs fixing.

Agreed. This is potentially a major, almsot show-stopping problem with
Mozilla/Thunderbird. It's not enough to say this is not Mozilla's problem,
anything which means that Mozilla appears not to work is a PR disaster for Mozilla.

(In reply to comment #90)
> (In reply to comment #89)
> ...> Possible solutions:
> > 
> >  - Don't download attachments until the user tries to open them (as a pref)
> 
> Please, no! This would be disastrous for those who download e-mail by sometimes
> expensive dial-up and read it later off-line. Anyway, I don't suppose POP3
> supports this.

Was a suggestion to provide a user pref so you could choose to do this, not
enforce it on all users.

On reflection I am fairly sure you are correct about POP3's lack of support for
this :(
Is it possible to download new messages into separate files in a tmp directory, 
then merge them into the mailbox?  This gives AV software an opportunity to 
detect problems and delete individual messages and attachments, but then 
Thunderbird can work with singular mail box files as designed.  This also gives 
Thunderbird a chance to insert a replacement message with a note about what 
happened where the offending message would have appeared.
yes, pop3 doesn't support not downloading attachments (imap does). But, you can
configure mozilla not to download pop3 messages over a certain size (not sure
how big the nevsky virus usually is).
(In reply to comment #93)
> yes, pop3 doesn't support not downloading attachments (imap does). But, you can
> configure mozilla not to download pop3 messages over a certain size (not sure
> how big the nevsky virus usually is).
Depending on which of the mails I got the sizes were 41.2K & 42.1K

So under the moz 50k limit

you can make the moz limit whatever you want - 50K is just the initial default
(In reply to comment #95)
> you can make the moz limit whatever you want - 50K is just the initial default

Fair point, however the people using mail that are showing the problems would
like to be able to accept word documents, pdfs and similar, which are usually as
large as the netsky option.

Certainly the suggested solution of using maildir or the suggestion from comment
92 would provide some resolution of this problem.

I don't think any normal user would see the whole mailbox going due to a vius
being recied but not run as good.

I agree that the AV vendours SHOULD understnad mbox, however the major ones
apppear not to, and as such no only is this issue causing data loss and user
problems, it's also generating some very bad PR for mozilla.

I'd just like to point out the I'm using McAfee 4.5.1, which is a fairly old
version (kept up to date of course), but it has *NEVER* tried to delete my
inbox, and I have had numerous virii sent to me. It only prompted after *I*
saved the attachment.

So McAfee used to do the right thing...
(In reply to comment #97)
> I'd just like to point out the I'm using McAfee 4.5.1, which is a fairly old
> version (kept up to date of course), but it has *NEVER* tried to delete my
> inbox, and I have had numerous virii sent to me. It only prompted after *I*
> saved the attachment.
> 
> So McAfee used to do the right thing...

Only if it's really the right thing to allow your inbox to remain infected. I
would expect anti-virus software to find a virus even when it is in encoded form
in my inbox - and to do something sensible when it finds it. Norton knows not to
delete an entire zip archive, or even quarantine it, when it finds a virus in
one, so it and McAfee could use a similar strategy for viruses in mailboxes.

> Only if it's really the right thing to allow your inbox 
> to remain infected. 

Note that your mailbox is _not_ infected.   Is anything 
executing the virus code when the virus data is sitting 
in your mail file?  No.  (The virus can't get executed 
unless the virus is decoded (from e.g., MIME) into some 
executable form (e.g., saving as an executable file).)


> I would expect anti-virus software to find a virus even when 
> it is in encoded form in my inbox - 

Okay, _detecting_ the virus data is reasonable.


> and to do something sensible when it finds it. 

Like what?  If you find a virus attachment in an e-mail file
while some application (Mozilla) is using the file, what's a 
sensible thing to do?  Unless you know something specific 
about the application, what can you do?

Moving the file out from under the application (e.g., 
quarantining it by renaming it or otherwise making it appear
to be deleted) is NOT a reasonable thing to do.  


> Norton knows not to delete an entire zip archive, or even 
> quarantine it, when it finds a virus in one, so it and 
> McAfee could use a similar strategy for viruses in mailboxes.

That's what Norton doesn't do, but what does it do?  (What
is its strategy for viruses (virus data) in mailboxes?
(What can it be other than to leave the mailbox files alone
and catch the viruses at a subsequent stage?))


Of course, even though Mozilla can't be responsible for 
working perfectly if someone comes along and hides its files
while it's running, Mozilla should be more robust given
the severity of deleting the entire mailbox file's contents.

Mozilla's file-compaction logic should probably be a little
paranoid, checking things every step along the way (checking
that all reads and writes worked, verifying that the 
compacted-to file has the expected structure and contents
before actually deleting the original file, etc.)


> I agree that the AV vendours SHOULD understnad mbox, 
> however the major ones apppear not to, ...

Are you sure?

The fact that a virus is detected in Mozilla's mailbox
file means that the AV software _does_ understand the 
format enough to detect the virus (e.g., find the 
message headers, find the MIME header and 
boundary, base-64-decode the virus attachment, etc).




RE: comment 100 
> The fact that a virus is detected in Mozilla's mailbox
> file means that the AV software _does_ understand the 
> format enough to detect the virus (e.g., find the 
> message headers, find the MIME header and 
> boundary, base-64-decode the virus attachment, etc).

OK so they are decoding it, so obviously understand the encoding method. However
they blatantly don't realise that the code is only a small part of the mbox file.

Ideally they shoul.d do what I did by hand:

 - Find the infected message.
 - Delete the section of the infected message and replace it with a string such
   as "Some idiot sent you something nasty so we removed it for you" in it's
   place .

A program called mailscanner does this when it scans incoming messages so it
can't be that hard.

The problem is that all current and 2005 releases that are out there will trash
a mozilla mailbox. Even though it isn't directly mozilla mails fault, it looks
bad on mozilla and a work around should be used.

In relation to my previous comment about maildir, this could cause problems on
file systems with an insufficient number of inodes, so it appears that saving an
attachment as a temporary file, letting the AV do it's evil, and if it's gone
replacing it with a simple message of:

"The attachment '<FILE NAME>' was removed by a third party product (possibly
Anti Virus)."

Anyway fingers crossed for a solution soon or I may have to seriously consider
outlook again :(
Don't suppose anybody can tell me in which files the code that handles the
downloading of mail is.

I found a few bits but couldn't find anything that appeared to do this. Guess I
must be looking in the wrong place :(
(In reply to comment #99)
> > Only if it's really the right thing to allow your inbox 
> > to remain infected. 
> 
> Note that your mailbox is _not_ infected.   Is anything 
> executing the virus code when the virus data is sitting 
> in your mail file?  No.

Well, this is a matter of definition, but as I understand it, something which
contains a viable virus etc (whether medical or software) is considered
infected, whether or not the virus etc is active at the time.
...
> 
> Like what?  If you find a virus attachment in an e-mail file
> while some application (Mozilla) is using the file, what's a 
> sensible thing to do?  Unless you know something specific 
> about the application, what can you do?
> 
> Moving the file out from under the application (e.g., 
> quarantining it by renaming it or otherwise making it appear
> to be deleted) is NOT a reasonable thing to do.  
> 
Agreed. One sensible strategy is described in comment 101.
> 
> > Norton knows not to delete an entire zip archive, or even 
> > quarantine it, when it finds a virus in one, so it and 
> > McAfee could use a similar strategy for viruses in mailboxes.
> 
> That's what Norton doesn't do, but what does it do?  (What
> is its strategy for viruses (virus data) in mailboxes?
> (What can it be other than to leave the mailbox files alone
> and catch the viruses at a subsequent stage?))
> 
In fact what Norton does with infected zip archives is only warn the user. It
could do the same with mailboxes, as an alternative strategy. This is a lot
better than doing nothing and also a lot better than deleting or quarantining
the whole mailbox.
> 
> Of course, even though Mozilla can't be responsible for 
> working perfectly if someone comes along and hides its files
> while it's running, Mozilla should be more robust given
> the severity of deleting the entire mailbox file's contents.
> 
> Mozilla's file-compaction logic should probably be a little
> paranoid, checking things every step along the way (checking
> that all reads and writes worked, verifying that the 
> compacted-to file has the expected structure and contents
> before actually deleting the original file, etc.)
> 
Indeed. The initial problem which I saw was not that anti-virus was deleting the
mailbox, but that it was hidden from Mozilla and so Mozilla simply overwrote it.

> > > Only if it's really the right thing to allow your inbox 
> > > to remain infected. 
> > 
> > Note that your mailbox is _not_ infected.   Is anything 
> > executing the virus code when the virus data is sitting 
> > in your mail file?  No.
> 
> Well, this is a matter of definition, but as I understand it, something which
> contains a viable virus etc (whether medical or software) is considered
> infected, whether or not the virus etc is active at the time.
> ...

I tend to agree that if the code is there, even in a dormant form, it's still a
virus. It's likle dormant smallpox is still smallpox and hence still nasty.

> Indeed. The initial problem which I saw was not that anti-virus was deleting 
> the mailbox, but that it was hidden from Mozilla and so Mozilla simply 
> overwrote it.

Yes it does quarantene the file, however on some versions of norton at least it
says could not fix, shall I delete the virus infected file?

Again a request to anybody who knows in which file the code that handles
download of e-mails & parsing of MIME are as I can not find them in the source
tree :( 
the code that writes to the mailbox file is in
mozilla/mailnews/local/src/nsPop3Sink.cpp - in particular,
nsPop3Sink::WriteLineToMailbox writes to m_outFileStream.

see http://lxr.mozilla.org/mozilla/source/mailnews/local/src/nsPop3Sink.cpp#647

Optionally using a temp file is going to be a non-trivial amount of work, though
it's doable...you'll have to deal with message filters moving the message, among
other things...
I'd like to point out that this particular problem has exposed a somewhat larger
issue: 

We need a How To or FAQ for installing Mozilla, Thunderbird and Firefox in
corporate environments with some tips on dealing with other applications that
may need to be tuned to peacefully coexist with something other than Outlook and
IE. For example, there is a known method of excluding the Inbox from scanning by
Symantec AV or Norton Corporate AV (same product - names recently changed). As
was pointed out some months ago, other email clients use similar indox file
structures, so it isn't an entirely new issue for the AV vendors (even if they
choose not to fix it but offer a workaround).

This would be very helpful for IT people attempting to build desktop packages
for testing and mass deployment (including documentation of installation
procedures), as well as those of us who build 3 or 4 systems at a time once
every few months.

And it should anticipate questions like: if I build a Ghost image with Mozilla
as the default browser and mail client, will I need to change the id number of
the default client on each cloned computer to avoid potential identity problems
between 2 clients sharing the same id number?
(In reply to comment #104 and comment #105)

Thanks, I'll have a look at the code, I'm probably getting in a bit deep as Java
is my main language rather than C, however I'll have  look over it and see whats
what and if I can do anything with it.

(In reply to comment #106)

> For example, there is a known method of excluding the Inbox from scanning by
> Symantec AV or Norton Corporate AV (same product - names recently changed). As
> was pointed out some months ago, other email clients use similar indox file
> structures, so it isn't an entirely new issue for the AV vendors (even if they
> choose not to fix it but offer a workaround).
> 
> This would be very helpful for IT people attempting to build desktop packages
> for testing and mass deployment (including documentation of installation
> procedures), as well as those of us who build 3 or 4 systems at a time once
> every few months.

The issue is probably more severe if your building 10+ machines at once, and
rolling out few hundred in a matter of days. The FAQ can be useful, however the
comment about not scannig certain fails for worms & viruses is not really a safe
option.

I am sure it wouldn't take much for somebody to write something that was
downloaded and run (by the user clicking) and then looking over inbox, MIME
Decoding the viral code and then running it, posibly even hidden from the AV.

That would be a disaster and is probably a good enough reason to find a
permanent solution.
A footnote on the behavior of Symantec Anti Virus 9.0 when you manually exclude
the inbox (and enter the incoming port of your email feed if different from the
default). This is different from that of it's predecessor - Norton Corporate AV 7.6.

Upon opening the message in Mozilla, SAV displays the message:

"Symantec Email Proxy deleted the following email message:
From: xxx@xxx.com
To: yyy@yyy.net
Subject: Mail Delivery (failure yyy@yyy.net)"

It isn't clear when it actually deletes the virus payload, but I suspect it is
upon reciept, as it seems to be inserting itself as a proxy (since it requires
the incoming port #) before Mozilla. If so, this may be of some help to those
who fear leaving a virus payload intact if it can be verified.


Norton 2004 etc are supposed to install a proxy mail server for pop3 whereby it
recieves the message and then scans it and passes it on to the mail program. 
Have to say I am not sure why it works with netscape, but appears to not be set
up  with the install of mozilla mail I have.

The message you have does sound like it has removed a nasty attachment or mail
mind :)


(In reply to comment #105)
> the code that writes to the mailbox file is in
> mozilla/mailnews/local/src/nsPop3Sink.cpp - in particular,
> nsPop3Sink::WriteLineToMailbox writes to m_outFileStream.

I've had a look over the code and don't currently trust my C skills to be
sufficient to not create a buffer overflow or some odd pointer error while
hacking at that. (Sorry, looks like I got too used to Java managing memory etc
for me, so I will have to brush my C up).

Regarding the comment about message filters moving the message, how about
grabbing the message before any of the filters have a chance to do things with
it and then popping it back where it came from?
To at least spare users from the worst data loss, would auto-creating a backup
of the mail account on startup be an easy enough work around? I realize that
would be a lot of data for some people, but still. That way, if users experience
this bug the damage is somewhat limited.
This seems to be a fork in the road branding issue - there are workarounds, but
they require our users to be sufficiently motivated to seek them. 

BUT, if Mozilla/Thunderbird is going to compete in the consumer market against
IE, it must come out of the box in such a way as to be able to cope with this
issue. Otherwise Moz will not retain the perception of quality that people who
want something better than IE will expect. 

I realize this is a really tough issue that is going to take some really serious
work by some highly skilled people (I'm certainly not that good) on a volunteer
basis. This is one of the weaknesses of the open source paradigm - but it can be
a real strength as well if our collective passion to build a better mousetrap is
sincere!
(In reply to comment #112)
... 
> I realize this is a really tough issue that is going to take some really serious
> work by some highly skilled people (I'm certainly not that good) on a volunteer
> basis. This is one of the weaknesses of the open source paradigm - but it can be
> a real strength as well if our collective passion to build a better mousetrap is
> sincere!

It is certainly a weakness of the open source paradigm, or at least of Mozilla's
implementation of it, that a "critical" issue like this one is in fact not being
dealt with at all because the person it is assigned to, Seth Spitzer, is "not
reading bugmail". No doubt Seth has good reasons, but if his absence is not
temporary this bug needs to be reassigned. And since there are those who have a
commercial interest in the success of Mozilla, and since this is an issue which
threatens that success and is officially listed as "critical" and causing data
loss, perhaps it is a candidate for reassignment to one of the few who are not
volunteers.
Symantec/Norton Anti Virus corporate (7+) and Intel LanDesk (6+) store thr
exclusion directories/extentions in the registry at the following locations:

HKEY_LOCAL_MACHINE\SOFTWARE\INTEL\LANDesk\VirusProtect6\CurrentVersion\Storages\Filesystem\RealTimeScan
 - keyname "ExcludedExtensions" collon separated list of extentions (all caps)
eg. "MSG,TST,EXT"

HKEY_LOCAL_MACHINE\SOFTWARE\INTEL\LANDesk\VirusProtect6\CurrentVersion\Storages\Filesystem\RealTimeScan\NoScanDir
 - key name is the full path eg "c:\dir\subdir" and the value is 0
(non-recursive) 1 (recursive)

It would not be hard to detect the existence of this key and add the profiles
directory upon installation.

I know similar registry locations exist for McAfee (I'll look them up tomorrow
at work) and other Anti-Virus programs. 

The least we can do is warn the user on startup that their mailbox needs to be
excluded if the exclusion is not detected. Offering to do this automatically and
 supply a link to the knowledgebase of the AV supplier, or a FAQ page on the
Thunderbird homepage.

This would also be a great addition to the (upcoming) msi package.
(In reply to comment #109)
> Norton 2004 etc are supposed to install a proxy mail server for pop3 whereby it
> recieves the message and then scans it and passes it on to the mail program. 
> Have to say I am not sure why it works with netscape, but appears to not be set
> up  with the install of mozilla mail I have.

I can confirm that NAV 2004 (and various earlier versions) use a proxy scanner
that scans all inbound and outbound e-mail (even if it's not going through the
Windows registered mail client).  However, it doesn't always seem to catch stuff
when it comes in.

My mail provider scans and cleans before delivery so I don't normally get to see
how NAV responds to e-mail worms, but in September a new batch of worm-spawn
edged out the filters and made it to my inbox.  Strangely, NAV didn't alert me
until I had the message open for about 15 seconds (either to read or forward). 
For what it's worth, here's the excerpt from the NAV log:

========== BEGIN CUT N' PASTE ==========
Source: Unknown00000000.data 
Description: The compressed file Unknown00000000.data within
Unknown0000288A.data within E:\winutils\Thunderbird profile\Default
User\Mail\pop.sbcglobal.net\After Thunderbird.sbd\Spam.sbd\Contain virus is
infected with the Download.Trojan virus. 
Click for more information about this virus : Download.Trojan
========== END CUT N' PASTE ==========


I don't know if it wasn't detected until that point because it was "compressed"
or what, but I was frustrated to find that the scan didn't catch it as it
entered my system.


My $0.02US.

-Neil R.
(In reply to comment #115)

> I can confirm that NAV 2004 (and various earlier versions) use a proxy scanner
> that scans all inbound and outbound e-mail (even if it's not going through the
> Windows registered mail client).  However, it doesn't always seem to catch
stuff when it comes in.

Thats quite interesting. I wonder if it only recognises certain versions that
possibly co-incide with Netscapre releases. Prehaps my proxy got dumped during
an upgrade.

Certainly however the AV needs to understand mbox, however Mozilla/firefox
should not leave itself open to this.

Regarding the option of just excluding the mailbox, I feel that is unacceptable
as we still have dormant viral code left on the computer.

What are the work arounds that are refered to?
Are they proper solutions or just quick and dirty hacks?

I'd like to point out that many users, such as a co-worker and myself, might use
Thunderbird in Corporate environments to check their personal mail. I have no
access to the corporate settings (well, technically I do, but I'm not about to
explain to the CTO that I need to exclude a file on my machine) and therefore
are in a situation where they probably need to use Outlook or Eudora to check
their mail. I've lost my inbox twice in the last month alone. Just food for
thought for those trying to determine if this is worth fixing or not ... 
I sufered the same problem, and i am sure this is not AV problem, but
Thunderbird's. Until recently I used Mozilla mail and I also use Outlook
Express. I had no problem of this kinf with them. I use Kaspersky Antivirus set
to report virus only. Kaspersky locks file and reports that it found virus.
Nothing more.

Thunderbird has problem with locked file and for some reason it truncates it.

In other mail readers, when virus occures, I turn of antivirus, delete virus
from mailbox, pack mail database and turn Antivirus On. With Thunderbirt it does
not work. When file is locked for the first time, and I turn off antivirus, file
remains locked for Thunderbird until i exit and start it again.

So, now, when Antivirus reports virus, I have to disable Antivirus, exit
Thunderbird, start it again, erase virus mail, pack database and then enable
antivirus. If I fail doing that it is likely that mailbox will be truncated by
Thunderbird.

I am experienced user and I can bet this is Thunderbird's bug. It seems it tries
to ignore the fact that file is locked which results in unwanted consequences.

What it should do:
- when it finds mailbox locked it should say so and do not remember that state
- it should check if file is locked each time it tries to access it, not to
remember the first occurence of lock
- if file is locked, Thunderbird should awoid any operation that may truncate
file. It should report the problem to user, instead, like saying "Hey, I cannot
do that, mailbox is locked, possibly by your antivirus"
*** Bug 270839 has been marked as a duplicate of this bug. ***
Product: MailNews → Core
*** Bug 233223 has been marked as a duplicate of this bug. ***
*** Bug 271578 has been marked as a duplicate of this bug. ***
I second (or third, fourth... :-)) the suggestion of switching to maildir
format. Not only does it provide security against the entire mailbox being
deleted when a subsequent AV signature update causes a previously undiscovered
virus to be detected, it is also significantly faster: the real-time scanners
cause a significant CPU load and delay when saving mails to a large inbox or
other folder, because they scan the entire file. This problem is magnified when
the profile or mail directory is on a network drive.

To those who suggest excluding the profile directory from AV scanning:
1. What if I choose to save my mail in a different directory? I can do this on
Account Settings / Server Settings / Local directory.
2. What if I add another profile? Do I have to manually add the exclusion to the
AV program?
3. (Perhaps a little contrived, but given that we're aiming for millions of
users, possible:) What if a user manages to save an attachment with a not yet
detected virus to their profile folder, and later (after an AV update would
detect it) open the file? They then have a virus on their system which would
otherwise have been caught.

I am not at all familiar with the Thunderbird codebase, but surely it cannot be
an immense problem to switch the mail storage system. There must be some storage
subsystem which tells the front end which messages are in a folder, and delivers
the message content, because IMAP works completely differently to POP3. Looking
at an existing IMAP account, the .msf files exist, but no mbox files until the
folder is marked for offline use.

There's probably enough mail program code with a BSD (or Apache?) license that
could be used as a starting point for maildir support.
my $0.02: this is a data loss bug, there should be no known data loss bugs in
any shipping software.  Whatever it takes to fix it, workaround it, whatever,
needs to be addressed ASAP.

I would also think maildir format would be better for this bug as well as I
would assume there would be performance/memory advantages of being able to
access individual emails not indexing into a large (many, many MB in my case) file. 
First let me say that I applaud the work so far of the Thunderbird developers. 
You have the makings of a potentially great mail client here.  With the release
of 1.0 I was getting ready to recommend it to people I know as well as making it
the standard mail client in the school district in which I work as head of IT.  

Then I found this bug, and I've had to stop dead in my tracks.  It is almost
unbelievable to me that a critical dataloss bug three years old has not been
fixed.  I've been looking at the posts here, and I have seen lots of excuses. 
"It's the AV companies' fault."  You're not going to change them all, if any,
especially with TB's small market share.  "It's ignorant users' fault."  The
world is full of ignorant end users, or corporates who can't control their
antiviruses.  Do you want to write off all of these people?

There have been some fixes suggested here.  Changing the mailbox format?  I
don't know if it will work.  How about changing the mailbox file extension?  My
NAV, for instance, doesn't scan .mbx (OE) mailboxes by default.  How about
changing to that?

The problem doesn't have to be solved tomorrow, and it may not be easy. 
However, it will never be solved by talking it to death.  C'mon, coders, let's
please get an solution to this problem so that I and others can use Thunderbird
and recommend it as what it should be - a world-class e-mail client.




Sorry, but in my last comment I meant that by default NAV does not scan .dbx
files (not .mbx) and that using .dbx on the mailbox files might be considered.


Changeing the mailbox format by stripping out attachments is probably not a good
idea, it potentially breaks other software. 

E.g. what would happen to imap folders if  I read the mail from another imap
client. Then the attachment would already be stripped out and residing on some
other client unavailable to me. 

If this can be fixed without any such side effects fix it, if not, mark it as
wontfix. By the way don't forget to complain to your AV software vender if you
are affected by this.
(In reply to comment #126)
> Changeing the mailbox format by stripping out attachments is probably not a good
> idea, it potentially breaks other software.  
> ...
> If this can be fixed without any such side effects fix it, if not, mark it as
> wontfix. ...

I agree that changing the mailbox format is not a good idea, at least certainly
not as the best quick way to fix this urgent bug. But fortunately there are ways
to fix it, or at least greatly reduce its severity, without changing the format.
One of those is the one I mentioned in comment 79: to save each attachment to an
incoming message as a temporary file, and then reopen it and save the reopened
version in the mailbox. This should force the virus scanner to scan the
attachment before it ever reaches the mailbox file. If this has a serious
performance hit, it can be made optional.

I know there can still be problems if a virus gets into the mailbox before the
anti-virus data is updated to catch it. But this should catch the great majority
of incoming viruses.
(In reply to comment #126)
> E.g. what would happen to imap folders if  I read the mail from another imap
> client. Then the attachment would already be stripped out and residing on some
> other client unavailable to me. 

What other mail client would be accessing Thunderbird's LOCAL cache of IMAP
folders?  Other IMAP client would acces the server's version which would still
have the attachments on each message.  We're talking about the local cache of
the IMAP folders and they are in the TB profile folder so no other program
should be touching them.
A lot has been said in this bug and people are beginngin to repeat what others 
have said.

There are a couple of solutions that will help solve this problem, one is 
better than the other and at present there is no solution that has been 
decided on.

To recap:
Add a extension to the mbox files (be it .bdx, .mbx, .mbox, .tbm wtc)
pro's
  -  The extension can easily be added to a list of excluded filetypes in all
     antivirusapplications
con's
  -  The virus would remain in the file unnoticed until it is finally
     extracted by a user to a location on the harddrive/floppydrive.
  -  Large coorporations are probably not willing to exclude extensions for
     different kinds of files because of the above.

Switch to maildir format
pro's
  -  each file itself would be scanned upon access and cleaned or removed. 
     This prevents the whole mailfile from being deleted.
  -  Compacting of folders will be a thing of the past (unless an index of 
     files is kept for other performance enhancements.
con's
  -  Lots of files on a harddrive cause defragmentation and thereby 
     performance degradation of the whole system. On windows systems
     the swapfile is configured to reside on the same disk as the maildir
     would be lowering performance even more.
  -  Complete change in format breaks compatibility with older versions and 
     would require a lot of extraction/conversion upon upgrades.

Extact binaries to a different folder altogether
pro's
  -  Infected attachments would reside in a different folder, virusscanners
     would not delete the mailfolder if a virus is detected.
  -  Changes to the mailformat would be small and can probably be worked 
     around by inserting file:// references to the attachments so that older 
     clients still know where to find the attachments.
con's
  -  Many plain text pieces in messages are detected as virusses (scripting 
     that manipulates outlook into doing bad things for example). These would 
     still reside in the mailbox file. Detection of these would still cause 
     a virusscanner to delete the whole maildir file.
  -  Naming issues or the attached files (what to do with attachments that 
     contain files with the same name), this could be solved by using a 
     foldername that is unique for each message, potentionally causing a lot
     of folders to be created.
  -  Small breakage of the current format
  -  Still need to extract all attachments upon upgrade.

Extracting an attachment (or save the whole message), see what the 
virusscanner does and re-include the attachment into the mbox file
pro's
  -  Known virusses cannot end up in the mbox file.
  -  Known scripted messages could also be deleted by the virusscanner 
     before infection of the mbox file occurs.
  -  No changes to the file format causing no breakage with older versions.
     delete
con's
  -  Virusses that are not yet known (due to older definition files) will
     still be included into the mbox file causing the mbox file to be deleted
     upon next access.

Nag the viruscompanies to fix their scanners till they've done so
pro's
  -  No changes to the file format
con's
  -  Needs coorporation of all anti-virus companies (which they've been very
     reluctant to give)
  -  Does not solve the problem for people stuck on older versions of the 
     virusscanner software

Encrypt the mbox format
pro's
  -  encrypted format is probably undetected by the antivirus software
con's
  -  Virus is still dormant in the mbox file
  -  Needs conversion upon upgrade
  -  Breaks compatibility with the older versions

Integrate with anti-virus software using MS-office integration
pro's
  -  Uses a way to scan the mbox file using a documented? format
  -  directly integrates with current versions of anti-virusscanners
con's
  -  Currently unknown how to do this integration
  -  Currently unknown if this will actually work.

Split the mbox into partitioned files of x Mb each
pro's
  -  Format is relatively unchanged
  -  Only part of mail history is lost
  -  Defragmentation of prevented due to larger filesize
con's
  -  Requires conversion upon upgrade
  -  breaks compatibility with older versions
  -  Dataloss still occurs

To conclude:

Safest method is to save messages in seperate files, but this will cause 
performance loss due to defragmentation.

Easiest to implement is add an extension to the mbox files and adding those to 
the exclusionlist (which is unlikely to happen in corporate environment).

Also easy is to implement is to encrypt the mbox files with a simple 
encryption/decryption stream.

Whichever method is chosen, none of them are perfect and none of them will 
completely solve all problems. It is however very important that a solution is 
chosen that at least minimizes dataloss.
(In reply to comment #129)
...
> 
> Switch to maildir format
...
> con's
>   -  Lots of files on a harddrive cause defragmentation and thereby 
>      performance degradation of the whole system. On windows systems
>      the swapfile is configured to reside on the same disk as the maildir
>      would be lowering performance even more.
...
> 
> Safest method is to save messages in seperate files, but this will cause 
> performance loss due to defragmentation.

Another con here, if you really mean saving each message (and not just each
attachment) as a separate file, is that it can be very wasteful of disk space in
some filesystems. A high proportion of the many thousands of messages in my
folders are 1 KB or 2 KB. But the minimum file allocation is 16 KB I think. The
result could be many thousands of wasted blocks of 14-15 KB. Not everyone has
gigabytes to spare on their disks.
...
> 
> Whichever method is chosen, none of them are perfect and none of them will 
> completely solve all problems. It is however very important that a solution is 
> chosen that at least minimizes dataloss.

Agreed. But who is in a position to choose and implement a solution? It seems
that Seth Spitzer has taken responsibility for this bug and gone away (or been
sent). Is there any way the assignment can be reversed - if Seth is not coming
back? Is the situation being officially monitored e.g. by our QA contact Esther?
Believe me, this is high on my priority list. My plan is to do the temp file
thing first, since that's fairly easy. It will probably be optional - I'll have
to see. I'm also thinking of doing the single file per message thing, though
that will definitely be optional, and is a lot more work. The rest of the
approaches (storing attachments separately, etc) are not likely at this point
given our mailbox format. Though I'd also like to add support for removing
attachments for other reasons, while still keeping our mailbox format.
Assignee: sspitzer → bienvenu
(In reply to comment #129)
 
> There are a couple of solutions that will help solve this problem, one is 
> better than the other and at present there is no solution that has been 
> decided on.

Bearing in mind comments #123 and #124 (i.e. the release change from 0.x to 1.0
and the effects this will have on expectations and production environments),
wouldn't it be pragmatic step forward if there was a new warning screen in the
install wizard, to tell the user to configure the virus scanner to quarantine
files instead of deleting them?

It might not help every layman but at the very least this would alert system
administrators to the risks involved giving them the chance to postpone (#124)
or configure the virus scanners properly.

Bearing in mind the consequences of this issue (meltdown) and the length of time
that developers have been aware of the potential disaster - anything short of
such a warning would be cavalier.

Alan
*** Bug 273969 has been marked as a duplicate of this bug. ***
(In reply to comment #132)
...
> wouldn't it be pragmatic step forward if there was a new warning screen in the
> install wizard, to tell the user to configure the virus scanner to quarantine
> files instead of deleting them?

Unfortunately this doesn't work round the problem properly. The problem is that
a quarantined file is unreadable. And at least some parts of Mozilla, including
the routine to compact a folder and maybe also the one to append a new incoming
message to it, seem to assume that an unreadable file is empty and so overwrite it.

This suggests another relatively quick fix, or partial fix: if a folder file is
found to be unreadable (but the file does exist), do something sensible like
flagging an error, but do not attempt to write anything to that file. See
comment 82.
> configure the virus scanner to quarantine files instead of deleting...
> Unfortunately this doesn't work round the problem properly. 
Not wanting to distract from your quick-fix I'd just like to point out that
configuring the virus-checker to quarantine files is the suggestion from the
issues FAQ and it's disappointing to hear that this doesn't solve the problem
after all.
How about a instal warning saying that the following anti-virus programs CAN be
used with thunderbird ... and that others may lead to lost inboxes?
Summary: Deleted inbox after receiving virus infected mail (McAfee, Norton) → Deleted inbox after receiving virus infected mail (McAfee, Norton, AntiVir)
(In reply to comment #129)
> Switch to maildir format
> pro's
>   -  each file itself would be scanned upon access and cleaned or removed. 
>      This prevents the whole mailfile from being deleted.
>   -  Compacting of folders will be a thing of the past (unless an index of 
>      files is kept for other performance enhancements.
> con's
>   -  Lots of files on a harddrive cause defragmentation and thereby 
>      performance degradation of the whole system. On windows systems
>      the swapfile is configured to reside on the same disk as the maildir
>      would be lowering performance even more.

What if the maildir was zipped? Most virus scanners can scan inside a zip file
-- can they clean/delete a file within a zip file?  Just put compression on low
for the zip (for better performance) and there could even be a user option for
the compression level so they can trade-off performance<--->disk-space.

>   -  Complete change in format breaks compatibility with older versions and 
>      would require a lot of extraction/conversion upon upgrades.

Why do we need to maintain backwards compatibility with older versions?  It's
not like a document editor where we have older clients opening files distributed
by newer clients.  We're (discussing) making a one-time conversion to a new
format. I don't think it's too much to require an initial conversion on an
upgrade either.


>>   -  Complete change in format breaks compatibility with older versions and 
>>      would require a lot of extraction/conversion upon upgrades.
>
> Why do we need to maintain backwards compatibility with older versions?  It's
> not like a document editor where we have older clients opening files distributed
> by newer clients.  We're (discussing) making a one-time conversion to a new
> format. I don't think it's too much to require an initial conversion on an
> upgrade either.

Thunderbird isn't the oinly client using the mbox format. A host of unix based
emailclients use this. I know a lot of people who use Thunderbird under
X-Windows and a text/commandline client when connected through SSH. So it is not
only a backwards compatibility issue. You must also keep in mind that if a newer
version isn't as much liked as an older version, or if the newer build contains
a nasty bug, people will want to be able to downgrade until a
workaround/fix/enhancement is made. Completely changing the format makes this
impossible.
(In reply to comment #137)
> >>   -  Complete change in format breaks compatibility with older versions and 
> >>      would require a lot of extraction/conversion upon upgrades.
> >
> > Why do we need to maintain backwards compatibility with older versions?  It's
> > not like a document editor where we have older clients opening files distributed
> > by newer clients.  We're (discussing) making a one-time conversion to a new
> > format. I don't think it's too much to require an initial conversion on an
> > upgrade either.
> 
> Thunderbird isn't the oinly client using the mbox format. A host of unix based
> emailclients use this. I know a lot of people who use Thunderbird under
> X-Windows and a text/commandline client when connected through SSH. So it is not

good point, I know that many others use mbox but I hadn't considered that other
clients might share mbpx folders with TB.  It does warrent consideration,
however, how many other mainstream email programs take that case (other email
programs sharing thier data) into consideration?  By mainstream I'm mainly
referring to Eudora, Outlook, whatever AOL uses. Maybe Evolution too, not sure
what it does.

TB (well the community) has to decide whether they are attempting to have TB
become widespread (like Firefox) or remain an email program mainly for
experienced users.  If we are targeting the general computer users, then we
can't be concerned with storing our email in a non-standard format and can't
worry about if some other program wants to share our data without going through
our APIs.  I'm not saying we shouldn't TY to play nice like that, however I
think it's more important that the bugs get fixed (especially data loss bugs)
then it is to worry about other programs directly accessing and using our data
without going through TB.

Note: obviously we document the storage format completly so anyone can read the
data if they want to add code for that

> only a backwards compatibility issue. You must also keep in mind that if a newer
> version isn't as much liked as an older version, or if the newer build contains
> a nasty bug, people will want to be able to downgrade until a
> workaround/fix/enhancement is made. Completely changing the format makes this
> impossible.

Well, what if TB introduces a bug that corrupts my data or gets all my data
wiped by, oh, I dunno, a virus scanner.  I can't downgrade to fix that.  I'm not
trying to be annoying, I'm serious.  Right now, there's a data loss bug in the
software. I'll be the first person to step up and say 'assume you won't be
perfect' and would normally be all over that agreeing with you.  However, in
this case, the tradeoff is fix a bug where you could permanently lose all your
data but people may have to deal with not being able to downgrade, or take the
chance of complete data loss.  

We have a large pool of alpha/beta testers and recent experience has shown that
the testing process works, and builds released for public consumption (0.8, 0.9,
etc.) have not had the major bugs in them that force people to downgrade --
that's why we test before releasing a milestone -- and fixes to major bugs have
landed quickly.

So, I would say, yes, I agree that that tradoff (not supporting downgrades) is
there, but I think the positives of fixing that bug outweight the negatives of
disrupting downgrades.


*** Bug 250401 has been marked as a duplicate of this bug. ***
(In reply to comment #130)
> (In reply to comment #129)

While using Mdir could be a good idea, a person would have to configure file
systems such as ext2/3 to cope with a very large number of nodes.

If they don't do this they waste a very large amount of disk space and if they
do do this then the filesystem will be very slow.

As suggested mbox with save to temp files, then merge scanned file into mbox
seems like a safe idea for now.

In reply to people saying that this would allow undiscovered code/scripts in and
then loose data afterwards consider that:

 - Not scanning will let said code trash your system.
 - Any form of scanning could trash your system.
 - Realistically, if the code got past your AV you probably have more things to
worry about.
*** Bug 275027 has been marked as a duplicate of this bug. ***
Ok..

Had it another 2 times...

Really frustrating and a good waste of my time reconstructing mail boxes.
I am affraid that mozilla/thunderbird is just not worth it at present.

Will be changing peoples mail clients, and will revisit the options when this is
fixed.
*** Bug 275372 has been marked as a duplicate of this bug. ***
(In reply to comment #142)
I was up to the point to leave TB as mail client due to this bug, when I
installed the new Symantec Norton Antivirus version 9 yesterday.
It installs a mailproxy that intercepts the mailstream and deletes viruses in
attachments on the fly. The result is a message like:
****** quote cleaned mail *******
Symantec Email Proxy deleted the following email message:

From: re-mail_system@hotmail.com
To: Mail-User@planet.nl
Subject: illegal signs in your mail
.
****** end quote ******

This suggests that you can still use TB or Mozilla Mail as your preferred mail
client, as long as you keep your antivirus program up to date :-)

Harold

*** Bug 275649 has been marked as a duplicate of this bug. ***
*** Bug 276186 has been marked as a duplicate of this bug. ***
This patch adds support for a (currently) hidden pref
"mailnews.downloadToTempFile" which makes pop3 download each message to a file
before appending to inbox - if a virus checker deletes the temp file before we
can append it to the inbox, we ask the user if they want to skip this message.
If they skip it, we won't try to download it again (and if leave on server
isn't set, we will delete the message from the server). We may want to add a UI
for this in advanced settings...but I don't think we want to default it to
true.
Attachment #170512 - Flags: superreview?(mscott)
Blocks: 241894
David - importing mail also triggers this - see bug 241894.
*** Bug 247223 has been marked as a duplicate of this bug. ***
*** Bug 235232 has been marked as a duplicate of this bug. ***
Same thing:  t-bird 0.9  incoming mail infected with Netsky_32  Norton report,
complete inbox deleted without a trace....
*** Bug 278580 has been marked as a duplicate of this bug. ***
I have also installed Symantec AntiVirus 9 and without even realising it at 
first have since noticed it intercepts all my mail as it comes in. I didn't 
have to configure anything.

However, even though this is the case, with the "mailnews.downloadToTempFile" 
idea, if an email passes through the scanner at first but is then detected a 
week later due to newer definition files, we are back in the same spot where 
we'll lose our Inbox. Likewise with Symantec AntiVirus 9... If mail passes 
through at first and is then detected as infected later, we too will lose our 
Inbox.

Regardless of the mail scanner we use, we are in a bad situation if we keep all 
our mail in the one file. To address this situation once and for all we need to 
change the way mail is handled to be either individual files, or encrypted so 
as to avoid initial detection. Then when the mail is re-opened, you could copy 
the email to a temp file, let the automatic virus scanner have it's way and 
amend the database accordingly if it gets removed.
Encryption makes the mailboxes much less portable.  The ultimate solution to
this problem, I think, is maildir.  However, maildir has it's own problems one
of them being ease of implementation.

I think the most reasonable course of action until maildir can be implemented
would be the file extension idea.  Granted, there will be messages in the box
that have viruses, however they will be detected if the attachments are saved or
launched.

It probably wouldn't be too difficult to have an option that allows the user the
specify the extension, with the default being the same as Outlook's mailbox
extension.

(In reply to comment #153)
Encryption makes the mailboxes much less portable.  The ultimate solution to
this problem, I think, is maildir.  However, maildir has it's own problems one
of them being ease of implementation.

I think the most reasonable course of action until maildir can be implemented
would be the file extension idea.  Granted, there will be messages in the box
that have viruses, however they will be detected if the attachments are saved or
launched.

It probably wouldn't be too difficult to have an option that allows the user the
specify the extension, with the default being the same as Outlook's mailbox
extension.

(In reply to comment #153)
I think the most reasonable course of action is to separate the attachments from
the messages so that if a virus is sent, only the attachment is deleted and not
the message.  This has other advantages too, like, I can delete large
attachments but save the message it came with.

Eudora does this and has been very successful.  

Just put a marker with a reference to the attachment in the message that is
stored in the mbox file.  This should be a portable, backwards compatible
solution if the reference is just a small text line/file inserted where the
attachment would be in the mbox file.  Other mail readers would just see a text
attachment and Moz would interpret that attachment as a refrerence to the actual
file somewhere else on disk. Moz can support both systems (actual file and
reference) simultaniously.
(In reply to comment #156)
> I think the most reasonable course of action is to separate the attachments from
> the messages so that if a virus is sent, only the attachment is deleted and not
> the message. ...

Although this would reduce the frequency of the bug, it doesn't actually fix it.
For a virus may be included in the main text portion of the message. I'm not
sure if it could be active in such a case (it would be if encoded with something
like UUENCODE), but it would still be detected by anti-virus scanners and as a
result the inbox or folder would be quarantined - although not usually deleted
completely.
Actually I hate that feature of Eudora.

In my experience, the attachments always end up disassociated with the messages
and either the user deletes the message and the attachment stays...or the user
clicks on the attachment and cannot pull it up...or the program forgets which
messages have attachments because the TOC gets corrupted.

This feature also makes it very difficult to move away from Eudora if you decide
you do not like it.

Also, as someone else indicated, what if the message itself contains something
the virus checker thinks is a problem?  An example of this is a paypal phishing
scam.

(In reply to comment #156)

Also see 260539 - a warning when this happens, so the user can retrieve the
inbox from quarantine.
(In reply to comment #159)
> Also see 260539 - a warning when this happens, so the user can retrieve the
> inbox from quarantine.

I managed to sucessfully retrieve tha mailbox from quarenteen many times, but if
the user has large mailboxes it can become a pain in the a**e.

I'm going to apply the patch to the latest mozilla and see where I get from there.

With regard to maildir, as I have pointed out before, this causes problems when
the file system hasn't been created for this, eg Maildir will use lots of inodes
to store a medium size mailbox. This could result in the FS running out of
inodes. If enought inodes are created then the FS will become slow for normal
operations.

I think the current save to temp file first is a good option.
*** Bug 280052 has been marked as a duplicate of this bug. ***
Comment on attachment 170512 [details] [diff] [review]
add (currently) hidden pref "mailnews.downloadToTempFile" which makes pop3 download each message to a file before appending to inbox

How about simplifiying the wording for the text. I don't think end users need
to be informed about the temporary file.

%brandname% encountered an error trying to download the following message: \n  
  From: \n%S Subject: %S\n

This message may contain a virus or there is not enough disk space. Skip this
message?

(Note the spaces before the From so we can indent the message info a little
bit).

When getting the prompt service in HandleTempDownloadFailed, shouldn't we just
get the  nsIPromptService, passing in the dom window from nsIMsgWindow. Or are
we supposed to go through the docshell? I'm not sure.
Attachment #170512 - Flags: superreview?(mscott) → superreview+
I'll change the message...re the doc shell window, your way does seem the
"right" way, passing in the true parent, etc, though I think it's more code :-)
I'll switch over to it anyway...the code I copied was probably older code.
patch checked in - the hidden pref is "mailnews.downloadToTempFile". If you set
it to true in prefs.js or user.js, pop3 mail will get downloaded to a temp file
first. If anyone wants to try it out with a virus checker and virus e-mails, and
let me know how it goes, that would be great. If it works OK, adding a UI for
this would be the next step (probably under advanced?)
Flags: blocking-aviary1.1?
I tried the new pref and it causes some problems when trying to download more
than one mail at once. For each mail following the first one I receive the error
message as decribed in bug 166111.
Sven, what platform are you running on?
I'll look into the problems with this pref set tomorrow.
(In reply to comment #167)
> I'll look into the problems with this pref set tomorrow.

Well, the problem is, the temp file is deleted in IncorporateComplete() after
handling the first message. But we only go through BeginMailDelivery() (where
the file is created) for each retrieve session, not for each message.
So when trying to write the first line of the second message
(WriteLineToMailbox() from IncorporateBegin()), we fail because the temp file
doesn't exist anymore.
(In reply to comment #168)
...
> Well, the problem is, the temp file is deleted in IncorporateComplete() after
> handling the first message. But we only go through BeginMailDelivery() (where
> the file is created) for each retrieve session, not for each message. ...

We really need separate temporary files for each message, as we don't want a
whole batch to be deleted or quarantined because one has a virus. So the problem
is not with the deletion, but that a new file is not created for each message.

thx, Christian. Yes, of course the idea is that we'd have a separate temp file
for every message. I just need to make sure it gets created for every message.
truncate temp file instead of deleting it after incorporate complete. We'll
still delete the temp file when done with mail delivery.
Attachment #173650 - Flags: superreview?(mscott)
Attachment #173650 - Flags: superreview?(mscott) → superreview+
There is another case this patch will not fix. Take a user whose virus
definitions auto update and does an automatic system scan weekly. If the user
gets an infected email before the new virus definition, his inbox still gets
nuked at the end of the week. 

The same happens to users who only scan weekly instead of every incoming email,
either because of misconfigured software or expired subscription. This is the
users fault but still makes Mozilla look like the bad guy.
(In reply to comment #172)
I think this issue can not be fixed 'cause of the internal design - all messages
are stored in UNIX mailbox format, which is a single file per folder.
(In reply to comment #172)
> There is another case this patch will not fix. Take a user whose virus
> definitions auto update and does an automatic system scan weekly. If the user
> gets an infected email before the new virus definition, his inbox still gets
> nuked at the end of the week. 
> 
> The same happens to users who only scan weekly instead of every incoming email,
> either because of misconfigured software or expired subscription. This is the
> users fault but still makes Mozilla look like the bad guy.

This scenario seems less likely to look like Mozilla/Tbird is the guilty party,
since the deletion happens as a direct result of the virus scan.  The data loss
I hit due to this bug was worst when a wave of several back-to-back virus
arrivals caused a month of inbox data to get shuffled to Norton's quarantine
area in chunks: the inbox was quarantined, new inbox was auto-created, mail
arrived, inbox was quarantined, new inbox was created, mail arrived, etc.

If the scan is a once-per-X (week, day, month) incident, it is unlikely to have
a 2nd incoming virus lead to inbox files getting repeatedly quarantined or even
overwritten (depending on how recklessness the antivirus software is). 
Hopefully, a user would notice the disappearing inbox before the next
full-system scan.

If, on the other hand, they are scanning all incoming email, but are just
getting updates on a once-per-week basis, they'll get the scanner update, lose
their virus-laden inbox, then they'll start losing each infected temp file.

Last, anyone relying on weekly virus updates is getting close to unsupportable
for being too far from following best practices.  While fixing that issue might
be possible, it's a much more pressing issue to create support/KB documents that
talk users through reconfiguring A/V software or using a different A/V package.

But hey, that's just my opinion-- I'm probably wrong.  Oh, and 1e6 *KUDOS* for
the design/development of the tempfile workaround!!!
*** Bug 282898 has been marked as a duplicate of this bug. ***
Quick question.

Does the final patch save a per e-mail temporary file or a per retrieve
temporary file. If the first is true then that is all good, if the second is the
case then surely the whole retrieval will be lost if any virus is found in any
of the downloading mails.

Also I agree with the comment that the person who scans once a week is outside
of supportability. If you scan once a week, then you know that you stand a
chance (a very high one these days) of getting infected in that week and having
to do some cleaning up. If a person is doing this they are probably aware of the
risks and also probably know what they are doing with the AV as they will
regularly encounter viral files.

Thanks for the fix mind, looks like I will be revisiting mozilla/thunderbird as
a mail client again :)
Sorry brain dead moment.

Regarding the pref, could it be made to default to true, or provide a UI dialog
at some realistic point in the future.
I just feel this feature would be useful to the homoe users.

Thanks
we create a temp file per message. Or to put it another way, there's never more
than one message in the temp file. Yes, a UI for this would be useful. I was
hoping to get some testing with the hidden pref first...
(In reply to comment #178)
> we create a temp file per message. Or to put it another way, there's never more
> than one message in the temp file. Yes, a UI for this would be useful. I was
> hoping to get some testing with the hidden pref first...

No problem I'll test it on some of the systems I have access to and give feed
back  after a week of running or so.
Well it's now running on a system I have with AV and there were no problems
getting mail. Havn't had anything for the AV to shout and scream about so far,
but seems good enough.

Mark



Geez! Some debate going here! I'm afraid it's too much tech talk for me, but
perhaps you're able to help us?

I still haven't deleted my email bow, however Kaspersky AVP has detected a nest
of worms in : 
C:\WINDOWS\Profiles\max\Application
Data\Thunderbird\Profiles\default.pac\Mail\mail.esmedia.cz\Inbox/[From "Rita 

I've deleted the independent emails from my email, however they still show in
the virus scan, and it seem like they are spreading. I'm not happy about
deleting the intire inbox of a few 1000 mails, just because of a few infected
mails...besides; they keep coming every week, so it's an ongoing problem for all
of us?!

I've also mailed Kaspersky, as I'm not sure whether it's the anti-virus
programmers who should improve, or whether it's a specific [silly] Mozilla issue...
Any ideas?  The worms are: Email-Worm.Win32.NetSky.x,
Email-Worm.Win32.NetSky.af, Email-Worm.Win32.Bagle.at, Email-Worm.Win32.Mydoom.m....

Thank you for your interest!
Max
Max, you need to compact the folder in question to actually remove the deleted
e-mails, right click on the folder and pick "compact this folder"
(In reply to comment #178)
> we create a temp file per message. Or to put it another way, there's never more
> than one message in the temp file. Yes, a UI for this would be useful. I was
> hoping to get some testing with the hidden pref first...

Ok had a week of testing and nothing seems to have fallen over.
Mails have been fetched ok and I purposely sent a copy of on of the various
netsky worms to one of the machines without the inbox being deleted.

So far looks good from where I am. Unfortuanately the Test PC is going to be
rebuilt soon, so that's about all the testing I can do for now
I believe this is normal and expected function.

When NAV finds a virus (etc), it can only work at the file level. Emails are 
stored in a single, large file, not individually.  So, NAV can't simply remove 
the one affected email.  It has to remove the entire file which in this case is 
the entire, full, mail folder (one example, the INBOX file).
I thought this one was working fine, but now today my inbox is gone!

PLEASE FIX ASAP!!!!!!!
I thought this one was working fine, but now today my inbox is gone!

PLEASE FIX ASAP!!!!!!!

Thunderbird 20050309
> I thought this one was working fine, but now today my inbox is 
> gone!

Did your anti-virus software simply delete or quarantine your 
Inbox file, or did it confuse Mozilla and cause Mozilla to wipe 
out some data (including preventing restoring a quarantined
file)?
 


(Note that if it's that your anti-virus software is configured 
to simply delete files it thinks are infected, there's not much 
Mozilla can do about it (besides going to an unrecognized and 
less-standard file format).  Besides, that's a pretty bad way to 
configure anti-virus software (deleting instead of quarantining 
or whatever)--complain to your system administrator, software 
vendor, or yourself.

If the case is that your anti-virus software is configured to 
quarantine files it thinks are infected, then clean the file if 
possible and unquarantine it.  If the anti-virus software can't 
clean it, then try to configure your anti-virus software to 
exclude the Inbox file from checking.  (You'll also need to
exclude the Trash folder's file and probably the Junk folder's
file too.)  If you can't change the anti-virus software's 
configuration so that you can use the original application
(Mozilla) to remove the virus (delete the mail message carrying 
the virus), that's an irresponsible restriction too--complain 
to your system administrator (or software vendor).  

However, if the anti-virus software quarantines the file, 
causes Mozilla to get confused, and Mozilla does something that 
prevents the anti-virus software from restoring the file (e.g., 
Mozilla creates a corrupted partial copy of the Inbox file), 
then that is clearly something that Mozilla could handle better. 

(I wouldn't argue that it's a full Mozilla bug (Mozilla's fault),
since no software (e.g., the anti-virus software) should ever be 
f***ing around with and deleting private files out from under a 
running application in the first place.  (Few applications accept
the burden of running without problems if their files are 
asynchronously deleted.)  However, since Mozilla does seem
to get confused and possibly prevent restoring of a 
quarantined file, I would call it a robustness bug.)
 

Daniel

(In reply to comment #187)
> ...
> 
> However, if the anti-virus software quarantines the file, 
> causes Mozilla to get confused, and Mozilla does something that 
> prevents the anti-virus software from restoring the file..., 
> then that is clearly something that Mozilla could handle better. 
> 
> (... I would call it a robustness bug.)
>  
I wonder if Sasquatch's problem is related to what I wrote in comment 82 to this
bug. This problem is that in at least one case, compaction, when Mozilla cannot
read an inbox because it has been locked or quarantined by anti-virus software,
Mozilla deletes that inbox. I don't think this part of this bug has been fixed,
or addressed at all. And, while arguably it is not Mozilla's problem, it is
indeed a robustness issue, and also one of public relations: if Sasquatch's
problem gets wide publicity, Mozilla's reputation could be seriously damaged.

The fix should be a fairly simple one, but maybe at some deep level in the code:
if a file which Mozilla expects to find exists but is unreadable, it should
never be deleted or overwritten, but probably the activity should be aborted
with an error message.
(In reply to comment #186)
> I thought this one was working fine, but now today my inbox is gone!
> 
> PLEASE FIX ASAP!!!!!!!
> 
> Thunderbird 20050309

Is this patch included in any builds of Thunderbird, as the bug appears to be
marked as NEW and not confirmed or resolved.


I think part of the problem is that the NAV and Thunderbird are both working at
the same time. The mail comes in, one with a virus gets picked up, then the
inbox gets deleted, then another bunch come in (making a new inbox!), then
another virus gets the inbox deleted again, etc.

Not good.
(In reply to comment #190)
> I think part of the problem is that the NAV and Thunderbird are both working at
> the same time. The mail comes in, one with a virus gets picked up, then the
> inbox gets deleted, then another bunch come in (making a new inbox!), then
> another virus gets the inbox deleted again, etc.
> 
> Not good.

Ther eis a patch attached which saves the mail to a temporary file which then
allows the AV to do it's evil and not delete the entire inbox.

From the symptoms your getting I would say that whatever build of T-Bird  your
using doesn't have this patch applied. I don't know if it has actually made it
into nightly builds yet, so you may need to check out of cvs or patch the source
& build it yourself. I know this isn't much help, but it's all I can think of at
the moment.

The patch doesn't fix what was already described in comment #52 (virus already
in your inbox and (days) later the virus scanner finds it). The only way
to fix this would be to change the mailbox storage format to maildir instead of
mbox (quite a major change but not impossible). This is best discussed by the
drivers or those with authority on the future of thunderbird/mozilla. - IMHO I'd
rather go maildir to fix this and other issues. I think this horse was whipped
some time ago but I'm not sure where I read it (as to mbox vs maildir).

Would someone on the thunderbird/mozilla mail team like to comment and put us all
in our place?  ;-)
I believe the better option is to provide alternative storage methods, than
replacing mbox with something else. 

Pegasus supports two alternative storage methods - its default system or mbox. 

Note further that there are four different incompatible mbox formats
http://en.wikipedia.org/wiki/Mbox
, so mbox is not only one standard supported by all linux formats...

My recommendation would be to make mailbox formats plugin based, giving room for
all types of formats.

Thunderbird to have at least two alternative box formats. Note that there are
not only one common mbox format, but four different incompatible mbox formats!

I think the better option of this reason would be to make storage formats plugin
based adding two plugins by default - one for the current mbox format and one
that would store each message in individual files - for example as a file-based
xml database. This way we add flexibility for adding new formats with unique
capabilites - some maybe storing the mail on a server and only maintaining a
local index for quick searches, some encrypted mailboxes, some compressed
formats, and whatever other mailbox format people are willing to make a plugin for. 
*** Bug 256043 has been marked as a duplicate of this bug. ***
Whiteboard: [asaP1]
sounds like the work for this fix is completed.  blocking1.1-  

renominate if this is incorrect.
The patch does work for me, and I have no reason to believe that moz/TB can be
expected to do anythign about a virus that has come in and got through the AV
and is later deleted by the AV.

This fix prevents Moz from looking bad, by having it trash the inbox when a
virus comes in. I addition it also focuses the problem back onto the AV vendors
who should not be carrying out whole file deletes/quarantine for a commonly used
file format.

Anyway fingers crossed this work around should protect anyone with an up to date
AV where a mail happens to get past the now common mail proxy systems.
This is still not marked as fixed. Is it or isn't it fixed? Thanks.
u can tell this has been resolved by going to privacy>anti-virus. now, av can
delete individual messages instead of the whole mailbox. this should be marked
fixed.
OK, marking fixed. We've already got a bug open for supporting maildir.
Status: NEW → RESOLVED
Closed: 19 years ago
Resolution: --- → FIXED
(In reply to comment #198)
> u can tell this has been resolved by going to privacy>anti-virus. now, av can
> delete individual messages instead of the whole mailbox. this should be marked
> fixed.

I turned on on the option to quarantine individual messages, and now when I am
prompted to skip a message that might have a virus in it, I sometimes get this
error:

Unable to write the email to the mailbox. Make sure the file system allows you
write privileges, and you have enough disk space to copy the mailbox.

It doesn't happen all the time, but when it happens, it stops downloading all
other messages as well. I have tons of disk space available, and I do have write
privs to that folder.
Using Thunderbird 20050413. Just got bit by this again, and this time I had
"Allow anti-virus clients to quarantine individual incoming messages" turned on.

Virus came in, my entire Inbox got quarantined.

I don't think this is really fixed.
(In reply to comment #201)
> Using Thunderbird 20050413. Just got bit by this again, and this time I had
> "Allow anti-virus clients to quarantine individual incoming messages" turned on.
> 
> Virus came in, my entire Inbox got quarantined.
> 
> I don't think this is really fixed.


same thing for me with the last build and "Allow anti-virus clients to
quarantine individual incoming messages" turned on too.
huge problem for professional use.
Flags: blocking-aviary1.1?
*** Bug 296043 has been marked as a duplicate of this bug. ***
Is it worth reopening this bug to avoid a flood of duplicates.

It does appear that there is still an issue with the original fix.

I'm sorry, but is this one fixed in 1.0.6 (last official "release" version, right?)?

I just had someone who had this error in this version today. I've been using
trunk for a while, but my PC is in for repairs right now. I can't seem to find
the setting anywhere in 1.0.6. Is this trunk only, or am I missing something?

Thanks.
It happened to me for the 3d time now, PLEASE fix it properly (its over 1 year I
reported it) Stopped using Thunderbird for the moment.
I still don't understand why there can't be an option to add a file extension to
the mailbox.  This way to virus scanner could be configured to ignore the
mailboxes all together.

The solution of having a temp file simply doesn't make sense.  This is always
going to result in having the mailbox quarantined at some point.  Unless you
subscribe to the conspiracy theory that anti virus companies write the the
viruses, it should seem pretty obvious that the temp file solution is not going
to work in the long term (or in some cases the short term).  

After all, scanners can only check for viruses after they are known.  Therefore
there is a window between when you can receive the virus and when the virus
scanner will detect the virus.  If the virus comes in during this window, your
entire mailbox will get quarantined.  This seems rather obvious to me and lot
less sensible then simply excluding the mailboxes all together.
To: junk@path.berkeley.edu

Most virus scanners use huristic scanning which will try to detect new viruses 
based on previous viruses. Sure, this won't catch all, but this is the way life 
goes. We won't be able to change this and so there's no point dwelling on it.

Some virus scanners allow you to exclude entire directories. Why not try 
excluding your mail directory. This way it's safe. Then, when you try gaining 
access to an attachment, it'll be saved to a temporary location and your virus 
scanner will then pick up on it.

Cheers,
Jeremy
Marking this one to block any future possible branch builds. This is a dataloss
and possibly could even be considered a security issue.
Flags: blocking-aviary2.0-
Flags: blocking-aviary1.0.7?
> Some virus scanners allow you to exclude entire directories. Why not try 
> excluding your mail directory. This way it's safe. Then, when you try gaining 
> access to an attachment, it'll be saved to a temporary location and your virus 
> scanner will then pick up on it.

The reason why is because I manage a large number of workstations.  The virus
scanning is centrally managed.  Yes I can exempt specific directories, however,
since the mail directories change on a per machine/user basis I would have to
handle exclusions for all the machines individually.  With a file extension
based scheme, I can globally tell the virus checker to ignore the files across
all machines.  In addition, I would not consider excluding an entire directory
the best practice.  It gives viruses a very easy place to hide.

I'm of the opinion that most viruses should be stopped before the user's machine
if possible anyway.  Our site uses virus scanning at the mail server, and in
addition uses attachment file renaming and mime retyping for things like .exe,
.scr, .pif, etc.  Most viruses don't make it to the machines, and of the ones
that do, almost all of them are neutralized.  In order for a user to execute a
virus, they have to save it and rename it thus forcing them to think about it
instead of just clicking on them.

Granted it's not great to have viruses sitting in the mailbox, however, it's not
too bad if they can't be clicked on and executed.  Having the mailbox
quarantined, however, is unacceptable.  This issue is preventing me from
deploying Thunderbird on a wide scale within my organization (something I'd
really like to do because of it's cross platform compatability, junk filtering
capabilities...and it's not outlook).

> Most virus scanners use huristic scanning which will try to detect new viruses 
> based on previous viruses. Sure, this won't catch all, but this is the way life 
> goes. We won't be able to change this and so there's no point dwelling on it.

Exactly.  This is why the current solution is no solution.


(In reply to comment #210)
...
> The reason why is because I manage a large number of workstations.  The virus
> scanning is centrally managed.  Yes I can exempt specific directories, however,
> since the mail directories change on a per machine/user basis I would have to
> handle exclusions for all the machines individually.  


This is even more pronounced because of these bugs:
Bugzilla Bug 247973
Bugzilla Bug 233746 (if it also applies to Thunderbird=?)


> all machines.  In addition, I would not consider excluding an entire directory
> the best practice.  It gives viruses a very easy place to hide.

So does a filename extension...

Sry for this agression, but wtf is going on here?

There is an extremely heavy bug in Thunderbird, that makes it unusable and stops
it from getting the chance to ever run in a corporate environment.
Just marking this bug as fixed doesn't make it true.
It's a fact, that if your virus definitions get updatet after you recieved a
virus-infected mail, your mailbox gets deleted. That's not acceptable, so this
bug is definitely not fixed.
It's a fact that very few virus-scanners implement a feature to exclude a
directory, but nearly every one of them has an "exclude extension" feature.
Everyone here knows, that the profile-paths include the username, so it is
impossible to set up a virus-scanner to exlude all the profile-directorys for a
whole company.

It's not our problem if the solution is secure or not, because it's the problem
of the anti-virus supplier. They seem to believe in the sercurity of their
"exclude extension" feature, otherwise they wouldn't include it in their
scanners. If they want a more secure solution, they will have to build in
support for every mailbox format out there. But as said before, that's not our
problem.

So why can't just someone give the mailbox files an extension, so thunderbird
can finally compete with other email-clients in corporate environenments?
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
re comment #211: Those bugs have got nothing to do with this

re comment #213: So exclude the whole of %appdata%\Thunderbird\Profiles (or
wherever the profiles are stored for your machine/company), or simply disable
the scanning of mime-encoded files. Or get better AV software.

re the issues of adding an extension or switching to maildir: If they aren't
already, file enhancement requests. They are beyond the scope of this bug.

Anyone who still wants to moan about the way this has been fixed: read
https://bugzilla.mozilla.org/etiquette.html
Status: REOPENED → RESOLVED
Closed: 19 years ago → 19 years ago
Flags: blocking-aviary2.0-
Flags: blocking-aviary1.0.7?
Resolution: --- → FIXED
Excellent. So pointing out that a bug marked as FIXED isn't really fixed is now
considered "moaning?" The point is that the "save each message as a temporary
file so the AV software can scan it and potentially quarantine it" approach
still results in the entire In box getting quarantined on occassion. 

Given this:

> It's a fact that very few virus-scanners implement a feature to exclude a
> directory, but nearly every one of them has an "exclude extension" feature.

Your solution is this:

> exclude the whole of %appdata%\Thunderbird\Profiles (or
> wherever the profiles are stored for your machine/company), or simply disable
> the scanning of mime-encoded files. Or get better AV software.

? 

If you want Thunderbird to be taken seriously in the corporate world, you can't
seriously expect everyone to upgrade their AV software simply so they can use
your software. 

Actually, if you want Thunderbrid to be taken seriously in the consumer world,
you can't expect that either.

This bug is extremely serious because it appears to be (and in some ways is)
data loss. There's no indication that your In box get eaten by your AV software.
It looks like Thunderbird lost it. 

Refusing to fix this bug isn't going to help anyone. I mean, look at Comment
#206. You've already lost one user. And that person was helping to test.

Done.  I did a search and didn't find it so I added it.
https://bugzilla.mozilla.org/show_bug.cgi?id=305354

I honestly am surpised at this attitude.  I cannot understand how anyone could
consider the temp file fix to be a real fix to this problem, and thus making the
matter resolved.  It's obvious it isn't.

> re the issues of adding an extension or switching to maildir: If they aren't
> already, file enhancement requests. They are beyond the scope of this bug.
> 
> Anyone who still wants to moan about the way this has been fixed: read
> https://bugzilla.mozilla.org/etiquette.html


re comments 215, 2nd part of 216

Please remember that this entire problem is caused by bug(s) in certain AV
software. The fix checked in worksaround the problem for the case of a virus
being detected as it arrive, which is the most common complaint in this bug by
far. The real fix can only come from the makers of the affected AV software.

For the case of the AV software detecting a virus after it's been sitting on
your system for a while, I'd say you've got more serious problems.

A more comprehensive Mozilla 'fix' for this *AV* problem, would require
significantly more work, work that obviously isn't felt worth it at this moment
of time by the *2* people who currently do work on Thunderbird.

As always in open source - patches are welcome.
> For the case of the AV software detecting a virus after it's been sitting on
> your system for a while, I'd say you've got more serious problems.

Um...virus comes in...virus definition comes in.  It doesn't have to be a while.
 In fact the time differential may be an hour or less.  There's not much that
can be done about that.  We're not talking about viruses that have been there a
month...or week...or even a day.

Got stung by this bug 2 more times this past week!!!!!

Mozilla Corp looks like fools.

 )-:
We can't fix this "properly" (this means a way to have only the infected mail
quarantized). That's correct - that's the job of the AV-supplier. BUT we CAN
give thunderbird the abillity to compete with other email clients, wich all
suffer from the same problem.
You understand? We can't fix it - all we need to do is to get on the same level
with the other email clients. This can be done by simply adding a file extension
to the mailboxes. And please don't tell me that this would be too much work.
(In reply to comment #220)
> We can't fix this "properly" (this means a way to have only the infected mail
> quarantized). That's correct - that's the job of the AV-supplier. BUT we CAN
> give thunderbird the abillity to compete with other email clients, wich all
> suffer from the same problem.
> You understand? We can't fix it - all we need to do is to get on the same level
> with the other email clients. This can be done by simply adding a file extension
> to the mailboxes. And please don't tell me that this would be too much work.

See comment #39, by the developer assigned to this bug. If you think you know
better than please submit a patch.

Please - no more comments in this bug!
*** Bug 264936 has been marked as a duplicate of this bug. ***
It seems someone had a similar problem with a nightly branch and "NOD".

http://forums.mozillazine.org/viewtopic.php?t=330908


Can't there be a new beta release to help these poor people? Anyone think
Thunderbird should release BEFORE Firefox?
What if the "message filter" feature be extended in such a way that an external program can scan individual emails and give yes or no answers according to whatever criterion it carries. Then we can do virus-scaning like this:

  if Message-Body  get-yes-when-scanned-by  "Virus-Scanning-Prog-A"
    move Message to "mail with virus" folder

Even better, it could be this (requires further extending, though):

  if Message-Body  get-yes-when-scanned-by  "Virus-Scanning-prog-A"
    process Message with "Virus-killing-prog-B"

I'm proposing this because message filtering can be run both on message retrieval and later on user demand ("Run filters on folder"). Users are given a chance to identify infected mails ALREADY IN THE MBOX FILE.

I really need this feature. I've used Thunderbird for more than a year without doing any virus scanning on mails until recently. (Yes I use linux and I'm very lucky.) I installed clamav only a few days ago to find 2 virus in my inbox. Without this feature I can't imagine a way to pick out the 2 infected mails from a 200M+ mbox.
This one seems to have come back recently with a 1.5 version of Thunderbird and Norton/Symantec anti-virus. 

Anyone else? 

I restore from the av software, and the inbox comes back, but then gets taken out again after a new scan.

Help!
(In reply to comment #225)
> This one seems to have come back recently with a 1.5 version of Thunderbird and
> Norton/Symantec anti-virus. 

You may see, for Thunderbird and the Mozilla Suite:

Antivirus software - MozillaZine Knowledge Base
http://kb.mozillazine.org/Antivirus_software

(with the sections "Keeping your antivirus software from deleting your Inbox", "Antivirus programs with compatibility problems", etc.).
The point is that I thought that quarantining individual message IS the fix, and that setting has been and still is checked off on the Thunderbird in question. The trouble is that it isn't working.
No.  The "fix" stores a copy of the message outside the inbox before adding it to the inbox to give a virus scanner the chance to scan the message before being put in the inbox.

As is mentioned many times earlier in this thread, this really isn't a fix at all because it depends on virus scanners being able to detect the message when it arrives.  If it gets detected later, it eats your inbox.

Thunderbird really needs a method of adding a file extension to mailboxes that can be exluded from a centralized virus scanner on a global level.  Exclusion is currently very difficult because profiles have random names and this makes deployment across many machines very difficult.

In your case you can probably tell the scanner to avoid your mail directory (kind of cripples the effectiveness of scanning though).
So then what is the point of quarantining individual messages? For that matter, what is the point of antivirus software if you can't run it on your email which is the most likely point of entry lately?
Good questions.  Ideally your virus scanner would understand the mbox format and quarantine only the infected messages.  Your (most) virus scanners are dumb and don't do this.  

Scanning the message when it comes in, gives you a chance to prevent quarantining the inbox.  If used with something that would exclude inboxes from being scanned it would be helpful.

In my opinion, you have to rely on user education once the message makes it past the scanner.  After all, if the scanner misses it (Even if it had the ability to quarantine individual messages instead of the whole box) the message will sit in the user's box until the scanner was updated.  Therefore the user has to know not to click it and to delete it, anyway.
I know this has been raised before (e.g., comment #2).  However, viruses do seem to be spread not in the body of messages but in their attachments.  This whole mess could be resolved by separating out the attachments the way some other E-mail clients do (e.g., Eudora).  Until this is at least an option, this problem will not go away.  

Note that Norton Anti-Virus notifies the user every two weeks to update the virus definitions if the user has not enabled automatic live update.  Except when there is a serious attack, daily updates are really not necessary.  Thus, the recently added capability to place new incoming messages in a separate temporary file will mean either a proliferation of such files until the next virus definition update or else a loss of the entire inbox upon the next update even if those temporary files were retained for a week.  

There is a recommendation to keep the inbox almost empty to avoid serious loss from this problem.  Be honest.  How many of you have fewer than 100 messages in your inbox?  (No, don't answer here.  Just think about it.)  I keep filing messages in other folders and deleting messages, but my inbox has over 400 messages.  In any case, it's possible that the loss of only one very important message could be catastrophic.  And anyway, this recommendation could merely shift the loss to other mail folders.  

I won't use Thunderbird for E-mail until a real fix is implemented.  


I personally hate that feature of Eudora.  It makes it difficult to migrate to another mail client, Eudora often loses its association between messages and attachments, and often when you delete the message the attachments stay.

Maildir is a much better option where each message is saved as a separate file.  I think implementing maildir in thunderbird would be a significant undertaking though.
Has it been seriously considered to develop an "Antivirus API" for Thunderbird? The minimal requirements for such an API would include being able to:

* Scan individual e-mail messages every time the user accesses them.

* Scan the raw e-mail as well as individual parts/attachments as they are parsed by Thunderbird.

* Add, read and modify certain headers in the messages.

* Remove individual parts/attachments.

* Modify individual parts/attachments, including changing the name and MIME type.

* Instruct Thunderbird to remove/delete the entire message (even based on scanning just one part/attachment of the message) and compact the folder.

* Instruct Thunderbird to block access to individual parts of the message or the entire message and inform the user of the reason (infected message).

I'm aware of the setting [Tools->Options->Privacy->Anti-virus], but neither a POP3/IMAP scanner nor individual spool files are a good solution.  (Why is this setting under Privacy and not Advanced?  What's the privacy implication?)

If you look at [http://kb.mozillazine.org/Thunderbird_:_FAQs_:_Anti-virus_Software] then one of the recommendations are "[C]onsider waiting a while before opening the attachment. This gives your AV program's manufacturer a chance to provide a perhaps necessary new update."

To do this for the entire e-mail (since there is HTML/Javascript-based malware) the message would need to be rescanned every time it is accessed after a virus signature file update.  For that an API or a method to plug into / hook into Tunderbird is necessary.  Scanning the entire mbox is a waste of CPU and has performance issues if only a few e-mails in a large mailbox are being accessed.

We at FRISK Software are interested in developing an F-PROT Antivirus plugin for Thunderbird that would be a part of our product, similar to our current Microsoft Outlook plugin.

P.S.  Also posted to http://forums.mozillazine.org/viewtopic.php?p=2740502 and discussions on similar topics can be found in bugs 247223 and 369847.  Sorry for the spamming.
(In reply to comment #236)
> Has it been seriously considered to develop an "Antivirus API" for Thunderbird?...

> I'm aware of the setting [Tools->Options->Privacy->Anti-virus], but neither a
> POP3/IMAP scanner nor individual spool files are a good solution.  (Why is this
> setting under Privacy and not Advanced?  What's the privacy implication?)
...

Or a junk/virus/spam API.
The quickest, simplest, safest solution is to give all mbox files a ".pst" or ".dbx" file extension so that they are, by default, excluded by most old and new antivirus software.

With such a solution in place, can we later worry about evangelizing Thunderbird to the AV developers to fix their mbox scanner.
Product: Core → MailNews Core
Keywords: relnote
Blocks: 1144999
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: