169991 - News (usenet) posts with .au domains display corrupt QuickTime plugin instead of text.

Status: RESOLVED FIXED

(Core Graveyard :: Plug-ins, defect, P2)

Description

[Dash]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.2b) Gecko/20020920
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.2b) Gecko/20020920

Random (to each execution of Mozilla), some plain-text Usenet posts display a
QuickTime plugin that failed to load, instead of the content.

Source of the message (have to use View.Message Source, Ctrl+U fails on these
messages) is seemingly normal with nothing out of the "norm".  Message often
viewable after restarting Mozilla.

QuickTime 5.0.2 Pro installed.

Reproducible: Sometimes

Steps to Reproduce:
1. Load Usenet.
2. Subscripe to any newsgroup.
3. Browse messages until locate.

Actual Results:  
See details.

Expected Results:  
Display a plain-text message.
Unable to test binary messages.

Comment 1

[K Chayka]

win32 trunk build 2002092404, win98se

I see this, too, particularly on posts from one particular person ("rf", a
regular poster in alt.html.critique and some c.i.w.a.* groups).  Pretty much all
keyboard shortcuts appear to be disabled while this message is in view, too, but
the mouse works OK, so View Source is available from the menu.  Will attach the
source of one of these posts.  The posts from this individual display just fine
in the 2002091008 build.  The saved .eml file also displays fine in the browser
window, even on the 0924 build.

Comment 2

[K Chayka]

Attachment: source of an offending message

Comment 3

[Dash]

Attachment: Another sample

This is a random message that generates the bug, I can't see anything wrong
with it's source.

Comment 4

[K Chayka]

Attachment: screenshot showing the problem

Looks like the problem starts with the 2002091808 build.  0917 looks fine.

I see the broken QT plugin instead of the message text on all posts for
particular individuals.  The common denominators between them are:

1. Newsreader MS OE v6
2. NNTP posting host somewhere in Australia

Probably coincidence, but it was pretty odd.

Comment 5

[K Chayka]

Seems the Aussie connection is more than just coincidence.  Go into QuickTime
Preferences, Browser Plug-in MIME settings and uncheck Audio type:
uLaw/AU audio file (au,snd,ulw)

The message text for the broken posts starts displaying normally.
BTW, I also have QuickTime 5.02.

So, steps to reproduce are:
1. Install QT plug-in, check Browser plug-in Audio MIME type for "au"
2. Start mail-news (plugins disabled in mail-news may also be relevant)
3. View a post that originated on an Australian server*.
The expectation is that the message text displays, actual result is a broken QT
plug-in.

* This probably happens with mail or news, but I don't correspond with anyone
from Australia so I can't verify mail.  I found 3 Australian posters, including
"rf", on Usenet newsgroup alt.html.  All of their posts showed the broken QT
plug-in before changing the MIME type, all displayed normally afterwards.

BTW, I would change this bug's status from Unconfirmed to New, but I am not
authorized to do so.

Comment 6

[[e-mail] (gone - use [e-mail] instead)]

Confirming.  I can't even read the message in the client, although I can see the
message in the source.

news://news.mcom.com:119/ks9h9.9695$[e-mail] is an
example of one.

Comment 7

[jay garcia]

Confirmed Win98se Mozilla build 2002092904

Also confirming the Australian connection as well for whatever odd reason.

Comment 8

[[e-mail] (gone - use [e-mail] instead)]

Jay, it's most likely because the top level domain of .au is being read as the
Quicktime file extension .au.  See comment 5.

Comment 9

[Mike Kaply [:mkaply]]

I see this on OS/2 and can confirm that the issue is the ".au" extension.

On OS/2, we have a separate audio plugin looking for .au and it gets loaded in
these cases.

Comment 10

[Mike Kaply [:mkaply]]

This happened between 9/17 and 9/18

The bug referenced in peterl's checkin to nsPluginHostImpl.cpp is incorrect.

Looks suspicious.

copying peterl

Comment 11

[Peter Lubczynski]

possible regression from bug 163568?

Comment 12

[Hermann Schwab]

Mozilla build ID 2002092704 on Win98 SP1

one more news from australia displays corrupt QuickTime plugin:
snews://secnews.netscape.com:563/[e-mail]

BTW, context menu is missing some functions with snews://
i.e. Open Link in New Window, Open Link in New Tab, Save Link Target As ...
Ctrl-U doesn´t work, but View/PageSource does.

Comment 13

[WD]

Peter:   Yes, backing out the patch for bug 163568 does indeed fix this bug.

Comment 14

[Matthias Versen [:Matti]]

-> Plugins

Comment 15

[rubydoo123]

hmm, handing this over to Peter since the patch back out fixes the problem

Comment 16

[Mike Kaply [:mkaply]]

I'm a little disappointed in the target here. This is a pretty huge regression.

Comment 17

[Phil "clef" Sweeney]

Ummm yes I agree with Michael.  This bug prevents me from using any trunk
builds, and if it continues I won't even be able to use 1.2beta.  Why isn't the
bad code being backed out?  I don't think anyone really needs to go to
http://www.exoticeroticball.com/ =)

Comment 18

[Peter Lubczynski]

Attachment: patch v.1

Here's the fix. Please review.

Comment 19

[av (gone)]

Comment on attachment 101798 [details] [diff] [review]
patch v.1

r=av

Comment 20

[Peter Lubczynski]

Hm...this seems to be working just fine without the last patch in the build I
just made. I think something checked in this weekend, possibly the fix from bug
157135, fixed this. Marking WFM.

Comment 21

[Phil "clef" Sweeney]

Still seeing it on nightly build 2002100714 (winxp)

Comment 22

[Dash]

I'm also still seeing it, using the same build (2002100714) Win32.

Comment 23

[[e-mail] (gone - use [e-mail] instead)]

*** Bug 173207 has been marked as a duplicate of this bug. ***

Comment 24

[[e-mail] (gone - use [e-mail] instead)]

Reopening.  

Comment 25

[Peter Lubczynski]

Attachment: patch v.2

I must have had a bad build. I see the problem again today.

Here's a new patch that makes http://www.exoticeroticball.com/ and
news://news.mcom.com:119/ks9h9.9695$[e-mail] work
in todays build.

Comment 26

[av (gone)]

Comment on attachment 102197 [details] [diff] [review]
patch v.2

So, the patch leaves out those schemes which do not implement |nsIStandardURL|,
r=av

Comment 27

[Darin Fisher]

Comment on attachment 102197 [details] [diff] [review]
patch v.2

peterl:

why is that you want to only do this for URLs that happen to implement
nsIStandardURL?  i don't understand the connection.  doing this kind of
filtering seems very fragile.  what are you really trying to get at here? 
would a whitelist of URI schemes make more sense?

i would only consider this patch as a temporary, last-resort sort of solution.

Comment 28

[[e-mail] (gone - use [e-mail] instead)]

*** Bug 173730 has been marked as a duplicate of this bug. ***

Comment 29

[Arthur]

What bothers me is that the plugin programm is started even though my
preferences say that plugins shouldn't be started for mail and news. Is this
addressed with the current patch or could some evil person still manage to get a
plugin executed?

Comment 30

[[e-mail] (gone - use [e-mail] instead)]

Edit | Advanced | Scripts & Plugins.

Is "Mail & Newsgroups" checked under "Enable Plugins for"?

For me, it is, which is no surprise why this is then behaving like this. 
However, I thought that we wanted to disable plugin support in mail.  I verified
the bug on disabling JS in mail, but not plugins (if there indeed was such a bug).

Yulian/Scott, do you remember such a bug?

Comment 31

[WD]

I believe Plugins are disabled for Mail/News by default.
That's how it is on my machine (unchecked)

Yet the newsposts from .au will all cause the Quicktime Plugin to load.

Comment 32

[Arthur]

It isn't checked, so plugins shouldn't be enabled.

Comment 33

[Frank Widmaier]

additional to comment 30 : I haven't activated plugins (and JS) for Mail&News
and use "show message body"=>"Plain text".. I see QT too. Another person of
de.comm.software.mozilla sees another plugin that's related to .au - files  on
his computer.
(using nightly 20021008-08)

Comment 34

[Michael Carman]

It appears that the underlying problem is not restricted to QuickTime.
This message: <[e-mail]> tried to load a plugin (LA? some
sort of audio file) as well. This is with build 2002092408.

Comment 35

[Peter Lubczynski]

Attachment: patch v.3

This patch limits the fix from bug 163568 to only HTTP channels that have
succeeded so mail/news souldn't be effected. File channels are taken care of
somehwere else and we can add other type as needed.

Comment 36

[gsanft]

I just had a look over at bug 163568 and the "fix" offered there seems to be the
cause of this problem.  Did this whole problem occur because of a mis-configured
web server?  

If I understand that correctly then I don't understand.  Why would any effort be
expended on the part of the browser to attempt to compensate for a misconfigured
web server?

The proper action would be to *always* rely on the content type as specified by
the server for any items "served".  The much less reliable name extension should
only be used in the case of local items for which there is no other source for
determining the content type.  

If a user is inconvenienced by a misconfigured web server it's not the concern
of the browser in any way.  The web server is the offender, and the problem
should be fixed on that end.  

Trying to do what I think I see being done here is just begging for trouble. 
We've already seen one unfortunate aspect of it.  What happens when malicious
payload is suitably named and served?

Comment 37

[Peter Lubczynski]

Every other browser works fine with the testcase in bug 163568 so we should too
rather than trying to evanglize. 4.x plugins specifically list which URL
suffixes to look for. The last patch here only does that for http channels which
I think will fix both of these bugs.

Comment 38

[Darin Fisher]

Comment on attachment 103017 [details] [diff] [review]
patch v.3

ok, this is marginally better.	i still think the correct solution is to fix
the mailnews URL parsing, but i'm OK with this patch in light of the fact that
the mailnews changes are likely non-trivial.  please make sure there is a bug
filed against mailnews to cleanup their URL parsing.  thx!!

sr=darin

Comment 39

[Peter Lubczynski]

thanks for the review darin! bug 175344 has been opened

Comment 40

[av (gone)]

Comment on attachment 103017 [details] [diff] [review]
patch v.3

r=av

Comment 41

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Why in the world are we ignoring MIME types in favor of extensions?

Comment 42

[Boris Zbarsky [:bzbarsky]]

Sorry, even with that patch this whole thing is unacceptable due to the
regression bug 175368.

We should only be triggering off the extesion if there is no registered handler
(content handler, plugin, helper application, etc) for the content type.  That
solution would still fix bug 163568 without breaking the world.

If we have to not trust the content-type, we should be trusting sniffing of
content data over extension, imo.

Comment 43

[Peter Lubczynski]

Attachment: patch v.4

bug 163568 has started working without this code and since it seems to break
stuff, here's a patch to back it out

Comment 44

[Boris Zbarsky [:bzbarsky]]

Comment on attachment 103384 [details] [diff] [review]
patch v.4

sr=bzbarsky.

Perhaps there's a way we can have the external helper app service detect the
plugin and do the right thing?	Not sure how possible or not that is...

Comment 45

[Darin Fisher]

Comment on attachment 103384 [details] [diff] [review]
patch v.4

r=darin for backout

Comment 46

[Brion Vibber]

*** Bug 175433 has been marked as a duplicate of this bug. ***

Comment 47

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Comment on attachment 103384 [details] [diff] [review]
patch v.4

a=dbaron for trunk checkin.

Please make sure there's a bug filed on the bad parsing of the mailnews URI
types as well.

Comment 48

[Peter Lubczynski]

Filed bug 175344 about the urls. 

patch in trunk, marking FIXED

Comment 49

[Phil "clef" Sweeney]

Thanks, it WFM now in nightly build 2002102104.

Comment 50

[WD]

*** Bug 175923 has been marked as a duplicate of this bug. ***

Comment 51

[Dan Tobias]

It sounds like the Mozilla developers have snuck in a form of MIME-type
second-guessing, a la MSIE, something that is very much against the standards
and can have nasty consequences.  In this case, the presence of anything at the
end of a URL that looks like the file extension of a plugin data type triggers
the plugin whether appropriate or not.  The misinterpreting of the ".au" TLD is
among the sillier consequences.  One can wonder / fear that a future version
might attempt to execute anything from a .com domain under MS-DOS, since .com is
an extension for DOS executables.

I've got a test script in my site that detects some of this sort of MIME
second-guessing (it trips up MSIE sometimes); the output of the following URLs
is always of type text/plain, and a standards-compliant browser should show it
that way:

http://webtips.dan.info/cgi-bin/plaintext.pl
http://webtips.dan.info/cgi-bin/plaintext.pl/test.html
http://webtips.dan.info/cgi-bin/plaintext.pl/test.gif
http://webtips.dan.info/cgi-bin/plaintext.pl/test.au
http://webtips.dan.info/cgi-bin/plaintext.pl/test.swf
http://webtips.dan.info/cgi-bin/plaintext.pl/test.exe

Comment 52

[Boris Zbarsky [:bzbarsky]]

It sounds like _a_ mozilla developer snuck something like that in, yes... did
you even read this bug?

(Though we're about to run into the issue of whether text/plain for .dmg.gz
files is or is not really text/plain... this may become a necessity because no
web server I have ever seen serves OSX binaries with the right mime type.... 
standards are nice, but not if they mean you can access 0% of the content out
there.  As I mentioned in this bug already, if we have to guess at the type, we
should look at magic numbers, not extensions.)

Comment 53

[Dan Tobias]

In standard Apache setups, a webmaster can set up MIME types for any particular
subdirectory via .htaccess files.  While some hosting sites disable this, they
are in the minority.  Thus, the majority of site developers are capable of
serving their sites with proper MIME types regardless of the content types or
file extensions they may wish to use.  If they don't trouble themselves to do
so, is it the job of the browser to second-guess it even though this may well
cause other perfectly legitimate sites to break, or even worse, provide a back
door for virus or trojan-horse programs to get themselves executed?

Comment 54

[Boris Zbarsky [:bzbarsky]]

This is all true.  And if people actually update their mime types, we don't have
to do anything.  That would be very nice, since I would much prefer not to have
to muck with this.

Comment 55

[Peter Lubczynski]

Although some new servers come nicly configured for some mime types, that's
usually not that case for plugins which have more exotic mime types and
especially not the case with some really old non-configurable servers or web
applications which are probably not going to change for a very long time. The
web is not perfect and we've got to adapt rather than encourage behavior that
*forces* the average user to switch to another browser to view the page.

At the macdev meeting, someone suggested sniffing for a binary null in the first
1k of a stream for displayable mime types as a possible means of falling back to
the extension.

Comment 56

[Dan Tobias]

Here's what the W3C has to say about MIME-type second guessing:

The architecture of the Web depends on applications making 
dispatching and security decisions for resources based on their 
Internet Media Types and other MIME headers. It is a serious 
error for the response body to be inconsistent with the 
assertions made about it by the MIME headers. Web software SHOULD 
NOT attempt to recover from such errors by guessing, but SHOULD 
report the error to the user to allow intelligent corrective 
action.
http://www.w3.org/2001/tag/2002/0129-mime

Here's somebody's test suite for MIME type handling:
http://entropymine.com/jason/testbed/mime/

I hope Mozilla doesn't plan on turning over to the Dark Side of the Force by
imitating the Evil Empire's browser in its disregarding of standards, something
that has the pernicious consequence of encouraging sloppy server configuration,
where webmasters say "It works in MSIE, so it must be the other browser's fault
if it won't render."  Even worse, it creates openings for malicious content to
slip through firewalls and proxies that screen based on MIME headers.  It also
frustrates Web developers whenever they are attempting to do a valid but rare
thing, like serve plain text via a URL ending in some extension other than .txt
(especially common for output of CGI scripts), and the browser refuses to take
it at face value.

Comment 57

[Boris Zbarsky [:bzbarsky]]

Dan, please stop ranting.  We all know what that part of the spec says.  We all
know about the entropymine test suite (certainly anyone who's working on MIME
stuff in Mozilla does).  You obviously failed to read the part of my comment
that said that _if_ (and that's a huge if) we start not trusting content-types
then we should use data-sniffing, not extensions.

Get a grip man.  There is no sinister conspiracy to do anything.  There are just
people who want the browser to 1) follow the standards and 2) be usable.  When
those two are in direct conflict, one must tread carefully.

Comment 58

[Dash]

Confirmation that everything works in Win32 build 2002102204.

The major problem with second guessing is the security issue.  I feel my
computer is safe when I use Mozilla.  Mainly because it doesn't second guess -
or so I thought!

Comment 59

[Dan Tobias]

I apologize if I'm offending anyone by my rants, which show that in my own
psychology I do think of this sort of thing as a religious issue... but I guess
I shouldn't try to impose my religion on others.

I'd like to know more about this "OSX binary" problem alluded to... the
execution of binary programs of any sort is obviously a significant security
risk, so if the capability of doing so is to be present in a Web browser it
needs to be implemented very, very carefully... any setup that might cause the
browser to start executing something when and where the user does not expect it
can be a severe problem.

And I will note that my observations of MSIE are that it, too, attempts some
sort of "smart" content sniffing, for which the file extension is not the only
(or even the most important) factor, though it is one of the things looked at. 
Thus, Mozilla would be heading down the same road as MSIE if it were to start
analyzing data content and disregarding MIME types in some cases, and this
approach does cause MSIE to fail in some cases where a standards-compliant
browser succeeds.

Comment 60

[Boris Zbarsky [:bzbarsky]]

These are not executable we would execute but ones the user wants to download
(the standard format for OSX packages are these .dmg.gz files that Apache will
serve as text/plain with content-encoding:gzip).  The Windows equivalent would
be showing all .exe files as plaintext.  The Linux equivalent would be showing
all .tar.gz files as plaintext (or all .rpm files).

See bug 175848 for the discussion on that, and please do not spam it with "this
is evil" comments... everyone involved already knows that.  The question is how
to resolve a real-life problem at the same time.

Comment 61

[Matthias Versen [:Matti]]

*** Bug 177673 has been marked as a duplicate of this bug. ***

Comment 62

[WD]

*** Bug 178673 has been marked as a duplicate of this bug. ***

Comment 63

[Matthias Versen [:Matti]]

*** Bug 179101 has been marked as a duplicate of this bug. ***

Comment 64

[R.K.Aa.]

*** Bug 179345 has been marked as a duplicate of this bug. ***

Comment 65

[Jo Hermans]

*** Bug 179806 has been marked as a duplicate of this bug. ***

Comment 66

[Torben]

*** Bug 172536 has been marked as a duplicate of this bug. ***

Comment 67

[Matthias Versen [:Matti]]

*** Bug 180511 has been marked as a duplicate of this bug. ***

Comment 68

[R.K.Aa.]

*** Bug 181433 has been marked as a duplicate of this bug. ***

Comment 69

[Boris Zbarsky [:bzbarsky]]

*** Bug 181439 has been marked as a duplicate of this bug. ***

Comment 70

[Matthias Versen [:Matti]]

*** Bug 181718 has been marked as a duplicate of this bug. ***

Comment 71

[Jo Hermans]

*** Bug 182120 has been marked as a duplicate of this bug. ***

Comment 72

[WD]

*** Bug 188589 has been marked as a duplicate of this bug. ***