336900 - "*.tgz.md5" files not saved after download.

Status: RESOLVED INCOMPLETE

(Tech Evangelism Graveyard :: English US, defect)

Description

[Eddie Maddox]

User-Agent:       Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3
Build Identifier: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3

Example: http://civicspacelabs.org/home/releases/civicspace/civicspace-0.8.3.tgz.md5

Observe: Firefox 1.5.0.3 error message box: "... the source file could not be read. ..."


Reproducible: Always

Steps to Reproduce:
0. Firefox 1.5.0.3 (Mac blue G3, OSX 10.3.latest-update)

1. Goto: http://CivicSpaceLabs.org/

2. Select: "GET CIVICSPACE", upper right

3. Select: "1. _Download_ the software for free if you have the technical resources to install and configure it on a server."

4. Select: "Version 0.8.3 (March 14th, 2006)
... (9MB, _md5-5946bf12d6dc80302304cecb5713c24b_)",
- file name: http://civicspacelabs.org/home/releases/civicspace/civicspace-0.8.3.tgz.md5

5. Observe: Firefox 1.5.0.3 error message box: "... the source file could not be read. ..."

Actual Results:  
Firefox 1.5.0.3 error message box: "... the source file could not be read. ..."

Expected Results:  
"*.tgz.md5" files successfully saved to local file system after download.


IRC, #QA, with Tracy, ~ 0800-0900 FRI 5 MAY 2006 PT:

	<eddie-MacG3>	Hello.. I have a Steps-to-Reproduce prepared for a possible Ff bug.
	<tracy>	eddie-MacG3: excellent, do you know how to file a bug?


	<eddie-MacG3>	I've done it once.
	<tracy>	go for it again. if you need assistance, just ask. :-)


	<eddie-MacG3>	For this one: http://eddiemaddox.com/Mozilla-QA/ , Ff-bug.txt
	I would just like a second opinion that it really is a Ff bug.
	Then I'ld be happy to file it.
	<tracy>	eddie-MacG3: hmmm, I don't think Firefox is supposed to be able to open such a file. but I was able to download it with Firefox.


	<eddie-MacG3>	tracy: Kieran Lal, who leads CS, gets the same error I do. That's why he states the MD% sums on the page.
	tracy: MD% -> MD5
	<tracy>	I saw the same error If I tried to load http://civicspacelabs.org/home/releases/civicspace/civicspace-0.8.3.tgz.md5 into the browser.
	but clicking the link at the page you gave brings up the file handling dialog: open with.. or save.


	<eddie-MacG3>	So, select "Save link as...". It downloads, but never gets Saved!
	<tracy>	eddie-MacG3: what platform?


	<eddie-MacG3>	Mac blue G3, OSX 10.3.latest-updates
	<tracy>	interesting... it works for me on a 10.4 Mac.


	<eddie-MacG3>	Also notice: the file dialog box highlights the file name Except for the ".md5" extension.
	The highlighting ends after ".tgz".
	<tracy>	did you try to "open with" Stuffit Expander?


	<eddie-MacG3>	Nothing ever brings up the dialog box that offers Stuffit for expansion.
	<tracy>	It works for me, that's all I can offer. You may file a bug if you think it's handling this incorrectly.

Comment 1

[Ben Fowler]

Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.9a1) Gecko/20060506 Minefield/3.0a1

I am seeing the same as you. Using your URL,
http://civicspacelabs.org/home/releases/civicspace/civicspace-0.8.3.tgz.md5
I get a zero length file named pvu28rt7.md5 in the downloads folder, which cannot
be read.

I wonder if this relates to one of the downloads.rdf bugs.

Comment 2

[Eddie Maddox]

(In reply to comment #1)
> I wonder if this relates to one of the downloads.rdf bugs.

I am not a developer, so I'll just watch from the sidelines...if you don't mind.

I do appreciate your work on this.

Thank you very much,
Eddie

Comment 3

[Eddie Maddox]

Firefox 1.5.0.4.RC2:
Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4

Same problem.

Eddie

Comment 4

[Eddie Maddox]

(In reply to comment #3)
> Firefox 1.5.0.4.RC2:

Also: Bon Echo/Firefox 2.0.A2

> Same problem.

Eddie

Comment 5

[Eddie Maddox]

"Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8.1a3) Gecko/20060526 BonEcho/2.0a3"

Same problem.

Eddie

Comment 6

[Eddie Maddox]

(In reply to comment #1) Ben Fowler:
> ...
> I am seeing the same as you. Using your URL,
> ... I get a zero length file ... which cannot be read.

Therefor, should the Status be changed from "UNCONFIRMED" to "NEW"?

For that matter, can someone confirm Ben's hypothesis? Or offer a better explanation/patch?


Thank you very much.
Eddie

Comment 7

[Ben Fowler]

(In reply to comment #6)
> (In reply to comment #1) Ben Fowler:
> Therefore, should the Status be changed from "UNCONFIRMED" to "NEW"?

In theory, I could request CANCONFIRM powers and do just that, but I am not 
yet ready or perhaps happy to be quite that intimate with this Bugzilla; in 
practice, rather than deal with the 'paperwork' I either submit a patch or 
just leave things.

Do you really think that this bug would get more attention if it were labelled 
NEW? - and if the answer to that is 'yes', then I do see your point.

Anyway, I will see whether I can create a patch, or tell you why I couldn't.

Comment 8

[Ben Fowler]

(In reply to comment #6)
> [ snip ]
> 
> For that matter, can someone confirm Ben's hypothesis? Or offer a better
> explanation/patch?

I'm wondering whether Firefox has somehow got it into its head that the data was 
being sent 'gzip encoded', and of course got zero bytes back when trying to 
decode (inflate) a file that had never been gzip'd in the first place.

Comment 9

[Eddie Maddox]

(In reply to comment #7)
> (In reply to comment #6)
> > (In reply to comment #1) Ben Fowler:
> > Therefore, should the Status be changed from "UNCONFIRMED" to "NEW"?
> ...
> Do you really think that this bug would get more attention if it were labelled 
> NEW? - and if the answer to that is 'yes', then I do see your point.

I had not thought that far ahead, but that would be a desirable sideffect
of an accurate Status value.

> Anyway, I will see whether I can create a patch, or tell you why I couldn't.

Thank you very much.

I wonder if there is a whole class of bugs like this,
with other endings: "*.tgz.*", not just "*.tgz.md5"
that get mishandled. Perhaps even a broader class
than "*.tgz.*" with other values for "tgz".

Thanks again,
Eddie

Comment 10

[Ben Fowler]

Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.9a1) Gecko/20060529 Minefield/3.0a1

After turning on logging with: 
export NSPR_LOG_MODULES=nsHttp:5,uriloader:5,HelperAppService:5

I get some lines of output like:

[thread_id a000ed68]: Found extension 'md5' (filename is 'civicspace-0.8.3.tgz.md5', handling attachment: 0)
[thread_id a000ed68]: HelperAppService::DoContent: mime 'application/x-tar', extension 'md5'
[thread_id a000ed68]: Getting mimeinfo from type 'application/x-tar' ext 'md5'
[thread_id a000ed68]: Mac: HelperAppService lookup for type 'application/x-tar' ext 'md5' (IC: 0x1e910fc0)
[thread_id a000ed68]: OS gave us: By Type: 0x0 By Ext: 0x0 type has default: false
[thread_id a000ed68]: Creating new mimeinfo
[thread_id a000ed68]: OS gave back 0x2207c890 - found: 0
[thread_id a000ed68]: Data source: Via type: retval 0x00000000
[thread_id a000ed68]: Extension 'md5' matches mime info: 1
[thread_id a000ed68]: MIME Info Summary: Type 'application/x-tar', Primary Ext 'md5'
[thread_id a000ed68]: Type/Ext lookup found 0x2207c890
[thread_id a000ed68]:   After dispatch, m_targetStreamListener: 0x22ad8000, rv: 0x00000000
[thread_id a000ed68]: nsExternalAppHandler::OnStartRequest( 1f6da7e0, 0 ) setting ApplyConversion to [1] from "md5"/"application/x-gzip"
[thread_id a000ed68]: nsExternalAppHandler::OnStartRequest( 1f6da7e0, 0 ) setting encodingChannel's ApplyConversion to [1]

and I am fairly sure that Firefox is using the fact that the filename has 
a gizp-matching extension (.tgz) and an unknown one (.md5) (= a recognised
MIME type "application/x-tar" and an unknown extension) to fail to switch 
off channel conversion. This seems surprising, and is arguably very wrong -
I would wager money that you thought that with HTTP, unknown files were
treated as plain text; but it is possibly Apple's bug.

I would suggest that the ApplyConversion switch should be set to off when
there are either unknown extensions or MIME types in the picture.

It would be a good idea to check what Safari and Konqeror do, and, if you 
are able to consult with someone who knows the code, why.

I'll sleep on it.

The problem is not located in just one part of the code, but you might
want to look at 
http://lxr.mozilla.org/seamonkey/source/uriloader/exthandler/nsExternalHelperAppService.cpp#687
for the function:

NS_IMETHODIMP nsExternalHelperAppService::ApplyDecodingForExtension(const nsACString& aExtension,
                                                                    const nsACString& aEncodingType,
                                                                    PRBool *aApplyDecoding)
{
  *aApplyDecoding = PR_TRUE;
  PRUint32 i;
  for(i = 0; i < NS_ARRAY_LENGTH(nonDecodableExtensions); ++i) {
    if (aExtension.LowerCaseEqualsASCII(nonDecodableExtensions[i].mFileExtension) &&
        aEncodingType.LowerCaseEqualsASCII(nonDecodableExtensions[i].mMimeType)) {
      *aApplyDecoding = PR_FALSE;
      break;
    }
  }
  return NS_OK;
}

where the ApplyDecoding is only turned off if there is a 
non-DecodableExtension and a matching MIME type.

Perhaps a more important piece of the jigsaw puzzle is to think about 
how a md5 checksum came to have that MIME Type. This is partly a server 
side setting, videlicet:

$ curl -I http://ben-fowlers-tiger.local/~bfowler/Firefox/Bug_336900/civicspace-0.8.3.tgz.md5
HTTP/1.1 200 OK
Date: Mon, 29 May 2006 21:44:51 GMT
Server: Apache/1.3.33 (Darwin) PHP/4.4.1
Last-Modified: Mon, 29 May 2006 10:08:47 GMT
ETag: "1b4a50-37-447ac82f"
Accept-Ranges: bytes
Content-Length: 55
Content-Type: application/x-tar
Content-Encoding: x-gzip

and you could probably ameliorate the problem for yourself by telling 
Apache to send a suitable MIME type (try "text/plain" to start with) for 
all files whose last extension is .md5 . Obviously you can't rely on 
changing the configuration of every server on the planet, which is why I 
suggest comparing other browsers and possibly other platforms.

Hope this helps, but there may not be a fix for Firefox that works in
all circumstances and is acceptable to the wider community

Comment 11

[Eddie Maddox]

(In reply to comment #1)
> ...
> I am seeing the same as you.
> ...
> I get a zero length file named pvu28rt7.md5 in the downloads folder, which
> cannot
> be read.

By this comment From Ben Fowler  2006-05-06 this bug is confirmed.
So, how is it that the  Status:   	UNCONFIRMED
has not been updated to  Status:   	NEW ?


Thank you,
Eddie

Comment 12

[Ben Fowler]

(In reply to comment #11)
> (In reply to comment #1)
> > ...
> 
> By this comment From Ben Fowler  2006-05-06 this bug is confirmed.
> So, how is it that the  Status:         UNCONFIRMED
> has not been updated to  Status:        NEW ?

My comment 7 still applies. You could come round and beat me into submission, 
or just accept the fact that there are not that many Mac devs at all working 
on Firefox.

Literally, there are just the two of us chatting here.

Could I repeat some of my requests for help:

    What does Safari do?
    What does Konqueror do?

It is possible that Firefox is wrong in ever thinking that this is an encoded 
channel, but unless someone else in the team fixes that, I think that the best 
that we can aim at is parity with one of those applications. Which (if either)
handles this situation the better?

Ben.

Comment 13

[Eddie Maddox]

Still not fixed and
still being encountered ("CONFIRMED") in the wild:

Puppy Linux Discussion Forum
Forum index » Taking the Puppy out for a walk » Misc
SeaMonkey 1.1.8 candidates - please help test
http://www.murga-linux.com/puppy/viewtopic.php?t=25948
...
MU Posted: Sat Feb 02, 2008 6:19 pm
...
btw., small bug (also in Firefox2.0.0.11 and seamonkey 1.1.2):
http://distro.ibiblio.org/pub/linux/distributions/puppylinux/test/puppy-3.02alpha1/
Click on:
http://distro.ibiblio.org/pub/linux/distributions/puppylinux/test/puppy-3.02alpha1/puppy-unleashed-3.02alpha1.tar.gz.md5.txt
It is empty.
But in reality, there is text in it.
You can try with wget to check it.
I encountered the same with own files on my server.
It happens, when a file
TESTFILE.tar.gz
has a
TESTFILE.tar.gz.md5.txt
If I rename it to
TESTFILE.tar.gz.txt
then it works as expected.
Mark
###

Mark's: "*.tar.gz.md5.txt"
CivicSpace's: "*.tgz.md5"

Both: After the source file is downloaded/read,
the source file local copy gets truncated to zero length,
whether that file is rendered or saved to disk.

Is this a "data corruption" bug?

Gecko(?) is (apparently)
corrupting the data in the source file local copy,
after it downloads/reads it,
by deleting all data in the source file local copy,
leaving a zero length source file local copy erroneously.

Thank you,
Eddie Maddox

Comment 14

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

Can you reproduce this with FF3B3?

Comment 15

[Boris Zbarsky [:bzbarsky]]

The URL in the URL field sends:

  Content-Encoding: x-gzip

but the data it sends is not valid gzip data, so we bail out from trying to decompress it.  Using "save link as" should work fine, but clicking on the link and then saving will give the error, as you discovered.

Have you reported the problem to the site?  They're just sending completely incorrect headers given their data.  Ben caught on to the fact that we're trying to decompress because the file extension is not that of a compressed file, but the only reason we're decompressing to start with is that the site claims the data IS compressed.

This bug should probably be moved to tech evang: it's just a server misconfiguration.

Comment 16

[Eddie Maddox]

(In reply to comment #15)
> Have you reported the problem to the site?
> ...  Ben caught on to the fact that ...
> 
> This bug should probably be moved to tech evang:
> it's just a server misconfiguration.

I did not fully comprehend all of what Ben Fowler
was talking about. Now maybe I understand a little better.

So, maybe the Mozilla code is not at fault,
but the Mozilla process is!

Mozilla products have a feature for
reporting broken website code
so it can get fixed.

Is there a similar Mozilla "feature" for also
reporting broken server software configs "upstream",
whether it's a buggy default server config or not?

The people who simply use Firefox/SeaMonkey
to get work done should not have to be
told to do "tech evengilism" as a bug fixing strategy.

Mozilla should participate in FOSS server software projects,
like Apache, to contribute patches when needed to fix things
that impact Mozilla products or Mozilla users.

This may be a little further than the Mozilla Manifesto had in mind,
but, as a consumer, things need to "just work",
whether that is code, or process/evangelism, or whatever.

So, maybe there needs to be an FOSS Manifesto?
And some inter-project, inter-organization
collaboration to get things like this Bug 336900 fixed?

Thank you,
Eddie Maddox

Comment 17

[Eddie Maddox]

(In reply to comment #14)
> Can you reproduce this with FF3B3?

I use Mac OSX 10.3, which is not supported in
Firefox3/Gecko 1.9.

Thank you,
Eddie Maddox

Comment 18

[Boris Zbarsky [:bzbarsky]]

> Is there a similar Mozilla "feature" for also
> reporting broken server software configs "upstream",

No.

> The people who simply use Firefox/SeaMonkey
> to get work done should not have to be
> told to do "tech evengilism"

I don't seen anyone telling them to.

Comment 19

[Eddie Maddox]

I filed a separate bug to address the
source file local copy data corruption issue
whenever the source file server is mis-configured:

Bug 418571 – Data corruption: source file local copy truncated to zero length.
https://bugzilla.mozilla.org/show_bug.cgi?id=418571

Thank you,
Eddie Maddox

Comment 21

[Chris Lawson (gone)]

INCOMPLETE due to lack of activity since the end of 2009.

If someone is willing to investigate the issues raised in this bug to determine whether they still exist, *and* work with the site in question to fix any existing issues, please feel free to re-open and assign to yourself.

Sorry for the bugspam; filter on "NO MORE PRE-2010 TE BUGS" to remove.