Closed Bug 184452 Opened 22 years ago Closed 12 years ago

Necko - Allow handling of files > 2gig (>2 GB)

Categories

(Core :: Networking, defect)

Product:
Core
Component:
Networking
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED
Milestone:
Future

People

(Reporter: dougt, Unassigned)

References

Details

(Whiteboard: DO NOT COMMENT HERE (see also comment 54))

Attachments

(2 files)

DOwnload Screenshot
16.12 KB, image/jpeg
Details
Firefox 3.5 passed the 2GB mark successfully
18.34 KB, image/png
Details 
we may need to create a 64 bit version of the necko interfaces.
Target Milestone: --- → Future
the only way to really support >2G downloads would be to switch all interfaces
over and make those interfaces be the only way to do things.  this means
deprecating several key xpcom/necko interfaces (namely nsIInputStream and
nsIStreamListener).
Someone needs to really see how common these large files are now, and/or gauge
when they will be common. I've seen one game demo that broke the 1gig mark, and
I suspect this to be more and more common. That's still not 2gig's but I'm sure
it's comming in the not too distant future.

Another alternative is to do something similiar to what Microsoft did. Provide
alternatives that allow a secondary 32 bit value to be passed. This isn't as
clean, but allows existing code to work unchanged if it's known it doesn't have
to deal with such large files.
darin: are those interfaces frozen?
biesi: last i checked ;-)
Shouldn't this really be "allow 2GB filesizes via 64 bit interfaces" ?
OS: Windows XP → All
QA Contact: benc → ashishbhatt
Hardware: PC → All
I guess it depends on what you mean 64 bit interfaces? Win32 a 32 bit interface
API provides 32 bit API functions that handle access to files larger than 2 gigs.
*** Bug 205443 has been marked as a duplicate of this bug. ***
*** Bug 215450 has been marked as a duplicate of this bug. ***
*** Bug 131439 has been marked as a duplicate of this bug. ***
Going to make this more general, so that it can apply to upload as well as download.
Summary: Necko - Allow downloads of > 2gig downloads → Necko - Allow handling of files > 2gig
Blocks: 215450
QA Contact: ashishbhatt → benc
Blocks: 207400
*** Bug 215091 has been marked as a duplicate of this bug. ***
*** Bug 225866 has been marked as a duplicate of this bug. ***
*** Bug 226391 has been marked as a duplicate of this bug. ***
Well I guess we've arrived at the not too distant future
Inf / 10 == Inf :(
ok, list of frozen interfaces that would require changes for files >2gb:

nsIChannel.idl:
  attribute long contentLength;

nsIStreamListener:
 67                          in unsigned long aOffset,
 68                          in unsigned long aCount);
(parameters of onDataAvailable)

nsIFile.idl: surprisingly this requires no changes.

nsIInputStream.idl: several functions + nsWriteSegmentFun
nsIOutputStream.idl: basically same as nsIInputStream

nsIScriptableInputStream:
 49     unsigned long available(); 
 55     string read(in unsigned long aCount); 

-afaik this is a complete list of the frozen interfaces that would require changes-

of the unfrozen ones, nsIWebProgressListener.idl comes to mind, but most likely
there are others.
Summary: Necko - Allow handling of files > 2gig → Necko - Allow handling of files > 2gig (>2 GB)
unfrozen ifaces in xpcom:
nsIAsync{Input,Output}Stream: in unsigned long aRequestedCount
nsIByteArrayInputStream.idl: (maybe)
NS_NewByteArrayInputStream (nsIByteArrayInputStream ** aResult, char * buffer,
unsigned long size);
(size would be the part to change, but who would create a byte input stream with
more than 4 GB?)

nsIObjectInputStream: unlikely (putBuffer(in charPtr aBuffer, in PRUint32 aLength))

nsIObservableOutputStream.idl:     void onWrite(in nsIOutputStream outStr, in
unsigned long amount);

nsIPipe.idl: segmentSize, segmentCount

nsISeekableStream.idl:
    void seek(in long whence, in long offset);
    unsigned long tell();

nsIStringStream.idl: similar to nsIByteArrayInputStream

*** Bug 215450 has been marked as a duplicate of this bug. ***
*** Bug 228968 has been marked as a duplicate of this bug. ***
I think were going to see more 2GB+ downloads in the future, as said earlier,
some gaming demo's are creeping up already to the 1GB mark.  2GB is only a
matter of time.

DVD images could also be that large.
Blocks: NegativeDownload
*** Bug 229979 has been marked as a duplicate of this bug. ***
I think that list of frozen interfaces is wrong - if unsigned longs are used,
that gives us 4GB, not 2GB. The content length is an issue, though happily, not
for mailnews, since we only open streams on parts of a file :-)
sure, 4 GB are better than 2 GB, but I don't think we should limit these APIs to
4 GB either.
2GB -> 4GB is a bandaid, not a fix
For starters a DVD image is easily above 4 GB...

48 or 64bit sizes feels like the way to go here, unless people want to keep
revisiting this issue every 6 months.
nsISeekableStream now supports 64 bit streams (though some implementations will
truncate at 32 bits and ASSERT)
*** Bug 242859 has been marked as a duplicate of this bug. ***
*** Bug 245115 has been marked as a duplicate of this bug. ***
No longer blocks: 243974
Depends on: 243974
Is bug 247599 related to this? Probably its the same issue of 32 vs 64 bits
representation, but maybe in a different part of Mozilla.
(In reply to comment #28)
> Is bug 247599 related to this? Probably its the same issue of 32 vs 64 bits
> representation, but maybe in a different part of Mozilla.

um, 4 MB can fit easily into a 32 bit variable. that bug is not related.
*** Bug 248482 has been marked as a duplicate of this bug. ***
*** Bug 256338 has been marked as a duplicate of this bug. ***
*** Bug 260859 has been marked as a duplicate of this bug. ***
here an example for an application:

- a web-site with a form to choose software/update packages
- after submit the server sends you an ISO-image ready for burning on a CD/DVD

Problems:

- content-length-header ist limited to 2GB in mozilla (some browsers 4GB)
- without content-length-header the download stops after 2.4GB (of 5.5GB)

so it's time for the future :) or we have to use CDs forever
I really think this needs some serious attention. This will be just another 
excuse for people not to use Gecko based browsers. I'm sure in intranet 
environments such large files are going to be more and more common.
*** Bug 266323 has been marked as a duplicate of this bug. ***
I just ran into this bug.

While I'm just a user, I'm commenting to agree with the fact that while
currently uncommon, files in excess of 2 GB (or 4 GB) will be seen with
increasing regularity.  In my case, FC3 DVD ISO at 2.4 GB.  I realize the bug is
mostly cosmetic, but everything else in Mozilla/Firefox is so polished that it
really stood out.
This probably will never get fixed, altough it's being constantly reported as a
new bug (even I couldn't find it on bugzilla for the first time I reported it)

So much for the 1.0 hype:
- Tabs still open to windows which have no toolbars or tab-bars making it
impossible to use them (i.e. opening links in a popup window)
- DM does not behave correctly with files larger than 4GB
- DM retry-function seldomly works
- Crashes with known _overflow_ exploits
(http://lcamtuf.coredump.cx/mangleme/gallery/)
- on win32 does not recover quickly from being minimized for few hours (over
night), it's rather funny how I can leave opera or ie windows open and start
using them in the morning with no lag whatsoever, but with mozilla? no no.
This is not a Firefox download manager tracking bug nor is it a place for you to
rant about the hype surrounding Firefox 1.0 and the problems you've had with it.
Oh, sorry. I thought this was the Firefox-product.. When I reported the bug I
put it under Firefox/DM, but this seems to be general Browser/Networking. 

Once again, I'm truly sorry for the confusion here.
*** Bug 272315 has been marked as a duplicate of this bug. ***
*** Bug 277785 has been marked as a duplicate of this bug. ***
Attached image DOwnload Screenshot — 
Screenshot of this bug in action
thewulf@gmail.com: That's Bug 228968 AND DON'T ATTACH ANY SCREENSHOTS ON THAT
BUG EITHER. We really know it's displaying negative values, no need for more
screenshots.
This bug is on way for very long time, it should be fixed _fast_

just my .5 snts
I just didn't see any existing SS so I figured might as well. No biggie.
*** Bug 286187 has been marked as a duplicate of this bug. ***
*** Bug 288939 has been marked as a duplicate of this bug. ***
*** Bug 288939 has been marked as a duplicate of this bug. ***
*** Bug 290236 has been marked as a duplicate of this bug. ***
I already run multiple times into this bug. once with fedora DVD image and one
wihth MacOS X DVD disk images. Both are around 2.5GB. So its definitively time
to fix this.

It is VERY ANNOYING if you download a 2.5GB file (took me 4h) and you end up
with a 2.0GB file without any error message whatsoever. You will burn it to DVD,
try to boot it. YOu will burn it again and again until you realize your image
file is too small for what it should be.

So you wasted already more time on this than what it takes to fix this :).

*** Bug 293036 has been marked as a duplicate of this bug. ***
*** Bug 293615 has been marked as a duplicate of this bug. ***
2.5 Years into this bug, and I still have to go use "some other browser" if I
want to download DVD ISOs (for things like Linux, etc.). 

Just another user voting that this really really needs to get fixed at some
point in the near future.
(In reply to comment #53)
> 2.5 Years into this bug, and I still have to go use "some other browser" if I
> want to download DVD ISOs (for things like Linux, etc.). 

that is fixed (in versions newer than 1.0.x). my understanding is that this bug
refers to other places as well, not just downloads, and not all of those are fixed.

(In reply to comment #54)
> (In reply to comment #53)
> > 2.5 Years into this bug, and I still have to go use "some other browser" if I
> > want to download DVD ISOs (for things like Linux, etc.). 
> 
> that is fixed (in versions newer than 1.0.x). my understanding is that this bug
> refers to other places as well, not just downloads, and not all of those are
fixed.

This is not true. it is not fixed. Of course you can download 2.5GB files with
Firefox. The download just stops after 2048MB and you THINK it has downloaded
everything. It would have been nice to have a dialogbox popping up in the
beginning saying the file is too big or such. But no, you have to wait hours and
hours to realize that all your download is wasted bandwith.

A very SERIOUS bug, especially its so old by now.
I had this again in Firefox 1.0.2. And I'm sure its still there in 1.0.4 (And
no, I wont try it until someone confirms its fixed, as downloading >2GB takes an
awful lot of time for me).
He said newer than 1.0.x, which means 1.1, which is still in alpha
yes, I can confirm that it's fixed for 1.1 tree , I am using nightly snapshot of
Deer Park alpha 1 (2005.05.30) and I've just donwloaded 2.5 GB iso which passes
its own crc test ok. However I haven't found  > 4GB file on fast nearby network
so I don't know anything about > 4GB files yet, but if there are 64bit
interfaces already (and it seems like that from source codes to me) it will be
ok too.
> but if there are 64bit
> interfaces already (and it seems like that from source codes to me)

yeah, for downloads there are. not for uploads... and maybe some other stuff too. 
(In reply to comment #55)
> And I'm sure its still there in 1.0.4

I am using 1.0.4, and I just downloaded FC3 via FTP, which is 2.3GB. The
download counter went negative near the end, but the download was still valid.
Firefox produced a 2.3GB file that contains data throughout the entire file, so
I believe it's legit. That would suggest the bug is /partially/ fixed.
I tried the same but in my case it used HTTP instead of FTP.
I have a file of 2117734496 bytes (1.97GB) instead of 2.7GB as it should be.
So the protocol type does make a difference.
(In reply to comment #57)
> yes, I can confirm that it's fixed for 1.1 tree , I am using nightly snapshot of
> Deer Park alpha 1 (2005.05.30) and I've just donwloaded 2.5 GB iso which passes
> its own crc test ok. However I haven't found  > 4GB file on fast nearby network
> so I don't know anything about > 4GB files yet, but if there are 64bit
> interfaces already (and it seems like that from source codes to me) it will be
> ok too.

Debian 3.1 is >4 GB. See:
http://cdimage.debian.org/debian-cd/3.1_r0a/i386/iso-dvd/debian-31r0a-i386-binary-1.iso

(or use a mirror)

Firefox 1.0.4 stopps at 2 GB.
(In reply to comment #61)
> Firefox 1.0.4 stopps at 2 GB.

yes, indeed, but use 1.1 version tree (
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/ ), which
has it fixed by now, and you'll see that it work for > 4 GB with FTP or HTTP
(I've tried your iso link from both and run verification on files,they are
absolutely ok and over 4 GB)
Whiteboard: DO NOT COMMENT HERE
It appears that the download manager.. or at least support for byte-range-resume
does not work yet. When I try to resume ('retry' as download mgr calls it), the
download manager app does not actually send the new (range) request...

sample: http://up.ascentmedia.com/upweb/test.jsp?file=FC4-i386-DVD.iso
(In reply to comment #63)
> .. or at least support for byte-range-resume does not work yet.

Seems to be true for files less than 2GB as well.
*** Bug 301543 has been marked as a duplicate of this bug. ***
*** Bug 308424 has been marked as a duplicate of this bug. ***
*** Bug 231788 has been marked as a duplicate of this bug. ***
I think there is another serious problem - and reason for fixing this bug. When
trying to download files > 4GB (for example
ftp://sunsite.informatik.rwth-aachen.de/pub/Linux/suse/i386/9.3/iso/SUSE-9.3-Eval-DVD.iso)
the browser (firefox 1.0.7, running under SuSE Linux 9.1) crashes when reaching
the 4GB border. I suppose the reason might be a "division by zero" as the
average transfer rate swapped to neg. numbers after passing the 2GB and grew
afterwards until it reached zero - and the browser crashed.
Heinz: that's a different problem.  please filea new bug if you can reproduce it
with firefox 1.5 beta.
*** Bug 311344 has been marked as a duplicate of this bug. ***
Firefox RC1 (Mac) cuts files of length over 4 GB to 4 GB when "downloading" from a  file in the harddisk. The Debian ISOs are over 4 GB.

Bojan
Firefox 1.0.7 (Mac) cuts files of length over 4 GB to 4 GB, like Firefox 1.5 RC1, when "downloading" from a file in the harddisk. It shows the negative size of the downloaded part as described elsewhere in this bug report. 

The local download is a good trick to test the download manager. Can please someone, at best with a local network at home, retest the 4 GB limit with all downloading possibilites (HTTP, FTP, harddisk) before Firefox 1.5 is released? Tests with HTTP and FTP seemed to be made and work. However, this bug should be fixed, doesn't matter how curious the download possibilities are.

Bojan

PS: I meant Firefox 1.5 RC1 instead of Firefox RC1 on my previous post. :)
*** Bug 317085 has been marked as a duplicate of this bug. ***
Local-to-local downloads are still limited to 4 GB in Firefox 1.5.

Bojan
*** Bug 320136 has been marked as a duplicate of this bug. ***
This is still broken in Firefox 1.5... I just spent the past couple of hours trying to download the DVD image of Fedora Core 5 (~3GB in size), and it was running smoothly the entire time. Once it hit 2GB though, it just stopped and Firefox said the download was finished when in fact it really wasn't. The download manager showed the correct status and file size, but gave me no indication that it would quit automatically after 2GB.
I'm using FireFox 1.5.0.1 on Windows, and was downloading the purchase of Oblivion (which is slightly bigger than 4 GB). After 4 GB (and three hours), it stopped with a write error that suggested I try saving the file somewhere else.

Two problems:

1) It really needs to support > 32 bits. fpos_t exists for a reason.

2) As long as it doesn't support > 32 bits, it needs to tell me that it won't work, with a good error message, ideally before it tries to download.

this is the wrong bug. but as everyone keeps posting to it...

What filesystem are you saving to when it doesn't work in 1.5?
biesi, what's the correct bug?  I noticed that after I downloaded the FC5 iso images that the DVD iso doesn't even show up in the directory lists.  Via http, this could very well be a bug in apache 2.0 but via file://, it's a moz bug.  In fact, via file://, ff 1.5 won't even load the directory that contains the image.  The FS is ext3.  
I'm saving to NTFS when it doesn't work in 1.5.0.1, running Windows XP SP2.

Btw: If this is the wrong bug, then what bug should I re-direct to? This bug was what came up when doing a search.
Yes - if this is the 'wrong bug', which one is correct? Is there one that is general to the file & stream interfaces and not just networking or Necko? The interfaces listed earlier appear to be core? I could not find a 'closer' bug...

Or do we need a new bug/RFE for the core file/stream (not network) interfaces? One that gets the attention of the specific owners? My searching did not find another bug/RFE for file/stream that suggests (rightfully) depecating ALL 32 bit file/stream interfaces (deprecate 32 bit and adjunct new 64 bit interfaces - not change existing ones as seems to have been implied here and the reaons for no action).

Otherwise this seems to be closet match for well-doers (who are not product specialists) to express frustration (it is 3-1/2 years old and 'new', after all).
bug 243974 is right, and FIXED, which is what I thought the state of that issue is. the relevant download interfaces do support this. in fact last I tested this it worked for me. new issues for specific download problems should get new bugs.

comment 54 describes what this bug is about.
Whiteboard: DO NOT COMMENT HERE → DO NOT COMMENT HERE (see also comment 54)
This is not the right bug for file download disk writing problems with files > 4 GB. Also, bug 243974 was not consistent with the symptoms I saw in 1.5.0.1.

See new bug: https://bugzilla.mozilla.org/show_bug.cgi?id=331647

*** Bug 299598 has been marked as a duplicate of this bug. ***
until this bug is directly linked from kernel.org FAQ, we got a lot of request to fix.
http://www.kernel.org/faq/#largefiles

This 4GB limit size is a problem that I've run into in developing FireFTP.  It seems the problem lies with nsIBinaryOutputStream.writeBytes:

Error: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIBinaryOutputStream.writeBytes]" nsresult: "0x80004005 (NS_ERROR_FAILURE)" location: "JS frame :: chrome://fireftp/content/js/connection/dataSocket.js :: anonymous :: line 258" data: no]
*** Bug 350903 has been marked as a duplicate of this bug. ***
*** Bug 350903 has been marked as a duplicate of this bug. ***
Besides DVDs and video media files, I keep running into this bug downloading the English wikipedia for offline processing.  The compressed version of the database (one entry per article without revision history) has been over 2 Gbytes for some time.  A recent image is at http://download.wikipedia.org/enwiki/20060920/enwiki-20060920-pages-meta-current.xml.bz2

I hope SeaMonkey/Firefox/Mozilla would be able to download this before Microsoft fixes XP's builtin ftp utility....
Hello ,Give my vote to you
Sites that post multi-gig data files do commonly become the reality.
Firefox cannot handle them gracefully (neither can IE, but who cares?).
As of today's mainstream browsers (not concerning command line tools)
only the latest Opera was able to download this file correctly
(i.e. completely):

ftp://ftp.ncbi.nih.gov/pub/geo/DATA/supplementary/series/GSE2109/GSE2109_RAW.tar
(and there are many more similar huge files there)

So I am voting for this bug!

My download (MSDN Servicepack 1)
http://download.microsoft.com/download/8/5/4/854f7409-47bd-41a2-b3b2-1a4875294550/MSDVDEUDVDX1370478.img
stopped at 2 GB (90 %).
But this download has 2,320,840,704 Bytes.
So I cannot download files greater than 2 GB by Firefox.
 
mass reassigning to nobody.
Assignee: dougt → nobody
(In reply to comment #94 by Josef Goldbrunner)
> http://download.microsoft.com/download/8/5/4/854f7409-47bd-41a2-b3b2-1a4875294550/MSDVDEUDVDX1370478.img

This download (2.2G) works for me without problems (Firefox 2.0.0.6, actually Debian's Iceweasel). The download completes without errors, and the DVD image is ok. Which version of Firefox were you using?
Can this possibly be a built difference between Josef's version and mine (Windows vs. Linux)?

Also, I have no problems downloading files larger than 4G via http and ftp. Why am I not hitting this bug?
I am having this problem with both firefox 3.0 and 2.0.0.14 for windows.  truncates them to 2GiB.  
somebody's using a signed 32-bit integer somewhere...
one of the places I am having this problem with is www.opensuse.com trying to download opensuse 11.0 DVD which is 4.3GB.
I have files on my web site which are also 4.6GB.

*please* fix!
Running Firefox 3.5.3, I was able to download the 10.1 Gentoo Live DVD, a 2.6 GB file found at http://distfiles.gentoo.org/releases/amd64/10.1/livedvd-amd64-multilib-10.1.iso

User Agent string: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.3)  Gecko/20090910 Ubuntu/9.04 (jaunty) Firefox/3.5.3

Package information: 3.5.3+build1+nobinonly-0ubuntu0.9.04.2

This package was built from http://archive.ubuntu.com/ubuntu/pool/universe/f/firefox-3.5/firefox-3.5_3.5.3+build1+nobinonly.orig.tar.gz patched with http://archive.ubuntu.com/ubuntu/pool/universe/f/firefox-3.5/firefox-3.5_3.5.3+build1+nobinonly-0ubuntu0.9.04.2.diff.gz with cosmetic local modifications following the steps on the first post of http://ubuntuforums.org/showthread.php?t=1225754

Is it safe to mark this bug as fixed?
I know the FF 3.52 download manager handles files over 2GiB.  I just tested it with a local web page and a DVD ISO file.
Agreed.  Just downloaded the 10,181.73 MiB (9.94 GiB) file from http://download.wikimedia.org/enwiki/20091009/enwiki-20091009-pages-meta-current.xml.bz2 using FF 3.5.3 on Windows XP SP2.  It's fixed.  It works great.
Looks like all the bugs this is depending on are fixed and so this is now working. Marking fixed.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
I always saw this bug to be about all parts of necko that handle 32-bit file sizes, and not all of those are fixed. in particular, upload isn't...
Oh, true, completely forgot about upload. Can't seem to find any place where I could even begin to upload >2gig file.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
You do it locally, with a .html page saved on your disk (or a local webserver) that has some sort of upload system on it. Unfortunately the only upload system I know is Uber-Uploader, and I've never checked if it handles 2 GB files.
(In reply to comment #104)
> Oh, true, completely forgot about upload. Can't seem to find any place where I
> could even begin to upload >2gig file.

You can upload multi-GB files on YouTube (through regular POST, not flash upload): http://www.youtube.com/my_videos_upload?nobeta

Cheers from the YT team - we hope this can be fixed soon obviously :P


For reference, to make large files:
Linux:
dd if=/dev/zero of=4gbfile bs=1024 count=4194304
Windows:
fsutil file createnew d:\temp\4gbfile.txt 4294967296
That page specifically states that the files are to be up to 2 GB.
(In reply to comment #107)
> That page specifically states that the files are to be up to 2 GB.

So, if you try testing with a larger than 2GB file, yes, it will be marked as 'too big'.  But the POST should upload completely (that is, if this bug were fixed :P)

If you need an additional partner account to test with, let me know (those are up to 20GB - email me directly).  Although, I do see that you guys have the http://www.youtube.com/firefox  channel - maybe you can use that one for testing?  (obviously making the test videos just private ones)
(In reply to comment #106)
> For reference, to make large files:
> Linux:
> dd if=/dev/zero of=4gbfile bs=1024 count=4194304

You really want to use:
dd if=/dev/zero of=/tmp/4gbfile bs=1024 seek=4194304 count=1
faster, and doesn't actually require 4 GB of disk space :)
So what's up with this bug?

Firefox 15 still truncates downloads to 4 GB

Chromium downloaded all the 4.2 GB. I'm not trolling anything here, just trying to prove that it wasn't a server or filesystem/OS problem (openSUSE 12.3 Factory)
This might work in Firefox 18 now that bug 784912 has been fixed.
Also probably relevant that bug 215450 was fixed recently.
Guys, downloads are supposed to work for a long time now. If they're not, please do file a new bug with steps to reproduce.

Since uploads are also fixed, resolving this bug.
Status: REOPENED → RESOLVED
Closed: 15 years ago → 12 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: