395330 - Active downloads don't appear in download manager

Status: VERIFIED FIXED

(Toolkit :: Downloads API, defect)

Description

[Hussam Al-Tayeb]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a8pre) Gecko/2007090705 Firefox/3.0a8pre
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a8pre) Gecko/2007090705 Firefox/3.0a8pre

After the checkin for cross session download resuming. active downloads no longer appear in the download manager but are correctly downloading in the background.
After the download is completed, the download manager does not show anything as completed until manually I close the download manager and reopen it.

Reproducible: Always

Steps to Reproduce:
1. Close the download manager.
2. Set Firefox to not close the download manager when a download is complete.
2. Right click on an link for a 1+MB file from any website and 'save link as' to your desktop. 
3. Manually open the download manager. If it is already open, close it then reopen it again (this is an important step).
4. Notice that the file isn't in the download manager.
5. The file is now finished downloading in the background and is on your desktop.
6. The download manager is still showing an empty 'active' and 'completed' downloads.
7. Close the download manager and reopen it.
8. The file is now correctly listed as 'completed'

Comment 1

[Hussam Al-Tayeb]

The issue here is that since I can't see the progress of the file while downloading, I can't actually pause or cancel it.

Comment 2

[Hussam Al-Tayeb]

Attachment: Screenshot of download manager while downloading some 600KB tarball file.

Comment 3

[Stephen Donner [:stephend] Not actively reading bugmail]

Which file exhibits this bug, specifically?  Note that most files populate the Active area for me using Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9a8pre) Gecko/2007090604 Minefield/3.0a8pre (and I tried this on Ubuntu earlier today and it worked there too).

Also, are you downloading from an FTP or an HTTP server?

Comment 4

[Hussam Al-Tayeb]

(In reply to comment #3)

Please open this website and scroll down to the buttom. 
http://ftp.gnome.org/pub/GNOME/sources/libgsf/1.14/

Then download the file libgsf-1.14.6.tar.bz2 

Comment 5

[Stephen Donner [:stephend] Not actively reading bugmail]

I can't reproduce on Windows (it always works for me), but I'll check my Fedora Core 7 and Ubuntu installs at work tomorrow.

You *might* be running into 393821; do you have a reasonably fast broadband connection?

Comment 6

[Hussam Al-Tayeb]

Not really, It's a 128kbit connection so the file takes around 40 seconds to download. That's enough time to test.

Comment 7

[Stephen Donner [:stephend] Not actively reading bugmail]

not to overload mardak, but maybe he can help?

Comment 8

[Stephen Walker]

Is a proxy being used? I've been seeing a problem of downloads not being shown as active, even though they are downloading in the background, with a local squid proxy. The download status will show up if a search is entered but only with a starting... status.

Comment 9

[Hussam Al-Tayeb]

Yes, I am behind a squid proxy server.
If the version of squid makes a difference, I can contact them and ask.

Comment 10

[Stephen Walker]

Hopefully the problem isn't squid specific but narrowing down the version(s) affected could be useful. I'm hitting this bug with the 2.6.STABLE16 windows binary.

Comment 11

[Ed Lee :Mardak]

Can you install the mozStorage Explorer extension and do the following..
http://shawnwilsher.com/archives/118

1) Open up the extension from Tools -> mozStorage Explorer
2) Start a download
3) While it's downloading, select "downloads.sqlite" and paste the following to the top box and hit "execute statement"..
SELECT name, source, state FROM moz_downloads ORDER BY id DESC LIMIT 1
4) Check if the name and source match up with what you expect from the file you're downloading and jot down the state
5) Let the download finish and do the same query, and note what the new state is
6) Post the 2 state numbers back here

Comment 12

[Hussam Al-Tayeb]

(In reply to comment #10)
> Hopefully the problem isn't squid specific but narrowing down the version(s)
> affected could be useful. I'm hitting this bug with the 2.6.STABLE16 windows
> binary.
> 

My ISP is still using 2.5.STABLE11 

Comment 13

[Hussam Al-Tayeb]

(In reply to comment #11)
> Can you install the mozStorage Explorer extension and do the following..
> http://shawnwilsher.com/archives/118
> 
> 1) Open up the extension from Tools -> mozStorage Explorer
> 2) Start a download
> 3) While it's downloading, select "downloads.sqlite" and paste the following to
> the top box and hit "execute statement"..
> SELECT name, source, state FROM moz_downloads ORDER BY id DESC LIMIT 1
> 4) Check if the name and source match up with what you expect from the file
> you're downloading and jot down the state
> 5) Let the download finish and do the same query, and note what the new state
> is
> 6) Post the 2 state numbers back here
> 

I tried your procedure with 2 files
with both the state while downloading is 5 and after download is finished state becomes 1.

Comment 14

[Hussam Al-Tayeb]

I forgot to add. In all cases 'name' and 'source' are correct.

Comment 15

[Hussam Al-Tayeb]

Attachment: Download manager while searching for a file that is currently downloading.

On a similar note, if I use search for the file that is currently downloading, it will show as 'starting' throughout all the download process even if the file is correctly downloading in the background.

Comment 16

[Ed Lee :Mardak]

(In reply to comment #13)
> with both the state while downloading is 5 and after download is finished state
> becomes 1.

(In reply to comment #15)
> Created an attachment (id=280925) [details]
> Download manager while searching for a file that is currently downloading.
> 
> On a similar note, if I use search for the file that is currently downloading,
> it will show as 'starting' throughout all the download process even if the file
> is correctly downloading in the background.

Makes sense. 5 is "queued" so basically the download manager isn't getting any data until it's finished (1).

If you checked the file size of these downloads, I'm assuming they stay 0 until they finish and jump to the right file size.

Seems like the proxy is holding back the data until it has it all then making it available all at once. If so, bug 393821 will make the download show up in the dlmgr, but it won't solve the problem of no progress updates when using the proxy. (With bug 393821 fixed, the download will show up as "starting..." then change to "Done".)

Comment 17

[Hussam Al-Tayeb]

(In reply to comment #16)
> (In reply to comment #13)
> > with both the state while downloading is 5 and after download is finished state
> > becomes 1.
> 
> (In reply to comment #15)
> > Created an attachment (id=280925) [details] [details]
> > Download manager while searching for a file that is currently downloading.
> > 
> > On a similar note, if I use search for the file that is currently downloading,
> > it will show as 'starting' throughout all the download process even if the file
> > is correctly downloading in the background.
> 
> Makes sense. 5 is "queued" so basically the download manager isn't getting any
> data until it's finished (1).
> 
> If you checked the file size of these downloads, I'm assuming they stay 0 until
> they finish and jump to the right file size.
> 
> Seems like the proxy is holding back the data until it has it all then making
> it available all at once. If so, bug 393821 will make the download show up in
> the dlmgr, but it won't solve the problem of no progress updates when using the
> proxy. (With bug 393821 fixed, the download will show up as "starting..." then
> change to "Done".)
> 

No, that's not correct. If I check the file size in filemanager while downloading, it is correctly incrementing from 0 to 700KB.

Also, it is a transparent proxy. I don't actually enter proxy settings.
And there is 128KBit cap on transfer rate, so even if the proxy has the file cached, I will still download it at around 12kbytes/sec.

Comment 18

[Hussam Al-Tayeb]

I just tried it again using a different connection (dialup) and I got the same results. Similarly to the previous case, the file does download in the background at 3kbytes per sec. The downloading file increments in size gradually according to nautilus. but it doesn't show as downloading in the download manager regardless how many times I close and reopen the download manager.
In the end, I have to close and reopen the download manager after it has finished downloading then it shows as downloaded.

Comment 19

[Ryuichi KUBUKI]

I saw it on WindowsXP Minefield. It seems to be like this on servers that use chunked HTTP.

Comment 20

[Elmar Ludwig]

I see this consistently for downloads from rapidshare.com, see for example the game patches linked from here:

http://games.rapidshare.com/games/2007/05/two_worlds/en/#download

Comment 21

[Ed Lee :Mardak]

It doesn't seem to be just "chunked" that's causing the problem to show up.

http://ed.agadak.net/chunk.php

That should send a 3.3MB file with transfer encoding chunked, but that'll correctly show up -- only difference is the size is unknown.

Comment 22

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to comment #21)
> It doesn't seem to be just "chunked" that's causing the problem to show up.
> 
> http://ed.agadak.net/chunk.php
> 
> That should send a 3.3MB file with transfer encoding chunked, but that'll
> correctly show up -- only difference is the size is unknown.

While that's true, I can confirm Elmar's problem...I had wanted to report it, but the files I was downloading from RapidShare were not so savory :-) 

Comment 23

[Ed Lee :Mardak]

Does anybody have this working on a public download site where I don't need to wait 15 minutes between each time I try downloading? I've looked at the http headers and duplicated them on that chunk.php page, but I don't get the same results..

Comment 24

[Ed Lee :Mardak]

Figured out why downloads weren't doing progress updates. POST requests aren't resumable, so GetEntityID returned not resumable causing NS_ENSURE_SUCCESS to panic.

So reason why rapidshare caused the problem is that you need to submit a captcha for the download.

I'm guessing for the proxy case (if this does solve the proxy case)... does squid use a HTTP version of 1.0? That would cause GetEntityID to return NOT_RESUMABLE as well.

[e-mail]: Next time please comment why you add a blocking bug. I was wondering why you added bug 377243 with no comment, and only now I realize you put it as the regression bug. (And mark as regression :))

biesi: There's nothing we can really do about resuming POST downloads, right? Unless we really want to do what is done for webpages.. "resuming will be resending POST data.. blablabla"

stephend: For a litmus testcase, it's just starting a download that required a POST form submission.. basically find a download that requires a button push instead of a link. ;)

Comment 25

[Ed Lee :Mardak]

Attachment: v1

Comment 26

[Hussam Al-Tayeb]

(In reply to comment #25)
> Created an attachment (id=282881) [details]
> v1
> 

Thanks Edward, your patch fixes this bug for me. 

Comment 27

[Christian :Biesinger (don't email me, ping me on IRC)]

(In reply to comment #24)
> biesi: There's nothing we can really do about resuming POST downloads, right?
> Unless we really want to do what is done for webpages.. "resuming will be
> resending POST data.. blablabla"

I'd be very surprised if servers supported range requests for POST

Comment 28

[Stephen Donner [:stephend] Not actively reading bugmail]

Looks like the Apple.com download page for QuickTime uses POST, but it appears in the download manager; seems like there's a more-specific testcase that can be written, instead?

Comment 29

[Ed Lee :Mardak]

The quicktime download page loads a new page that has an iframe that loads the download. Check URL for testcase.

Comment 30

[Stephen Donner [:stephend] Not actively reading bugmail]

in-litmus+

http://litmus.mozilla.org/show_test.cgi?id=4701

Thanks for the testcase, Mardak!

Comment 31

[Shawn Wilsher :sdwilsh]

Comment on attachment 282881 [details] [diff] [review]
v1

r=sdwilsh

If this gets checked in today, we don't have to get approval ;)

Comment 32

[Ed Lee :Mardak]

Checking in toolkit/components/downloads/src/nsDownloadManager.cpp;
/cvsroot/mozilla/toolkit/components/downloads/src/nsDownloadManager.cpp,v  <--  nsDownloadManager.cpp
new revision: 1.125; previous revision: 1.124
done

Comment 33

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED using:

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a9pre) Gecko/2007100205 Minefield/3.0a9pre

and

Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.9a9pre) Gecko/2007100204 Minefield/3.0a9pre