201910 - Page load over 56k modem fails if page(s) contain lots of objects

Status: RESOLVED WONTFIX

(Core :: Networking, defect)

Description

[Hogarth]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-AU; rv:1.4b) Gecko/20030410
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-AU; rv:1.4b) Gecko/20030410

If you have one page with lots of objects (like the one above) and/or are
loading lots of pages at once (say through going through a page and
middle/ctrl-clicking on links or selecting a group bookmark) Mozilla spams the
connection with a whole heap of requests. When the connection is slow (like a
56k modem) this leads to increadibly slow load times (it took about 40 mins for
the above page to stop loading) and even when it does finish there's no
guarantee that the page will be complete (even after 40mins the above had
missing and half-loaded images).


Reproducible: Always

Steps to Reproduce:
1. Have a slow connection
2. Get lots of pages loading at the same time


Actual Results:  
The page did not fully load. Images were missing. In other cases even the base
html page does not finish loading and so you wind up with missing chunks that way.

Expected Results:  
The page to load fully.

It'd probably be a good idea to put load requests into a queue and have mozilla
take x items off of it at a time allowing people to select connection speed,
thereby changing the value of x. The default could be 'fscking fast connection'
and as such everything and anything that hits the queue would be loaded as soon
as it hits the queue. A selection of 56k modem would lead to say x = 5 so that
no more then 5 items are loading in parallel at any one time. Otherwise, mozilla
winds up timing out on its load requests and the above happens.

As for my environment. Well, obviously, I'm behind a 56k modem. I also have
http/1.1 and pipelining selected and I have a proxy before the modem.

Comment 1

[Boris Zbarsky [:bzbarsky]]

> It'd probably be a good idea to put load requests into a queue and have mozilla
> take x items off of it at a time

If pipelining is off, that's what we do.  x == 8.  The point of pipelining is
that you can make multiple parallel requests on the same server connection,
which should improve life... but it sounds like it's backfiring in this case?

Or you're just seeing a pipelining bug...

Comment 2

[tenthumbs]

I tried this on a modem link using a local squid proxy. It took about 15 minutes
to load. There were 110 requests related to this page to a number of different
locations. There were 12 404 responses so not all objects will appear.

The modem connected at 44000 so if it took the reporter ~40 minutes then there's
either a really bad connection or something else is going wrong, perhaps the proxy.

Comment 3

[Hogarth]

On the offchance that it may have been squid screwing things up I did the following:

1. I shut down squid
2. I removed the firewall redirect
3. I cleared mozilla's cache
4. I hit shift-reload on the page

~20mins later (dunno why it took longer before but oh-well) mozilla finished
with the page and I stopped counting at 26 images not loaded. I know at least
some of these exist because earlier I used wget to suck them down and if I go to
the image location manually I get the image.

I shall now turn pipe-lining off (keeping http 1.1 and keep-alive) and try again.

Comment 4

[Hogarth]

Ok.

1. killed the tab
2. cleared the cache
3. made a new one and started loading the url
4. stopped it
5. hit shift-reload
6. waited...
7. found network was screwing up so hit esc
8. hit shift-reload
9. waited...

About 20 mins later, mozilla stopped trying to load the page with about the same
result as before (except some of the image that I remember loading before didn't
this time).

so as near as I can tell, it's not my cache (squid) as that's off and pipelining
doesn't make much of a difference (similar result with it both on and off). My
PC has 256MB of ram, a p3 700, mozilla has 50MB of cache, its link is a 100mb
ethernet link to my gw server whose link to the internet is a modem.

My ISP has proxies but they have a 40mbit or so link to the innanet so they
wouldn't be getting choked. No way to avoid unless someone puts up a testpage on
a port other then 80 (we're transparently proxied).

Is there any other testing I can do?

Comment 5

[Darin Fisher]

i get broken images on that site as well w/ pipelining disabled and no proxy. 
linux build 2003041108.  investigating...

Comment 6

[Darin Fisher]

i'm confused.  the broken image icons result from 404 responses from the server.
 where's the bug here?  do you have any other testcases?

Comment 7

[Hogarth]

As I stated previously, I could go to one of the not-shown images, get its
location, open a new tab and have mozilla load it (maybe I wasn't clear) so it's
not all 404s.

As for other testcases, I can whip out my group bookmarks. You can either have
the 30 or so news webpages group bookmark or my comics group bookmark. Which
would you like? :) (both?)

*ponders his ability of making a testcase*

Comment 8

[tenthumbs]

I'm not saying that this is the issue here but I've long thought that 
mozilla malfunctions when it's doing a lot of work on a "slow" machine. 
The page here has links to many sites and the reporter's machine is < 
1/3 my speed so I guess it qualifies as slow (sorry about that). The 
usual symptom is that some objects fail to load. At least that's what I 
would occasionally see when I still had a slow machine. Personally, I 
would look at timers. Still, this bug may be domething completely 
different.

Comment 9

[Darin Fisher]

Hogarth: yes, please provide several URLs.  that way, i'll have a better chance
of seeing what you're seeing.  thx!

Comment 10

[Darin Fisher]

-> default owner

Comment 11

[A.D.F.]

(In reply to comment #8)
> I'm not saying that this is the issue here but I've long thought that 
> mozilla malfunctions when it's doing a lot of work on a "slow" machine. 
> The page here has links to many sites and the reporter's machine is < 
> 1/3 my speed so I guess it qualifies as slow (sorry about that). The 
> usual symptom is that some objects fail to load. At least that's what I 
> would occasionally see when I still had a slow machine. Personally, I 
> would look at timers. Still, this bug may be domething completely 
> different.

Here there can be at least 4 different problems.

1) Cache problem, when you load lots of big images browser tries to keep them in memory along with the previous ones (it does not purge the older ones if they are in the short history), so the whole thing sucks a lot because it consumes much more memory than needed and usually OS starts to swap and do other unpleasant things; in the past I have seen this behaviour on a machine PIII 733 Mhz, 384 MB RAM with Linux and Mozilla 1.7.12, I could try to reproduce this with newer versions (SeaMonkey).

2) Interrupted transfers (timeout), many sites which serve big images tend to force send socket buffer size to a very high value, let's say between 128 KB and 512 KB and with these values modem connections can timeout;  if a page has links to tens or even hundreds of different servers then connections to those servers should not be kept alive nor pipelined unless there are at least 3 or more images for the same server;  other servers have a lower limit on throughput; if there are 16 - 24 connections which are all downloading images, then on a modem with 5 KB/sec. it means 200 - 300 bytes / sec. and many servers have a protection against very slow connections, so even if they don't use big send socket buffers they may detect the fact that client is too slow for them (and so they fire it).

The solution is to limit the number of parallel connections when the available bandwidth to Internet is less than 256 Kbit/sec.

3) Throttling issues, some servers (and or load balancers, etc.) may respond very slowly to first request when they are overloaded; besides this they can ignore too many requests coming from the same (IP) address.

4) Pipelining code, maybe some detection logic for broken chains (proxy + server) and some smart tricks could be deployed here to avoid timeouts and restart problems.

Anyway in this case I think that most (but not all) of the problems comes from slow / non answering servers, unreacheable domains, slow DNS and files not found.