380435 - archive.mozilla.org can be horrendously slow

Status: RESOLVED DUPLICATE of bug 360397

(Release Engineering :: General, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

As many including myself have noticed over time, archive.mozilla.org is often limited in its usefulness by the fact that it can be horribly slow to access, sometimes even timing out.

From IRC:
<gavin|> I have theorized that it's drive take a long time to spin up after getting a request after long periods of inactivity
<gavin|> that is purely speculation, however!

Whether or not that's the issue, there's clearly a big problem with it that should be fixed.

Comment 1

[Noah (NoahSUMO/SolidSnake) [:Noah]]

I've been meaning to suggest this idea for sometime now but never thought to file a bug to get a response. Nice idea. ;)

Anyway, I believe it is due to the numerous volume of files trying to be displayed. Maybe that's wrong since a server is supposed to send out information as quickly as it can be downloaded and a broadband connection vs. a 56k user [me] should not have any trouble quickly loading that index. So it could be a server problem. But that page is pretty large/long, never checked it's file size tho.

Suggestion: Divide the archive into years. ex. 2004 | 2005 | 2006 | 2007
IF this can be done. Much more organized, and no needless downloading of previous years' entries.

Comment 2

[Dave Miller [:justdave]]

The disks in question are on a NetApp network storage device, and are really really speedy (typically faster than a local hard drive) as a network mount.

And any other directory on the site seems to be really speedy to me.  So yes, the huge list of files in that directory is probably related.

[root@manna archive.mozilla.org]# cd pub/mozilla.org/firefox/nightly/
[root@manna nightly]# ls -1 | wc -l
7384

Comment 3

[Dave Miller [:justdave]]

OK, the netapp gods say there's nothing else that can be done on the server end, this is purely an artifact of the large number of files in the directory.  If anyone cares badly enough about it, I guess the directory structure needs to be rearranged, which is probably a build thing since tinderboxes will have to change where they post their nightlies to keep them lined up (unless maybe we only move old ones to subdirectories)

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

I'm hoping to resolve this problem in the near term. We'll have to do something when we rearrange how the ftp servers are set up.