Closed Bug 453584 Opened 17 years ago Closed 17 years ago

Severe load time performance regression with the Bugzilla update

Categories

(bugzilla.mozilla.org :: General, defect)

Product:
bugzilla.mozilla.org
Component:
General
Type:
defect
Priority:
Not set
Severity:
major

Tracking

()

Status:
RESOLVED WORKSFORME

People

(Reporter: bzbarsky, Unassigned)

References

Details

As far as I can tell, the post-update Bugzilla takes a lot longer to load pages. The bug reporting form at https://bugzilla.mozilla.org/enter_bug.cgi?product=mozilla.org, for example, just took about 10 seconds to go from "initial request sent" to "something other than a blank screen shows". I'm on a 3Mb connection, so it's not that the pipe is slow or anything. As far as I can tell, we're at least partly network-bound nevertheless. One possible reason is the number of scripts and stylesheets the pages load (7 external scripts and 16 stylesheets for the bug-entry page, but that includes the alternate sheets, which we load at slightly lower priority). One thing that would help here in the short run would be to put the stylesheets before the scripts (since we block the parser on the latter but not the former). Past that, I'm not sure what else is going on. Doesn't seem to be any XMLHttpRequest going on, for example... I do see something odd: I never get read access to the cache entries for this stuff, only write access (in nsHttpChannel). So I'm downloading all this stuff all the time.
OK, for other sites I do get read access to the cache entries. Something is special about Bugzilla here.
(In reply to comment #0) > As far as I can tell, the post-update Bugzilla takes a lot longer to load > pages. The bug reporting form at > https://bugzilla.mozilla.org/enter_bug.cgi?product=mozilla.org, for example, > just took about 10 seconds to go from "initial request sent" to "something > other than a blank screen shows". That sounds like it could be some problem with your connection--others were having strange problems with other sites if they were coming from the internal network, today. (In other words, on Firefox 3 here on Comcast at my home in Mountain View I'm not seeing anything like a 10-second page load time on that link.) > One thing > that would help here in the short run would be to put the stylesheets before > the scripts (since we block the parser on the latter but not the former). Sure, that's harmless and we can do that easily. > I do see something odd: I never get read access to the cache entries for this > stuff, only write access (in nsHttpChannel). So I'm downloading all this stuff > all the time. My guess would be that that's something that the Netscaler is doing, or Apache is doing?
Hmm... So we have a stream-based cache entry (since that's what HTTP does). It's stored in memory (no Cache-control: public in the response headers). We open it in Connect(), get write-only access (it's a brand-new entry). Then we get some data, and call nsCacheEntryDescriptor::nsOutputStreamWrapper::OnWrite as part of reading the data into the consumer. This calls RequestDataSizeChange() on the cache entry descriptor, which calls nsCacheService::OnDataSizeChange. That calls OnDataSizeChange() on the memory cache device. But the memory cache device mSoftLimit is 0 in my case, so we abort the whole thing.
Max, I'm not on the MoCo internal network. I'm in Boston, using Speakeasy. And the problem is definitely limited to bugzilla from what I can tell, and reasonably new. I'll dig some more to see what's going on with the memory cache, but did Bugzilla use to send cache-control: public headers?
Hmm. And now a restart later I can't reproduce the mSoftLimit issue (mSoftLimit and mHardLimit were both being set to 0 at startup, fwiw). Maybe I should try restarting my non-debug build too and seeing whether that helps....
Doesn't seem to make much of a difference. On the other hand, we seem to only set up a 25M memory cache on a box with 4 gigs of RAM, and it's full over here. So if we're OK with disk caching bugzilla (at least all the scripts and stylesheets), we should really allow that.
(In reply to comment #4) > Max, I'm not on the MoCo internal network. I'm in Boston, using Speakeasy. > And the problem is definitely limited to bugzilla from what I can tell, and > reasonably new. I'm also not seeing this problem connecting from Michigan on Charter cable. What IP address are you getting for Bugzilla? IIRC we have a caching proxy for it in Beijing, and may also have one in Amsterdam. I've seen east coast US get routed to Amsterdam before (the transatlantic hop is actually shorter than the transcontinental one quite often). > I'll dig some more to see what's going on with the memory cache, but did > Bugzilla use to send cache-control: public headers? Bugzilla has never sent any cache-control headers at all, and I just confirmed that Apache isn't adding them on the webheads, nor is the Netscaler adding any. If you're seeing any you must have another proxy server between you and Bugzilla.
(In reply to comment #7) > What IP address are you getting for Bugzilla? IIRC we have a caching proxy > for it in Beijing, and may also have one in Amsterdam. Just checked the geodns config, we do NOT have that set up in Amsterdam currently, just Beijing. And the Beijing IP only gets fed to IPs in China.
I've also noticed that everything gets slower with the upgrade. I'm in Germany and as far as I can tell there isn't any problem with my provider.
OS: Mac OS X → All
Hardware: PC → All
(In reply to comment #4) > Did Bugzilla use to send cache-control: public headers? No, not upstream, at least as far as I know. We probably could, on non-secure bugs. We have a bug filed about using cache-control headers somewhere.
> What IP address are you getting for Bugzilla? 63.245.209.72 > If you're seeing any I'm not. I was just trying to figure out the reasons why the new setup could be slower than the old one. The old one having such headers was a possibility. Is there any place where I could test the two versions on the same server? That might let me get closer to figuring out where time is spent...
I did some more testing. Bumping the size of the memory cache seems to help some. There's still a 3-4 second lag between me hitting enter and anything happening, with the status bar set to "waiting for bugzilla.mozilla.org..." but after that things seem to get loaded from cache. I also checked, and my ping latency to bugzilla is about 115ms, which should be fine.
(In reply to comment #0) > One thing > that would help here in the short run would be to put the stylesheets before > the scripts (since we block the parser on the latter but not the former). How hard would it be to do that and get it pushed? At this point any little bit of potential speedup would be helpful (turning off JavaScript does help a little bit, but the UI and functionality degradation sort of negates the little speed-up on loading).
(In reply to comment #13) > How hard would it be to do that and get it pushed? Already done and committed to CVS: bug 453585.
bz, is it still an issue? justdave updated b.m.o a day or two ago.
Things are a lot better now (though I think part of that was the YUI version rollback, perhaps).
FYI, the bfcache fix is now applied to b.m.o. Is this bug fixed now? Is there anything else which slows down Bugzilla?
The bfcache fix doesn't affect this bug, since this was about non-history loads. Nothing's changed here since the last time I commented. It takes 4-5 seconds to load a small bug page, which is about as fast as it used to be, I think.
I'm a user with a 56 Kbps modem for a dial-up connection. Today, I ran a query on bugzilla.mozilla.org. From the resulting bug list, I selected four bugs for display, each in a separate tab. (Before selecting any of the bugs, I terminated my E-mail and NTP clients and waited until all modem activity ceased.) It took exactly 10 minutes (from 12:30:00 PDT to 12:40:00 PDT) for the bugzilla.mozilla.org server to send me 2,912,082 bytes. That's less than 39 Kbps, definitely less than the rated speed of my modem. Note that I timed and counted bytes only for the four selected bug reports. There was no query activity other than resolving the "?id=nnnnn" input. The query to generate the bug list was completed before I started timing and counting. If what I observed is not relevant to this bug report, please let me know. I will then submit a new bug report. (People ask me why I don't have broad-band. Why should I pay extra for that when servers don't deliver at broad-band speeds?)
According to the numbers you posted, your average transfer rate for that 10 minutes was 38 Kbps. That's actually a very good speed (and probably maxing your throughput) for a 56 Kbps modem. The 56 Kbps is a maximum *theoretical* speed, and I've never seen a 56K modem actually manage to go that fast in the wild. In fact, 38400 baud (38.4 Kbps) is a fairly typical connect rate from what I've seen. I can assure you that bugs download much faster over broadband (I can dump my browser cache and reload this bug, and I have the entire thing including the banner graphics in under 5 seconds).
Just out of interest why the js function initKeywords() is inline and served with each page load? Shouldn't it be better stored in an external js file? That's really a lot of content which is loaded again and again.
(In reply to comment #21) > Just out of interest why the js function initKeywords() is inline and served > with each page load? Shouldn't it be better stored in an external js file? See bug 457116 about this topic.
I notice that my pageload times feel faster (closer to pre-upgrade times) when I'm not logged in, which means bug 364162 could be a factor here (as well as bug 457116 already mentioned); I have 31 searches.
Re comment #20: Yesterday, I downloaded a new version of Acrobat Reader from an Adobe server. I got speeds in excess of 40 Kbps, often reaching more than 43 Kbps with my dial-up modem. In terms of time to download a given number of bytes, the Adobe server was 22% faster than the bugzilla.mozilla.org server; that is, it would take 22% less time to download the same number of bytes from Adobe's server than from bugzilla.mozilla.org.
Overall reports indicate that this is no longer a problem.
Status: NEW → RESOLVED
Closed: 17 years ago
Resolution: --- → WORKSFORME
Do you still see perf regressions while being logged in since the recent upgrade? If I guessed correctly what was slowing down Bugzilla, this should be fixed now, see bug 364162.
Component: Bugzilla: Other b.m.o Issues → General
Product: mozilla.org → bugzilla.mozilla.org
You need to log in before you can comment on or make changes to this bug.