726168 - The web interface to hg.mozilla.org is too slow

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[Jeff Muizelaar [:jrmuizel]]

time wget https://hg.mozilla.org/try/file/e41e54cb6090/xpcom/base

does not complete in finite time (>44s)

time wget https://hg.mozilla.org/try/file/e41e54cb6090/xpcom/base
--2012-02-10 17:23:00--  https://hg.mozilla.org/try/file/e41e54cb6090/xpcom/base
Resolving hg.mozilla.org (hg.mozilla.org)... 63.245.209.160
Connecting to hg.mozilla.org (hg.mozilla.org)|63.245.209.160|:443... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

--2012-02-10 17:23:44--  (try: 2)  https://hg.mozilla.org/try/file/e41e54cb6090/xpcom/base
Connecting to hg.mozilla.org (hg.mozilla.org)|63.245.209.160|:443... connected.
HTTP request sent, awaiting response... 

After 44 seconds wget retries.

Comment 1

[Jeff Muizelaar [:jrmuizel]]

It completed after 4m16.490s

Comment 2

[Shyam Mani [:fox2mike]]

Jeff, can you reproduce this now? It seems to be fine..

real	0m3.656s

Comment 3

[Jeff Muizelaar [:jrmuizel]]

I get 3 seconds now as well. But that's still really slow.

Comment 4

[Ben Kero [:bkero]]

The entire hg infrastructure has been migrated to a new datacenter with freshly installed machines.  Do you still experience this problem?

Comment 5

[Jeff Muizelaar [:jrmuizel]]

It's still not great:

time wget https://hg.mozilla.org/mozilla-central/file/3a9629092a5f/tools/performance/memtest/cgi/list_reflow_form.cgi
--2012-04-17 13:45:55--  https://hg.mozilla.org/mozilla-central/file/3a9629092a5f/tools/performance/memtest/cgi/list_reflow_form.cgi
Resolving hg.mozilla.org (hg.mozilla.org)... 63.245.215.25
Connecting to hg.mozilla.org (hg.mozilla.org)|63.245.215.25|:443... connected.
HTTP request sent, awaiting response... 200 Script output follows
Length: unspecified [text/html]
Saving to: `list_reflow_form.cgi'

    [   <=>                                 ] 15,917      25.8K/s   in 0.6s    

2012-04-17 13:45:56 (25.8 KB/s) - `list_reflow_form.cgi' saved [15917]


real	0m1.547s
user	0m0.018s
sys	0m0.008s

but it's much better than it was and is now usable again. I'd be ok with the bug being resolved if you want to, but it would still be nice for it to be faster.

Comment 6

[Ben Kero [:bkero]]

Are you running this test remotely?  Perhaps your ISP does not have very good routes to us.  I have better results from attempts inside the datacenter (with mirrored results from the other sjc1 datacenter):

[[e-mail]3 ~]# time curl -s https://hg.mozilla.org/mozilla-central/file/3a9629092a5f/tools/performance/memtest/cgi/list_reflow_form.cgi > /dev/null

real	0m0.558s
user	0m0.023s
sys	0m0.041s
[[e-mail]3 ~]# time curl -s https://hg.mozilla.org/mozilla-central/file/3a9629092a5f/tools/performance/memtest/cgi/list_reflow_form.cgi > /dev/null

real	0m0.536s
user	0m0.025s
sys	0m0.037s
[[e-mail]3 ~]# time curl -s https://hg.mozilla.org/mozilla-central/file/3a9629092a5f/tools/performance/memtest/cgi/list_reflow_form.cgi > /dev/null

real	0m0.544s
user	0m0.027s
sys	0m0.035s
[[e-mail]3 ~]# time curl -s https://hg.mozilla.org/mozilla-central/file/3a9629092a5f/tools/performance/memtest/cgi/list_reflow_form.cgi > /dev/null

real	0m0.527s
user	0m0.028s
sys	0m0.033s

Comment 7

[Ben Kero [:bkero]]

I also get shorter response times over the bad wireless in my apartment.  This problem isn't going to be fixable without some deep investigation into performance, and quite frankly we haven't had any other performance issues yet.

bkero@Archimedes:~$ time curl -s https://hg.mozilla.org/mozilla-central/file/3a9629092a5f/tools/performance/memtest/cgi/list_reflow_form.cgi > /dev/null

real	0m0.660s
user	0m0.000s
sys	0m0.004s
bkero@Archimedes:~$ time curl -s https://hg.mozilla.org/mozilla-central/file/3a9629092a5f/tools/performance/memtest/cgi/list_reflow_form.cgi > /dev/null

real	0m0.729s
user	0m0.004s
sys	0m0.000s
bkero@Archimedes:~$ time curl -s https://hg.mozilla.org/mozilla-central/file/3a9629092a5f/tools/performance/memtest/cgi/list_reflow_form.cgi > /dev/null

real	0m0.815s
user	0m0.004s
sys	0m0.000s
bkero@Archimedes:~$ time curl -s https://hg.mozilla.org/mozilla-central/file/3a9629092a5f/tools/performance/memtest/cgi/list_reflow_form.cgi > /dev/null

real	0m0.688s
user	0m0.004s
sys	0m0.000s
bkero@Archimedes:~$ time curl -s https://hg.mozilla.org/mozilla-central/file/3a9629092a5f/tools/performance/memtest/cgi/list_reflow_form.cgi > /dev/null

real	0m1.087s
user	0m0.004s
sys	0m0.000s
bkero@Archimedes:~$ time curl -s https://hg.mozilla.org/mozilla-central/file/3a9629092a5f/tools/performance/memtest/cgi/list_reflow_form.cgi > /dev/null

real	0m0.754s
user	0m0.000s
sys	0m0.004s
bkero@Archimedes:~$ time curl -s https://hg.mozilla.org/mozilla-central/file/3a9629092a5f/tools/performance/memtest/cgi/list_reflow_form.cgi > /dev/null

real	0m0.767s
user	0m0.000s
sys	0m0.004s

Comment 8

[Shyam Mani [:fox2mike]]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #5)
> It's still not great:

That's 1.5 seconds, out of which 0.6 seconds were spent downloading the file :) The rest probably went in DNS, TCP and all the stuff your packets have to go through to reach us and back to you.

From 44 seconds and 4 mins (where it was truly problematic) to less than 2 seconds, is a massive improvement. Calling that not great is confusing to me :)