Closed Bug 1214797 Opened 10 years ago Closed 10 years ago

Investigate slow/breaking connection to AMO

Categories

(Cloud Services :: Operations: AMO, task)

Product:
Cloud Services
Component:
Operations: AMO
Platform:
x86_64
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: TheOne, Assigned: jason)

Details

For several months now, I have been experiencing very bad performance to AMO. Symptoms: * Page-loading times of 30+ seconds for https://addons.mozilla.org/en-US/editors/ * Connection-reset errors when submitting an add-on review * Super slow downloading of files (e.g. https://addons.mozilla.org/firefox/downloads/source/1767134), speed < 200kbit/s I have a 16Mbit connection and no other consumers on the line. $ traceroute addons.mozilla.org traceroute to addons.dynect.mozilla.net (63.245.216.132), 64 hops max, 52 byte packets 1 fritz.box (192.168.1.1) 0.445 ms 0.314 ms 0.274 ms 2 217.0.117.16 (217.0.117.16) 7.957 ms 6.407 ms 6.356 ms 3 217.0.81.170 (217.0.81.170) 10.709 ms 8.422 ms 8.576 ms 4 217.239.48.146 (217.239.48.146) 12.244 ms 12.016 ms 12.920 ms 5 ffm-b12-link.telia.net (213.248.103.169) 35.191 ms 34.940 ms 35.688 ms 6 ffm-bb2-link.telia.net (62.115.142.44) 113.928 ms ffm-bb2-link.telia.net (62.115.142.58) 51.006 ms ffm-bb1-link.telia.net (62.115.142.18) 36.195 ms 7 ash-bb3-link.telia.net (80.91.246.60) 134.151 ms ash-bb4-link.telia.net (213.155.135.59) 130.415 ms ash-bb4-link.telia.net (80.91.246.64) 127.001 ms 8 dls-b21-link.telia.net (213.155.137.111) 177.662 ms dls-b21-link.telia.net (213.248.80.142) 164.626 ms dls-b21-link.telia.net (213.155.137.53) 160.134 ms 9 nyk-bb1-link.telia.net (62.115.135.94) 124.239 ms phx-b1-link.telia.net (62.115.138.201) 184.617 ms phx-b1-link.telia.net (62.115.136.219) 188.212 ms 10 ash-bb3-link.telia.net (62.115.134.111) 138.090 ms mozilla-ic-140268-phx-b1.c.telia.net (213.248.104.202) 168.051 ms 168.378 ms 11 v-1057.core1.market.phx1.mozilla.net (63.245.216.50) 184.782 ms mozilla-ic-140268-phx-b1.c.telia.net (213.248.104.202) 198.363 ms 169.841 ms 12 addons-mozilla-org.mktns.services.phx1.mozilla.net (63.245.216.132) 185.893 ms 186.858 ms 183.516 ms $ traceroute addons.cdn.mozilla.net traceroute to d1sp2sgy246t7c.cloudfront.net (54.192.47.30), 64 hops max, 52 byte packets 1 fritz.box (192.168.1.1) 0.534 ms 0.383 ms 0.315 ms 2 217.0.117.16 (217.0.117.16) 7.215 ms 7.483 ms 6.989 ms 3 217.0.81.170 (217.0.81.170) 7.967 ms 10.009 ms 7.052 ms 4 f-ed5-i.f.de.net.dtag.de (217.5.95.26) 12.739 ms 12.058 ms 12.043 ms 5 ffm-b12-link.telia.net (213.248.103.169) 32.248 ms 34.149 ms 34.131 ms 6 a100row-ic-151139-ffm-b12.c.telia.net (213.248.96.82) 36.927 ms 36.998 ms 36.114 ms 7 * 54.239.4.248 (54.239.4.248) 35.621 ms * 8 54.239.4.223 (54.239.4.223) 47.331 ms * 36.159 ms 9 54.239.5.29 (54.239.5.29) 40.657 ms * 35.882 ms 10 * * * 11 * * * 12 * * * ...
I'm seeing the same thing whenever I load https://addons.mozilla.org/en-US/editors/ as well. AMO itself is mostly performant, but editors frequently results in connection resets. trace to addons.mozilla.org is: 3 * * 192-171-60-26.cpe.pppoe.ca (192.171.60.26) 29.359 ms 4 ae1_2552-agg01-tor2.teksavvy.com (192.171.63.12) 28.943 ms xe-0-0-1_0-agg01-tor2.teksavvy.com (69.165.174.249) 30.735 ms xe-0-0-0_0-agg01-tor2.teksavvy.com (69.165.219.105) 24.307 ms 5 ae0_2160-bdr04-tor.teksavvy.com (192.171.59.67) 23.783 ms 26.312 ms ae1_2170-bdr04-tor.teksavvy.com (192.171.59.99) 24.991 ms 6 ae6.tor10.ip4.gtt.net (173.205.58.185) 24.535 ms 28.002 ms ae7.tor10.ip4.gtt.net (173.205.58.189) 25.100 ms 7 xe-3-3-0.chi12.ip4.gtt.net (141.136.105.33) 60.656 ms xe-10-3-1.chi12.ip4.gtt.net (141.136.110.198) 37.213 ms xe-8-2-0.chi12.ip4.gtt.net (141.136.105.197) 41.010 ms 8 1-2-2.er4.ord7.us.zip.zayo.com (128.177.105.185) 50.063 ms 37.392 ms 36.841 ms 9 ae2.er5.ord7.us.zip.zayo.com (64.125.20.150) 39.680 ms 41.744 ms 41.862 ms 10 ae10.er2.ord7.us.zip.zayo.com (64.125.20.145) 38.988 ms 39.470 ms 34.598 ms 11 ae14.cr2.ord2.us.zip.zayo.com (64.125.31.69) 38.992 ms 37.598 ms 36.072 ms 12 v23.cr2.ae29.iah1.us.zip.zayo.com (64.125.31.241) 66.813 ms 75.174 ms 65.918 ms 13 ae6.mpr4.phx2.us.zip.zayo.com (64.125.31.66) 94.006 ms 93.740 ms 80.812 ms 14 64.124.201.182.mozilla.com (64.124.201.182) 82.383 ms 87.294 ms 88.509 ms 15 v-1058.core1.market.phx1.mozilla.net (63.245.216.54) 87.147 ms 89.504 ms 82.809 ms 16 addons-mozilla-org.mktns.services.phx1.mozilla.net (63.245.216.132) 83.689 ms 83.041 ms 86.481 ms trace to the cdn is 3 192-171-60-26.cpe.pppoe.ca (192.171.60.26) 23.384 ms 192-171-63-10.cpe.pppoe.ca (192.171.63.10) 26.743 ms * 4 xe-0-0-0_0-agg01-tor2.teksavvy.com (69.165.219.105) 24.934 ms ae1_2553-agg01-tor2.teksavvy.com (192.171.60.28) 25.923 ms ae1_2552-agg01-tor2.teksavvy.com (192.171.63.12) 23.137 ms 5 ae1_2170-bdr04-tor.teksavvy.com (192.171.59.99) 28.779 ms ae0_2160-bdr04-tor.teksavvy.com (192.171.59.67) 28.937 ms ae1_2170-bdr04-tor.teksavvy.com (192.171.59.99) 30.145 ms 6 ae3_10-bdr01-nyc.teksavvy.com (206.248.155.94) 38.135 ms ae4_10-bdr01-nyc.teksavvy.com (206.248.155.98) 35.694 ms 32.927 ms 7 nyiix1-peering.amazon.com (198.32.160.244) 37.067 ms 35.378 ms 42.680 ms 8 178.236.3.183 (178.236.3.183) 36.036 ms 178.236.3.181 (178.236.3.181) 37.405 ms 54.240.229.118 (54.240.229.118) 37.659 ms 9 54.240.228.49 (54.240.228.49) 37.343 ms 54.240.202.49 (54.240.202.49) 39.521 ms 54.240.202.51 (54.240.202.51) 37.565 ms
We made changes in where AMO is hosted and I expect throughput to a little bit better. Can we test and see how it is? Can you give me more details about the connection timeouts? How long does it take for to happen? Can you give me URLs to pages that fail to load so I can investigate further?
Assignee: nobody → jthomas
I haven't seen any of the mentioned symptoms after the switch to AWS and jasons tweaks. Thanks everyone!
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.