562986 - Traffic to mirrors from bouncer way below average

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Reed Loden [:reed]]

In the last few hours, two different mirror admins have reported to #mirrors that their traffic levels have dropped off dramatically. I checked my mirror's levels, too, and I'm seeing the same thing. I'm still getting some traffic, but it's ridiculously low (only 130.68Mbps at weight 25000). Something is wrong...

Comment 1

[matthew zeier [:mrz]]

This should be back - I was testing something and added it with a mirror weight of 50,000.

Are  you still seeing depressed traffic?

Comment 2

[Reed Loden [:reed]]

(In reply to comment #1)
> Are  you still seeing depressed traffic?

Yeah, I'm down to 121.96Mbps right now.

Comment 3

[matthew zeier [:mrz]]

What are you expecting to see?  I topped out at just under 2Gbps earlier which matches what I've seen in the past on a Friday past peak during a non-release.

Comment 4

[Reed Loden [:reed]]

(In reply to comment #3)
> What are you expecting to see?  I topped out at just under 2Gbps earlier which
> matches what I've seen in the past on a Friday past peak during a non-release.

Generally, 1-1.5Gbps on average.

Comment 5

[matthew zeier [:mrz]]

I'm still seeing a bunch of traffic to this Ankeena box even though it's marked as disabled (and no longer showing up on the sentry page).

Comment 6

[Reed Loden [:reed]]

Attachment: traffic stats over last 24 hours

Comment 7

[Levent YILDIRIM (:Ankara)]

Attachment: ftp.ankara.edu.tr

It's my network traffic last 24 hours

Comment 8

[Jeremy Orem [:oremj]]

I've been watching one webhead for about 8 min now and here are the top 10 mirrors the app is selecting:

956, http://www.mirrorservice.org
1019, http://pv-mirror01.mozilla.org
1109, http://mozilla2.snt.utwente.nl
1147, http://mirror.leaseweb.com
1159, http://mirror.yellowfiber.net
1198, http://mozilla3.snt.utwente.nl
1205, http://ftp.sunet.se
1409, http://hyperion.zih.tu-dresden.de
2255, http://ftp.jaist.ac.jp
2434, http://mozilla.mirror.ac.za

Comment 9

[Jeremy Orem [:oremj]]

Top 10 after 40 min:

4634, http://www.mirrorservice.org
4755, http://pv-mirror01.mozilla.org
5283, http://mirror.leaseweb.com
5300, http://mozilla2.snt.utwente.nl
5374, http://mirror.yellowfiber.net
5439, http://mozilla3.snt.utwente.nl
5703, http://ftp.sunet.se
6570, http://hyperion.zih.tu-dresden.de
10276, http://ftp.jaist.ac.jp
11185, http://mozilla.mirror.ac.za

Comment 10

[Reed Loden [:reed]]

So, why am I (the top mirror at weight 25000) not in that list?

Comment 11

[matthew zeier [:mrz]]

s/you/voxel/ :)

Comment 12

[matthew zeier [:mrz]]

12:55 < oremj> mrz: in the database voxcdn is not active for 3.6.3

I did note this earlier:

Log entry for  [257] () at 2010-04-30 17:17:29 UTC

Note: a FAILED/404 result on a file which is not included in the mozilla-current
      module is okay if you are only rsyncing mozilla-current.

Checking mirror 3347-mozilla.voxcdn.com ...
http://3347-mozilla.voxcdn.com/pub/mozilla.org sent no response after 5 seconds!  Checking recent history...
**** weight 25000 active 0 for http://3347-mozilla.voxcdn.com/pub/mozilla.org
**** weight 25000 active 1 for http://3347-mozilla.voxcdn.com/pub/mozilla.org
**** weight 25000 active 1 for http://3347-mozilla.voxcdn.com/pub/mozilla.org
**** weight 25000 active 1 for http://3347-mozilla.voxcdn.com/pub/mozilla.org
Pattern OK, leaving weight unchanged.

Comment 13

[Reed Loden [:reed]]

(In reply to comment #12)
> 12:55 < oremj> mrz: in the database voxcdn is not active for 3.6.3

Uh, why not? I'm pulling mozilla-releases, and I have 3.6.3. http://3347-mozilla.voxcdn.com/pub/mozilla.org/firefox/releases/3.6.3/

> Log entry for  [257] () at 2010-04-30 17:17:29 UTC

Looks like it's missing some information there?

> Checking mirror 3347-mozilla.voxcdn.com ...
> http://3347-mozilla.voxcdn.com/pub/mozilla.org sent no response after 5
> seconds!

Dunno... Other mirrors have been having sporadic red blocks as well... but seems everything is stable on my end.

Comment 14

[Reed Loden [:reed]]

(In reply to comment #11)
> s/you/voxel/ :)

Eh, Voxel just donates the bandwidth and the CDN network. The origin server that serves the CDN its content is my personal server (for which I personally pay).

Comment 15

[Jeremy Orem [:oremj]]

Found the problem.  It's a bug in sentry.

Comment 16

[Jeremy Orem [:oremj]]

Fixed in r66666.

Comment 17

[Fred Wenzel [:wenzel]]

Quick 411 on the bug: If you run a query with the Perl MySQL interface that returns the column "id" from two different tables, the latter will overwrite the former. Makes me want to abandon Perl sooner rather than later.

Fix is here: http://github.com/fwenzel/tuxedo/commit/545dc87 and in r66666.