596857 - samo traffic has tripled since two weeks ago

Status: RESOLVED INVALID

(mozilla.org Graveyard :: Server Operations, task)

Description

[Jeff Balogh (:jbalogh)]

These are wc -l counts from samo's logs, and we log the response time for just about every request that goes through zamboni.

We noticed we were getting DDOS'd on Sunday and banned some IPs, but traffic still seems unreasonably high.

2010-09-01.bz2  436539
2010-09-02.bz2  429703
2010-09-03.bz2  411821
2010-09-04.bz2  362534
2010-09-05.bz2  372631
2010-09-06.bz2  425186
2010-09-07.bz2  459941
2010-09-08.bz2  608335
2010-09-09.bz2  767475
2010-09-10.bz2  801628
2010-09-11.bz2  787208
2010-09-12.bz2  1121917
2010-09-13.bz2  1044026
2010-09-14.bz2  1045495

Comment 1

[Wil Clouser [:clouserw]]

We've been shipping Fx betas that have been taking more advantage of SAMO, but I don't think it would be that much of an increase.  It will account for some of the traffic though.

Comment 2

[Jeremy Orem [:oremj]]

Overall traffic hasn't changed much:


[root@dm-stats01 services.addons.mozilla.org]# zcat access_2010-09-01* | wc -l
21341360
[root@dm-stats01 services.addons.mozilla.org]# zcat access_2010-09-14* | wc -l
23321631

If you want to take a look at the access logs it's dm-stats01:/data/stats/logs/im-log02/services.addons.mozilla.org/

Comment 3

[Jeff Balogh (:jbalogh)]

(In reply to comment #2)
> Overall traffic hasn't changed much:

I think that means that a) we're getting a lot more logged-in users or b) zeus isn't picking up as much as it should.

Did the zeus switch affect how much we're caching on SAMO?  I didn't look at how many of the requests were logged-in, but I'd be surprised this kind of increase.

Comment 4

[Jeremy Orem [:oremj]]

We haven't switched SAMO yet, so nothing changed there.

Comment 5

[Jeremy Orem [:oremj]]

Can you check your logs again? Is traffic to zamboni still up?

Comment 6

[Jeff Balogh (:jbalogh)]

Looks like it's coming from the new add-on manager.

$ bzgrep -c 'api/1.5/search/guid:' 2010-09-*
2010-09-01.bz2:20406
2010-09-02.bz2:22202
2010-09-03.bz2:22058
2010-09-05.bz2:22131
2010-09-06.bz2:25840
2010-09-07.bz2:37262
2010-09-08.bz2:137626
2010-09-09.bz2:301021
2010-09-10.bz2:379194
2010-09-11.bz2:403047
2010-09-12.bz2:635676
2010-09-13.bz2:592658
2010-09-14.bz2:596365
2010-09-15.bz2:611121
2010-09-16.bz2:621478
2010-09-17.bz2:610504
2010-09-18.bz2:560207
2010-09-19.bz2:607619

I'm running some of my stats to see how fast we're handling search/guid since it appears to be a really popular page.  This is all traffic from beta5+ users.