512354 - Get an 2 hours of logs between peak times (7am-12pm) for AMO

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Hi Jeremey,

Wil suggested that I talk to you about setting up and configuring the AMO test cluster mentioned in 504701 so we can replay log files.

I specifically want to monitor the boxes that will be running Sphinx's searchd.

I think if we can piggy back searchd and the indexers on two Mysql slaves this should be easy enough.

Ultimately I want to see that sphinxd doesn't keel over, and hopefully see less strain on mysql than usualy.

Cheers,
-d

Comment 1

[Wil Clouser [:clouserw]]

This is going to be important for our search rewrite in our next milestone.  The timeline isn't set but it's going to happen in the next month or so.  Does that work for you?

Comment 2

[Jeremy Orem [:oremj]]

How big of a cluster do you need (app servers? db slaves?) Should this be on physical boxes?

Comment 3

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

We can use:

https://intranet.mozilla.org/Webdev:Test_Cluster

I think we could just do 1 master, 1 slave with sphinx/searchd and indexer on it.  1 app server.

Then we can just replay 1 single production app servers worth of logs into it.

Comment 4

[Jeremy Orem [:oremj]]

I think webdev runs that cluster now. Is this something you can set up? What do you need help with?

Comment 5

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

oremj,

Can I get a single days worth of access logs from a single frontend node?

I also heard you've got a log replay script... if you do I'll use it.

-d

Comment 6

[Jeremy Orem [:oremj]]

I do have a log replay script(http://code.google.com/p/logreplay/), but it isn't very well documented. You can also use siege(http://www.joedog.org/index/siege-home) which has better docs and also takes a file full of URLs.

Comment 7

[Jeremy Orem [:oremj]]

Dropped addons_logs.tar in khan:/home/ddash

Comment 8

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Is this across all boxes?  Doesn't matter if it is or not, but I'll want to know to figure out how much load I'm pushing.

Comment 9

[Jeremy Orem [:oremj]]

Those logs are from a single frontend node over 2 hours.