Get an 2 hours of logs between peak times (7am-12pm) for AMO

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
RESOLVED FIXED
9 years ago
3 years ago

People

(Reporter: davedash, Assigned: oremj)

Tracking

Details

Hi Jeremey,

Wil suggested that I talk to you about setting up and configuring the AMO test cluster mentioned in 504701 so we can replay log files.

I specifically want to monitor the boxes that will be running Sphinx's searchd.

I think if we can piggy back searchd and the indexers on two Mysql slaves this should be easy enough.

Ultimately I want to see that sphinxd doesn't keel over, and hopefully see less strain on mysql than usualy.

Cheers,
-d
Summary: Setup AMO test cluster for search load testing → Set up AMO test cluster for search load testing
This is going to be important for our search rewrite in our next milestone.  The timeline isn't set but it's going to happen in the next month or so.  Does that work for you?
Component: Public Pages → Server Operations
Product: addons.mozilla.org → mozilla.org
QA Contact: web-ui → mrz
Version: unspecified → other
(Assignee)

Comment 2

9 years ago
How big of a cluster do you need (app servers? db slaves?) Should this be on physical boxes?
We can use:

https://intranet.mozilla.org/Webdev:Test_Cluster

I think we could just do 1 master, 1 slave with sphinx/searchd and indexer on it.  1 app server.

Then we can just replay 1 single production app servers worth of logs into it.
(Assignee)

Comment 4

9 years ago
I think webdev runs that cluster now. Is this something you can set up? What do you need help with?
oremj,

Can I get a single days worth of access logs from a single frontend node?

I also heard you've got a log replay script... if you do I'll use it.

-d
(Assignee)

Comment 6

9 years ago
I do have a log replay script(http://code.google.com/p/logreplay/), but it isn't very well documented. You can also use siege(http://www.joedog.org/index/siege-home) which has better docs and also takes a file full of URLs.
Summary: Set up AMO test cluster for search load testing → Get an 2 hours of logs between peak times (7am-12pm) for AMO
(Assignee)

Comment 7

9 years ago
Dropped addons_logs.tar in khan:/home/ddash
Status: NEW → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED
Is this across all boxes?  Doesn't matter if it is or not, but I'll want to know to figure out how much load I'm pushing.
(Assignee)

Comment 9

9 years ago
Those logs are from a single frontend node over 2 hours.
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.