528381 - Socorro Staging improvements

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Austin King [:ozten]]

Currently our stage server is in much better shape that it's ever been... but we can tweak a few things to make it better.

1) Volume of data
There is very little recent data in stage.
Comparing /topcrasher/byversion/Firefox/3.5.3 on stage and prod the #1 crash has
56 crashes in stage and 4,759 in prod

How can we get more data? Do we need better hardware?

2) Type of crashes
Currently we push deferred crashes into stage for processing.
A side effect is that there is no 'developer' releases, no crashes with comments. We want  a data set that is similar to production...
Devs can manually submit jobs to stage, but is there an automated way to tweak this?

Comment 1

[Dave Miller [:justdave]]

is this for crash-stats?  It doesn't actually say what site this is for anywhere here.

Comment 2

[Aravind Gottipati [:aravind]]

Yeah, this is for the breakpad staging env.

Comment 3

[Austin King [:ozten]]

Testing this release has been difficult on stage due to the limited number size of the database. We're enhancing our trend reports, but there isn't enough data for interesting aggregate numbers to trend on.

We have dm-breadpad-stage, which I did most of the testing (on my dev env) since it has a snapshot of production. This is good, but it isn't the same grade of hardware, so we can't make accurate performance tests. This means when we hit a slow query, we'll have to test and optimize it in production (if that is even possible). This slows down IT and development.

We either need to improve how we populate stage or we need another set of production grade hardware for our staging environment.

Comment 4

[Austin King [:ozten]]

Some crons and configs were out of data / missing in stage. We need stage to be exactly like production so that we can do integration testing between components.

Comment 5

[Aravind Gottipati [:aravind]]

I have got production dumps (from the active pool) feeding into stage now.  Please take a look at the environment now and let me know if you guys think it needs any more changes.

Comment 6

[matthew zeier [:mrz]]

(need to move off VM to physical hardware)

Comment 7

[Austin King [:ozten]]

raw_adu tables in stage aren't populated.

Comment 8

[matthew zeier [:mrz]]

Next steps - 
- rebuild old ESX DL385, 64bit RHEL5
- Vlan81
- breakpad-stage01

Comment 9

[matthew zeier [:mrz]]

I put this on vlan74 like the other breakpad boxes.

10.2.74.61 / dm-breakpad-stage01

Comment 10

[Aravind Gottipati [:aravind]]

Staging (the monitor and processor) is now running on a super powerful box with crazy ram and more cpus than it has use for!  thanks for all the help mrz!

Comment 11

[Austin King [:ozten]]

Awesome!