620146 - set up area for more storage of crash analysis experiments

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[chris hofmann]

I'm starting to use more space and cpu cycles at

http://people.mozilla.com/~chofmann/crash-stats/
and
http://people.mozilla.com/crash_stacks/

and could use even more to do some advanced analysis of crash data.

mrz and others have suggested putting this another vm somewhere.

It just needs easy access to /var/www/html/crash_analysis/ so I can get easy access to some of the data files there.

Comment 1

[Aravind Gottipati [:aravind]]

chofmann wants a physical box to run cpu intensive jobs - something with cpu similar to people (Intel(R) Xeon(TM) CPU 3.40GHz) would be good.  I can get him nfs space off cm-ixstore01, so space is not that big a deal.

Comment 2

[Aravind Gottipati [:aravind]]

@chofmann: What kind of connectivity will this box need to have?  

Does it need to be in a publicly visible location?  Or can it be on an internal network, that you can get to (after you vpn in)?

Comment 3

[chris hofmann]

(In reply to comment #2)
> @chofmann: What kind of connectivity will this box need to have?  
> 
> Does it need to be in a publicly visible location?  Or can it be on an internal
> network, that you can get to (after you vpn in)?

It needs to be a publically visable location.  The analysis is derived from data thats all visible on crash-stats and the public version of the crash summary .csv files.

Comment 5

[matthew zeier [:mrz]]

https://inventory.mozilla.org/systems/show/1587/

Comment 6

[Corey Shields [:cshields]]

(In reply to comment #5)
> https://inventory.mozilla.org/systems/show/1587/

Logged in to this system, it appears to be up and serving dxr.mozilla.org, we need inventory updated appropriately.

Jeremy?

Comment 7

[Aravind Gottipati [:aravind]]

@mrz and @cshields: I went though every single box on that chasis - but was unable to find MXQ95006BC.

@chofmann: This machine will be in Phoenix, so I will not be able to mount the nfs home directories from people onto this box.

Comment 8

[Aravind Gottipati [:aravind]]

(In reply to comment #7)
> @mrz and @cshields: I went though every single box on that chasis - but was
> unable to find MXQ95006BC.
> 

Nevermind, mrz tells me the serial is incorrect.

Comment 9

[Aravind Gottipati [:aravind]]

This last attempt to use that server was a bust.

@phong: need a 64 bit rhel6 server (a physical box) to run this stuff on.

Comment 10

[Phong Tran [:phong]]

We'll need to find another server.  Someone used that blade already and didn't rename it in inventory.

Comment 11

[Corey Shields [:cshields]]

Phong, use https://inventory.mozilla.org/systems/show/2815/

Comment 12

[Phong Tran [:phong]]

I have RHEL 6 installed.

netops: Can we get this these ports moved over to vlan74 and hand this back?

bsx-b09 eth0:Gi5/0/13 eth1:Gi6/0/13

Comment 13

[Ravi Pina [:ravi]]

Work for comment 12 is complete.

Comment 14

[Phong Tran [:phong]]

server is online.

Comment 15

[chris hofmann]

how do I get to it?

$ ssh chofmann@bsx-b09
ssh: Could not resolve hostname bsx-b09: nodename nor servname provided, or not known

Comment 16

[Justin Dow [:jabba]]

Sorry, I haven't had a chance to get to this yet. I think there are just a few tweaks and network routes left. I'll have Noah give it a look over to find out what is still needed and get it online and your user account on it.

Comment 17

[Noah Meyerhans [:noahm]]

This box is apparently not in puppet and not installed with any root password that I know...

Comment 18

[Phong Tran [:phong]]

(In reply to comment #15)
> how do I get to it?
> 
> $ ssh chofmann@bsx-b09
> ssh: Could not resolve hostname bsx-b09: nodename nor servname provided, or not
> known

That's the switch name and not the name of the host.  We are not done setting it up yet.

Comment 19

[Phong Tran [:phong]]

(In reply to comment #17)
> This box is apparently not in puppet and not installed with any root password
> that I know...

just puppetized it and your keys should be on there now.

Comment 20

[Noah Meyerhans [:noahm]]

Hi Chris. I'm a bit unsure about your answer in Comment 3. What actually needs to be publicly accessible on this host? Does the host actually need to run a publicly accessible service, or are you just concerned about publishing the output of your experiments on the web?  We were thinking that we could publish the results via a virtual host (maybe crashanalysis.mozilla.org). The web server itself wouldn't actually run on this host, but on existing web server hosts.  Input for your experiments would automatically be copied to your host, and output would automatically be copied to a web server.

Comment 21

[Justin Dow [:jabba]]

chofmann: See bug 636230 for clarification on how we want to achieve this. Will that work for you?

Comment 22

[chris hofmann]

>Does the host actually need to run a publicly accessible service

no

> or are you just concerned about publishing the output of your 
> experiments on the web?

yes.

the other reason for this bug is that some of the in depth analysis I'm running soaks up a lot of cpu cycles on people.m.o .   we wanted to isolate that processing on its own box so we can actually expand it to look at more data.

As long as I can:
- get a login to a beefy linux box to create and modify some shell scripts, 
- won't interrupt the work of others, 
- and have some place where its easy for some automated shell scripts to publish the reports I'm generating, 

then I'll be a happy camper.

a lot of the scripts use simple gzip's of data on people under /var/www/html/crash_analysis/

and they gather and scrape data from the web interface of socorro
( https://crash-stats.mozilla.com/ )

Comment 23

[Justin Dow [:jabba]]

details sent per e-mail, but tl;dr is: box is up and reachable, with chofmann's user account at crashanalysis.dmz.phx1.mozilla.com. I've moved the location of all the crash_analysis and crash_stacks files from people to an NFS mount on that box, that gets served by apache on the socorro webheads.

Comment 24

[chris hofmann]

also need to get crashanalysis.dmz.phx1://home/chofmann/public_html/crash-stats

visible on the web somewhere. 
suggest maybe something like https://crash-experiments.mozilla.com/chofmann

then we could put other folks under that root too if anyone else develop something they want to share.

Comment 25

[chris hofmann]

ok, jabba set up a better structure for reaching all the various reports under a single host.  everything is now under.

https://crash-anlysis.mozilla.com/

next set of things are that:

- mime type for .csv files needs to be set to text to allow easy reading of some of the reports.

- looks like performance of network requests between crashanalysis.dmz.phx1 and
crash-stats.mozilla.com is extreeeemely slow.  I need reach that server and have reasonable performance to do some of the analysis.  check ping, or do a simple wget of a page on crash-stats to see the problem.

Comment 26

[Justin Dow [:jabba]]

I think we've resolved all the outstanding issues.