860157 - Would like a hosted machine to handle some mercurial repository analysis

Status: RESOLVED WONTFIX

(Infrastructure & Operations :: Virtualization, task)

Description

[Dave Townsend [:mossop]]

I've been working on a project that seems to be useful to many developers in Mozilla for generating logs of changes to entire directories: https://www.oxymoronical.com/blog/2013/04/Get-notifications-about-changes-to-any-directory-in-mercurial

Right now I have a periodic update script running on my webhost but sometimes my webhost kills this for using too many resources (it's a mercurial pull plus some mysql database access essentially). When that happens everything falls over and I have to manually recover it which is irritating. I;d like to just get access to a hosted machine to do this instead of abusing my webhost.

My requirements are to be able to use python 2.7 so I can set up a python virtualenv to install django and mercurial. Plus some space for the repository clones and the ability to set cron jobs.

Comment 1

[Dave Townsend [:mossop]]

ping?

Comment 2

[Shyam Mani [:fox2mike]]

What sort of specs are you looking at?

Comment 3

[Shyam Mani [:fox2mike]]

Also, is this going to be publicly accessible?

Comment 4

[Dave Townsend [:mossop]]

(In reply to Shyam Mani [:fox2mike] from comment #2)
> What sort of specs are you looking at?

I don't have a great feel for this. A 2GHz CPU with 4GB RAM would probably be plenty for my needs, I don't even need X running on it.

(In reply to Shyam Mani [:fox2mike] from comment #3)
> Also, is this going to be publicly accessible?

It doesn't need to be, it just needs to be able to access the web since it will be pulling from hg.mozilla.org and pushing data into a remote mysql database.

Comment 5

[Shyam Mani [:fox2mike]]

Opsec had some comments here, I'll wait for them to respond.

Comment 6

[Joe Stevensen [:joe]]

(In reply to Dave Townsend (:Mossop) from comment #4)
> It doesn't need to be, it just needs to be able to access the web since it
> will be pulling from hg.mozilla.org and pushing data into a remote mysql
> database.

Dave, where are you pushing data to?

Comment 7

[Dave Townsend [:mossop]]

(In reply to Joe Stevensen [:joes] from comment #6)
> (In reply to Dave Townsend (:Mossop) from comment #4)
> > It doesn't need to be, it just needs to be able to access the web since it
> > will be pulling from hg.mozilla.org and pushing data into a remote mysql
> > database.
> 
> Dave, where are you pushing data to?

A mysql database hosted at Dreamhost.

Comment 8

[Shyam Mani [:fox2mike]]

Opsec has no concerns.

Virtualization team, this will be a 2 CPU, 4 GB, standard disk node in scl3. Ack?

Comment 9

[Greg Cox [:gcox]]

The CPU, diskspace, and RAM seems acceptable.  The "webhost kills this for using too many resources" from comment 0 gave me pause.  What resources was it using up/binding on?

Our main concern is around disk IO.  Even if it's not a lot of space in use, with this pulling from hg, massaging, and pushing back out, I could see this getting very thrashy (since hg's already a high-turnover filespace).

What frequency of updates-kicking-off would this be doing?
How much data gets shoveled around on each update?
What kind of duty-cycle would each run have (e.g. it kicks off every 4 hours, but takes 2 minutes to run).
Are there any config options in the app that can be set to essentially 'ionice' it?

Comment 10

[Dave Townsend [:mossop]]

(In reply to Greg Cox [:gcox] from comment #9)
> The CPU, diskspace, and RAM seems acceptable.  The "webhost kills this for
> using too many resources" from comment 0 gave me pause.  What resources was
> it using up/binding on?

I only have shared hosting with my webhost so they're pretty jumpy at killing processes. I don't know for sure what it was doing with disk IO but I suspect it was more memory and CPU usage that caused the problems.

> Our main concern is around disk IO.  Even if it's not a lot of space in use,
> with this pulling from hg, massaging, and pushing back out, I could see this
> getting very thrashy (since hg's already a high-turnover filespace).

> What frequency of updates-kicking-off would this be doing?
> How much data gets shoveled around on each update?
> What kind of duty-cycle would each run have (e.g. it kicks off every 4
> hours, but takes 2 minutes to run).
> Are there any config options in the app that can be set to essentially
> 'ionice' it?

Basically the code runs as a python script, so if there are things we can do to python then that's easy enough. I already nice the process to try to make it play fair.

Right now it runs every 10 minutes. If no changes have landed to the repositories then it takes less than 10 seconds to complete. For a lot of changes it may take longer, likely into a few minutes for a big merge from inbound for example. There are two phases to this, first it pulls the new updates into the repository, this is just a regular hg pull, then it analyses the changes and sends a summary of them to the remote mysql server, no data is written to disk during that part, it should just be file reads from the hg repository.

I don't know how to quantify the amounts of data here, is there some way I could measure that?

Comment 11

[Greg Cox [:gcox]]

`sar -d` and `iotop` are decent starts.  The former telling you sectors written per second (which can get you a good long-term trend) and eyeballing the latter gives more instantaneous/highwater marks on data throughput.

Comment 12

[Dave Townsend [:mossop]]

(In reply to Greg Cox [:gcox] [OOO, back 2013-09-12] from comment #11)
> `sar -d` and `iotop` are decent starts.  The former telling you sectors
> written per second (which can get you a good long-term trend) and eyeballing
> the latter gives more instantaneous/highwater marks on data throughput.

Unfortunately neither of these work on the dreamhost server I use as they require kernel features that aren't enabled.

Comment 13

[Shyam Mani [:fox2mike]]

This has kind of stalled :| Dave, how can we help you move forward with this?

Comment 14

[Dave Townsend [:mossop]]

Well I'm a bit stumped right now. You (understandably) want to understand the performance costs of this work but I don't know how to quantify that and the only tools you've thought of so far don't work in the current environment it runs in. I did do a bunch of work to try to get this up and running in PaaS but unfortunately the database size restrictions there made it difficult.

Is there a quick and easy way to spin up a VM that I could play with to get those numbers?

Comment 15

[Shyam Mani [:fox2mike]]

Meh, slipped through the cracks. The VM folks should be able to spin up something you need.

Comment 16

[Chris Knowles [:cknowles]]

spoke with gcox about this.  We're happy to provide a RHEL6, 2CPU 4G VM.

Only question that I can see remaining - what would you like the hostname to be?  XX.private.scl3.mozilla.com 

CJK

Comment 17

[Chris Knowles [:cknowles]]

Just checking in, can you provide the hostname that you'd like?

XX.private.scl3.mozilla.com ??

Once that's provided we can get started on prepping out that VM.

Comment 18

[Dave Townsend [:mossop]]

At this point changes I made making the system more resumable have meant I probably don't need this anymore.