1074213 - Increase number of workers on dev

Status: RESOLVED INCOMPLETE

(Tree Management :: Treeherder: Infrastructure, defect, P3)

Description

[Mauro Doglio [:mdoglio]]

The log parsing queue on both dev and stage is increasing in size at a constant pace (about 1k per hour). At least for stage, we need to either add a worker machine or (better) create an environment like production. In that way we could have enough computing power to parse the logs.
If we decide to replace stage with a prod-like environment, we could be able to better test a deployment via chief.

Comment 2

[James Lal [:lightsofapollo]]

I think its worth mentioning that TC basically has "the scale of AWS" on spot for any input/output tasks (log parsing)... It may be more cost/time effective to run that on TC rather then manually provision machines since we are essentially designed for this busty load (and will continue get better over time cost wise).

At some point in the near future I would love to sit down and chat about how we can build a prototype and deploy that to -dev or staging.

Comment 3

[Ed Morley [:emorley]]

Now that the new stage VMs have been set up in bug 1079230, is there more work we need to do to get them up and running?

Comment 4

[Ed Morley [:emorley]]

In fact, let's handle this over in bug 1064438

Comment 5

[Ed Morley [:emorley]]

Let's make this bug just about dev, since stage is being switched to an IT managed instance with greater capacity in bug 1064438 and deps.

Comment 6

[Ed Morley [:emorley]]

We seem to be managing ok for now, and seeing as we're mainly using stage for the real-world-testing use cases, this seems redundant for now. We can reopen in the future if needs be.