1079114 - virtualize mxr-web1.webapp.scl3 and mxr-processor1.private.scl3

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[Greg Cox [:gcox]]

mxr-web1.webapp.scl3 and mxr-processor1.private.scl3: We'd like to p2v these seamicros.
The web node looks to need about 1 vCPU and 2G vRAM.
The processor looks to need about 2 vCPU and 8G vRAM.

I heard a rumor that mxr may be going away; the warranty deadline is around Feb 2015, and we'd like to knock things out in Q4 if possible.  If it'll be longer, let's p2v; if shorter, this can just be a noop.

Comment 1

[Ed Morley [:emorley]]

(In reply to Greg Cox [:gcox] (plz don't needinfo me) from comment #0)
> I heard a rumor that mxr may be going away

That's the long term plan from what I've heard, though from an end user perspective there are many issues that need fixing in terms of DXR reaching parity with MXR [1], that I imagine it will be several quarters more work - but you'll need to ask the DXR team (I couldn't find a meta bug for MXR eol).

[1] eg indexing the 30+ repos available on MXR, when there are only 2 currently on DXR, and DXRs entire model is based around compilation, and many of those 30 repos won't be compilable by DXR, so it's going to have to support normal indexing too.

Comment 2

[Kendall Libby [:fubar] (he/him)]

:edmorley, dxr does support text indexing; we're currently doing that for a few repos under hg.m.o/build/.

there is currently no time frame on mxr going away, and almost certainly not before 2/2015.

Comment 3

[Ed Morley [:emorley]]

(In reply to Kendall Libby [:fubar] from comment #2)
> :edmorley, dxr does support text indexing; we're currently doing that for a
> few repos under hg.m.o/build/.

Ah great :-)

Comment 4

[Greg Cox [:gcox]]

OK, taking decom off the table since we're looking at the warranty timing as the blocker.  Let us know about timeframes for p2v when you have a chance, please and thank you.

Comment 5

[Greg Cox [:gcox]]

Courtesy reminder that warranty is running out 2015-02-01.  We can p2v at your leisure.

Comment 6

[Kendall Libby [:fubar] (he/him)]

meh. what's the total downtime window for cutting over? if it's on the order of a couple minutes, doing it early EST is probably fine for the web head. the processor can go whenever; have fun with all those little files. ;-)

Comment 7

[Greg Cox [:gcox]]

Our standard quote is "1 hour".  On good days it can be as fast as 30mins, but it's hard to come in faster than that.

If you want us to do an early EST on the web with that, we certainly can, just let us know how early, when, and if there's any special prep work (maybe "shut down apache first"?).

The processor one looks easy, we can do that.  If you're serious about "whenever" we'll throw that on the pile, too, probably later this week.

Comment 8

[Kendall Libby [:fubar] (he/him)]

Processor is totally "whenever." It's just pulling repos and running glimpse on them, on an NFS mount.

Will need to look at the web head logs and see how busy it is on EST mornings and get back to you.

Comment 9

[Greg Cox [:gcox]]

mxr-processor1.private.scl3 p2v'ed this morning.  After looking at the usage pattern and the crons, I gave it 4 cores and 16G.  That's a net loss of 4 cores and no ram shrink, as it was closer to going into swap than I remember seeing before.

Whoever wrote /etc/cron.d/mxr.mozilla.org and included "parallel -j+0" in those jobs, I want to buy you a drink.
Nightly and weekly processing may take longer to pump those threads through, yell if there's a problem.

Comment 10

[Kendall Libby [:fubar] (he/him)]

Traffic-wise, the web head is pretty popular. The low points are midnight EST (wat) and weekends. I don't think you guys are night owls, and I doubt you have other weekend work planned (the next TCW isn't until Mar 7, afaik).

Of the work days, it looks like Friday mornings tend to be lightest, so if you want to roll this Friday, 8AM EST ish, just hardhat it in zeus, ping me in #vcs and make it happen.

Comment 11

[Greg Cox [:gcox]]

Per consulting with :fubar in IRC, took the midnight train going anywhere^w^w^wwindow and p2v'ed with hardhat.  Back up at around 0030 EST.

RAM usage looked much worse than when I looked a few months back, so I shrunk the CPU but left the RAM at 16G since it was using most of that, even with an allowance for caching.

Reopen if you see anything amiss.