Closed Bug 622263 Opened 15 years ago Closed 15 years ago

rename and move bm-l10n-dashboard01, bm-l10n-db out of build.m.o

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Type:
task
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: Pike, Assigned: zandr)

References

Details

I can't ssh into bm-l10n-dashboard01 or bm-l10n-db right now. I have the MPT tunnel open. wokbok:buildbotcustom axelhecht$ ssh root@bm-l10n-dashboard01 ssh: connect to host bm-l10n-dashboard01 port 22: Operation timed out wokbok:buildbotcustom axelhecht$ traceroute bm-l10n-dashboard01 traceroute to bm-l10n-dashboard01.mozilla.org (10.2.90.10), 64 hops max, 52 byte packets 1 10.2.21.1 (10.2.21.1) 213.563 ms 211.721 ms 211.773 ms 2 * * * 3 * * * ^C wokbok:buildbotcustom axelhecht$ host bm-l10n-dashboard01 bm-l10n-dashboard01.mozilla.org has address 10.2.90.10
Severity: normal → major
Seems to be working fine for me from IT jumphost. [aravind@boris ~]$ ssh -A root@bm-l10n-dashboard01 ls awstats-6.95.zip conf glusterfs [aravind@boris ~]$ @Axel: Did this work from cm-vpn01 in the past? or are you trying this from some other jumphost?
Assignee: server-ops → aravind
I'm going through 63.245.208.142 aka mpt-vpn.mozilla.com, and that used to work. I'd have one suspect, did those vm fall behind the buildvpn? O'Duinn doesn't want folks like me to have access to that, so it shouldn't, if possible.
The box is in the build network. Has this changed recently? You say that you were able to access this box in the past and per this bug, this is no longer the case. If all that is correct, my guess is that some network rules were changed to limit access to these boxes. Copying network folks so they can list any network rules that might have changed and caused this.
Yes, I was able to access the server just a few days ago, through the MPT tunnel.
This is as expected. See bug 602741 for details. Coping John.
FWIW, I'm not sure why bm-l10n-* are in the build realm, but I wouldn't call the l10n team not having access "expected". All of me, sethb, gandalf, stas need access to those machines. This set of people will grow larger as soon as we hire webdev resources for the l10n dashboard.
To clarify the impact here, on the next failure on the infrastructure, we'll stop taking any new updates on any localization of any mozilla product. This is to be expected to happen any day, as our japanese contributor rather reliable breaks the varnish cache, i.e., his commits are not accessible through http. https://bugzil.la/ALL%20reporter:l10n@mozilla.com%20[varnish has examples of the manual interaction that requires. For the record, the initial VM request is in bug 512754, the WONTFIXed bug for access to build-vpn is bug 600916.
Depends on: 512754, 600916
Rolling back the firewall change to log for a bit longer. See bug 602741 for details.
(In reply to comment #5) > This is as expected. See bug 602741 for details. Coping John. Axel: this is the build-vpn change I discussed with you before+during the allhands in December. I thought you said you were ok with this change, as your systems were only relying on the json file. Assuming these 2 machines are only using the json file, is the problem here that bm-l10n-dashboard01 and bm-l10n-db machines were incorrectly in the build network?
(In reply to comment #9) > Assuming these 2 machines are only using the json file, is the problem here > that bm-l10n-dashboard01 and bm-l10n-db machines were incorrectly in the build > network? If the solution is to move these machines, we'll need to keep logging after they've moved to see if they have any other dependencies on the build network.
I can say this much: None of our systems need access to internals of the releng automation at this point. The l10n team needs access to the VMs in SJ. The l10n team needs access to the l10n community server VM in AMS, where the scraper of the build json sits, which is totally independent of the stuff on the SJ VMs. Really, the two aren't even remotely related. The AMS setup used to benefit from offering links to the real buildbot output for individual builds, but that data is clobbered now, and thus that benefit is void. I am currently taking notes of a few lessons that we got from the current SJ setup, which may yield a new setup in the not too distant future. Which is to say, I'd rather not take risks on our ability to ship Fx4 updates whenever in order to get to a clean and properly designed solution. Right now, I'm thinking about what data we should be collecting, and if to add more slave power to the mix, etc. It may very well be a lesson-learned from this bug that our security setup of that system should get some thought, given that it might turn out to be a blocker to ship. I can't comment on "implementation details" like where a VM gets hardware resources from, and what that means to network setup. I lack any education on what anything I'd say here means practically.
Summary of irc discussion with zandr, mrz, pike, joduinn. 1) these machines are being switched out of the build.m.o network. These VMs will continue to live in MPT, but will be assigned new IPs and new hostnames. Zandr will advise once he verifies the new network location. 2) This requires stop/restart of these two machines, which will require a brief downtime. This also requires config changes in daemons running on these machines. This does *not* require a tree closure. 3) This requires a matching change in zeus.
Summary: SSH times out when trying to connect to bm-l10n-dashboard01 and bm-l10n-db → rename and move bm-l10n-dashboard01, bm-l10n-db out of build.m.o
Assignee: aravind → zandr
Axel: Can you clarify who needs access to this machine and on what ports? Is everyone who is accessing this coming in through MPT-VPN, or is there public access? If so, what is the public hostname? (I'm new to this part of the infra, but I'm not finding any Zeus interfaces on 10.2.90, so I'm a bit confused)
I don't know jack about Zeus, Jeremy set that up in bug 540714. Anyone who has access to the VMs is coming through MPT-VPN. We need ssh access to both machines, mysql to bm-l10n-db, and http[s] to bm-l10n-dashboard01. FWIW, bm-l10n-dashboard01 needs http access to hg.m.o. Oh, also the ganglia setup needs to be adjusted, http://im-ganglia01/ganglia/?c=Localization&m=load_one&r=day&s=descending&hc=4&mc=2 (bug 573442). FWIW, I don't think that my head is the right place to document the setup here. I'm happy to dig up bugs, but I don't understand what's what in those bugs. This feels more brittle to me than it should.
(In reply to comment #14) > I don't know jack about Zeus, Jeremy set that up in bug 540714. > > Anyone who has access to the VMs is coming through MPT-VPN. > > We need ssh access to both machines, mysql to bm-l10n-db, and http[s] to > bm-l10n-dashboard01. FWIW, bm-l10n-dashboard01 needs http access to hg.m.o. > > Oh, also the ganglia setup needs to be adjusted, > http://im-ganglia01/ganglia/?c=Localization&m=load_one&r=day&s=descending&hc=4&mc=2 > (bug 573442). > > FWIW, I don't think that my head is the right place to document the setup here. > I'm happy to dig up bugs, but I don't understand what's what in those bugs. > This feels more brittle to me than it should. axel, zandr: Its been a week since the last update - any progress?
Axel: I need to know either: a) how do I reconfigure -dashboard01 and -db to know about each other if the IP addresses change, or b) who should I be working with to coordinate this move?
a) The infrastructure on dashboard01 references db by name. To prepare the move, I'd need to know the new name of db. b) with me.
(In reply to comment #17) > a) The infrastructure on dashboard01 references db by name. To prepare the > move, I'd need to know the new name of db. As a hostname? If that's the case, then no reconfig of the app will be necessary, since I can leave a CNAME behind. > b) with me. OK, let me get some other ducks in a row and we can orchestrate a brief downtime to do this network change.
(In reply to comment #18) > (In reply to comment #17) > > a) The infrastructure on dashboard01 references db by name. To prepare the > > move, I'd need to know the new name of db. > > As a hostname? If that's the case, then no reconfig of the app will be > necessary, since I can leave a CNAME behind. Axel would a CNAME work for you and mean you could avoid a reconfig? > > b) with me. > > OK, let me get some other ducks in a row and we can orchestrate a brief > downtime to do this network change. cool! Define "brief downtime". Approx how long? What type of downtime do you need? * a full downtime for all trees * just l10n trees because localizers would not be able to checkin? * no downtime, because localizers could still checkin, but just briefly not see dashboard status of next l10n builds?
(In reply to comment #19) > Define "brief downtime". Approx how long? What type of downtime do you need? > * a full downtime for all trees > * just l10n trees because localizers would not be able to checkin? > * no downtime, because localizers could still checkin, but just briefly not see > dashboard status of next l10n builds? On the order of 10 minutes. (Therefore plan an hour.) What I'll need to do is shut down those two VMs and reconfigure them for a new VLAN. I do not know the operational impact of downing those VMs.
There's no releng-tree downtime for this. The downtime is to be coordinated with me, as I need to shut down and start up services in proper order. A CNAME is a technical detail that I can't evaluate, as that doesn't say anything to me in terms of what that means in practice.
(In reply to comment #21) > A CNAME is a technical detail that I can't evaluate, as that doesn't say > anything to me in terms of what that means in practice. OK, I don't appear to have access to this host to investigate, so... Please paste the config line that bm-l10n-dashboard01 uses to find bm-l10n-db (and vice-versa if it exists)
(In reply to comment #22) > (In reply to comment #21) > > > A CNAME is a technical detail that I can't evaluate, as that doesn't say > > anything to me in terms of what that means in practice. > > OK, I don't appear to have access to this host to investigate, so... > > Please paste the config line that bm-l10n-dashboard01 uses to find bm-l10n-db > (and vice-versa if it exists) Axel: can you give zandr the data he needs, or maybe give him temp access to the machines to self-serve for a few days? To minimize disruption, I'm hoping we can get this done a day/two after the "go to build beta10" and before beta11 load ramps up.
There are a host of IT folks that have ssh access anyway, so adding zandr doesn't make a big difference. Attach your public key here? I'll be traveling Sun-Mon, fwiw, crossing my fingers that it's just that.
There has been conversation directly in email, in which I believe I have addressed all the issues outstanding. Waiting for Axel to get a window scheduled for the move.
Depends on: 615584
Zandr, can you make an hour on Wednesday morning? Seems like that would work out with B12. If you give me a timewindow, I can announce it my time Tuesday.
Sorry for the delay getting back to you. If you'd still like to go ahead tomorrow AM, that's fine, or we could slip to Thursday. Wednesday AM is clear except 1000-1100 PST. Thursday AM is completely clear. I'll announce this as a normal IT downtime, and leave it to you to announce to your users.
That's 1000 PST, the one hour I *don't* have on Wednesday.
We hit networking issues this morning that fouled up this migration. I have confirmed that these issues are now sorted, and we'll be ready to go again when a window can be scheduled (this probably means next week/post b12). 1hr window again.
Can we get a bit of detail on what actually happened? The downtime was estimated to be 30mins, scheduled was 1h, de-facto it was 2:30. This bug is competing with fx4-shipping-relevant interactions with our l10n community, so we need to be really cautious.
(In reply to comment #32) > Can we get a bit of detail on what actually happened? 1) The new vlans were not actually available on the build ESX host. 2) The proposed solution was to move to a more appropriate ESX host. 3) There were firewall rules in place to prevent transferring the image files to the second ESX host. 4) The resources beyond myself to deal with these issues did not share my sense of urgency or were unavailable entirely once things started to go sideways. I believe we have now cleared the hurdles necessary to move these hosts, and many of the changes to support the switch are already in place. We also did complete the disk expansion in this downtime. So, we're ready to try again when I can get Axel and Phong available at the same time. At that point, the process is: 1) Shut down the VMs. 2) Replace the bm-l10n DNS A records with CNAMEs to the new names 3) Copy the VMs to the new cluster. 4) Restart the VMs.
(In reply to comment #32) > Can we get a bit of detail on what actually happened? > > The downtime was estimated to be 30mins, scheduled was 1h, de-facto it was > 2:30. :-( > This bug is competing with fx4-shipping-relevant interactions with our l10n > community, so we need to be really cautious. Yep. Axel: After talking with zandr, some questions: 1) when to try this again? (If it helps, we're looking at "go to build beta12" likely Monday - is it better to try this again before the weekend, or else after beta12 is built and handed to QA.) 2) who needs to be there for this? zandr again, obviously. Phong for VM move? Axel (or someone else in l10n) for dashboard stop/start?
(In reply to comment #34) > (In reply to comment #32) > > Can we get a bit of detail on what actually happened? > > > > The downtime was estimated to be 30mins, scheduled was 1h, de-facto it was > > 2:30. > > :-( > > > > This bug is competing with fx4-shipping-relevant interactions with our l10n > > community, so we need to be really cautious. > Yep. > > > Axel: > > After talking with zandr, some questions: > 1) when to try this again? (If it helps, we're looking at "go to build beta12" > likely Monday - is it better to try this again before the weekend, or else > after beta12 is built and handed to QA.) Axel, Zandr: Can we try this again during the upcoming Thurs morning PST downtime for linux resolution change? I know this dashboard move does not actually need a full tree closure, but if we have one happening anyway, it might make it easier? > 2) who needs to be there for this? zandr again, obviously. Phong for VM move? > Axel (or someone else in l10n) for dashboard stop/start?
I don't want that downtime to run into various IT folks having better things to do again.
Adding Phong, since I'll need him around to migrate the VM. Phong- What works for you? Can we try another 9am move?
... also, there's no ETA for b12 around, let's not confuse things.
Axel says not until b12 ships. I will arrange a time as soon as that happens.
To clarify, b12 *shipping* is not the right milestone here. My understanding is that it's the b12 builds starting that opens our window to do work here. We should plan to make this change as soon as the b12 builds are underway.
Per conversation on irc, we'll make the next downtime tomorrow, Thursday, feb 24, 9-10am PST. Plan of action is still in comment 33, AFAIK.
Downtime took a good deal longer again, with a few extra bumps. Filed bug 636721 on getting the ganglia coverage back up. Zandr, did you confirm that the backups complete and are useful? The other thing, at the start of this bug, I received a bunch of questions that I was unable to answer. Can we get the new setup documented, on the intranet or somewhere, so that next time we know what we're dealing with? Resolving FIXED, as I guess it is.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
Blocks: 637606
Depends on: 657863
No longer depends on: 657863
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.