692101 - l10n.mozilla.org server is down

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Pascal Chevrel (relman team) -> :pascalc]

All sites on l10n.mozilla.org are not available, I can't connect via ssh either, I get this error message:

ssh: connect to host l10n.mozilla.org port 22: Connection timed out

This is severely impacting my and localizers work.

Thanks for getting the server back up.

Comment 1

[Shyam Mani [:fox2mike]]

This is not a server IT manages. Our keys aren't even on the box.

Comment 2

[Axel Hecht [:Pike]]

This is a VM and nothing responds, including ssh. So folks with admin rights in that VM can't do anything. I do hope that IT's keys are on the box that hosts the VM, so moving back to IT to figure out how to get that VM back to talk to the world.

Comment 3

[Corey Shields [:cshields]]

No, nobody who works in IT today has keys on this box, and it is not a part of our centralized management to ensure that keys are up to date.  We had issues with this last week when Arky needed something changed and could not find the right people who had root.

We can reset the VM but that's about it.

Comment 4

[Shyam Mani [:fox2mike]]

And I've already mentioned this to Reed before, that if there's a situation and IT needs to help out then we need to have access on the machine but he didn't seem receptive to that idea.

Comment 5

[Corey Shields [:cshields]]

(In reply to Shyam Mani [:fox2mike] from comment #4)
> And I've already mentioned this to Reed before, that if there's a situation
> and IT needs to help out then we need to have access on the machine but he
> didn't seem receptive to that idea.

This is going to change..  Pascal, do you have root access to this vm?  As soon as it is back up we need you to add our keys to this VM.

thanks!

Comment 6

[Zibi Braniecki [:zbraniecki][:gandalf]]

I think there were some IT changes. The server works properly, but the access chain fails.

In my case:
zbraniecki@Zbigniews-MacBook-Air:~$ ping l10n.mozilla.org
PING l10n.mozilla-community.org (63.245.212.100): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
Request timeout for icmp_seq 4

If I log in to the machine and wget it's 127.0.0.1 it works properly so it's probably something about moving l10n.mozilla.org to l10n.mozilla-community.org

Comment 7

[Pascal Chevrel (relman team) -> :pascalc]

I don't have root access

Comment 8

[Axel Hecht [:Pike]]

Let's keep this bug to what it is, please. I'm happy to discuss changes to how we service that VM, but please get that into a different bug/discussion.

Can you please give the VM a boot, as it's not working properly? No idea how gandalf gets there.

Comment 9

[Reed Loden [:reed]]

Let me know if you need any assistance here with the box... Since the box is completely inaccessible, I really can't do anything on my end. Need somebody to bounce from VMware console, though if you see anything weird on the console, that would be useful to know.

Comment 10

[Jake Maul [:jakem]]

It seems we've actually lost connectivity to the whole VMware server now... this was not the case on Fri/Sat when I was able to console it, and there was nothing interesting on the console... just the login prompt.

I'm working on this now.

Comment 11

[Jake Maul [:jakem]]

This should be resolved... we ultimately had to kick the whole vmware server this resides on. Please let us know if it's not.