896483 - l10n.mozilla-community.org was put offline, breaking many tools and workflows

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Pascal Chevrel:pascalc]

Bug 829298 put servers offline, one of them was the one hosting my tools (http://l10n.mozilla-community.org/webdashboard/) to track the work on many critical web parts for shipping Firefox key pages and Firefox OS.

Note that the fact that the server was hosting http://l10n.mozilla-community.org/ and that this was an important tool was well known (see https://bugzilla.mozilla.org/show_bug.cgi?id=829298#c7), a migration path was organized by the community but it seems that the server was shut down without any notice, I don't even know if Reed had finished rsyncing the data before the server was switched off...

Please make sure that all the data is backed up and restore the service asap.

Marking as critical given the number of web localizers (probably most of them) impacted. Other tools that were digging data from the Web dashboard via json/rss (such as the main l10n dashboard at l10n.mozilla.org) are also experiencing bugs and difficulties.

Thanks

Comment 1

[Shyam Mani [:fox2mike]]

Pascal,

I'll make sure this is turned back on, but if you notice the bug it's been open for ages and the due date is 3 days away. I apologise that this happened without any notice (I didn't expect that either) but we need to move out of there, quickly.

Comment 2

[Derek Moore [:dmoore]]

We're getting power restored to the facility immediately, but we'll need to move quickly as we have to vacate within a matter of days. I'll update again when power is restored.

Comment 3

[Derek Moore [:dmoore]]

Power is restored. SRE team will work on bringing the VM guests back online.

Comment 4

[Adrian J Fernandez [:Aj]]

As per irc, the l10n VM was back online at ~11:05 am PDT. The other vm konigseberg was online much earlier.

The issue with the l10n VM was with booting once the esx host was back online. Seems the new kernel did not rebuild initramfs which made the kernel not find /dev/sda1

We booted into a 2.6.x kernel (the other 3.x kernels were bad), automatic fsck took place, after fsck, rebooted into 2.6.x recovery mode and VM booted.