1287248 - Community websites hosted on OVH servers suspended for lack of payment

Status: RESOLVED FIXED

(Participation Infrastructure :: MCWS, task)

Description

[Francesco Lodolo [:flod]]

I'm used to receive notifications for services expiring on OVH, imagine my surprise this morning when I discovered that our VPS has been suspended (the website is unreachable right now) because the bill hasn't been paid.

I don't think is acceptable for something like this to happen. Either we fix this, or I'm going to pay out of my own pocket and host the website somewhere else.

I've just paid the bill myself for this month, hoping to see the website coming back online asap.

Comment 1

[Francesco Lodolo [:flod]]

@Tom
I'm not sure if you're the right contact, if not please direct the question to the right person.

Comment 2

[Francesco Lodolo [:flod]]

The l10n community server is down as well, and I can't help noticing that the IP is assigned to OVH.
https://l10n.mozilla-community.org/

Is there any relation?

Comment 3

[Francesco Lodolo [:flod]]

babelzilla.org is down as well

moz.fr was down and they redirected to a different server

(bug 1287273) https://l10n.mozilla-community.org/ is down, but I don't know if it's related (it would be a strange coincidence)

I assume there are more.

If the l10n community server is going to be wiped clean (the message I got talks about 5 days, and it's from 12 hours ago) it's going to take several hours of work to restore all services hosted there.

Who can find someone to fix this?

Comment 5

[Rubén Martín [:Nukeador] ❌ [away till Aug 31st]]

I've alerted Brian, Pierros and team, we are on it.

Comment 7

[Stefan Plewako [:stef]]

Something similar for *.aviary.pl, firefox.pl and for planet.mozillapl.org, suspended ~ 07:42 UTC, reactivated ~ 17:36 UTC and not for the first time (previously 02.08.2015, ~ 12 hours) without any explanation.

Comment 8

[Francesco Lodolo [:flod]]

Looks like all accounts are back up, as far as I can tell.

Not closing the bug, since this procedure *needs* a post-mortem (and changes). And thanks to all those I bothered on a Summer Saturday afternoon to get this fixed.

Comment 9

[tad]

Massive thank you to Kensie, Tanner, Brian, and Pierros for handling these renewals.

I was completely unable to handle this, but they stepped in to co-ordinate making this work.

As for the post-mortem, I'd leave it as this:

- Lack of documentation on how to handle a renewal
- Monthly, manual renewal process
- Volunteers responsible for co-ordinating this renewal, and both of them missed it. Brian should have been on the renewal list and had documentation on how to process the renewal, but this was not the case.
- After suspensions were issued, I was (working) at an event and thus couldn't follow up, other than checking that someone else could step in (which they did)

Overall, this is a problem that shouldn't have happened. We'd ideally be on the new infrastructure by now, since that would eliminate this kind of problem (these renewals are supposed to be a temporary measure). Regardless of that though, this needs to not happen again.

Apologies on behalf of myself and MCWS for the difficulties caused, and again, thank you to those who took time out of their day to co-ordinate resolving this.

For next steps to prevent this from happening again, I'd suggest this:
- Create a monthly calendar event with all relevant people with authority over billing (myself, Brian, Majken) to remind them in advance (emails look the same as the other hundreds of emails OVH send me from my inbox view)
- Write a quick documentation on how to create a renewal invoice, so that anybody (including himself) can create an invoice for Brian to handle
- Add Brian to the email group that receives renewal notifications

I've already started action on adding Brian to that group - a request has been made to Mozilla's Google Apps admin.

Please do let me know if people have other ideas on ways to action preventing this.

Comment 10

[Francesco Lodolo [:flod]]

I completely understand why the renewal has to be monthly (we want to move away, sooner than later, no point in paying for longer periods). Having said that, why does it have to be manually and not automated?

Comment 11

[Firefox Bug Husbandry Bot]

Bulk move of bugs