1230266 - Decommission badges.mozilla.org from WebOps (leaving AWS alone)

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Eric Ziegenhorn :ericz]

badges.mozilla.org uses Django and therefore needs to be migrated to the Python cluster so we can upgrade Django for security reasons.

Comment 1

[Shyam Mani [:fox2mike]]

(In reply to Eric Ziegenhorn :ericz from comment #0)
> badges.mozilla.org uses Django and therefore needs to be migrated to the
> Python cluster so we can upgrade Django for security reasons.

Need to figure out who're the owners here. I know the site is used but I don't want to take on "ownership" by migrating this to the python cluster.

According to https://github.com/mozilla/badges.mozilla.org, this isn't maintained on the dev side anymore. 

Switching this to a decom bug, let's get security involved and see what the next steps are.

Comment 2

[Shyam Mani [:fox2mike]]

April/Jeff - the GH repo for this site says it's not being actively maintained. I don't think the site should continue to remain online if it's not actively maintained. How can we move forward on this? 

CC'ing Gene here, since this is backed by persona as well.

Comment 3

[Gene Wood [:gene]]

Will, Les,
   Can you help :fox2mike determine who owns the badges.mozilla.org web property as he's looking for clearance to decompression it?

Comment 4

[Gene Wood [:gene]]

decommission it I mean (no need to decompress it) =)

Comment 5

[Les Orchard [:lorchard]]

As far as I understand, no one owns these days. I gave it over to the badges team in MoFo a few years ago, but I think that team is no more. That said, I'm CC'ing some folks who are talking about doing something with it right now.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

I think Les covered everything I know.

Comment 7

[Shyam Mani [:fox2mike]]

(In reply to Gene Wood [:gene] from comment #4)
> decommission it I mean (no need to decompress it) =)

Gene, we'll make sure to not decompress ;)

From a security standpoint - What's our way forward with badges in case no one speaks up?

Comment 8

[George Roter]

Hi All,

We've started putting together a plan on this here:
https://docs.google.com/document/d/17efYpvm4Khhx4YACIVQKBcfV07hcZOBxrdSguYG6nag/edit

It's not detailed and we haven't run it by people who need to be informed.

I need to free up some resources to deal with this in the weeks ahead. Can we have a quick check-in on this at the end of June?

Copying Pierros on this bug so it's in our queue to talk about. Also Arielle and Hermina who can help manage.

George

Comment 9

[Gene Wood [:gene]]

:fox2mike,
  My interest in the topic is related to the fact that with Persona going away Nov 30, badges will stop working at that point without a new auth system. I just want to make sure it's either decommed before then or has a plan for updating the auth system.

Comment 10

[Shyam Mani [:fox2mike]]

(In reply to groter from comment #8)
> Hi All,
> 
> We've started putting together a plan on this here:
> https://docs.google.com/document/d/
> 17efYpvm4Khhx4YACIVQKBcfV07hcZOBxrdSguYG6nag/edit
> 
> It's not detailed and we haven't run it by people who need to be informed.
> 
> I need to free up some resources to deal with this in the weeks ahead. Can
> we have a quick check-in on this at the end of June?

End of June check-in :) Thanks for taking some of this on, George.

Comment 11

[George Roter]

Hi Shyam,

I just booked a day of Nikos' time in the 3rd week of August to take care of this.

We will:
1) Turn the site into static content -- turn off all functions to grant or add new badges, login, etc.

2) Clearly communicate when the site will be shutdown for good (including all static content) -- aiming for Jan 2017.

3) Have a plan for backing up all of the content when it is fully shut down.

Cheers,
George

Comment 12

[April King [:April]]

Note that the site is nicely backed up in the Internet Archive, so we could always redirect there.

Comment 13

[Nikos Roussos [:comzeradd]]

I had a look at the website and the options we have.

My suggestion is this:
1. Create a static mirror of the website.
2. Upload it on AWS (probably on an S3 bucket)
3. Amend the front page with a notice that informs about the site being read-only
4. Part of that notice should also be timeline for a full decommission of the website/domain (eg. in 6 months)

Things to consider:
1. Some pages return 503 erros at the moment.
2. Some views are rendered dynamically from the API.
Bottom line: we may not be able to get everything.

Comment 14

[George Roter]

Great. What I'd like us to also do is to prepare email communication that we can send to badges issuers and ideally recipients, if we have a way to access their email addresses (cross post to Discourse). Is that possible?

One we know that we can prepare email communication -- will require sign-off from Chris Lawrence.

Thanks,
George

Comment 15

[Ed Morley [:emorley]]

(Making accessible per bug 1197381 comment 3)

Comment 16

[Nikos Roussos [:comzeradd]]

I started creating a static mirror.

Regarding email communication, that would require some work and input from the people that have access to the current deployment. No easy way to fetch emails without database access. Having said that, there is a minor privacy concern on emailing people that didn't opt-in for email communications from Mozilla.

Comment 17

[:Atoll]

If we're planning to send an email communication to issuers and recipients, we need to ensure some things:

1: We have clear permission to email the recipients. Issuers are vouched Mozillians, but recipients are not, necessarily. They are provided an invite code that lets them signup. Does either the invitation email or the signup process meaningfully convey to them that we're going to email them for this purpose?

2: We provide a clear takedown policy, verified to work in production, prior to the archival date. If we're going to email people and say "we're freezing the site on date X", we owe them an opt-out method to remove their data before date X.

Are we prepared to meet both of these requirements?

Comment 18

[April King [:April]]

As for #2, the site is archived multiple times in archive.org.  So scrubbing their information now won't remove the old information, unless we also kill the site from the internet archive.

My personal opinion on the matter is that we should replace the site with a 302 redirect to the proper URL on the internet archive, instead of trying to host a static copy in perpetuity.

Comment 19

[:Atoll]

+1 to archive.org, they do persistence and rewriting as a day job.

Comment 20

[Nikos Roussos [:comzeradd]]

Redirecting to archive.org is a good option too. Their most recent "mirroring" happened on May 3, but probably nothing changed since then.

In that case, do we still need an archiving notice? If yes, it should happen on the current website.

Comment 21

[April King [:April]]

Sure, I think having an archive notice for a month or two is probably good, especially if it gets picked up by archive.org so people have context for the redirect at some point in the future.

Comment 22

[George Roter]

MARSHALL --> need your thoughts on (2) below.

A few thoughts:

1) I think that we should do a static mirror of our own, hosted for 1-2 months (or enough time for the static snapshot to be captured by archive.org ... is there a way to make an ask for another snapshot?). This will communicate to people a staged wind-down of the site and that Mozilla is doing this responsibly -- it's not substantively different from archive.org (other than a few badges added since May), but it gives a different perception with little risk to Mozilla.

2) Then I think we should have the 302 redirect up for URLs for another 3-6 months ... gives the long tail of people who want to use this their badges the chance to change the URL they reference. We should have a well crafted message on the redirect page with instructions on how people can keep referencing their badges on archive.org.

The we should stop hosting this/just get rid of it.

3) Re: emailing -- Given that recipients specifically had to opt-in to receiving a badge, I believe there is enough permission to email them about information that the badge they have will change quite dramatically. But it would be good to get Marshall's opinion on this.

- email to issues --> should go out shortly before we move to the static site
- email to recipients --> should go out immediately following going to archive.org

Comment 23

[Nikos Roussos [:comzeradd]]

I triggered archive.org to take a new snapshot of the website (it may take some hours to be fully accessible).

In the meantime, I created a mirror of the website, that we can deploy as a static mirror. I'll amend the frontpage to remove forms and persona logins that won't work and add a notice.

Notice example:
> badges.mozilla.org is shutting down. This service is currently available in a read-only mode. No new badges can be issued.
> 
> This website will be unavailable after the end of February, 2017.
> 
> After that date you can browse and link badges directly on archive.org mirror.

Where we can just link to https://web.archive.org/web/*/https://badges.mozilla.org

After the shutdown date we can add a redirect rule for archive.org.
For instance this badge:
https://badges.mozilla.org/en-US/badges/badge/Nightly-User/
will redirect here:
https://web.archive.org/web/20151204091735/https://badges.mozilla.org/en-US/badges/badge/Nightly-User

Comment 24

[Nikos Roussos [:comzeradd]]

We deployed the static mirror on our infra, with a relevant notice. If everything looks good we'll open a bug for the dns change.
http://badges.mozilla.org.s3-website-us-east-1.amazonaws.com/

Comment 25

[April King [:April]]

Looks good to me!

Comment 26

[Ryan Kelly [:rfkelly]]

Removing this as a blocker for the Persona EOL; I don't think it will be too terrible if it's up for a brief while post Persona EOL.

Comment 27

[April King [:April]]

Can we possibly update the archive.org link to: https://web.archive.org/web/*/badges.mozilla.org?  Also, can I ask what the status of the DNS change is?  It's getting pretty close to EOY.  :)

Comment 28

[Yousef Alam [:yalam96] (use NEEDINFO)]

The amazon route for getting an SSL cert didn't end up working because it's a mozilla.org subdomain, so I'll file a new bug for a DigiCert one and hopefully we can wrap this up this week.

Comment 29

[Henrik Mitsch [:hmitsch]]

As bug 1321364 (the DNS change) was done yesterday, this can be closed.

Comment 30

[:Atoll]

Reopening; we still need to tear down he old Webops infrastructure that was left behind with this move, as part of the decom.

Comment 31

[Daniel Hartnell [:danielh]]

The WebOps infrastructure that atoll mentioned has been removed:

- 1332462 has been created to decom the database
- Removed cron jobs on admin node
- Removed log directory from web clusters and the Puppet manifests
- Removed dev/stage DNS entries
- Removed dev/stage/prod instances of the app on the admin node and each web head
- Removed www and src directories on admin node
- Removed /var/log/httpd/${site} on each web head
- Removed SSL certificate catalog for domain in Zeus

I can follow up once the database has been decommissioned.

Comment 32

[Daniel Hartnell [:danielh]]

I'm going to go ahead and mark this as solved. I don't think it's necessary to wait for the database decommission in bug 1332462 to complete.

As noted above, everything in the WebOps infrastructure has been decommissioned.