Decommission badges.mozilla.org from WebOps (leaving AWS alone)

RESOLVED FIXED

Status

Infrastructure & Operations
WebOps: Other
--
major
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: ericz, Assigned: danielh)

Tracking

(Blocks: 1 bug)

Details

(Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/2298] )

(Reporter)

Description

3 years ago
badges.mozilla.org uses Django and therefore needs to be migrated to the Python cluster so we can upgrade Django for security reasons.

Updated

3 years ago
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/2298]
(In reply to Eric Ziegenhorn :ericz from comment #0)
> badges.mozilla.org uses Django and therefore needs to be migrated to the
> Python cluster so we can upgrade Django for security reasons.

Need to figure out who're the owners here. I know the site is used but I don't want to take on "ownership" by migrating this to the python cluster.

According to https://github.com/mozilla/badges.mozilla.org, this isn't maintained on the dev side anymore. 

Switching this to a decom bug, let's get security involved and see what the next steps are.
Flags: needinfo?(april)
Summary: Migrate badges.mozilla.org to Python cluster → Decommission badges.mozilla.org
April/Jeff - the GH repo for this site says it's not being actively maintained. I don't think the site should continue to remain online if it's not actively maintained. How can we move forward on this? 

CC'ing Gene here, since this is backed by persona as well.

Comment 3

2 years ago
Will, Les,
   Can you help :fox2mike determine who owns the badges.mozilla.org web property as he's looking for clearance to decompression it?
Flags: needinfo?(willkg)
Flags: needinfo?(lorchard)

Comment 4

2 years ago
decommission it I mean (no need to decompress it) =)
As far as I understand, no one owns these days. I gave it over to the badges team in MoFo a few years ago, but I think that team is no more. That said, I'm CC'ing some folks who are talking about doing something with it right now.
Flags: needinfo?(lorchard)
I think Les covered everything I know.
Flags: needinfo?(willkg)
(In reply to Gene Wood [:gene] from comment #4)
> decommission it I mean (no need to decompress it) =)

Gene, we'll make sure to not decompress ;)

From a security standpoint - What's our way forward with badges in case no one speaks up?
Flags: needinfo?(gene)

Comment 8

2 years ago
Hi All,

We've started putting together a plan on this here:
https://docs.google.com/document/d/17efYpvm4Khhx4YACIVQKBcfV07hcZOBxrdSguYG6nag/edit

It's not detailed and we haven't run it by people who need to be informed.

I need to free up some resources to deal with this in the weeks ahead. Can we have a quick check-in on this at the end of June?

Copying Pierros on this bug so it's in our queue to talk about. Also Arielle and Hermina who can help manage.

George
Flags: needinfo?(gene)
Flags: needinfo?(april)

Comment 9

2 years ago
:fox2mike,
  My interest in the topic is related to the fact that with Persona going away Nov 30, badges will stop working at that point without a new auth system. I just want to make sure it's either decommed before then or has a plan for updating the auth system.

Updated

2 years ago
Blocks: 1197381
(In reply to groter from comment #8)
> Hi All,
> 
> We've started putting together a plan on this here:
> https://docs.google.com/document/d/
> 17efYpvm4Khhx4YACIVQKBcfV07hcZOBxrdSguYG6nag/edit
> 
> It's not detailed and we haven't run it by people who need to be informed.
> 
> I need to free up some resources to deal with this in the weeks ahead. Can
> we have a quick check-in on this at the end of June?

End of June check-in :) Thanks for taking some of this on, George.
Flags: needinfo?(groter)

Comment 11

2 years ago
Hi Shyam,

I just booked a day of Nikos' time in the 3rd week of August to take care of this.

We will:
1) Turn the site into static content -- turn off all functions to grant or add new badges, login, etc.

2) Clearly communicate when the site will be shutdown for good (including all static content) -- aiming for Jan 2017.

3) Have a plan for backing up all of the content when it is fully shut down.

Cheers,
George
Flags: needinfo?(groter)

Comment 12

2 years ago
Note that the site is nicely backed up in the Internet Archive, so we could always redirect there.
I had a look at the website and the options we have.

My suggestion is this:
1. Create a static mirror of the website.
2. Upload it on AWS (probably on an S3 bucket)
3. Amend the front page with a notice that informs about the site being read-only
4. Part of that notice should also be timeline for a full decommission of the website/domain (eg. in 6 months)

Things to consider:
1. Some pages return 503 erros at the moment.
2. Some views are rendered dynamically from the API.
Bottom line: we may not be able to get everything.

Comment 14

2 years ago
Great. What I'd like us to also do is to prepare email communication that we can send to badges issuers and ideally recipients, if we have a way to access their email addresses (cross post to Discourse). Is that possible?

One we know that we can prepare email communication -- will require sign-off from Chris Lawrence.

Thanks,
George
(Making accessible per bug 1197381 comment 3)
Group: infra → mozilla-employee-confidential
I started creating a static mirror.

Regarding email communication, that would require some work and input from the people that have access to the current deployment. No easy way to fetch emails without database access. Having said that, there is a minor privacy concern on emailing people that didn't opt-in for email communications from Mozilla.
If we're planning to send an email communication to issuers and recipients, we need to ensure some things:

1: We have clear permission to email the recipients. Issuers are vouched Mozillians, but recipients are not, necessarily. They are provided an invite code that lets them signup. Does either the invitation email or the signup process meaningfully convey to them that we're going to email them for this purpose?

2: We provide a clear takedown policy, verified to work in production, prior to the archival date. If we're going to email people and say "we're freezing the site on date X", we owe them an opt-out method to remove their data before date X.

Are we prepared to meet both of these requirements?

Comment 18

2 years ago
As for #2, the site is archived multiple times in archive.org.  So scrubbing their information now won't remove the old information, unless we also kill the site from the internet archive.

My personal opinion on the matter is that we should replace the site with a 302 redirect to the proper URL on the internet archive, instead of trying to host a static copy in perpetuity.
+1 to archive.org, they do persistence and rewriting as a day job.
Redirecting to archive.org is a good option too. Their most recent "mirroring" happened on May 3, but probably nothing changed since then.

In that case, do we still need an archiving notice? If yes, it should happen on the current website.

Comment 21

2 years ago
Sure, I think having an archive notice for a month or two is probably good, especially if it gets picked up by archive.org so people have context for the redirect at some point in the future.

Comment 22

2 years ago
MARSHALL --> need your thoughts on (2) below.

A few thoughts:

1) I think that we should do a static mirror of our own, hosted for 1-2 months (or enough time for the static snapshot to be captured by archive.org ... is there a way to make an ask for another snapshot?). This will communicate to people a staged wind-down of the site and that Mozilla is doing this responsibly -- it's not substantively different from archive.org (other than a few badges added since May), but it gives a different perception with little risk to Mozilla.

2) Then I think we should have the 302 redirect up for URLs for another 3-6 months ... gives the long tail of people who want to use this their badges the chance to change the URL they reference. We should have a well crafted message on the redirect page with instructions on how people can keep referencing their badges on archive.org.

The we should stop hosting this/just get rid of it.

3) Re: emailing -- Given that recipients specifically had to opt-in to receiving a badge, I believe there is enough permission to email them about information that the badge they have will change quite dramatically. But it would be good to get Marshall's opinion on this.

- email to issues --> should go out shortly before we move to the static site
- email to recipients --> should go out immediately following going to archive.org
Flags: needinfo?(merwin)
I triggered archive.org to take a new snapshot of the website (it may take some hours to be fully accessible).

In the meantime, I created a mirror of the website, that we can deploy as a static mirror. I'll amend the frontpage to remove forms and persona logins that won't work and add a notice.

Notice example:
> badges.mozilla.org is shutting down. This service is currently available in a read-only mode. No new badges can be issued.
> 
> This website will be unavailable after the end of February, 2017.
> 
> After that date you can browse and link badges directly on archive.org mirror.

Where we can just link to https://web.archive.org/web/*/https://badges.mozilla.org

After the shutdown date we can add a redirect rule for archive.org.
For instance this badge:
https://badges.mozilla.org/en-US/badges/badge/Nightly-User/
will redirect here:
https://web.archive.org/web/20151204091735/https://badges.mozilla.org/en-US/badges/badge/Nightly-User

Updated

2 years ago
Group: mozilla-employee-confidential

Updated

2 years ago
Flags: needinfo?(merwin)
We deployed the static mirror on our infra, with a relevant notice. If everything looks good we'll open a bug for the dns change.
http://badges.mozilla.org.s3-website-us-east-1.amazonaws.com/

Comment 25

2 years ago
Looks good to me!
Depends on: 1321364
Removing this as a blocker for the Persona EOL; I don't think it will be too terrible if it's up for a brief while post Persona EOL.
No longer blocks: 1197381

Comment 27

2 years ago
Can we possibly update the archive.org link to: https://web.archive.org/web/*/badges.mozilla.org?  Also, can I ask what the status of the DNS change is?  It's getting pretty close to EOY.  :)
The amazon route for getting an SSL cert didn't end up working because it's a mozilla.org subdomain, so I'll file a new bug for a DigiCert one and hopefully we can wrap this up this week.

Updated

2 years ago
Blocks: 1331703
As bug 1321364 (the DNS change) was done yesterday, this can be closed.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
Reopening; we still need to tear down he old Webops infrastructure that was left behind with this move, as part of the decom.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Summary: Decommission badges.mozilla.org → Decommission badges.mozilla.org from WebOps (leaving AWS alone)

Updated

2 years ago
Blocks: 1332026

Updated

2 years ago
Depends on: 1332462
(Assignee)

Updated

2 years ago
Assignee: server-ops-webops → dhartnell
(Assignee)

Comment 31

2 years ago
The WebOps infrastructure that atoll mentioned has been removed:

- 1332462 has been created to decom the database
- Removed cron jobs on admin node
- Removed log directory from web clusters and the Puppet manifests
- Removed dev/stage DNS entries
- Removed dev/stage/prod instances of the app on the admin node and each web head
- Removed www and src directories on admin node
- Removed /var/log/httpd/${site} on each web head
- Removed SSL certificate catalog for domain in Zeus

I can follow up once the database has been decommissioned.
(Assignee)

Comment 32

2 years ago
I'm going to go ahead and mark this as solved. I don't think it's necessary to wait for the database decommission in bug 1332462 to complete.

As noted above, everything in the WebOps infrastructure has been decommissioned.
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.