867982 - Hard Delete Makes after 14 days

Status: RESOLVED FIXED

(Webmaker Graveyard :: MakeAPI, defect)

Description

[Chris DeCairos (:cade)]

Makes marked as deleted should be removed Permanently every 14 days.

Comment 2

[Chris DeCairos (:cade)]

Attachment: https://github.com/mozilla/MakeAPI/pull/177

I wrote a script that can be used to remove all makes that were deleted 2 weeks or more before the scripts invocation.

How could we set this up to run periodically on our infra?

Comment 3

[Jon Buckley [:jbuck]]

Comment on attachment 8340034 [details] [review]
https://github.com/mozilla/MakeAPI/pull/177

This script won't exit properly

Comment 4

[Chris DeCairos (:cade)]

Comment on attachment 8340034 [details] [review]
https://github.com/mozilla/MakeAPI/pull/177

I forgot to remove a function from the script I copied from :D

Comment 5

[Jon Buckley [:jbuck]]

Comment on attachment 8340034 [details] [review]
https://github.com/mozilla/MakeAPI/pull/177

lookin' good

Comment 6

[[github robot]]

Commit pushed to master at https://github.com/mozilla/MakeAPI

https://github.com/mozilla/MakeAPI/commit/659324602ae7e93c934f890e2acf23f9bc2f9c72
Bug 867982 - Add script for deleting makes deleted more than 2 weeks prior to the current time at the scripts execution

Comment 7

[Chris DeCairos (:cade)]

Merged into master. Now we can set up cron jobs to execute the script. need-info-ing jp.

Comment 8

[JP Schneider :jp]

We can setup a worker node which receives MakeAPI but does not sit in the load balancer, and then runs a cron job.

Let me know what the cronjob is going to need to run, how often, and I'll get it setup.

Comment 9

[Chris DeCairos (:cade)]

(In reply to JP Schneider :jp from comment #8)
> We can setup a worker node which receives MakeAPI but does not sit in the
> load balancer, and then runs a cron job.
> 
> Let me know what the cronjob is going to need to run, how often, and I'll
> get it setup.

It should run the `scripts/cleanup.js` script.

The exit code will be 0 if successful, 1 if not.

Once a day seems fine. As for when, ideally when our traffic is lightest, but I cannot say with certainty when that is.

Comment 10

[JP Schneider :jp]

I am working on this here: https://github.com/mozilla/mofo_system_configurations/compare/webmakercontroller

I'm creating a new controller role / node, makeapi will be the first app I have put into that by creating a new nodetype variable which is set based on if something is a controller or web server in the MakeAPI manifest.

Comment 11

[JP Schneider :jp]

I've got it to a review-worthy state.  I have additionally added the mofo-techops/creds/app/staging(production)/webmaker-controller folder.  The lame part here will be ensuring if we update makeapi configs for usual nodes, we do the same for this folder.  I'll think on how to change that so it works better.

Tomorrow, I'll be ready to create IAM creds and rights for this new node type, and test it out while makeapi-production puppet is stopped.  Staging...of course. :)

Comment 12

[JP Schneider :jp]

PR with puppet diff is here: https://github.com/mozilla/mofo_system_configurations/pull/208

You can see I added a variable into Makeapi, 'nodetype', to determine which services/files each server had setup.

Comment 13

[JP Schneider :jp]

I've created a staging and prod node, in an autoscaling group (we want the node replaced if it dies).

In staging, I ran the job with these results:
root@i-834865fb:/var/www/makeapi# cd /var/www/makeapi && . ./.env && /usr/bin/node scripts/cleanup.js 
82 Makes deleted successfully.

I'm bringing prod up now, and it should log to this stream:
http://loggins.mofoprod.net/streams/52a0a7dce4b00e9f99fec7a8/messages?q=*&rangetype=relative&relative=300

Comment 14

[JP Schneider :jp]

The actual logging stream is here: http://loggins.mofoprod.net/streams/52a0c068e4b00e9f99fecc93/messages?q=*&rangetype=relative&relative=300

Closing this out!