996721 - LDAP protect mozillians stage database dump

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[Andrei Hajdukewycz [:sancus]]

The URL for this dump will be https://mozillians-dev.allizom.org/media/mozillians_org.latest.sanitized_stage.sql.gz

It should be put behind LDAP. We can also put this in its own directory if need be, I don't really know if you can LDAP protect individual files... maybe we should do a whole directory for the older files too, though.

Comment 1

[Andrei Hajdukewycz [:sancus]]

Given that we're going to have multiple files(the .latest, and then older ones) it probably makes more sense to have a directory for these, no?

Once we have the directory LDAP protected, then we can enable copying of the files to it, as well.

So maybe /media/stage-db/ ?

Comment 2

[Chris Turra [:cturra]]

i have created the media/stage-db/ as a volume on the mozillaians stage netapp volume and setup ldap auth as requested.

  https://mozillians.allizom.org/media/stage-db/cturra.txt

Comment 3

[Andrei Hajdukewycz [:sancus]]

Can you do this on mozillians-dev instead? I know it's kind of counterintuitive, but that's where all the db dumps are currently set to go.

Since we already have the copies figured out for that location, seems easiest.

Comment 4

[Chris Turra [:cturra]]

the dev/stage reference got me confused. i have created a media/dev-db/ directory you can use for these dumps. i don't want to have any stage references in dev since it will definitely cause more confusion down the road. you can however place stage db dumps in this directory if you'd like.

  https://mozillians-dev.allizom.org/media/dev-db/

Comment 5

[Andrei Hajdukewycz [:sancus]]

Yeah that's fine, thanks!

Comment 6

[Justin Crawford [:hoosteeno] [:jcrawford]]

There was a question in IRC about this. The question was, "why are we dumping stage data at all?".

I believe it's answered here:
https://bugzilla.mozilla.org/show_bug.cgi?id=933749#c18

Answer: Core devs use stage data occasionally to debug issues that are tied to prod data.

Comment 7

[Sheeri Cabral [:sheeri]]

reopening so I can put the right copy location for stage into the script...

Comment 8

[Giorgos Logiotatidis [:giorgos]]

Friendly ping! :)

Comment 9

[Sheeri Cabral [:sheeri]]

Pull request sent.

Comment 10

[Sheeri Cabral [:sheeri]]

It was approved!

And the script ran via cron a few minutes ago. I looked at the result, where the stage stuff should be, and it's here:

[[e-mail]1 ~]# ls -lrth /data/genericrhel6-dev/src/mozillians-dev.allizom.org/mozillians/media/dev-db/
total 2.4M
-rw-r--r-- 1 root root    5 Apr 15 12:54 cturra.txt
-rw-r--r-- 1 root root 2.4M Apr 29 14:20 mozillians_org.latest.sanitized_stage.sql.gz

Does it look good to you from an "I can download it after entering my LDAP credentials" point of view?

Comment 11

[Sheeri Cabral [:sheeri]]

just a ping on this - can you get to that file? If so, let's close this:

[[e-mail]1 ~]$ ls -lrth /data/genericrhel6-dev/src/mozillians-dev.allizom.org/mozillians/media/dev-db/
total 2.4M
-rw-r--r-- 1 root root    5 Apr 15 12:54 cturra.txt
-rw-r--r-- 1 root root 2.4M May  5 20:00 mozillians_org.latest.sanitized_stage.sql.gz

Looks like the daily cron job is running just fine.

Comment 12

[Andrei Hajdukewycz [:sancus]]

https://mozillians-dev.allizom.org/media/dev-db/mozillians_org.latest.sanitized_stage.sql.gz is reachable to me after entering my LDAP credentials.

This is fully fixed now, thanks :sheeri!