737674 - Populate mozillians stage server with production data

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P1)

Description

[tofumatt [:tofumatt]]

I know this is probably a PITA request, but would it be possible to copy over the production data (uploads + DB) to the stage server for Mozillians? We need to test a migration from a production state to verify it works 100%.

This has to do with photos changing, which required certain permissions changes to work if I recall correctly. I think we did something like this recently (prod > stage copy) but I can't find the relevant bug.

After copying the data, you should do a redeploy of mozillians stage from the prod branch to make sure code is up-to-date and the migrations run OK.

CC'ing timw in case I'm wrong about anything, missing things, or he has anything to add.

Comment 1

[Jake Maul [:jakem]]

Need infrasec ack on this.

@infrasec, please let us know what you want to see to let this happen. This can be a temporary copy of the data (less than 1 week), if that helps any.

Comment 2

[tofumatt [:tofumatt]]

This might not be needed after all, so hold off on it for a bit. Sorry :=/

Comment 3

[Tim Watts [:timw]]

Won't happen because of privacy. 
Generating fake data is out best option.

Comment 4

[Matt Brandt [:mbrandt]]

Bumping to verified wontfix - fake data ftw

Comment 5

[Tim Watts [:timw]]

Ok we still need sanitized production data. We are writing a program to clean the fields if you can tell us which ones need it. I'm assuming users email and the hashed pasword field, as far as I can tell, all other things are public already.

Comment 6

[tofumatt [:tofumatt]]

https://github.com/tofumatt/mysql-anonymous

If I can have a shot at sanitizing the production data with the above script that'd be useful. But it's based on stuff that worked for at least Zamboni and I believe Input, so running it on a mozillians-prod dump should work OK.

Comment 7

[Aakash Desai [:aakashd]]

Hey all, we're blocking a release because of this bug. 

I apologize for the notice on a Friday. Can we get someone on it today?

Comment 9

[Tim Watts [:timw]]

Ok I can see how it's confusing so here are the steps we need:
1) copy the mozillians.org database to staging.
2) run the following sql statements on staging.

=== SANITIZING SQL ===
SET FOREIGN_KEY_CHECKS=0;
TRUNCATE django_admin_log;
TRUNCATE django_session;
UPDATE auth_user SET email = CONCAT(id, '@mozilla.com'), username = CONCAT('_user_', id);
UPDATE invite SET recipient = CONCAT(id, '@mozilla.com');
SET FOREIGN_KEY_CHECKS=1;
=== END SQL ===

Comment 11

[Sheeri Cabral [:sheeri]]

Done!


mysql> use mozillians_stage;
Database changed
mysql> SET FOREIGN_KEY_CHECKS=0;
Query OK, 0 rows affected (0.00 sec)

mysql> TRUNCATE django_admin_log;
Query OK, 0 rows affected (0.05 sec)

mysql> TRUNCATE django_session;
Query OK, 0 rows affected (0.04 sec)

mysql> UPDATE auth_user SET email = CONCAT(id, '@mozilla.com'), username = CONCAT('_user_', id);
Query OK, 2250 rows affected (0.19 sec)
Rows matched: 2250  Changed: 2250  Warnings: 0

mysql> UPDATE invite SET recipient = CONCAT(id, '@mozilla.com');
Query OK, 1171 rows affected (0.10 sec)
Rows matched: 1171  Changed: 1171  Warnings: 0

mysql> SET FOREIGN_KEY_CHECKS=1;
Query OK, 0 rows affected (0.00 sec)

Comment 12

[Tim Watts [:timw]]

thank you

Comment 13

[Matt Brandt [:mbrandt]]

+1 the thank you - thanks everyone that worked to get stage into it's happy place again.