725403 - [mozillians] Push 1.3.1 to stage (and then prod)

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

1.3.1 is a do-over of 1.3:

https://bugzilla.mozilla.org/show_bug.cgi?id=724723

Please push to stage as before and run the ./manage.py cron flee_ldap command.

After the command is run:

SELECT count(*) FROM auth_user WHERE last_name='';

Should be about 39 (before ~1500+)

Comment 1

[Brandon Burton [:solarce]]

Will do shortly after 11AM

Comment 2

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Cool, Matt and Aakash will give yay or nay, and then we'll try to push live to prod this afternoon if that works.

Comment 3

[Brandon Burton [:solarce]]

Attachment: flee errors

Comment 4

[Brandon Burton [:solarce]]

SQL output on stage

mysql> use mozillians_stage
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> SELECT count(*) FROM auth_user WHERE last_name='';
+----------+
| count(*) |
+----------+
|       33 |
+----------+
1 row in set (0.00 sec)

Comment 5

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

So stage is problematic at the moment.  I don't feel like we can properly write to elastic search via the admin nodes.

Can we spend some time tomorrow debugging this?  I can hop on vidyo in the morning.

I can run commands against ES from the stage heads, but not the admin node.

We'll need to double check this on prod's admin node as well before moving forward.

-d

Comment 6

[Jason Thomas [:jason]]

Bounced celery on stage and it ingested tasks related to elastic search. Queried each elastic search host and returned proper results.

Comment 7

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Sorry everyone, we're finding serious bugs late in the process, we'll try again with a 1.3.2 push bug hopefully today, and push to prod tomorrow... but I feel like a broken record.