[mozillians] Push 1.3.1 to stage (and then prod)

RESOLVED FIXED

Status

--
critical
RESOLVED FIXED
7 years ago
5 years ago

People

(Reporter: davedash, Assigned: bburton)

Tracking

Details

Attachments

(1 attachment)

1.3.1 is a do-over of 1.3:

https://bugzilla.mozilla.org/show_bug.cgi?id=724723

Please push to stage as before and run the ./manage.py cron flee_ldap command.

After the command is run:

SELECT count(*) FROM auth_user WHERE last_name='';

Should be about 39 (before ~1500+)
(Assignee)

Updated

7 years ago
Assignee: server-ops → bburton
(Assignee)

Updated

7 years ago
Status: NEW → ASSIGNED
(Assignee)

Comment 1

7 years ago
Will do shortly after 11AM
Cool, Matt and Aakash will give yay or nay, and then we'll try to push live to prod this afternoon if that works.
(Assignee)

Comment 3

7 years ago
Created attachment 595468 [details]
flee errors
(Assignee)

Comment 4

7 years ago
SQL output on stage

mysql> use mozillians_stage
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> SELECT count(*) FROM auth_user WHERE last_name='';
+----------+
| count(*) |
+----------+
|       33 |
+----------+
1 row in set (0.00 sec)
So stage is problematic at the moment.  I don't feel like we can properly write to elastic search via the admin nodes.

Can we spend some time tomorrow debugging this?  I can hop on vidyo in the morning.

I can run commands against ES from the stage heads, but not the admin node.

We'll need to double check this on prod's admin node as well before moving forward.

-d
Bounced celery on stage and it ingested tasks related to elastic search. Queried each elastic search host and returned proper results.
Sorry everyone, we're finding serious bugs late in the process, we'll try again with a 1.3.2 push bug hopefully today, and push to prod tomorrow... but I feel like a broken record.
Status: ASSIGNED → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.