Closed Bug 725403 Opened 9 years ago Closed 9 years ago

[mozillians] Push 1.3.1 to stage (and then prod)

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

All
Other
task
Not set
critical

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: davedash, Assigned: bburton)

References

Details

Attachments

(1 file)

1.3.1 is a do-over of 1.3:

https://bugzilla.mozilla.org/show_bug.cgi?id=724723

Please push to stage as before and run the ./manage.py cron flee_ldap command.

After the command is run:

SELECT count(*) FROM auth_user WHERE last_name='';

Should be about 39 (before ~1500+)
Assignee: server-ops → bburton
Status: NEW → ASSIGNED
Will do shortly after 11AM
Cool, Matt and Aakash will give yay or nay, and then we'll try to push live to prod this afternoon if that works.
Attached file flee errors
SQL output on stage

mysql> use mozillians_stage
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> SELECT count(*) FROM auth_user WHERE last_name='';
+----------+
| count(*) |
+----------+
|       33 |
+----------+
1 row in set (0.00 sec)
So stage is problematic at the moment.  I don't feel like we can properly write to elastic search via the admin nodes.

Can we spend some time tomorrow debugging this?  I can hop on vidyo in the morning.

I can run commands against ES from the stage heads, but not the admin node.

We'll need to double check this on prod's admin node as well before moving forward.

-d
Bounced celery on stage and it ingested tasks related to elastic search. Queried each elastic search host and returned proper results.
Sorry everyone, we're finding serious bugs late in the process, we'll try again with a 1.3.2 push bug hopefully today, and push to prod tomorrow... but I feel like a broken record.
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.