1305703 - please deploy balrog 2.10.1 to stage/prod

Status: VERIFIED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[[e-mail] (:bhearsum)]

Balrog version 2.10 is ready to be pushed to stage. Please deploy the new Docker image (master-2016-09-27-12-57) the admin and public webapps, and the new Docker image for the Agent (master-2016-09-27-12-57).

We'd like the production push for this to happen sometime between 11am and 1pm pacific on September 28th.

Comment 1

[Benson Wong [:mostlygeek]]

This release has a few notable backend changes: 

- Bug 1304082: SMTP notifications for balrog-admin
- Bug 1304496: switch SSL cert on aus4-admin.mozilla.org to a digicert SHA2 cert
- Tweaked gzip on web cluster. Balrog responses were too small to qualify with our default config. Custom config qualifies anything 256 bytes and larger

Comment 2

[Benson Wong [:mostlygeek]]

Deployed to stage.

Comment 3

[[e-mail] (:bhearsum)]

Just finished testing in stage, and found two issues:
1) A regression in the webapp code. Can we pick up 2.10.1 (master-2016-09-28-13-50) to grab the fix?
2) I couldn't make change notification work in stage. I made a few changes that should've sent mail to [e-mail], but I haven't seen any messages from Balrog on https://groups.google.com/a/mozilla.com/forum/#!forum/junk-balrog-db-changes - just my own test messages.

Comment 4

[Benson Wong [:mostlygeek]]

I sent a test email from the staging server from the CLI and it appears to have made it through. 

/var/log/maillog shows these errors messages: 

Sep 28 14:28:31 ip-172-31-0-141 postfix/smtpd[25423]: lost connection after EHLO from localhost[127.0.0.1]
Sep 28 14:28:31 ip-172-31-0-141 postfix/smtpd[25423]: disconnect from localhost[127.0.0.1]


There's no error logs that came out of balrog-admin

Comment 5

[[e-mail] (:bhearsum)]

(In reply to Benson Wong [:mostlygeek] from comment #4)
> I sent a test email from the staging server from the CLI and it appears to
> have made it through. 
> 
> /var/log/maillog shows these errors messages: 
> 
> Sep 28 14:28:31 ip-172-31-0-141 postfix/smtpd[25423]: lost connection after
> EHLO from localhost[127.0.0.1]
> Sep 28 14:28:31 ip-172-31-0-141 postfix/smtpd[25423]: disconnect from
> localhost[127.0.0.1]
> 
> 
> There's no error logs that came out of balrog-admin

We debugged on IRC and discovered that Balrog was trying to send STARTTLS, which the mail server doesn't support. I'll need a code change to fix that, but I don't think we need to block on that today. 2.10.1 is ready to go to prod as far as I'm concerned.

Comment 6

[Benson Wong [:mostlygeek]]

Deployed to prod.

Comment 7

[[e-mail] (:bhearsum)]

Looks good to me, thanks!