Closed Bug 1304082 Opened 8 years ago Closed 8 years ago

enable change notifier for balrog admin app

Categories

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bhearsum, Assigned: bhearsum)

References

Details

Attachments

(1 file)

This was requested way back when we originally talked about the migration. I ended up implementing support for it in https://bugzilla.mozilla.org/show_bug.cgi?id=1251338, but we didn't enable it. It should work if you set the following environment variables:
SMTP_HOST
SMTP_PORT
NOTIFY_TO_ADDR
NOTIFY_FROM_ADDR
SMTP_USERNAME (optional)
SMTP_PASSWORD (optional)

NOTIFY_TO_ADDR should be balrog-db-changes@mozilla.com. We should probably test it out in stage first, but it would be best if it were disabled there once we know it works (too spammy).
No longer blocks: 1304054
Blocks: 1246675
We decided that we should have a separate list for stage, to make sure we don't bust it in the future. I created junk-balrog-db-changes@mozilla.com for this.
Depends on: 1305703
We tried enabling this in stage today, and got an exception:
{"EnvVersion": "2.0", "Type": "Rules", "Severity": 3, "Fields": {"message": "Failed to connect to SMTP server:", "traceback": "Uncaught exception:\n  File \"./auslib/db.py\", line 1790, in send_email\n    conn.starttls()\n  File \"/usr/local/lib/python2.7/smtplib.py\", line 644, in starttls\n    raise SMTPException(\"STARTTLS extension not supported by server.\")\n<class 'smtplib.SMTPException'>\nSMTPException('STARTTLS extension not supported by server.',)\n", "requestid": 140010317872208, "error": "SMTPException('STARTTLS extension not supported by server.',)"}, "Timestamp": 1475080251884000000, "Logger": "Balrog", "Hostname": "ip-172-31-48-255", "Pid": 8}

Looks like I'll need to tweak the code to make it possible to disable starttls.
We also hit another error, which happens prior to the smtp connection in certain cases:
ion=transaction)\n  File \"./auslib/db.py\", line 1530, in addLocaleToRelease\n    transaction=transaction)\n  File \"./auslib/db.py\", line 498, in update\n    return self._prepareUpdate(transaction, where, what, changed_by, old_data_version)\n  File \"./auslib/db.py\", line 456, in _prepareUpdate\n    self.onUpdate(self, \"UPDATE\", changed_by, query)\n  File \"./auslib/db.py\", line 1838, in bleet\n    row = table.sel$

I'll steal this bug for now while I work out fixes for both issues.
Assignee: bwong → bhearsum
(In reply to Ben Hearsum (:bhearsum) from comment #3)
> We also hit another error, which happens prior to the smtp connection in
> certain cases:
> ion=transaction)\n  File \"./auslib/db.py\", line 1530, in
> addLocaleToRelease\n    transaction=transaction)\n  File \"./auslib/db.py\",
> line 498, in update\n    return self._prepareUpdate(transaction, where,
> what, changed_by, old_data_version)\n  File \"./auslib/db.py\", line 456, in
> _prepareUpdate\n    self.onUpdate(self, \"UPDATE\", changed_by, query)\n 
> File \"./auslib/db.py\", line 1838, in bleet\n    row = table.sel$
> 
> I'll steal this bug for now while I work out fixes for both issues.

Ben - any chance you can dig up the full traceback for this one? It looks like the pastebin this came from has expired. It should be from around 4pm eastern on the 28th.
Flags: needinfo?(bwong)
This is the long line you are looking for...

{
    "EnvVersion": "2.0",
    "Fields": {
        "error": "SMTPException('STARTTLS extension not supported by server.',)",
        "message": "Failed to connect to SMTP server:",
        "requestid": 140372048409168,
        "traceback": "Uncaught exception:\n  File \"./auslib/db.py\", line 1790, in send_email\n    conn.starttls()\n  File \"/usr/local/lib/python2.7/smtplib.py\", line 644, in starttls\n    raise SMTPException(\"STARTTLS extension not supported by server.\")\n<class 'smtplib.SMTPException'>\nSMTPException('STARTTLS extension not supported by server.',)\n"
    },
    "Hostname": "xx",
    "Logger": "Balrog",
    "Pid": 8,
    "Severity": 3,
    "Timestamp": 1475092806206509056,
    "Type": "Rules"
}
Flags: needinfo?(bwong)
(In reply to Benson Wong [:mostlygeek] from comment #5)
> This is the long line you are looking for...
> 
> {
>     "EnvVersion": "2.0",
>     "Fields": {
>         "error": "SMTPException('STARTTLS extension not supported by
> server.',)",
>         "message": "Failed to connect to SMTP server:",
>         "requestid": 140372048409168,
>         "traceback": "Uncaught exception:\n  File \"./auslib/db.py\", line
> 1790, in send_email\n    conn.starttls()\n  File
> \"/usr/local/lib/python2.7/smtplib.py\", line 644, in starttls\n    raise
> SMTPException(\"STARTTLS extension not supported by server.\")\n<class
> 'smtplib.SMTPException'>\nSMTPException('STARTTLS extension not supported by
> server.',)\n"
>     },
>     "Hostname": "xx",
>     "Logger": "Balrog",
>     "Pid": 8,
>     "Severity": 3,
>     "Timestamp": 1475092806206509056,
>     "Type": "Rules"
> }

Actually, I've got this full traceback in comment #3. The one I'm looking or should have "addLocaleToRelease" in the traceback.
[pid: 8|app: 0|req: 350/359] 127.0.0.1 () {42 vars in 628 bytes} [Wed Sep 28 19:40:44 2016] HEAD /api/csrf_token => generated 0 bytes in 1 msecs (HTTP/1.1 200) 6 headers in 384 bytes (0 switches on core 0)
{"EnvVersion": "2.0", "Type": "auslib.admin.base", "Severity": 3, "Fields": {"message": "Exception on /releases/Thunderbird-45.4.0-build1/builds/Linux_x86-gcc3/en-US [PUT]", "traceback": "Uncaught exception:\n  F
ion=transaction)\n  File \"./auslib/db.py\", line 1530, in addLocaleToRelease\n    transaction=transaction)\n  File \"./auslib/db.py\", line 498, in update\n    return self._prepareUpdate(transaction, where, what
{"EnvVersion": "2.0", "Type": "auslib.admin.base", "Severity": 3, "Fields": {"message": "Caught ISE 500 error.", "requestid": 140372046519120}, "Timestamp": 1475091644615927808, "Logger": "Balrog", "Hostname": "i
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
return self.wsgi_app(environ, start_response)
File "./auslib/admin/base.py", line 37, in __call__
return self.app(environ, start_response)
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1577, in make_response
rv = self.response_class.force_type(rv, request.environ)
File "/usr/local/lib/python2.7/site-packages/werkzeug/wrappers.py", line 847, in force_type
response = BaseResponse(*_run_wsgi_app(response, environ))
File "/usr/local/lib/python2.7/site-packages/werkzeug/test.py", line 871, in run_wsgi_app
app_rv = app(environ, start_response)
TypeError: 'exceptions.IndexError' object is not callable
Rail, I'm hoping to get this in stage today. Can you have a look if you have time?
Attachment #8799754 - Flags: review?(rail)
Attachment #8799754 - Flags: review?(rail) → review+
Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/aa85ffe8361514818e41211c5a283c21641e9ee4
bug 1304082: fix issues with change notifier in production (#142). r=rail
Depends on: 1311036
This is in production and working. Prod sends notifications to https://groups.google.com/a/mozilla.com/forum/#!forum/balrog-db-changes.

Thanks to Ben & others for helping resolve the issues found in stage.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: