Last Comment Bug 1304082 - enable change notifier for balrog admin app
: enable change notifier for balrog admin app
Status: RESOLVED FIXED
:
Product: Cloud Services
Classification: Client Software
Component: Operations: Deployment Requests (show other bugs)
: unspecified
: Unspecified Unspecified
-- normal (vote)
: ---
Assigned To: Ben Hearsum (:bhearsum)
:
:
Mentors:
Depends on: 1305703 1311036
Blocks: 1246675
  Show dependency treegraph
 
Reported: 2016-09-20 09:30 PDT by Ben Hearsum (:bhearsum)
Modified: 2016-10-19 12:16 PDT (History)
3 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
fix issues with change notifier (42 bytes, text/x-github-pull-request)
2016-10-11 07:03 PDT, Ben Hearsum (:bhearsum)
rail: review+
Details | Review | Splinter Review

Description User image Ben Hearsum (:bhearsum) 2016-09-20 09:30:20 PDT
This was requested way back when we originally talked about the migration. I ended up implementing support for it in https://bugzilla.mozilla.org/show_bug.cgi?id=1251338, but we didn't enable it. It should work if you set the following environment variables:
SMTP_HOST
SMTP_PORT
NOTIFY_TO_ADDR
NOTIFY_FROM_ADDR
SMTP_USERNAME (optional)
SMTP_PASSWORD (optional)

NOTIFY_TO_ADDR should be balrog-db-changes@mozilla.com. We should probably test it out in stage first, but it would be best if it were disabled there once we know it works (too spammy).
Comment 1 User image Ben Hearsum (:bhearsum) 2016-09-21 12:45:58 PDT
We decided that we should have a separate list for stage, to make sure we don't bust it in the future. I created junk-balrog-db-changes@mozilla.com for this.
Comment 2 User image Ben Hearsum (:bhearsum) 2016-09-28 09:37:28 PDT
We tried enabling this in stage today, and got an exception:
{"EnvVersion": "2.0", "Type": "Rules", "Severity": 3, "Fields": {"message": "Failed to connect to SMTP server:", "traceback": "Uncaught exception:\n  File \"./auslib/db.py\", line 1790, in send_email\n    conn.starttls()\n  File \"/usr/local/lib/python2.7/smtplib.py\", line 644, in starttls\n    raise SMTPException(\"STARTTLS extension not supported by server.\")\n<class 'smtplib.SMTPException'>\nSMTPException('STARTTLS extension not supported by server.',)\n", "requestid": 140010317872208, "error": "SMTPException('STARTTLS extension not supported by server.',)"}, "Timestamp": 1475080251884000000, "Logger": "Balrog", "Hostname": "ip-172-31-48-255", "Pid": 8}

Looks like I'll need to tweak the code to make it possible to disable starttls.
Comment 3 User image Ben Hearsum (:bhearsum) 2016-09-28 13:27:06 PDT
We also hit another error, which happens prior to the smtp connection in certain cases:
ion=transaction)\n  File \"./auslib/db.py\", line 1530, in addLocaleToRelease\n    transaction=transaction)\n  File \"./auslib/db.py\", line 498, in update\n    return self._prepareUpdate(transaction, where, what, changed_by, old_data_version)\n  File \"./auslib/db.py\", line 456, in _prepareUpdate\n    self.onUpdate(self, \"UPDATE\", changed_by, query)\n  File \"./auslib/db.py\", line 1838, in bleet\n    row = table.sel$

I'll steal this bug for now while I work out fixes for both issues.
Comment 4 User image Ben Hearsum (:bhearsum) 2016-10-07 06:54:24 PDT
(In reply to Ben Hearsum (:bhearsum) from comment #3)
> We also hit another error, which happens prior to the smtp connection in
> certain cases:
> ion=transaction)\n  File \"./auslib/db.py\", line 1530, in
> addLocaleToRelease\n    transaction=transaction)\n  File \"./auslib/db.py\",
> line 498, in update\n    return self._prepareUpdate(transaction, where,
> what, changed_by, old_data_version)\n  File \"./auslib/db.py\", line 456, in
> _prepareUpdate\n    self.onUpdate(self, \"UPDATE\", changed_by, query)\n 
> File \"./auslib/db.py\", line 1838, in bleet\n    row = table.sel$
> 
> I'll steal this bug for now while I work out fixes for both issues.

Ben - any chance you can dig up the full traceback for this one? It looks like the pastebin this came from has expired. It should be from around 4pm eastern on the 28th.
Comment 5 User image Benson Wong [:mostlygeek] 2016-10-07 09:57:51 PDT
This is the long line you are looking for...

{
    "EnvVersion": "2.0",
    "Fields": {
        "error": "SMTPException('STARTTLS extension not supported by server.',)",
        "message": "Failed to connect to SMTP server:",
        "requestid": 140372048409168,
        "traceback": "Uncaught exception:\n  File \"./auslib/db.py\", line 1790, in send_email\n    conn.starttls()\n  File \"/usr/local/lib/python2.7/smtplib.py\", line 644, in starttls\n    raise SMTPException(\"STARTTLS extension not supported by server.\")\n<class 'smtplib.SMTPException'>\nSMTPException('STARTTLS extension not supported by server.',)\n"
    },
    "Hostname": "xx",
    "Logger": "Balrog",
    "Pid": 8,
    "Severity": 3,
    "Timestamp": 1475092806206509056,
    "Type": "Rules"
}
Comment 6 User image Ben Hearsum (:bhearsum) 2016-10-07 10:19:21 PDT
(In reply to Benson Wong [:mostlygeek] from comment #5)
> This is the long line you are looking for...
> 
> {
>     "EnvVersion": "2.0",
>     "Fields": {
>         "error": "SMTPException('STARTTLS extension not supported by
> server.',)",
>         "message": "Failed to connect to SMTP server:",
>         "requestid": 140372048409168,
>         "traceback": "Uncaught exception:\n  File \"./auslib/db.py\", line
> 1790, in send_email\n    conn.starttls()\n  File
> \"/usr/local/lib/python2.7/smtplib.py\", line 644, in starttls\n    raise
> SMTPException(\"STARTTLS extension not supported by server.\")\n<class
> 'smtplib.SMTPException'>\nSMTPException('STARTTLS extension not supported by
> server.',)\n"
>     },
>     "Hostname": "xx",
>     "Logger": "Balrog",
>     "Pid": 8,
>     "Severity": 3,
>     "Timestamp": 1475092806206509056,
>     "Type": "Rules"
> }

Actually, I've got this full traceback in comment #3. The one I'm looking or should have "addLocaleToRelease" in the traceback.
Comment 7 User image Benson Wong [:mostlygeek] 2016-10-07 13:55:42 PDT
[pid: 8|app: 0|req: 350/359] 127.0.0.1 () {42 vars in 628 bytes} [Wed Sep 28 19:40:44 2016] HEAD /api/csrf_token => generated 0 bytes in 1 msecs (HTTP/1.1 200) 6 headers in 384 bytes (0 switches on core 0)
{"EnvVersion": "2.0", "Type": "auslib.admin.base", "Severity": 3, "Fields": {"message": "Exception on /releases/Thunderbird-45.4.0-build1/builds/Linux_x86-gcc3/en-US [PUT]", "traceback": "Uncaught exception:\n  F
ion=transaction)\n  File \"./auslib/db.py\", line 1530, in addLocaleToRelease\n    transaction=transaction)\n  File \"./auslib/db.py\", line 498, in update\n    return self._prepareUpdate(transaction, where, what
{"EnvVersion": "2.0", "Type": "auslib.admin.base", "Severity": 3, "Fields": {"message": "Caught ISE 500 error.", "requestid": 140372046519120}, "Timestamp": 1475091644615927808, "Logger": "Balrog", "Hostname": "i
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
return self.wsgi_app(environ, start_response)
File "./auslib/admin/base.py", line 37, in __call__
return self.app(environ, start_response)
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1577, in make_response
rv = self.response_class.force_type(rv, request.environ)
File "/usr/local/lib/python2.7/site-packages/werkzeug/wrappers.py", line 847, in force_type
response = BaseResponse(*_run_wsgi_app(response, environ))
File "/usr/local/lib/python2.7/site-packages/werkzeug/test.py", line 871, in run_wsgi_app
app_rv = app(environ, start_response)
TypeError: 'exceptions.IndexError' object is not callable
Comment 8 User image Ben Hearsum (:bhearsum) 2016-10-11 07:03:32 PDT
Created attachment 8799754 [details] [review]
fix issues with change notifier

Rail, I'm hoping to get this in stage today. Can you have a look if you have time?
Comment 9 User image [github robot] 2016-10-13 07:52:29 PDT
Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/aa85ffe8361514818e41211c5a283c21641e9ee4
bug 1304082: fix issues with change notifier in production (#142). r=rail
Comment 10 User image Ben Hearsum (:bhearsum) 2016-10-19 12:16:16 PDT
This is in production and working. Prod sends notifications to https://groups.google.com/a/mozilla.com/forum/#!forum/balrog-db-changes.

Thanks to Ben & others for helping resolve the issues found in stage.

Note You need to log in before you can comment on or make changes to this bug.