1407980 - [antenna] log something when can't HEAD bucket

Status: RESOLVED WORKSFORME

(Socorro :: Antenna, task)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

When Antenna starts up, the AntennaAPI is created, which creates a BreakpadSubmitterResource which creates a S3CrashStorage which creates a S3Connection.

When the S3Connection is created, the first thing it does is try to verify its configuration by HEADing the S3 bucket. If that fails, then it throws an exception that bubbles up all the way to the top, but never gets printed out.

Thus you end up with mysterious logs like this:

"""
Attaching to socorro_antenna_1
antenna_1        | [2017-10-12 12:30:08 +0000] [7] [INFO] Starting gunicorn 19.7.1
antenna_1        | [2017-10-12 12:30:08 +0000] [7] [INFO] Listening at: http://0.0.0.0:8000 (7)
antenna_1        | [2017-10-12 12:30:08 +0000] [7] [INFO] Using worker: gevent
antenna_1        | [2017-10-12 12:30:08 +0000] [10] [INFO] Booting worker with pid: 10
antenna_1        | {"Timestamp": 1507811408511663872, "Type": "antenna.app", "Logger": "antenna", "Hostname": "e3a8ffd5765d", "EnvVersion": "2.0", "Severity": 6, "Pid": 10, "Fields": {"host_id": "e3a8ffd5765d", "message": "BASEDIR=/app"}}
antenna_1        | {"Timestamp": 1507811408511835136, "Type": "antenna.app", "Logger": "antenna", "Hostname": "e3a8ffd5765d", "EnvVersion": "2.0", "Severity": 6, "Pid": 10, "Fields": {"host_id": "e3a8ffd5765d", "message": "LOGGING_LEVEL=INFO"}}
antenna_1        | {"Timestamp": 1507811408511947520, "Type": "antenna.app", "Logger": "antenna", "Hostname": "e3a8ffd5765d", "EnvVersion": "2.0", "Severity": 6, "Pid": 10, "Fields": {"host_id": "e3a8ffd5765d", "message": "METRICS_CLASS=antenna.metrics.LoggingMetrics"}}
antenna_1        | {"Timestamp": 1507811408512032000, "Type": "antenna.app", "Logger": "antenna", "Hostname": "e3a8ffd5765d", "EnvVersion": "2.0", "Severity": 6, "Pid": 10, "Fields": {"host_id": "e3a8ffd5765d", "message": "SECRET_SENTRY_DSN="}}
antenna_1        | {"Timestamp": 1507811408512110336, "Type": "antenna.app", "Logger": "antenna", "Hostname": "e3a8ffd5765d", "EnvVersion": "2.0", "Severity": 6, "Pid": 10, "Fields": {"host_id": "e3a8ffd5765d", "message": "HOST_ID="}}
antenna_1        | {"Timestamp": 1507811408512769792, "Type": "antenna.app", "Logger": "antenna", "Hostname": "e3a8ffd5765d", "EnvVersion": "2.0", "Severity": 6, "Pid": 10, "Fields": {"host_id": "e3a8ffd5765d", "message": "Setting up metrics: [<class 'antenna.metrics.LoggingMetrics'>]"}}
antenna_1        | [2017-10-12 12:30:08 +0000] [7] [INFO] Shutting down: Master
antenna_1        | [2017-10-12 12:30:08 +0000] [7] [INFO] Reason: Worker failed to boot.
socorro_antenna_1 exited with code 3
"""

This bug covers fixing it so that if get_app() throws an exception, something is logged.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

This situation is probably not as mysterious in server environments because if we're missing an S3 bucket, other things will yell at us loudly. It bit me just now when trying to get Antenna running in a local dev environment.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 4041 - use dev script to run antenna as a workaround

Comment 3

[[github robot]]

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/ab78d049a10c3bdfcd2665d2945f3d4b9e09cba6
bug 1407980 - use antenna dev script to alleviate missing bucket issue

https://github.com/mozilla-services/socorro/commit/7fd81adf4f2d4ac1a3070ce4388e3a44a133282d
Merge pull request #4041 from willkg/1407980-antenna-fix

bug 1407980 - use antenna dev script to alleviate missing bucket issue

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

This is no longer an issue. We redid how verification works so that it tries to upload a "test" file and if that fails, it errors and logs pretty loudly what's going on.

That was done in bug #1359147.

Marking as WORKSFORME.