1155021 - [basket] Add an exception class to New Relic ignore list

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Paul [:pmac] McLanahan]

Please add "ratelimit.exceptions:Ratelimited" to the New Relic config file for basket under the setting "error_collector.ignore_errors". According to the docs it's a space separated list.

Comment 1

[C. Liang [:cyliang]]

I've added that error to the list of errors for New Relic to ignore, then restarted celery for the dev, stage, and prod bedrock environments.   New Relic looks like it is still receiving information about celery transactions, so I think we're good.

Comment 2

[Paul [:pmac] McLanahan]

Did you also restart the wsgi processes? This particular error shouldn't happen in celery, only in the web app.

Comment 3

[Paul [:pmac] McLanahan]

Sorry. I should have been more specific; it's the webapp that requires this setting change, and not the celery workers. I do still see the exceptions coming through in NR.

Comment 4

[C. Liang [:cyliang]]

It's my bad for jumping to the conclusion that this was celery-related simply because there was a similar request in the past. =)  

As to control the possible ripple effects from doing this, I've tweaked thing in puppet so that the changes I made in generating the New Relic config file are only applied to the cluster holding basket.  I've made the changes to dev, stage, and prod, restarted the New Relic plugin agent, and issued a graceful restart to Apache which hopefully restarted the wsgi processes.

Comment 5

[C. Liang [:cyliang]]

Just a quick ping to see if the exceptions are being correctly blocked.  I don't see any ratelimit.exceptions appearing in New Relic over the past seven days for dev, stage, or prod but I'm not sure if any of these are currently running the code that raises those exceptions.

Comment 6

[Paul [:pmac] McLanahan]

Thanks for the ping. It looks good to me as well, and the code is there and I see the rate limit hits in graphite. Thanks again!