869213 - Enable NewRelic for Reps

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Engagement, task)

Description

[Justin Crawford [:hoosteeno] [:jcrawford]]

Reps dev (reps-dev.allizom.org) has code to support NewRelic, and it will be moving to stage/prod soon.

Please go ahead with the needed server changes and add them to NewRelic panel.

Related bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=860266

Thanks!

Comment 1

[Jake Maul [:jakem]]

Completed! Should be live in 30min or less... if not, let me know... we may need to manually kick Apache, I'm not entirely sure. :)

Comment 2

[Jake Maul [:jakem]]

Had to revert this, it broke reps.mozilla.org. Details to come.

Comment 3

[Jake Maul [:jakem]]

Attachment: reps-newrelic-traceback

Okay so... traceback fun attached.

One working theory is that New Relic, while not in itself dependent on unidecode or django-uuslug, being loaded early in the wsgi file is somehow causing that to break.

:pmac and I have guessed that moving New Relic further down in the wsgi file may alleviate the problem. Perhaps after line 24: https://github.com/mozilla/remo/blob/master/wsgi/playdoh.wsgi#L25. This is a guess, not a guarantee of success.

I can say that, at the system level, neither bedrock nor the generic cluster (basket, mozillians) has a system-level package installed to provide this library. New Relic does not seem to require it (upstream does not indicate it's a dependency, either). I haven't examined vendor-local for this projects, but I suspect we'll find they don't include django-uuslug, and hence don't run into this issue... whatever it is.

Comment 4

[Jake Maul [:jakem]]

CC'ing other folks that helped out with this troubleshooting, in case they have additional insight.

Comment 5

[Jake Maul [:jakem]]

Flagging NEEDINFO, either for additional troubleshooting that can definitively pin down the cause and solution, or confirmation that the theoretical fix is in place and we can install the package and restart apache to test it.

Sadly we cannot test on dev or stage. The Engagement cluster uses 32-bit Seamicro Atom nodes for dev and stage, and thus far I have not been able to package New Relic up to work on them. It's not out of the realm of possibility, I just haven't gotten it to work yet. Bug 788675 is on our plate to replace those nodes.

Comment 6

[Giorgos Logiotatidis [:giorgos]]

Moving newrelic import after 'import manage' as suggested in comment 3, seems to fix the problem. :nemo tested that on a similar to production environment with success.

New code in prod, stage and dev servers.

:jakem can we try enabling NewRelic again?

Comment 7

[Jake Maul [:jakem]]

Seems to be working!

Comment 8

[Giorgos Logiotatidis [:giorgos]]

Can we also get NewRelic for reps-dev.a.o and reps.a.o?

Thanks!

Comment 9

[Jake Maul [:jakem]]

Can't do this until engagement dev and stage are moved off of the Seamicro Atom platform and onto something 64-bit. Blocking on the appropriate bug and triaging to the right component.

Comment 10

[Giorgos Logiotatidis [:giorgos]]

Now that we moved to generic cluster I see reps.a.o and reps-dev.a.o in NR but they are not reporting any data.

Jake can you please take a look?

Comment 11

[Brandon Burton [:solarce]]

I'll look into this later today

Comment 12

[Brandon Burton [:solarce]]

It's getting data now that it's on generic, it just gets so few requests there isn't much data.

By hammering it with curl I am seeing a spike in request rate in https://rpm.newrelic.com/accounts/263620/applications/3650909 , you can see the spike at 14:40 PST in http://note.io/1fFowAo

Same thing on staging, https://rpm.newrelic.com/accounts/263620/applications/3689774 you can see me hammering it at 14:50 PST, http://note.io/1fFpSLt