Closed Bug 1135376 Opened 9 years ago Closed 9 years ago

[alerts] authorization isn't working in production

Categories

(Input Graveyard :: Backend, defect, P1)

Product:
Input Graveyard
Component:
Backend
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: willkg, Assigned: willkg)

References

Details

(Whiteboard: u=analyzer c=alerts p=1 s=input.2015q1) 

Authorization isn't working for posting alerts to production. It works fine on my local machine, but isn't working in production.

HTTP conversation looks ok with the appropriate HTTP header:

    Authorization: Token <TOKEN>

The server returns the appropriate WWW-Authenticate header:

    WWW-Authentication: Token

However, the Fjord code seems to see no Authorization header at all and returns:

    HTTP/1.1 401 UNAUTHORIZED

    {"detail": "Authentication credentials were not provided."}

This bug covers figuring out what's going on and fixing it.
Grabbing this to work on Monday morning.

I suspect the ZLB is doing something with that header--either swallowing it or passing it along with a different header name. To figure that out, we should make the code epically fail (raise Exception) so I get an email with the WSGIRequest in it from the point of view of the input web node. We can push that to -stage since -stage and -prod should be doing the same thing in this regard. That'll shed some light on whether the ZLB is fiddling with things and what I should do about it.

If that's not what's going on, then I'll have to mull.
Assignee: nobody → willkg
Status: NEW → ASSIGNED
I spent some time today on this. It's definitely the ZLB. If I send an "Authorization" header, it doesn't make it to Django running on the webnodes. If I send an "Athorization" (with no u), it makes it just fine.

I skimmed some Zeus docs. I think it's likely there's a PassEnvAuthorization configuration option which is either set to off or left as the default. That'll cause the symptoms we're seeing.

I think our options are these:

1. Open up a webops bug to see if they'll turn this on. I think this will take a long time to happen and since it affects other sites, it might not happen at all.

2. Additionally support a non-standard header. I'm not sure what the ramifications of this might be, but for our purposes, it's probably fine for now.

I'm going to go with item 2 and add support for a new header called Fjord-Authorization. Should be an easy fix.
I just thought: doesn't the admin panel of Input require a HTTP basic auth? I bet Zeus is where that authorization is configured. I can imagine this either interfering with passing tokens through to Fjord, or alternatively, being the reason why Zeus may be configured to not pass authorization.
I have no idea--could be. All I know is that the Authorization header is definitely not making it through.

In a PR: https://github.com/mozilla/fjord/pull/495
Also, isn't it like 6:30am there?
Whiteboard: u=analyzer c=alerts p= s=input.2015q1 → u=analyzer c=alerts p=1 s=input.2015q1
If by "there" you mean Portland, then yes. But I got in to Paris a few hours ago for the BuddyUp work week, so it is more like 3:30pm. I'm just trying to fight off the jet lag.
Ahh... Paris.

I somehow created the PR off of a non-master branch. So I closed #495 and created #496:

https://github.com/mozilla/fjord/pull/496
Pushed this out just now and verified it's working.

Marking as FIXED.
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
As a side note, the commits for this bug were all mis-labeled with bug #1130765. Oops. :(
Product: Input → Input Graveyard
You need to log in before you can comment on or make changes to this bug.