976101 - https://bounceradmin.allizom.org/api always returns 401

Status: RESOLVED FIXED

(Webtools :: Bouncer, defect)

Description

[Rail Aliiev [:rail]]

When I try to use https://bounceradmin.allizom.org/api I always get:
HTTPError: HTTP Error 401: UNAUTHORIZED

I double checked the credentials, they look correct.

The same code work fine against https://bounceradmin.mozilla.com/api

Ping me on IRC so we can debug this together.

Comment 1

[Laura Thomson :laura]

:rhelmer, can you give Rail a hand?

Comment 2

[Robert Helmer [:rhelmer]]

Hey is it possible for me to get access to the bouncer staging server(s)? I'd like to start debugging this.

Comment 3

[Chris Turra [:cturra]]

as discussed on irc, your account was already able to access the bouncer stage server. woohoo :)

Comment 4

[Robert Helmer [:rhelmer]]

(In reply to Chris Turra [:cturra] from comment #3)
> as discussed on irc, your account was already able to access the bouncer
> stage server. woohoo :)

This works but need a little more help please :) Can I have sudo and/or could you add me to the list of admins in:
https://bounceradmin.allizom.org/admin/

E.g. change to:

ADMINS = (
     ('cturra', '[e-mail]'),
     ('rhelmer', '[e-mail]'),
)

Comment 5

[Robert Helmer [:rhelmer]]

So, good news - I can repro the bug! :)

I was able to give myself an admin account (using django's manage.py) and login to https://bounceradmin.allizom.org/admin

When I am logged in via the browser I can hit the API just fine:
https://bounceradmin.allizom.org/api/mirror_list

But my username/password doesn't work with that same URL in e.g. curl

Poking around in the admin UI, my account seems to have all permissions and be in all groups.
Need to do some further debugging to figure out what's going on.

Comment 6

[Robert Helmer [:rhelmer]]

Oh could you post the production Apache config for the bounceradmin vhost also? And please let me know if there's a .htaccess file in /data/www/download.mozilla.org/tuxedo on prod?

From looking at the code, I can tell what's going on is that there's a header getting set by Apache, HTTP_AUTHORIZATION, which is kind of the old-school way of doing this:

https://github.com/mozilla/tuxedo/blob/master/apps/api/decorators.py#L94

I'm not sure why this isn't just looking directly at the HTTP "Authorization" header, since it looks like it's just expecting the same thing the client puts in there for HTTP Basic auth (base64'd "username:password"), I'd like to just make stage like prod and get it working before trying to understand why :) 

Maybe wenzel remembers ^ (not urgent so not needinfo'ing him, just curious)

Comment 7

[Robert Helmer [:rhelmer]]

Ah I bet we're missing this in stage:
http://modwsgi.readthedocs.org/en/latest/configuration-directives/WSGIPassAuthorization.html

Comment 8

[Chris Turra [:cturra]]

(In reply to Robert Helmer [:rhelmer] from comment #7)
> Ah I bet we're missing this in stage:
> http://modwsgi.readthedocs.org/en/latest/configuration-directives/
> WSGIPassAuthorization.html

`WSGIPassAuthorization On` was in the production apache config, but not stage. aside from that, they look identical. i just committed a change to add `WSGIPassAuthorization` to the stage bouncer apache config. it should be available within 30 minutes.

Comment 9

[Robert Helmer [:rhelmer]]

(In reply to Chris Turra [:cturra] from comment #8)
> (In reply to Robert Helmer [:rhelmer] from comment #7)
> > Ah I bet we're missing this in stage:
> > http://modwsgi.readthedocs.org/en/latest/configuration-directives/
> > WSGIPassAuthorization.html
> 
> `WSGIPassAuthorization On` was in the production apache config, but not
> stage. aside from that, they look identical. i just committed a change to
> add `WSGIPassAuthorization` to the stage bouncer apache config. it should be
> available within 30 minutes.

Awesome! Thanks, I bet that does it.

Comment 10

[Robert Helmer [:rhelmer]]

Hm, not quite yet.. prod must be doing something different here if the code is the same. 

cturra, can I get access to change the code on here? I'd like to do some further debugging.

[Wed Apr 09 20:46:31 2014] [error] ERROR:django.request:Internal Server Error: /api/mirror_list
[Wed Apr 09 20:46:31 2014] [error] Traceback (most recent call last):
[Wed Apr 09 20:46:31 2014] [error]   File "/data/www/download.allizom.org/tuxedo/vendor/lib/python/django/core/handlers/base.py", line 111, in get_response
[Wed Apr 09 20:46:31 2014] [error]     response = callback(request, *callback_args, **callback_kwargs)
[Wed Apr 09 20:46:31 2014] [error]   File "/data/www/download.allizom.org/tuxedo/vendor/lib/python/django/views/decorators/http.py", line 41, in inner
[Wed Apr 09 20:46:31 2014] [error]     return func(request, *args, **kwargs)
[Wed Apr 09 20:46:31 2014] [error]   File "/data/www/download.allizom.org/tuxedo/apps/api/decorators.py", line 42, in wrapper
[Wed Apr 09 20:46:31 2014] [error]     realm, *args, **kwargs)
[Wed Apr 09 20:46:31 2014] [error]   File "/data/www/download.allizom.org/tuxedo/apps/api/decorators.py", line 100, in _view_or_basicauth
[Wed Apr 09 20:46:31 2014] [error]     uname, passwd = base64.b64decode(auth[1]).split(':')
[Wed Apr 09 20:46:31 2014] [error] ValueError: too many values to unpack

Comment 11

[Robert Helmer [:rhelmer]]

(In reply to Chris Turra [:cturra] from comment #8)
> (In reply to Robert Helmer [:rhelmer] from comment #7)
> > Ah I bet we're missing this in stage:
> > http://modwsgi.readthedocs.org/en/latest/configuration-directives/
> > WSGIPassAuthorization.html
> 
> `WSGIPassAuthorization On` was in the production apache config, but not
> stage. aside from that, they look identical. i just committed a change to
> add `WSGIPassAuthorization` to the stage bouncer apache config. it should be
> available within 30 minutes.

Are stage and/or prod using Zeus? I just read that it does something very similar, controlled using the PassEnvAuthorization setting (hides HTTP Authorization header by default, and setting that causes it to populate HTTP_AUTHENTICATION header)

Comment 12

[Chris Turra [:cturra]]

all dev/stage/prod go through zeus and at first glance don't anything specific about setting headers. generally, from what i've seen elsewhere, zeus honors any headers set by apache. 

code pushes are already automated in dev (bounceradmin-dev.allizom.org). currently, there is a cron that deploys from origin/master every 15 minutes in that environment.

Comment 13

[Robert Helmer [:rhelmer]]

(In reply to Chris Turra [:cturra] from comment #12)
> all dev/stage/prod go through zeus and at first glance don't anything
> specific about setting headers. generally, from what i've seen elsewhere,
> zeus honors any headers set by apache. 
> 
> code pushes are already automated in dev (bounceradmin-dev.allizom.org).
> currently, there is a cron that deploys from origin/master every 15 minutes
> in that environment.

Thanks! bounceradmin-dev.allizom.org works fine using the same exact query :( not sure what's different about stage, but it seems to be getting something different in the HTTP_AUTHORIZATION header, somehow. This isn't the kind of thing I'd want to log under normal circumstances, let me take another look around before I start modifying the code.

Comment 14

[Robert Helmer [:rhelmer]]

(In reply to Robert Helmer [:rhelmer] from comment #10)
> [Wed Apr 09 20:46:31 2014] [error]     uname, passwd =
> base64.b64decode(auth[1]).split(':')
> [Wed Apr 09 20:46:31 2014] [error] ValueError: too many values to unpack

nthomas noticed that stage started working for release automation - the reason I got the above failure is because my (randomly-generated) password on stage contained a ':'!

I've filed bug 996408 to followup on that.

cturra, I was able to debug this using the email I am getting from you putting my email into the django settings, thanks!