739863 - server-full doesn't look up userid if hosted behind apache auth

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Server: Sync, defect)

Description

[Ryan Kelly [:rfkelly]]

If an instance of server-full is hosted behind apache, and apache is set up with do its own authentication, then server-full breaks.

It appears to that server-core will accept the REMOTE_USER variable forwarded by apache, but not look up the corresponding userid in the auth database.  This results in request.user["userid"] equal to None and all kinds of hilarious database-level errors.

This may be worth a fix in server-core, and at the very least needs documenting as a known issue with workaround.  We should also put sanity-checks in server-storage so that a userid of None never makes it down to the database.

Comment 1

[Ryan Kelly [:rfkelly]]

Attachment: patch to error out if passed REMOTE_USER

Another use was bitten by this last week.  I think the most sensible thing to do is to error out if REMOTE_USER is provided by the webserver - it indicates that some other authentication mechanism is also active on this url space, and that's very unlikely to work properly.

Comment 2

[Toby Elliott [:telliott]]

So, after working through this with Ryan, I think that there's a client problem, so I'm adding Nick and Richard.

In order for us to get the REMOTE_USER variable set before auth, it means that apache has had to go through basic-auth. However, we're using basic auth, so we should be consuming the same header as apache and reaching the same place. That was clearly not the case, based on the reported bug - the basic-auth header used the user's username, not a hash of their email.

Here is my best guess as to what happens:

Client sends a perfectly normal sync request, with hash and password

Apache (pre-sync) does a basic-auth lookup. However, it's not using our auth db, so it gets a bad username/password and returns a standard 401.

Firefox gets back a basic-auth request and overrides sync with a cached value in password manager, returning the request with that instead.

That passes apache basic-auth, so REMOTE_USER is set and we receive it. Boom


There's obviously a server bug that needs to be fixed - if there's not a numeric userid we shouldn't get that far - but I think we should be poking at client as well, since the above is the only explanation I've got. If it was a malformed header (two sets of basic auth credentials), it shouldn't go away when apache basic-auth is turned off.

Comment 3

[Richard Newman [:rnewman]]

I seem to recall that user mentioning in passing that they had logged in to their site in Firefox. Necko will certainly use stored credentials: Bug 646628.

Comment 4

[Ryan Kelly [:rfkelly]]

Committed in http://hg.mozilla.org/services/server-core/rev/de19cb96e1f1

Is it worth trying to anything more about this on the server side?  AFAICT this is the only place that a non-numeric userid might sneak into the system.

Comment 5

[Ryan Kelly [:rfkelly]]

*sigh* this patch breaks the metlog testcases in server-storage, since they use REMOTE_USER to circumvent authentication in the tests.  Should be a straightforward fix, but it can wait until I'm back from leave...

Comment 6

[Ryan Kelly [:rfkelly]]

NOTE TO SELF: I have disabled server-storage in jenkins until I get this fixed, this bug must not be closed until it's re-enabled

Comment 7

[Ryan Kelly [:rfkelly]]

Attachment: server-storage patch to avoid using REMOTE_USER in the tests

This fixes the server-storage tests now that environ["REMOTE_USER"] is not supported.

Comment 8

[Ryan Kelly [:rfkelly]]

Committed in http://hg.mozilla.org/services/server-storage/rev/a5d73030e7d1

Jenkins re-enabled for server-storage.  Tests passing.  Let the good times roll.

I don't think there's any more we can do server-side to mitigate this issue, so closing the bug.