728658 - (http-511) Handle HTTP error 511 Network Authentication Required (RFC 6585: standard secure proxy authentification/captive portal detection)

Status: RESOLVED WONTFIX

(Core :: Networking: HTTP, enhancement)

Description

[Nicolas Mailhot]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.1) Gecko/20100101 Firefox/10.0.1
Build ID: 20120209081558

Steps to reproduce:

Since bug #479880#c2

there is no clean way for a proxy or captive portal to get a browser to display an authentication dialog when user credentials expire while he is browsing on an https website.

(to be sure, the previous methods were insecure and hackish but they existed because nothing better was available)

The IETF finally set up to fix this problem and defined a standard HTTP error that let access control equipments tell the browser authentication or re-authentication is needed and where the authentication form is located.

http://tools.ietf.org/id/draft-nottingham-http-new-status-04.txt

(since error 511 uses out-of-band authentication it is possible for the browser to only trust specific certs on error 511 and protect the user)

Please add error 511 handling in Firefox, or have the ietf draft corrected if it's missing somethin

Comment 2

[Nicolas Mailhot]

Probably needs at least a transient bar telling the user something requested authentication to continue accessing the URL he just typed and (maybe) does he want to trust the cert presented by the captive portal in the future?

Comment 3

[Nicolas Mailhot]

As noted by Julian Reschke in bug #562917#c21

-> <http://www.rfc-editor.org/queue2.html#draft-nottingham-http-new-status> (so the spec is approved and in the queue for publication as RFC)

Comment 4

[Curtis Koenig [:curtisk-use curtis.koenig+bzATgmail.com]]]

sec traige, waiting for action

Comment 5

[Nicolas Mailhot]

A user-frinedly workflow would be:
1. on error 511 receival open a new tab, and focus it
2. let the user complete captive portal log-in, 
3. propose to trust (or not) the captive portal certificate later
4. return to the initial tab and retry the request that triggered the error 511 once authentication on the captive portal succeeded

(the captive portal can not issue redirects to do all this transparently since the browser won't honor them on https web sites anymore, so all the switching to and from the captive portal page needs to be managed by the browser now)

Comment 6

[Jesse Ruderman]

In reply to comment 0:

> there is no clean way for a proxy or captive portal to get a browser to display 
> an authentication dialog when user credentials expire while he is browsing on an 
> https website.

> The IETF finally set up to fix this problem

The RFC says nothing of the sort:

>  However, these risks are not unique to the 511 status code; in other
>  words, a captive portal that is not using this status code introduces
>  the same issues.

>  Also, note that captive portals using this status code on an SSL or
>  TLS connection (commonly, port 443) will generate a certificate error
>  on the client.

Doing something "useful" with 511-over-MITMed-SSL would mean a huge increase in attack surface:

* We'd have to poke a hole all the way through our TLS stack to even see the 511.

* We'd have to isolate the MITM page, both technically (e.g. cookies) and in the user interface. Maybe it could build on the work in bug 463027.

* Network attackers would have a chance to use browser exploits against users who are careful to stick with HTTPS.

I think operating systems are in a better position to help here.  They know which wifi network you're connected to, and can reject this kind of hostility on networks that were previously not hostile.  They can help even if you're playing a game that doesn't use HTTP.  For example, Mac OS X 10.7 tries connecting to http://www.apple.com/ and opens a special window if it doesn't get what it expects.

Comment 7

[Nicolas Mailhot]

(In reply to Jesse Ruderman from comment #6)

> I think operating systems are in a better position to help here.  They know
> which wifi network you're connected to, and can reject this kind of
> hostility on networks that were previously not hostile.

What you describe does not and can not work except on the simplest captive portal possible. The IETF spec is much nicer than the various approaches you propose  as it can be used to implement authentication on proxies in a reasonably standard and device-agnostic way (without requiring every possible web client to be in a single Active Directory domain, at which point you can as well use MS products everywhere).

Contrary to the other sorts of captive portals, a corporate proxy:
1. won't be accessed wirelessly only. 
2. won't try to hijack the whole DNS
3. won't block access to Intranet resources. It's not between the browser and the network, but between the internal and external network.
4. won't block access to every possible Internet resource (only those the corporate policy says is restricted to specific classes of users that need to authenticate)
5. will have some authentication expiry time: you will need to reconnect after a while (unlike most hotels eternal credentials)
6. won't be unique. Depending on the internal network size the load may be shared by several proxies, ideally placed on several physical sites (to avoid loss of connectivity in case of one physical site going down)

A consequence of 5. is that the proxy won't use a special private IP address (gateways on different physical sites won't even be in the same network ranges), and that auth may not be shared across sites (so the same URL can be freely accessible through proxy farm of site A, where the client already authenticated, and not proxy farm of site B, where it has not authenticated yet).

Therefore it is pretty much useless to hope to detect reliably the proxy by accessing another http URL in case of problems and hoping to get a captive portal. The other URL may not be filtered, or the browser/network stack may decide to go through a path where it's already authenticated, etc

This kind of setup worked fine in the past since IE6 was single-threaded, tried very hard to always pass through the same gateway IPs (leading to user frustration when the chosen path had availability problems and IE wouldn't switch to another), and didn't refuse to redirect https accesses.

A modern browser is much more likely to hit authentication requests, won't accept https redirections, so I don't see how it could be sanely handled except by design similar to the one the ietf specified. Please support that spec in firefox.

> The RFC says nothing of the sort:

The RFC is overly careful and self-deprecating.

> * We'd have to isolate the MITM page, both technically (e.g.
> cookies) and in the user interface.

But the whole point of the RFC is that as a browser vendor YOU CAN ISOLATE THE PORTAL PAGE and IT IS NOT TRYING TO PERFORM A MITM.

The portal is not trying to trick the browser in masquerading as something else, it declares to the browser what it is, and the browser is free to protect the user by wrapping the portal pages in all the warning chrome it wants, and not sending any session data to the page the error 511 redirects to.

So the portal is trying to play nice.

If playing nice is not supported properly by browsers, captive portals will continue to use trickery, going as far as emit fake certificates for the web site the user requested just to get display their login page (a mode which was the first suggestion of our proxy provider and you're proving it right here).

Do you really want proxy and other captive portal users to start breaking ssl sessions by default because you're leaving them no other choice?

Comment 8

[Jesse Ruderman]

> The RFC is overly careful and self-deprecating.

The entire "Security considerations" portion of the RFC is based on the assumption that it does not modify SSL.

Comment 9

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Jesse Ruderman from comment #6)
> Doing something "useful" with 511-over-MITMed-SSL would mean a huge increase
> in attack surface:
> 
> * We'd have to poke a hole all the way through our TLS stack to even see the
> 511.

I imagine it would work something like this:

1. We do the TLS handshake and verify the certificate.
2. The handshake fails with some kind of error (handshake timeout, OCSP timeout, untrusted issuer during cert validation).
3. We determine, from that error, whether we think we should try to detect the captive portal. If so, we issue a request to captive-portal-test-mozilla.org. If that response comes back as a 511, or with a wispr response, or some other indication that we're in a captive portal, then we kick into captive portal mode.

> * We'd have to isolate the MITM page, both technically (e.g. cookies) and in
> the user interface. Maybe it could build on the work in bug 463027.

That approach is too risky. It is safer to make a request to a dedicated domain with no cookies and no HTTP auth.

> I think operating systems are in a better position to help here.  They know
> which wifi network you're connected to, and can reject this kind of
> hostility on networks that were previously not hostile.  They can help even
> if you're playing a game that doesn't use HTTP.  For example, Mac OS X 10.7
> tries connecting to http://www.apple.com/ and opens a special window if it
> doesn't get what it expects.

Right, this is pretty much what I think we should do.

Comment 10

[Nicolas Mailhot]

(In reply to Jesse Ruderman from comment #8)

> 3. We determine, from that error, whether we think we should try to detect
> the captive portal. If so, we issue a request to
> captive-portal-test-mozilla.org. If that response comes back as a 511, or
> with a wispr response, or some other indication that we're in a captive
> portal, then we kick into captive portal mode.

As I explained in comment #7 that won't work as :
1. you're still assuming there is a single captive portal on the network and the first portal you find will be the one the user needs to authenticate at to reach the URL he tried to access

2. you're still assuming all URLs are treated the same by network gateways and that a request on your specific test URL will trigger the same reaction from the captive portal

3. you're still assuming the need-to-authenticate request will only occur only during the initial negotiation phase, and that the user portal auth will never expire and he'll never need to reauth in the middle of some browsing on an https website (which is false too both because load-balancing may move the user to another gateway at some point, and because even if he stayed on the same equipment forever and the equipment never needed resetting for maintenance any sane operator will require re-auth after a while to prevent permanent ssl tunneling)

The 511 error page itself won't be crypted however it most certainly will auto-refresh and redirect to an https captive portal as per the rfc example

> For example, Mac OS X 10.7
> tries connecting to http://www.apple.com/ and opens a special
> window if it doesn't get what it expects.

This is completely dumb in a corporate context as the user may be prevented from reaching apple but that does not mean he can not use Intranet apps.

The url windows uses for the same purpose is blacklisted on our corporate network as windows was overly keen to loop testing it and was just overloading network infrastructure for no good reason at all. So it will not succeed whether the user is authenticated on the proxy or not.

This kind of solution only works in an hotel room.

Comment 11

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Nicolas Mailhot from comment #10)
> 2. you're still assuming all URLs are treated the same by network gateways
> and that a request on your specific test URL will trigger the same reaction
> from the captive portal

AFAICT, there's no way we're going to send the request (even without cookies and HTTP auth) through a TLS connection where the certificate verification failed. But, we have to make *some* request to get the 511 response.

> 3. you're still assuming the need-to-authenticate request will only occur
> only during the initial negotiation phase, and that the user portal auth
> will never expire and he'll never need to reauth in the middle of some
> browsing on an https website

No, I am not.

If we're making an HTTPS connection, then we must get a valid handshake, including in particular a trusted certificate, from the connection, BEFORE we send ANY request. And, the captive portal requires us to send it a request BEFORE it can return the 511 response.

> This is completely dumb in a corporate context as the user may be prevented
> from reaching apple but that does not mean he can not use Intranet apps.

For non-HTTPS connections, we could get the 511 and process it like you expect. The problem is HTTPS connections; there's no way to get the 511 response because we would never send any request on the connection.

> The url windows uses for the same purpose is blacklisted on our corporate
> network as windows was overly keen to loop testing it and was just
> overloading network infrastructure for no good reason at all. So it will not
> succeed whether the user is authenticated on the proxy or not.

Interesting point. It seems like there should be a complementary (to 511) status code indicating this situation.

> This kind of solution only works in an hotel room.

Hotel rooms, airports, coffee shops, and similar places are the primary motivation for this feature. If there are other use cases to consider, let's consider them. But, I don't think we should avoid implementing a solution for the most common cases just because there are a few (or even many) cases where it wouldn't work.

Comment 12

[Nicolas Mailhot]

(In reply to Brian Smith (:bsmith) from comment #11)
> (In reply to Nicolas Mailhot from comment #10)
> > 2. you're still assuming all URLs are treated the same by network gateways
> > and that a request on your specific test URL will trigger the same reaction
> > from the captive portal
> 
> AFAICT, there's no way we're going to send the request (even without cookies
> and HTTP auth) through a TLS connection where the certificate verification
> failed.

But no one is asking you to send the request though the TLS connection where the certificate verification failed.

Look, the workflow is the following:

1. user requests an URL (TLS or not) that needs going through a gateway were he is not authenticated or no longer authenticated

2. the gateway emits an error 511 with some html code pointing to the authenticating portal it uses

3. the browser receives the 511. It may or may not cause a certificate failure (depending on if the request in 1. was crypted or not). If you try to process the 511 in the same connection yes that will be a security breach. But you don't need and indeed should not process it in the same connection. Because the fact it's a 511 tells you the error is not part of the dialog between the browser and the target web site, but a request to start a parallel dialog with an intermediary hop authentication gateway. So the correct way to proceed is to open a new browser tab, with the error 511 page, and no other data from the connection that triggered the 511

You may eventually leak some state because this error page corresponds to a specific URL request, but that's how authentication works. If you need to open 

> But, we have to make *some* request to get the 511 response.

Unfortunately, the way networks can be set up, you can not be sure another request will trigger the same 511. You have to use the 511 you already received.

 
> > 3. you're still assuming the need-to-authenticate request will only occur
> > only during the initial negotiation phase, and that the user portal auth
> > will never expire and he'll never need to reauth in the middle of some
> > browsing on an https website
> 
> No, I am not.
> 
> If we're making an HTTPS connection, then we must get a valid handshake,
> including in particular a trusted certificate, from the connection, BEFORE
> we send ANY request. And, the captive portal requires us to send it a
> request BEFORE it can return the 511 response.
> 
> > This is completely dumb in a corporate context as the user may be prevented
> > from reaching apple but that does not mean he can not use Intranet apps.
> 
> For non-HTTPS connections, we could get the 511 and process it like you
> expect. The problem is HTTPS connections; there's no way to get the 511
> response because we would never send any request on the connection.
> 
> > The url windows uses for the same purpose is blacklisted on our corporate
> > network as windows was overly keen to loop testing it and was just
> > overloading network infrastructure for no good reason at all. So it will not
> > succeed whether the user is authenticated on the proxy or not.
> 
> Interesting point. It seems like there should be a complementary (to 511)
> status code indicating this situation.
> 
> > This kind of solution only works in an hotel room.
> 
> Hotel rooms, airports, coffee shops, and similar places are the primary
> motivation for this feature. If there are other use cases to consider, let's
> consider them. But, I don't think we should avoid implementing a solution
> for the most common cases just because there are a few (or even many) cases
> where it wouldn't work.

Comment 13

[Nicolas Mailhot]

(In reply to Brian Smith (:bsmith) from comment #11)
> (In reply to Nicolas Mailhot from comment #10)
> > 2. you're still assuming all URLs are treated the same by network gateways
> > and that a request on your specific test URL will trigger the same reaction
> > from the captive portal
> 
> AFAICT, there's no way we're going to send the request (even without cookies
> and HTTP auth) through a TLS connection where the certificate verification
> failed.

But no one is asking you to send the request though the TLS connection where the certificate verification failed.

Look, the workflow is the following:

1. user requests an URL (TLS or not) that needs going through a gateway were he is not authenticated or no longer authenticated

2. the gateway emits an error 511 with some html code pointing to the authenticating portal it uses

3. the browser receives the 511. It may or may not cause a certificate failure (depending on if the request in 1. was crypted or not). If you try to process the 511 in the same connection yes that will be a security breach. But you don't need and indeed should not process it in the same connection. Because the fact it's a 511 tells you the error is not part of the dialog between the browser and the target web site, but a request to start a parallel dialog with an intermediary hop authentication gateway. So the correct way to proceed is to open a new browser tab, with the error 511 page, and no other data from the connection that triggered the 511

You may eventually leak some state because this error page corresponds to a specific URL request, but that's how authentication works. If you need to open a safe in Switzerland you have to divulge the safe number you want to open, and asking for a generic safe number to be let in the safe room won't work.

Now if 511 returned the location of the authentication portal instead of requiring the browser to process a whole HTML page with gods no what inside, things may be marginally easier and safer for you to process (non-browser http clients already complained at the ietf they could not process a whole html page just to scrap this location and display it to the user). But that's not how the rfc is written now. 

Though I'm sure the ietf would be more than happy to amend the rfc if a major browser like Firefox told them how network gateways can tell browsers they need to authenticate at a particular location (and committed to implement its part of the proposal), on any browser request (not just the simple case of first request or http-only request). IIRC the whole reason it has no location parameter is that it would look like a redirect and browser people make it understood they didn't want redirections of existing connections.

> But, we have to make *some* request to get the 511 response.

Unfortunately, the way networks can be set up, you can not be sure another request will trigger the same 511. You have to use the 511 you already received.
 
> > 3. you're still assuming the need-to-authenticate request will only occur
> > only during the initial negotiation phase, and that the user portal auth
> > will never expire and he'll never need to reauth in the middle of some
> > browsing on an https website
> 
> No, I am not.
> 
> If we're making an HTTPS connection, then we must get a valid handshake,
> including in particular a trusted certificate, from the connection, BEFORE
> we send ANY request. And, the captive portal requires us to send it a
> request BEFORE it can return the 511 response.

Again you assume a simplistic scenario.

If the user already authenticated fro browsing somewhere else, his authentication may still be valid so he won't get a 511 when negociating an https connection with another web site

If the gateway is reseted or if the load balancing moves the user from one gateway to another mid-browsing, the 511 will occur mid-session after the initial handshake.

You can not choose when the 511 will occur, and you can not make is so it occurs only over http.

And actually http is utterly uninteresting because the gateway can issue a redirect instead of a 511 now and the browser will happily execute it. The *only* interesting and useful case for 511 is how to handle https connections (including mid-session gateway re-auth)

> Hotel rooms, airports, coffee shops, and similar places are the primary
> motivation for this feature. If there are other use cases to consider, let's
> consider them. 

There are other use cases such as corporate networks (and a large part of the population spends most of its waken time on a corp network) and any big event (sports, fair, conference) where the host needs to provide network connectivity to a large number of unmanaged web clients but this connectivity is restricted to avoid legal problems or avoid that some bad apples pump all the bandwidth on items which have nothing to do with the event

> But, I don't think we should avoid implementing a solution
> for the most common cases just because there are a few (or even
> many) cases where it wouldn't work.

Handling the general case would handle both the simple and complex cases.

Most of the simple cases in your reduced scope don't need handling as redirects work just fine on http connections.

Refusing to handle the general case will only make people that need the general case go DPI and issue fake certs (in fact, this has already started; the information control infrastructure in police states was not developed for political control reasons it's a simple re-purposing of techniques developed for toll barriers and other captive portals). Because once you've set up this kind of infrastructure it is very easy to abuse — it's powerful enough to do much more than get a browser to display an auth page.

We arrived to this unpleasant situation because browser people never tried to tackle the captive portal case properly so everyone has to workaround browsers to make this use case work. Can we please stop the arms race now? Years of denials didn't make captive portals go away, just made lots of people miserable because their browser could not navigate the captive portals that were set up without browser cooperation.

BTW: last I've seen there was about the same number of people on our corp network than in a small country. You need to handle a lot of **** hotel portals to get close to it.

Comment 14

[Jesse Ruderman]

Another security problem with this proposal is that it allows any web site to open new windows/tabs (popups) without user interaction.  I might be okay with malicious wifi operators being able to do that, but not malicious web sites.

Comment 15

[Nicolas Mailhot]

So just add the usual white-listing error bar (as noted in comment #2)

Comment 16

[Nicolas Mailhot]

(In reply to Nicolas Mailhot from comment #15)
> So just add the usual white-listing error bar (as noted in comment #2)

(actually, no you can't because at this point you don't know the portal location to whitelist it, and you don't know the portal location because error 511 has no location target, and it has no location target because browser people didn't want to think it out and made the ietf define something that didn't look like a redirect. Oops)

Comment 17

[Julian Reschke]

(In reply to Nicolas Mailhot from comment #16)
> ...
> error 511 has no location target, and it has no location target because
> browser people didn't want to think it out and made the ietf define
> something that didn't look like a redirect. Oops)

Dunno what you're referring to here. I believe all the relevant history for the spec can be found in the mailing list archives (ietf-http and apps-discuss, methinks), and I don't think any requests like that ever were made.

Comment 18

[Nicolas Mailhot]

(In reply to Julian Reschke from comment #17)
> (In reply to Nicolas Mailhot from comment #16)
> > ...
> > error 511 has no location target, and it has no location target because
> > browser people didn't want to think it out and made the ietf define
> > something that didn't look like a redirect. Oops)
> 
> Dunno what you're referring to here. I believe all the relevant history for
> the spec can be found in the mailing list archives 

I think I was (mis?)remembering 
http://lists.w3.org/Archives/Public/ietf-http-wg/2011OctDec/0071.html

Anyway if Firefox devs are worrying about just anyone using 511 to trigger new tabs, they will want to provide the user a mean to whitelist 'good' 511 pages, and a 'good' 511 page is pretty much defined by the portal it will want the user to display, except this target is pretty hard to identify in the absence of a location header

So it seems browsers would have a use for a location info just like non-browser web clients

Comment 19

[Mark Nottingham]

The purpose of 511 as specified is to avoid having captive portal pages mistaken for content from the origin server, NOT to facilitate smoother login. I.e., it's there to limit the damage of captive portals (which we tried to make clear in the draft).

511 impacts a browser like Mozilla in a couple of ways --
  * It shouldn't cache .favicons that have a 511
  * It should consider a feed that returns 511 as temporarily unavailable, not corrupt (or authoritative)
  * It should retry automated downloads (like updates) later if receiving a 511

and so on. All of these *should* be the default actions anyway, based upon the generic HTTP semantics.

It might be possible to build better portal authentication / UX on top of 511, but that definitely isn't provided merely by the status code, and it wasn't in scope for this effort.

Hope this helps,

Comment 20

[Curtis Koenig [:curtisk-use curtis.koenig+bzATgmail.com]]]

still waiting

Comment 21

[Nicolas Mailhot]

(In reply to Brian Smith (:bsmith) from comment #11)

> AFAICT, there's no way we're going to send the request (even without cookies
> and HTTP auth) through a TLS connection where the certificate verification
> failed. But, we have to make *some* request to get the 511 response.

https://datatracker.ietf.org/doc/draft-ietf-httpbis-p1-messaging/?include_text=1 explicitly states (§ 2.7.2.  https URI scheme) that http and https are two different namespaces

“Resources made available via the "https" scheme have no shared identity with the "http" scheme even if their resource identifiers indicate the same authority”

Therefore you can not reliably downgrade an https connexion to http to reproduce errors received following an https request, to avoid having to handle them in tls context (and that is also how real-world equipments work)

Comment 22

[Honza Bambas (:mayhemer)]

Might be out of scope of this bug (511 response): Isn't there something in TLS, an extension we can define or just use, that can the proxy server use to identify it's not the target server before it sends the certificate response?

Like for HTTP it will use 511, for HTTPS (i.e. TLS+HTTP) it would use this new mechanism.

For HTTPS, there is no solution so far, as I can tell.  I think IETF or TLS group might want to define an extension for this.

Comment 23

[Yvan Boily [:ygjb][:yvan]]

Is there an ETA for when this might be implemented?  Are we still waiting on the RFC?

Comment 24

[Julian Reschke]

(In reply to Yvan Boily [:ygjb][:yvan] from comment #23)
> (...)  Are we still waiting on
> the RFC?

It's RFC 6585.

Comment 25

[Tej]

Has the 511 been implemented on Firefox yet? If not, what would be the ETA?

Comment 26

[Cork]

The bug isn't resolved so how could it be implemented? The standard isn't even completely hammered out yet.

Comment 27

[Tej]

Exactly, but I was implementing Transparent Proxy Authentication via Squid, and the latest Squid-3.2 does give me an option of a 511 like page for authentication to be sent.

Comment 28

[Julian Reschke]

(In reply to Cork from comment #26)
> The bug isn't resolved so how could it be implemented? The standard isn't
> even completely hammered out yet.

http://greenbytes.de/tech/webdav/rfc6585.html#rfc.section.6

Comment 29

[Nicolas Mailhot]

(In reply to Cork from comment #26)
>  The standard isn't
> even completely hammered out yet.

??? Since the opening of the bug the RFC has been finalized and published by the ietf

Comment 30

[Mark Nottingham]

Here are some useful things people could do to move this bug forward:

* Test to make sure that Mozilla will display the content of a 511 response
* Test to make sure that the pattern in 6.1 <http://greenbytes.de/tech/webdav/rfc6585.html#status-511> works
* Test to make sure that Mozilla doesn't cache / reuse a .favicon response with a 511 status code
* Test to make sure that Mozilla doesn't treat other automatically fetched responses as authoritative / cacheable when they have a 511 status code

Comment 31

[Curtis Koenig [:curtisk-use curtis.koenig+bzATgmail.com]]]

we will need to code review this when we have something to review

Comment 32

[Patrick McManus [:mcmanus]]

there is standards mumbling in this space.. so we should align with that if it mumbles something coherent