614565 - Lotus Notes webmail (at Hawaii DOE) broken by eTLD list update

Status: RESOLVED FIXED

(Core Graveyard :: Networking: Domain Lists, defect)

Description

[Mike Beltzner [:beltzner, not reading bugmail]]

Original report from Hendrix:

----

Aloha. The latest update of Firefox prevents hundreds of employees of
the Hawaii Department of Education (DOE) from accessing any DOE
web-based applications (i.e. email, student databases, professional
development registration, etc.). The DOE's technical support
branch--the Centralized Service Desk, (xxx) xxx-xxxx -- is advising all
employees to use an alternate browser.  Firefox is my preferred browser.
Is there anyway that you can rectify this matter since so many users
are being affected?

----

Cheng dug in with a username and password given by the contact, and said:

"I have headers (unfortunately they have username/password in plaintext so I won't attach).  Looks like it's trying to do a redirect to itself but failing to set a cookie (?)."

He then found this regression range:

http://hg.mozilla.org/releases/mozilla-1.9.2/pushloghtml?fromchange=95a205a82cb8&tochange=ea16213ee93e

in which I strongly suspect this as cause:

   bug 595300 - Update to NSS_3_12_8_RTM

Which was fixed in 3.6.11 and 3.5.14

Cheng can provide a test login/password to people who can debug what's happening here.

Comment 1

[:Gavin Sharp [email: [e-mail]]]

This was actually caused by http://hg.mozilla.org/releases/mozilla-1.9.2/rev/5d8b8a4a366a (confirmed via local bisect).

Comment 2

[:Gavin Sharp [email: [e-mail]]]

In particular, this addition:
http://hg.mozilla.org/releases/mozilla-1.9.2/rev/5d8b8a4a366a#l1.254

(the site in question is a k12.hi.us site)

Comment 3

[:Gavin Sharp [email: [e-mail]]]

The site is at http://lotusnotes.k12.hi.us/ .

In builds that work, the login process adds a cookie with the following attributes:

Name: LtpaToken
Domain: .k12.hi.us
Path: /

In builds that don't work, that cookie cannot be set (presumably because we now consider that domain to be "top level"), so the login fails.

Comment 4

[[e-mail]]

Gerv, any idea what we should do here? I'm really not sure if they actually mean to set a cookie for .k12.hi.us; if that domain is used as a US-wide k12 single signon portal or something, then we should fix on our end; if not, we should get them to fix theirs.

Comment 5

[[e-mail]]

(I guess what I'm really asking is whether someone wants to get in touch with them to work through this.)

Comment 6

[:Gavin Sharp [email: [e-mail]]]

I think it actually probably is intended as a *.K12.hi.us login, since presumably all of the infra under that domain is under the control of the Hawaii DOE (comment 0 suggests that it affects multiple systems).

Comment 7

[[e-mail]]

Oh, right, I misread as k12.us. So I think we probably want to remove all the k12.*.us rules...

Comment 8

[[:Cww]]

Ok, I followed up with the tech support desk there.  They would like us to remove k12.hi.us from the list of eTLDs.  I was wondering if we could get that in for 3.6.13 since it's really really low risk?

Comment 9

[[:Cww]]

(FWIW, it may be that we need to look at this on a state-by-state basis as it seems different states expect different behavior)

Comment 10

[[e-mail]]

(In reply to comment #9)
> (FWIW, it may be that we need to look at this on a state-by-state basis as it
> seems different states expect different behavior)

Sounds fragile to me... the argument for keeping them all in there isn't fantasically strong, since different K12 schools in a given state aren't likely to try to mess with each other's logins. I'd just drop them all.

Thanks for doing the legwork Cheng!

Comment 11

[Daniel Veditz [:dveditz]]

presumptively assigning to gerv as the official Keeper of the Lists.

If we re-spin 3.5.16/3.6.13 it'd be nice to pick this up. Renominating to keep on the radar.

Comment 12

[LegNeato]

I actually want to pick this up for 3.6.13 and 3.5.16. I'll send an email with the plan, but this is what I want:

* Keep the current builds live on the beta channel
* Have Toronto RelEng create build #2 of both with the .edu stuff backed out (not sure if this is possible currently)
* Push the build #2s to the beta audience on Monday or Tuesday

Comment 13

[Gervase Markham [:gerv]]

Just catching up here: I'm very happy to remove them, even pending further investigation. Removing them is pretty low risk, as you say. So do that, and I can think about the larger issues at leisure :-)

Gerv

Comment 14

[LegNeato]

Sounds good. Can we get a patch added to the bug so I can approve and we can land on both the relbranches and default?

Comment 15

[LegNeato]

Attachment: 1.9.2 patch

Comment 16

[LegNeato]

Attachment: 1.9.1 patch

Comment 17

[Daniel Veditz [:dveditz]]

These patches essentially back out bug 428944. We'll fix Hawaii but re-break Indiana and other states (although the two breakages were not equal).

Is there going to be a trunk patch? Maybe the trunk could delete only HI and wait for feedback on the other states before removing them.

HI doesn't seem to have any cc.hi.us or lib.hi.us sites (according to a Google inurl: search). The states that do seem to have very local sites.

> since different K12 schools in a given state aren't likely
> to try to mess with each other's logins. I'd just drop them all.

The adults might not, but schools are full of students. I can easily imagine parts of a school website being created by students much the way the yearbook and weekly paper is. If messing with those pages could potentially gather credentials to a "student database" (grades?) that might be very very tempting.

Comment 18

[Daniel Veditz [:dveditz]]

Comment on attachment 493313 [details] [diff] [review]
1.9.2 patch

r=dveditz

Comment 19

[Daniel Veditz [:dveditz]]

Comment on attachment 493314 [details] [diff] [review]
1.9.1 patch

r=dveditz

Comment 20

[Daniel Veditz [:dveditz]]

For the branches just back out the whole thing. For trunk we should consider only backing out k12.HI.us and leave the rest until we get feedback.

Comment 21

[Daniel Veditz [:dveditz]]

1.9.2.x:  http://hg.mozilla.org/releases/mozilla-1.9.2/rev/69e253891ca3
1.9.2.13: http://hg.mozilla.org/releases/mozilla-1.9.2/rev/426f09f86038
1.9.1.x:  http://hg.mozilla.org/releases/mozilla-1.9.1/rev/26016fb65870
1.9.1.16: http://hg.mozilla.org/releases/mozilla-1.9.1/rev/612ad84a4004

Comment 22

[Daniel Veditz [:dveditz]]

Attachment: trunk patch, only removes k12.hi.us

trunk patch to remove Hawaii from the k12 list.

Comment 23

[Gervase Markham [:gerv]]

OK, so the tension here is between the two different uses of the PSL. There's cookie setting, for which Hawaii wants to be able to set a cookie for hi.k12.us, and there's SSL indicator display, for which Indiana want to display foo.in.k12.us rather than in.k12.us.

In general, I have more sympathy with the latter view, and I think states any larger than Hawaii are less likely to have state-wide single signon across schools, libraries or community colleges. But the compat issues are smaller with the SSL indicator than with cookies.

So my view is that on the branches, we should back out the patch (better compat), and on trunk we should take this on a case-by-case basis for now. Let's back out hi.k12.us and see if we get other complaints as well. (Note bug 608362, where the company fixed their over-expansive cookie-setting.)

So basically, we should do what we just did :-)

Gerv

Comment 24

[B.J. Herbison]

Is it really the job of the Firefox team, the IE team, the Chrome team, the Opera team, and so on to debate the cookie scope for every possible domain? Shouldn't that be a decision for each domain owner?

How about something in the DNS records for hi.k12.us that says "cookies can be set for this domain" and something in "in.k12.us" that says "cookies can not be set for this domain, only subdomains"? Browsers would keep current rules, and gradually strip them back as the domains took over the work.

Comment 25

[Gervase Markham [:gerv]]

B.J.: there are internet-drafts (from Opera) for that sort of idea, but they haven't been ratified. And even if they were, they would need to be implemented on a wide enough scale that we could drop the PSL. If that happens, great. In the mean time, we need a list.

Gerv

Comment 26

[Daniel Veditz [:dveditz]]

Comment on attachment 493828 [details] [diff] [review]
trunk patch, only removes k12.hi.us

I'm interpreting comment 23 as an r=gerv, clegnitto agreed.

Comment 27

[Daniel Veditz [:dveditz]]

http://hg.mozilla.org/mozilla-central/rev/1a149c7a4fa2

Comment 28

[Gervase Markham [:gerv]]

Yep, r=gerv.

Gerv