Last Comment Bug 651185 - double quotes aren't a legal delimiter for 2231/5987 encoding
: double quotes aren't a legal delimiter for 2231/5987 encoding
Status: REOPENED
: dev-doc-complete
Product: Core
Classification: Components
Component: File Handling (show other bugs)
: Trunk
: All All
: -- normal with 1 vote (vote)
: mozilla8
Assigned To: Julian Reschke
:
Mentors:
http://greenbytes.de/tech/tc2231/#att...
Depends on: 685060 702938 703015
Blocks: 609667
  Show dependency treegraph
 
Reported: 2011-04-19 10:39 PDT by Julian Reschke
Modified: 2014-02-13 02:14 PST (History)
22 users (show)
bzbarsky: in‑testsuite+
ioana_damy: in‑moztrap?
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
test case (1.77 KB, patch)
2011-06-10 06:09 PDT, Julian Reschke
no flags Details | Diff | Review
only process 2231/5987 syntax when the value comes from a token (as opposed to quoted-string) (1.81 KB, patch)
2011-06-12 12:36 PDT, Julian Reschke
no flags Details | Diff | Review
only process 2231/5987 syntax when the value comes from a token (as opposed to quoted-string) (2.99 KB, patch)
2011-06-12 12:39 PDT, Julian Reschke
bzbarsky: review+
Details | Diff | Review

Description Julian Reschke 2011-04-19 10:39:58 PDT
User-Agent:       Mozilla/5.0 (Windows NT 6.1; rv:2.0) Gecko/20100101 Firefox/4.0
Build Identifier: FF4

See test case at 

 <http://greenbytes.de/tech/tc2231/#attwithfn2231quot>

The only UA who accepts this notation is FF. Let's get rid of this.

Reproducible: Always

Steps to Reproduce:
1. Navigate to <http://greenbytes.de/tech/tc2231/#attwithfn2231quot>
2. Run test at <http://greenbytes.de/tech/tc2231/attwithfn2231quot.asis>

Actual Results:  
Offers download of "foo-รค.html"

Expected Results:  
Header fields or filename parameter should be ignored.
Comment 1 Adam Barth 2011-04-19 10:44:02 PDT
See also http://code.google.com/p/chromium/issues/detail?id=41308

Changing this behavior will break some versions of Outlook Web Access.
Comment 2 Julian Reschke 2011-06-10 01:03:20 PDT
(In reply to comment #1)
> See also http://code.google.com/p/chromium/issues/detail?id=41308

Marked "wontfix" as of June 2, 2011.

> Changing this behavior will break some versions of Outlook Web Access.

Yes.
Comment 3 Julian Reschke 2011-06-10 06:09:06 PDT
Created attachment 538495 [details] [diff] [review]
test case

This is a test which will fail for the current code.
Comment 4 Adam Barth 2011-06-10 10:24:48 PDT
> > See also http://code.google.com/p/chromium/issues/detail?id=41308
> 
> Marked "wontfix" as of June 2, 2011.

We marked that bug wont fix because the number of users complaining about the OWA breakage was very small and OWA has already fixed the issue in a service release.
Comment 5 Julian Reschke 2011-06-10 11:02:02 PDT
(In reply to comment #4)
> > > See also http://code.google.com/p/chromium/issues/detail?id=41308
> > 
> > Marked "wontfix" as of June 2, 2011.
> 
> We marked that bug wont fix because the number of users complaining about
> the OWA breakage was very small and OWA has already fixed the issue in a
> service release.

Yep, and thanks for dong the right thing.

Now let's fix Firefox instead :-)
Comment 6 Julian Reschke 2011-06-12 12:36:18 PDT
Created attachment 538774 [details] [diff] [review]
only process 2231/5987 syntax when the value comes from a token (as opposed to quoted-string)
Comment 7 Julian Reschke 2011-06-12 12:39:24 PDT
Created attachment 538775 [details] [diff] [review]
only process 2231/5987 syntax when the value comes from a token (as opposed to quoted-string)
Comment 8 Boris Zbarsky [:bz] 2011-06-19 21:48:59 PDT
When was the OWA service release that fixed this?  Do we have any data on the uptake rate for it?

Or put another way, how many users would we break if we fixed this now, and how many fewer would it be if we waited a few months?
Comment 9 Adam Barth 2011-06-19 21:56:48 PDT
The Chromium bug says that MS fixed the issue in Exchange Server 2010 SP1 (X-OWA-Version 14.1.x.x), but I haven't verified that myself.  There's also some indication that OWA has User-Agent dependent behavior in this area, so it might be worth testing with a Firefox User-Agent.

I don't have data as to update rates for Exchange Server service packs, but I can tell you that the support volume for this issue is very low.
Comment 10 Boris Zbarsky [:bz] 2011-06-19 22:10:07 PDT
> so it might be worth testing with a Firefox User-Agent.

Julian, do you have a way of doing that?

> but I can tell you that the support volume for this issue is very low.

Can you at least quantify "very low" in a way that makes sense (e.g. estimated fraction of Chrome's users affected)?  I understand if you can't, but I'd really like to avoid jumping into changes that have known compat impact without understanding the size of that impact and how to minimize it.

Note that I agree that this is a good change to make; the only question is the timing.
Comment 11 Julian Reschke 2011-06-19 22:22:16 PDT
(In reply to comment #10)
> > so it might be worth testing with a Firefox User-Agent.
> 
> Julian, do you have a way of doing that?

NO, I don't have any access to OWA.

> > but I can tell you that the support volume for this issue is very low.
> 
> Can you at least quantify "very low" in a way that makes sense (e.g.
> estimated fraction of Chrome's users affected)?  I understand if you can't,
> but I'd really like to avoid jumping into changes that have known compat
> impact without understanding the size of that impact and how to minimize it.
> 
> Note that I agree that this is a good change to make; the only question is
> the timing.

There'll be ~6 months before this would go into a release; isn't that sufficient to monitor the situation?
Comment 12 Boris Zbarsky [:bz] 2011-06-19 22:48:14 PDT
> NO, I don't have any access to OWA.

OK.  Johnny, do we have an internal OWA install?  Or some other way to do some testing here?

> There'll be ~6 months before this would go into a release

If you checked this in today, it would be in aurora on July 5, beta on August 9, and release on Sept 20.  3 months from today.  I have no idea where your "~6 months" number comes from....

> isn't that sufficient to monitor the situation?

That depends on the size of the aurora and beta testing audiences and on how their rates of use of OWA compare to the release audience.  For this last, I can pretty much guarantee they're lower, given the various OWA issues we didn't find in betas in the past.
Comment 13 Julian Reschke 2011-06-19 23:13:50 PDT
(In reply to comment #12)
> > There'll be ~6 months before this would go into a release
> 
> If you checked this in today, it would be in aurora on July 5, beta on
> August 9, and release on Sept 20.  3 months from today.  I have no idea
> where your "~6 months" number comes from....
> ...

Uh. Apparently I still haven't understood the new release model. Sorry.
Comment 14 Adam Barth 2011-06-19 23:39:00 PDT
> Can you at least quantify "very low" in a way that makes sense (e.g.
> estimated fraction of Chrome's users affected)?

We asked the folks who run the help center and they said that there do exist some folks running into the issue, but not very many.  They basically said that it's not on their radar of issues they care about.  Sorry that's not overly quantitative, but that's the data I have.
Comment 15 Johnny Stenback (:jst, jst@mozilla.com) 2011-06-20 15:36:44 PDT
(In reply to comment #12)
> > NO, I don't have any access to OWA.
> 
> OK.  Johnny, do we have an internal OWA install?  Or some other way to do
> some testing here?

Not that I'm aware of :(

But if it'd be of significant help here then I can have one set up. I hear we could use a windows server for other testing every once in a while as well. Let me know...
Comment 16 Boris Zbarsky [:bz] 2011-06-20 21:21:19 PDT
I really think I've seen some of the necko folks mention testing OWA... unless I'm misremembering.
Comment 17 Jason Duell [:jduell] (needinfo? me) 2011-06-22 02:30:04 PDT
> I really think I've seen some of the necko folks mention testing OWA

Not that I've heard, but I only go back so far... :)
Comment 18 Julian Reschke 2011-06-22 02:40:16 PDT
In the meantime I have asked Exchange people for information about when 14.0 and 14.1 came out, but didn't get any reply that.

Absent additional information, maybe we can do this after July 5, so for Firefox 8?
Comment 19 Boris Zbarsky [:bz] 2011-06-22 12:21:03 PDT
That would make me happier, yes.
Comment 20 Boris Zbarsky [:bz] 2011-06-22 12:23:28 PDT
Comment on attachment 538775 [details] [diff] [review]
only process 2231/5987 syntax when the value comes from a token (as opposed to quoted-string)

r=me, but please do hold off until after the Aurora merge...  If you don't want to spend time thinking about that, please feel free to reassign to me and I'll land it after that.
Comment 22 Mounir Lamouri (:mounir) 2011-07-06 06:01:50 PDT
Merged:
http://hg.mozilla.org/mozilla-central/rev/c791539c83e8
Comment 23 Eric Shepherd [:sheppy] 2011-08-25 06:26:33 PDT
This was already added to Firefox 8 for developers; I've cleaned it up.
Comment 24 Matt Evans [:mevans] 2011-11-18 09:29:42 PST
OWA should be part of our web compatibility test matrix
Comment 25 Michael Lefevre 2011-11-21 09:16:32 PST
Not sure if the status of this bug should change, having made it into a release, but for the record, this fix is being reverted in bug 703015
Comment 26 Julian Reschke 2011-11-23 12:24:55 PST
This was backed out because of bug 703015.

Affected web sites:

1) Bug 703015: MS Exchange 2010 / Outlook Web Access; hopefully this will get fixed at some point in the future.

2) Bug 685060: PHP forum software from simplemachines.org; problem has been known since early September 2011, and it appears a bug fix is being made. We had only bug reports from this site and one other site using this software, though (Bug 701313)

Affected mail clients:

We heard about one mail server sending broken header fields which was fixed long ago (see Bug 703614).

My conclusion: we should try this again once we known Microsoft has fixed their problem (adding enough time for customers to upgrade)
Comment 27 Julian Reschke 2011-11-24 11:48:18 PST
Re-opened because the change was backed out for bug 703015.
Comment 28 rsx11m 2011-11-26 08:33:49 PST
Maybe I'm missing something, but in general I'm struggling with the rationale of why this bug should be fixed at all. I've learned that one should be pedantic when interpreting standards for writing code to communicate in a specific protocol, but to be tolerant on the reading end given that someone will always do something wrong (which is evident in the reported regressions).

The RFC 2231 headers in question are "quoted" but otherwise correct. Rejecting them entirely appears to be too strict and applies the "pedantic" rule to the wrong end. Even with the quotes, the information in the headers can still be recovered safely and unambiguously, and the patch actually increased code complexity.

Is there a requirement to ignore rather than to recover such insignificantly malformed headers? There sure is no benefit to the user in being strict here.
Comment 29 Julian Reschke 2011-11-26 09:09:20 PST
(In reply to rsx11m from comment #28)
> ...
> Is there a requirement to ignore rather than to recover such insignificantly
> malformed headers? There sure is no benefit to the user in being strict here.

The benefit is for the platform. FF is the only UA having this workaround. Not working around it makes it easier for producers to send valid values in the first place, and also removes the incentive for other UAs to copy the workarounds (which Chrome, thankfully, did not).
Comment 30 Jochen Roderburg 2011-12-22 12:40:03 PST
(In reply to Julian Reschke from comment #29)
> (In reply to rsx11m from comment #28)
> 
> The benefit is for the platform. FF is the only UA having this workaround.
> Not working around it makes it easier for producers to send valid values in
> the first place, and also removes the incentive for other UAs to copy the
> workarounds (which Chrome, thankfully, did not).

This might be correct for the Firefox/Browser case, but please don't forget again that this change has also created collateral damage in Thunderbird. And IMHO the situation in the area of mail programs is a bit different: So far I have not found any mail program that does not gracefully accept these slightly incorrect mime headers.
Comment 31 Julian Reschke 2011-12-22 12:51:34 PST
(In reply to Jochen Roderburg from comment #30)
> This might be correct for the Firefox/Browser case, but please don't forget
> again that this change has also created collateral damage in Thunderbird.
> And IMHO the situation in the area of mail programs is a bit different: So
> far I have not found any mail program that does not gracefully accept these
> slightly incorrect mime headers.

Which ones did you test? My understanding so far is that only one was sending the broken header fields, and that one was fixed long ago.
Comment 32 Jochen Roderburg 2011-12-22 13:39:23 PST
(In reply to Julian Reschke from comment #31)
> (In reply to Jochen Roderburg from comment #30)
> > This might be correct for the Firefox/Browser case, but please don't forget
> > again that this change has also created collateral damage in Thunderbird.
> > And IMHO the situation in the area of mail programs is a bit different: So
> > far I have not found any mail program that does not gracefully accept these
> > slightly incorrect mime headers.
> 
> Which ones did you test? My understanding so far is that only one was
> sending the broken header fields, and that one was fixed long ago.

I was talking about the receiving side, not the sending side.  ;-)
And I just was trying to explain that your argument that all browsers except Firefox treat the wrong headers as incorrect per RFC is IMO not transferable to mail programs where it looks to me as if all others accept the wrong headers.

About the problem itself:
Regarding the sender side: it was us (my colleague Sebastian Hagedorn) who reported that older versions (long ago was about one year ago) of the Horde/IMP webmail system created wrong headers. I have no idea how many of these program versions are still in use worldwide and don't know if there were others. But the problem with mail files is that correction of the sender programs do not change any existing mail files that people might have archived. So the problem of the disappearing attachments will still stay with us as long as the follow-up problems in Thunderbird are not correctly repaired (as described in the still open bug 705431).
Comment 33 Julian Reschke 2011-12-22 13:41:53 PST
(In reply to Jochen Roderburg from comment #32)
> ...
> archived. So the problem of the disappearing attachments will still stay
> with us as long as the follow-up problems in Thunderbird are not correctly
> repaired (as described in the still open bug 705431).
> ...

Yes, and I have an eye on that.

When we get back to this and this is not fixed in Thunderbird, we may have to special-case the call from the mail client. But of course every single special case is one too much and to be avoided.

Note You need to log in before you can comment on or make changes to this bug.