Closed Bug 392982 Opened 17 years ago Closed 17 years ago

Cannot add attachment in yahoo mail

Categories

(Core :: DOM: Core & HTML, defect)

Product:
Core
Component:
DOM: Core & HTML
Platform:
x86
Windows XP
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
VERIFIED FIXED
Milestone:
mozilla1.9beta1

People

(Reporter: polidobj, Assigned: david)

References

Details

(Keywords: regression)

Attachments

(3 files)

POST request sent from FF2
15.91 KB, text/plain
Details
POST request sent from FF3 trunk
17.21 KB, text/plain
Details
HTML source of Yahoo Mail attach files page
39.68 KB, text/html
Details
I was informed that adding attachments to Yahoo Mail (not the beta version) recently broke on the trunk. I found the regression range to be on Aug 10th. http://bonsai.mozilla.org/cvsquery.cgi?module=PhoenixTinderbox&date=explicit&mindate=1186789080&maxdate=1186793039 Boris I'm not sure which bug caused this problem. I see innerHTML in the javascript. But the attachment is added using a form submit. I see you were involved in both bugs. There are two errors from the console but this one seems to be the pertinent one: Error: unterminated string literal Source File: http://attach.re2.mail.yahoo.com/us.f519.mail.yahoo.com/ym/Attachments?YY=85850&y5beta=yes&y5beta=yes Line: 300, Column: 32 Source Code: path = UnescapeBackslash("ntent-Type: text/plain; charset=ISO-8859-1 The next line in the code has "); But I'm not sure if the server giving this bad line in the JS due to bad data being passed to it because of innerHTML or is the change in form submission messing up the server's response? Looking at bug 116346, I noticed browser.forms.submit.backwards_compatible was mentioned. Changing that has no effect on the problem.
There aren't any innerHTML changes in that range... It looks like yahoo's server-side code creates JS that tries to unescape the filename from the content-disposition header for display... But they somehow screw up the filename to unescape. Badly. Out of curiousity, what was the name of the file you were trying to attach? Does the JS error source code depend on that filename?
Blocks: 116346
Component: General → HTML: Form Submission
Product: Firefox → Core
QA Contact: general → form-submission
Need to sort out what's going on here and what we can do either on our end or on Yahoo's end to fix this. Can we get someone on the Yahoo end to look at this?
Flags: blocking1.9?
It doesn't look like it depends on the filename. I tried a filename with a different filename length but the bad line did not change. The first filename was 'team futures.txt' which is a file I had sitting on my desktop I happen to use as a guinea pig.
Steps: 1. Login to Yahoo Mail (using the default client *not* the beta client) 2. Compose a new message 3. Click "Attach Files" 4. Browse to and select a file (anything will do) 5. Click "Attach Files" Result: after pressing the "Attach Files" button, an error message appears: "There was a problem: Please try logging out and logging back in, or wait 10 minutes for the problem to be resolved." The page source for the error page is a bit strange, lots of form elements with like: <input type=hidden name=SaveCopy value="ntent-Type: text/plain; charset=ISO-8859-1 no"> Works: Mozilla/5.0 (Windows; U; Windows NT 5.1; ja; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 Doesn't work: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a8pre) Gecko/2007082019 Minefield/3.0a8pre
So it almost looks like the server code blindly assumes that the Content-Disposition header will always be the last (or only?) header and that after it there will be two bytes to skip (the blank line CRLF) and then you start reading data. That would explain where the "Co" from "Content-type" went. Compare bug 83065 comment 6, where PHP assumes the same thing about the Content-Type header. If servers are assuming both, then the only possible course of action is to only send one header... That said, I'd like to see whether this is a Yahoo-specific issue and if so whether they can fix it.
Yup, that looks like exactly what it's doing, argh. I traced the headers using tcpmon (Apache Axis version). POST request with FF 2.0.0.6 contains: -----------------------------7180180089615 Content-Disposition: form-data; name="SaveCopy" no -----------------------------7180180089615 Content-Disposition: form-data; name="Format" -----------------------------7180180089615 POST request with FF3 trunk contains: -----------------------------317622842327531 Content-Disposition: form-data; name="SaveCopy" Content-Type: text/plain; charset=ISO-8859-1 no -----------------------------317622842327531 Content-Disposition: form-data; name="Format" Content-Type: text/plain; charset=ISO-8859-1 -----------------------------317622842327531
Attached file POST request sent from FF2 —
John, could you attach the HTML of the page containing the form itself, please? - I'm referring to the web page with the "Attach Files" button; the compose message page, before the error occurs... Thanks! Robert
Something else that changed about the same time and might be related to this bug. In Yahoo mail (not beta) Click on bulk mail, click the delete all box and delete, a new window opens and says, "Are You Sure You Want To Delete All Items" "The check box for don't show again dose nothing" The check box for delete dose nothing. If you hit "delete" and close the window with the Red X, it will delete the files. The don't delete dose nothing, If you hit Don't delete and close the window with the Red X it will cancel the job. FF 2.06 dose not do this.
(In reply to comment #11) They're not related. bug 391777
David, this looks like this is caused by your change in bug 116346. We need to get this fixed before shipping Firefox 3.
Assignee: nobody → david
Flags: blocking1.9? → blocking1.9+
Jonas, my understanding of this problem is that it's a problem with the Yahoo server-side code that handles MIME attachments. Let me know if you feel otherwise. I have sent mail to the Yahoo classic mail folks regarding this, so hopefully we can get their input to resolve the problem.
It is a problem with yahoo, but it's due to a change on our side. We can't ship Firefox 3 not supporting yahoo mail. So we either have to get them, and everyone else that might be doing the same thing, to change their code, or revert our change.
Mail from Julia Lee at Yahoo: Hi John, I'll have someone on my team investigate. Julia ----- Original Message ----- From: "John Daggett" To: "Julia Lee" Sent: Wednesday, September 05, 2007 12:11 AM Subject: Re: engineering contact on Classic Mail > Hi Julia, > > I passed along Ratan's email to another developer here at Mozilla, I'm > not sure if anything happened after that. We bumped into what looks > like a bug in the server-side code for attachment handling in Yahoo > classic mail, the code is making assumptions about the structure of > MIME attachments. A change in the way we're encoding attachments in > Firefox 3 triggers this. We're hoping we can get this resolved. > > Thanks very much for your help, > > John Daggett > Mozilla Japan > > Details: > ====== > > When a user composes a mail, attaches a file to it and sends it, the > browser constructs a MIME-formatted POST request and passes it over to > the server. With Firefox 2.0, a snippet of this looks like: > > -----------------------------7180180089615 > Content-Disposition: form-data; name="SaveCopy" > > no > -----------------------------7180180089615 > Content-Disposition: form-data; name="Format" > > > -----------------------------7180180089615 > > In Firefox 3.0 we're going to add an extra header to this for each > part, so the POST request looks like: > > -----------------------------317622842327531 > Content-Disposition: form-data; name="SaveCopy" > Content-Type: text/plain; charset=ISO-8859-1 > > no > -----------------------------317622842327531 > Content-Disposition: form-data; name="Format" > Content-Type: text/plain; charset=ISO-8859-1 > > > -----------------------------317622842327531 > > The Yahoo mail server-side code that handles MIME attachments > formatted with two header lines throws an error. It looks like it is > expecting the content of the attachment to begin two lines after the > first header, so adding another header trips up this logic. > > The details are listed in our bug tracking system Bugzilla: > > https://bugzilla.mozilla.org/show_bug.cgi?id=392982 >
Sorry for not responding earlier (was in the middle of a move). Yeah, it seems clear that this is a server-side issue on Yahoo's side. If they are unable to resolve it on their end, the logical thing to do would be to roll back this change. I agree that we can't ship with Yahoo broken. It doesn't seem possible to create a general-case fix for this since Yahoo is assuming something very specific about these MIME headers. Anything other than the original (pre-bug 116346) headers would still break for Yahoo. I doubt many others are doing similar things, but all it takes is one high-profile one to throw a wrench in things.
So I think we have three options here: Option 1: Get Yahoo to change their code in time for beta. We can then get feedback from the beta on if this breaks other sites. If it does, I think we should simply revert the original patch. Whatever list of sites we find during beta is going to be a partial one, so if this is more than yahoo being a one-off I think we should revert. Option 2: Add an opt-in mechanism, such as an attribute on the form element. Option 3: Revert right away. Unless someone comes up with a patch for option 2, or manages to get yahoo to fix their stuff within a few weeks, I think we're forced to go with option 3. I don't want to ship a beta with yahoo broken, and probably many other sites too.
For purposes of the beta, would it make sense to just explictly check for the yahoo page in question and not do this on it? In e-miul, they said they expected it to be a month or so before they rolled out the fix, which might not be quick enough for beta.
To be honest, I very much expect a pile of other sites regressing over this. So I would much rather see a patch that implements an opt-in mechanism.
I don't like option 2 because it completely defeats the purpose of the feature. If I have complete control over my site, to "opt in" on my own forms, that buys me very little over just declaring UTF-8 everywhere. I can do that because it's my own site. But if I have 50 other sites doing POSTs to mine, I can't guarantee anything about the encoding they choose to use on those pages, much less whether or not they're going to choose to "opt in" to this feature or not. Option 3 is unacceptable to me because this feature has potential uses, and unless someone has a serious passion and/or funding for seeing it added to Firefox, by single-handedly testing (or arranging a testing effort) to hit every form on the Internet and make sure they all work, we'll always be in the same situation we are today: scared to implement because "we just don't know" if sites will break. This is exactly the reason we have beta testing. And while I sympathize with the perspective that we can't have beta testers unable to test because their most popular site(s) are broken with it, we should be working to facilitate changes on the site end to work with this browser change, not simply revert without a plan to move forward again. If Yahoo! fails to correct this issue in time for our release, I would vote for: a) reverting this change, for the release only (more defective site reports can trickle in from the nightly users, maybe enough to be more confident for the next release) b) implement an opt-OUT mechanism, track the broken sites, and remove it when the bulk of the broken sites get it together I like (b) because we'll be able to identify broken sites far more quickly with a beta cycle than we will with nightly users, and the beta users won't have an "unusable browser" if enough of their browsing activity occurs on a broken site.
I don't understand why an opt-in makes the feature pointless. Please describe what you mean by "just declaring UTF-8 everywhere" in more detail. If there are other sites posting to your server and have problems, why can't they opt-in too? They aren't going to be in any worse shape than they are today. Testing the whole web is of course never going to be done. And the main problem isn't really testing either, but having all the broken sites fix their server side parsing. An opt-out mechanism won't really help either since that won't fix any of the broken sites. Backing this out for now and attempting to reland it for the next release (Firefox 4) does seem like a valid strategy though, if you are willing to hound sites that we find that are broken?
Marking as blocking 1.9 Beta 1 via M9 target milestone.
Target Milestone: --- → mozilla1.9 M9
There are two ways you can (effectively) guarantee that your form submission will be in a known character encoding: 1. Have this feature implemented and inspect the encodings attached to each form parameter in the submitted data; or 2. Guarantee that the pages carrying the form are encoded and declared as UTF-8. Browsers will use the page's encoding when submitting the form. This is possible only on pages you have some degree of control over, and only if you know for sure that no one else's pages will contain forms POSTing to your application. Consider a search engine, for example. The search engine can slap UTF-8 all over its pages to ensure that searches made from those pages will be done in UTF-8, but there could be millions of forms on other peoples' web sites submitting searches. The operator of the search engine can't possibly ensure that all of these sites have pages encoded in UTF-8, which means searches from pages that don't will fail for (usually) non-ASCII characters. I consider "opt-in" to be pointless because no one will opt in except perhaps "power testers" sympathetic to this feature or MAYBE one or two closed environments where administrators deem this feature is a must-have. Consequently, we will never know which sites are broken because we'd be relying on the nightly users to discover them, without a concerted testing effort. It would take a long time (years?) of having it "opt in" before we'd reach a point where we'd be slightly more comfortable about turning it on by default. The opt-out approach won't FIX the sites, but it will get the ball going. We can't tell sites that they're broken until we know about them. Opt-out lets our beta testers discover them, and then flip the switch that lets them use the site anyway. I'm more than willing to hound the two sites/toolkits that I've seen break with this change. But I can't hound the rest until I know about them.
> Guarantee that the pages carrying the form are encoded and declared as UTF-8. Or that the <form> has accept-charset="UTF-8". Though this still has the problem you describe for document encodings...
I'm not expecting sites to opt-in during the beta period, but rather over the coming years, as they add support for parsing and using the header. In the current state we are adding no value (since currently no sites use the new header value) while knowingly breaking some sites and likely breaking many more. With opt-in sites that are having problems with getting files of unknown encodings can add support for the new header and use the opt-in mechanism. The only downside with this is that sites that get file submissions from many other sites will not get fixed automatically as they do this. First of all, file submissions across sites is fairly rare I think (would be happy to get data here). So the loss is likely very small. Second, there would be no actual regression with this, just slower pace of progress. Additionally, the sites that do add support for the new header, and that get submissions from external sites can put up documentation for how the external sites could improve their quality by using the opt-in. Using an opt-out mechanism buys us next to nothing. The only help would be that sites that do discover that they have problems with this would have an easier way to deal; adding the opt-out rather than fixing their code. Of course, this doesn't help in the case of submissions from external sites.
Jonas, file submissions are not the problem. We already send a Content-Type for those. The change we made was to also sent a Content-Type for the pure-text parts of the form submission. Those are the ones that you need to be able to convert from bytes to chars and hence need the charset.
Ah, true. Though multipart/form-data is mostly used when submitting files. Couldn't we fix the original bug by adding another http-header instead then? That should not affect parsing of the form data, and all pure-text parts will have the same encoding anyway.
I think we could, yes. That has the drawback of being a non-standard solution, but that might be preferable, depending on how much fallout we see from this. To be honest, I think we'll see less fallout than you seem to expect.. but maybe I'm just an optimist.
I'm generally optimistic too, until I see one site break. If one does, likely others do too. And this is a format that has been constant for over 10 years, so people has had plenty of time to assume it'll stay stable. And since we know sites will break by following the standard, I'm much more ok with doing an alternative solution like a header. And we're not making it harder for the web to move forward with following the standard either.
Note though, we'd need a patch fast as beta is coming up.
> over the coming years, as they add support for parsing and using the header. For the vast majority of sites, it's the frameworks they're using that will add this support. And I suspect they'll be doing this sooner rather than later if a browser out there supports it. > sites can put up documentation We actually have this situation today: if a form wants UTF-8, sites have the option of saying, "If you put a form on your own site that submits to us, please make sure your form submits in UTF-8." The fact that you never see this should tell you something. Yes, they can, but nobody is likely to. Nobody wants to deal with character encodings, and since many of these sites use frameworks that abstract all of that away from them, it's likely this feature will be implemented in those frameworks without the application developers even knowing about it. That makes it harder for them to push this awareness to other web developers POSTing to their applications. > And we're not making it harder for the web to move forward with following > the standard either. If we don't send the header per the standards, who's going to add support for the standard solution? I'm not sure I follow what you're saying here. Also, with regard to opt-out, my thought was that this would be a client option, not something in the HTML page. Server-side opt-in/opt-out makes less sense for precisely the reason you give: the issue is with 3rd-party sites posting to your applications, not with your own pages posting to themselves. Coordinating opt-in/opt-out instructions with 3rd-parties isn't going to be practical. For the purposes of the beta, it seems appropriate to let the user discover where things break, report the failures, and have a way to work around it. For a proper *release*, this approach is inappropriate, but it seems reasonable for a beta. The whole point of a beta is to see what breaks, right? FWIW, I've looked at a handful of toolkits/frameworks that do form submission processing, and I haven't found any (aside from the JIRA issue that was previously reported) that make Yahoo!-style assumptions about the data. I suspect most sites out there doing form processing are using toolkits like this and probably aren't rolling their own. I imagine sites like Yahoo! find the overhead of these toolkits to be excessive given their load/demands, which is why they choose to do their own thing. Only a handful of sites should have similar requirements, and THOSE we can pretty easily test. (I'd even go out on a limb and suggest that they all HAVE been tested by this point by our nightly users.)
(In reply to comment #33) > > And we're not making it harder for the web to move forward with following > > the standard either. > > If we don't send the header per the standards, who's going to add support for > the standard solution? I'm not sure I follow what you're saying here. The sites out there that need this would presumably add support for this and require it (and show sensible errors to the developers explaining what's needed to make things work). > Also, with regard to opt-out, my thought was that this would be a client > option, not something in the HTML page. Server-side opt-in/opt-out makes less > sense for precisely the reason you give: the issue is with 3rd-party sites > posting to your applications, not with your own pages posting to themselves. > Coordinating opt-in/opt-out instructions with 3rd-parties isn't going to be > practical. For the purposes of the beta, it seems appropriate to let the user > discover where things break, report the failures, and have a way to work around > it. > > For a proper *release*, this approach is inappropriate, but it seems reasonable > for a beta. The whole point of a beta is to see what breaks, right? Right, a client-side user out-out is unfortunately out of the question here. The vast majority of our users (99%+) have no idea what any of this means, or why something like this would break or fix some random site out there. And the danger (which I believe is pretty real in this case) is that even the testing we'll get out of our betas is unlikely to catch all the cases that break due to this. > FWIW, I've looked at a handful of toolkits/frameworks that do form submission > processing, and I haven't found any (aside from the JIRA issue that was > previously reported) that make Yahoo!-style assumptions about the data. I > suspect most sites out there doing form processing are using toolkits like this > and probably aren't rolling their own. I imagine sites like Yahoo! find the > overhead of these toolkits to be excessive given their load/demands, which is > why they choose to do their own thing. Only a handful of sites should have > similar requirements, and THOSE we can pretty easily test. (I'd even go out on > a limb and suggest that they all HAVE been tested by this point by our nightly > users.) I'm afraid I don't feel the same way about this myself. Having seen a number of form handling and submission problems over the years of my involvement in Mozilla development that went unrecognized until the changes went out into broad use (i.e. official release), this issue scares me to the point where I see no other sensible approach here than default to what the behavior has been for the last 15+ years, and opt-in in the HTML for this additional header.
Given that we know that sites break over this using the current approach (add header to each multipart section), and there is a proposed solution to the main problem that does not break sites (add a single http header) I don't see any reason to keep the current code. We're trying to solve a problem here, not promote a standard as a goal in itself. I agree that it would have been nice if we could follow the standard, but adding the http header isn't going to hurt anyone so why not? Again, time is of essence here if you want this in. We need to have this solved by beta.
> The vast majority of our users (99%+) have no idea what any of this means Why do we care about this if it's a beta release though? Am I misunderstanding the purpose of a beta cycle? For the purposes of Firefox, does beta = release candidate? If users aren't going to be able to handle something like this, they shouldn't be running beta releases of software. Server-side opt-in/opt-out is essentially useless. If client-side opt-in/opt-out is "out of the question", then our only remaining options are (a) a simple and indefinite rollback of the feature (maybe #ifdef it for those that want to press onward anyway), or (b) solve the problem a non-standard way, as you suggest. A non-standard HTTP header would seem to achieve our goals here, but some thought needs to go into it with (preferably) no reason for other browser vendors to disagree with the approach. I'd rather not rush such a thing in to meet the next beta release. So either way, it sounds like our options are constrained to the point where, for the next release, an effective rollback is the only acceptable option, regardless of what Yahoo! does.
Please see my bug 116346 comment 70.
Target Milestone: mozilla1.9 M9 → ---
Targeting M9 as this bug will block beta.
Target Milestone: --- → mozilla1.9 M9
So I talked with yahoo today, and they were not going to be able to have things fixed on their end in time for beta. So we're definitely going to need the http-header fix or a straight backout of bug 116346 by wednesday next week when we tighten up the check-in policy to prepare for shipping beta.
So wednesday we're locking down checkins tighter in order to get ready for the beta release. Is anyone going to be able to provide a new patch to implement the http-header before then? If not I think we're left with having to back out the current patch as only solution to have a working beta :(
We need to make a decision on this ASAP.
I haven't had time to explore other options (like the proposed non-standard HTTP header), and no one else has spoken up saying they're working on such a thing, so I don't think there's a decision to be made. It sounds like our options are constrained to the point where all we can do is roll back. I don't care for that option, but nobody has spoken up challenging the assertion that it's the only option we have, so let's do that. I'd still like to have a better understanding of what "beta" means, and the expectation of our beta test users, but that's probably a discussion that can happen somewhere else. What would people think about rolling this back ONLY for the beta release, and re-applying it afterward? That would remove the feature for the beta cycle while still keeping it around for additional testing. Since there are still concerns that we may never know how many sites will break with this change in place, it might help to have some concrete criteria that we can use to validate that concern. I hate to see a feature dropped on the floor because of some fuzzy, unquantifiable fear. I'm not trying to suggest the fear isn't unfounded, but it would be more useful if we could turn it into something a little more measurable. With a better idea of the work effort involved in testing this feature out, it might be possible for a unilateral effort to prove within a given margin of error that it's "safe enough" to go into the next beta. (I'm OK saying that Yahoo!'s continued failure to accommodate this MIME header is, alone, sufficient reason to keep the feature turned off, but I have to assume that Yahoo! will fix their code sooner or later.)
The problem is that we're past feature freeze so if we're taking this out it's not going back in if I understand the rules correctly. Unfortunately there aren't any hard rules for what is ok in a beta and what is not. The purpose of a beta is to get people to test the current state of the code. If there are bugs in there that cause them to go back to using the non-beta that would be the kind of bugs that we want to fixed before beta release. But it's always a judgment issue what will make users switch away and what won't.
I'm of the opinion that Yahoo Mail should be functioning in our beta. Unless we can get that fixed before the freeze, I think we should back out. And, yes, if we can't get this fixed before the beta, and we have to back out, I would argue that it's too late. We're at a point where we need to get a solid beta out to the world. There's a lot of awesome work people have done to get this far. Either way we choose on this bug, we're not going to hold up the release. If we back out, and then try to re-land, we risk regressing again. I believe we should be trying to minimize regressions and ship a good beta the feature set we have now.
I don't see this as something we can/should "fix". The regression here is that the addition of this feature broke Yahoo. But it's Yahoo that's doing the wrong thing, not us. Since Yahoo can't fix their site before the beta, and even if they could, there's a fear that "other sites" could be making similar mistakes, the onus is on us to ensure the beta is usable for these sites, and all we can do to do that is roll back. So let's roll back. Maybe we can re-approach this feature in the next major version of Firefox, either in its current standards-compliant form or some non-standard hack.
Is this backed out yet?
It is now. Marking this bug as fixed, and reopening the bug that caused this (bug 116346), whose patch got backed out.
Status: NEW → RESOLVED
Closed: 17 years ago
Resolution: --- → FIXED
Thanks, Johnny. :)
That patch only got partially backed out...
verified fixed using Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.9a9pre) Gecko/2007102504 Minefield/3.0a9pre. I can now successfully add an attachment to Yahoo mail.
Status: RESOLVED → VERIFIED
Component: HTML: Form Submission → DOM: Core & HTML
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: