138869 - yahoo (not .com) - Using HTTP/1.1, Mozilla can't search for "oeone"

Status: RESOLVED FIXED

(Tech Evangelism Graveyard :: English US, defect, P1)

Description

[Andrew Hagen]

The URL, http://search.yahoo.com/bin/search?p=akamait , does not work in Mozilla
Build ID: 2002041903 (0.9.9+) trunk Windows 98. 

Another way to reproduce the problem is to go to http://www.yahoo.com and search
for   akamait  .

The search works on IE 6. Mozilla can search for  akamait   on Google.

Expected results: Mozilla opens the URL.

Actual results: Mozilla does not open the URL

Comment 1

[Vadim Berezniker]

confirming linux build from this morning
pretty weird

os -> all

Comment 2

[Vadim Berezniker]

this works if you set HTTP to 1.0
breaks if you set HTTP to 1.1

Comment 3

[Andrew Hagen]

Confirming that. Adjusting component and summary.

Comment 4

[Tanyel]

Maybe somebody should set Mozilla's user-agent property to whatever Internet
Explorer uses and see if the search suddenly works.

Comment 5

[R.K.Aa.]

WFM on XP, 2002041603, HTTP1.1, so it's a recent regression if so.

Comment 6

[Bradley Baetz (:bbaetz)]

This is broken in RC1, so its not related to the piplining stuff

We send the http/1.1 request, and the server returns an http/1.0 response with a
302, no content length, and some data. The connection is clsoed, and we don't do
anything.

With 1.1, we follow the 302 - the server's response is the same. The only
difference is that with 1.0, we send connection: close

Darin: Why is http/1.0 disabling persistent connections?

HTTP/1.1 with persistent connections disabled doesn't work, though.

Comment 7

[Bradley Baetz (:bbaetz)]

reassigning to darin probably helps, too....

Darin: see previous comment.

Comment 8

[Bertilo Wennergren]

> Maybe somebody should set Mozilla's user-agent property
> to whatever Internet Explorer uses and see if the search
> suddenly works.

I did. It suddenly worked. It must be Yahoo doing some
weird stuff (not the first time...).

I use "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0rc1) Gecko/20020417".

Comment 9

[Bradley Baetz (:bbaetz)]

No, AFAICS, the response from the webserver is perfectly valid. Theres something
else going on.

yahoo is top100

Comment 10

[David Barnes]

There is a slight difference in the body of the response between Mozilla and IE
user agents. Seems like the Mozilla response is slightly corrupted by Yahoo
maybe? Look how the 'encoded' is somehow mangled.

    
HTTP/1.0 302 Found
Date: Sun, 21 Apr 2002 18:40:50 GMT
Location: http://google.yahoo.com/bin/query?p=akamait
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html; charset=iso-8859-1

The document has moved <A
HREF="http://google.yahoo.com/bin/query?p=akamait">here</A>.<P>
<P>Additionally, a 302 Found
error was enco3b     
hile trying to use an ErrorDocument to handle the request.

0

Comment 11

[Bradley Baetz (:bbaetz)]

Ooh. didn't notice that.

Yeah, the chunked encoding looks wrong. Hmmm. Should we care about that for
redirection responses? Why are we spending time parsing it anyway? I guess we
have to, to find the end...

What is the response with ie, then?

Comment 12

[Lorenzo Colitti]

Don't know if this helps, but it works through a HTTP 1.1 proxy (squid). Without
the proxy, it doesn't work.

This is with 2002041711 (1.0 RC1) on WinME.

Comment 13

[Antonio.Xu]

i only can use proxy,so i can not reproduce this bug.:-(

Comment 14

[Joseph Lindenberg]

Same thing happens with "akamaite", "akamaitec", "akamaitech", "akamaitechn",
"akamaitechno", "akamaitechnol", "akamaitechnolo", "akamaitechnolog",
"akamaitechnologi".

Also happens if you append various punctuation marks, like commas and periods,
to any of the above.

Interestingly enough, it doesn't happen with any _shorter_ substrings, like
"akamai", "akama", etc.

So, it only happens with substrings of length 7 - 16 characters, of the string
"akamaitechnologies".

Comment 15

[Joseph Lindenberg]

If, from www.yahoo.com, you click on "advanced search", then enter "akamait" in
the text box on that page, the same thing happens.  HOWEVER, if you choose
something other than "Yahoo!" from the pull-down menu, it works fine.

Comment 16

[Darin Fisher]

in linux build 2002042021, i'm able to repro this problem.  if i switch to
HTTP/1.0, then it works fine.  bbaetz: it is still using keep-alive
connections...  i don't see us sending "Connection: close".

this looks to me like a server bug.  what the heck is it doing sending a
Transfer-Encoding: chunked in a HTTP/1.0 response?!  i'm tempted to send this
over to evangelism.

Comment 17

[Darin Fisher]

here's what i get from a telnet session:

<telnet>
src> telnet search.yahoo.com 80
Trying 66.218.71.128...
Connected to search.yahoo.com.
Escape character is '^]'.
GET /bin/search?p=akamait HTTP/1.1
Host: search.yahoo.com

HTTP/1.0 302 Found
Date: Wed, 24 Apr 2002 05:31:24 GMT
Location: http://google.yahoo.com/bin/query?p=akamait
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html; charset=iso-8859-1

The document has moved <A
HREF="http://google.yahoo.com/bin/query?p=akamait">here</A>.<P>
<P>Additionally, a 302 Found
error was enco3b     
hile trying to use an ErrorDocument to handle the request.

0

</telnet>

notice that the chunked body is malformed.  there is no length indicator for the
first chunk.  one solution to this problem would be to ignore Transfer-Encoding
header if the response is HTTP/1.0.  that would be fairly simple and would do
the trick here since the server has sent a Connection: close header.

other than that, i can't think of a decent work around for this fck'd server.

-> mozilla 1.0

Comment 18

[Darin Fisher]

Attachment: v1 patch

this patch makes us ignore a Transfer-Encoding header if the response is
HTTP/1.0

Comment 19

[Darin Fisher]

bbaetz: can you please r= this patch?  thx!

Comment 20

[Lorenzo Colitti]

This looks like a high-risk change just to support one broken
server/load-balancer.  Is it the right thing to do for standards compliance? If
not, perhaps evangelism would be better... ?

Comment 21

[Bradley Baetz (:bbaetz)]

Why can't a 1.1 server send a chunked encoding? Our request advertises that we
support it....

Comment 8 suggests that changing the useragent to ie makes us work. Is this the
case? I'm still tempted to send this to evangalism.

However, what about this:

For a 302 response, do we need to decode the content? In theory, we need to do
so to find the end of the connection. For chunked we do, but why bother gzipping
stuff. Can't we just treat invalid content in a 3xx series response as wanting
us to close the connection? And if we get a connection: close, do we even need
to do that? Wouldn't that be faster. Or do we show this content in some case
(like if the site we're reditected to doesn't exist, or something)?

Comment 22

[Darin Fisher]

> Why can't a 1.1 server send a chunked encoding? Our request advertises that 
> we support it....

i assume you mean 1.0 server...

> For a 302 response, do we need to decode the content? In theory, we need to 
> do so to find the end of the connection. For chunked we do, but why bother 

we need to find the end of the transaction to support keep-alives and pipelining.

> gzipping stuff. Can't we just treat invalid content in a 3xx series response 

we don't handle "Transfer-Encoding: gzip" here... it'd be handled by the channel
if we ever needed to display the message body.

> as wanting us to close the connection? And if we get a connection: close, do >
we even need to do that? Wouldn't that be faster. Or do we show this content >
in some case
> (like if the site we're reditected to doesn't exist, or something)?

we could ignore a Transfer-Encoding header if the server sends "Connection:
close" ...that would work.

i'm happy to reassign this to evangelism, but it just seems odd to handle a
chunked response from a HTTP/1.0 server.  a HTTP/1.0 server presumably knows
nothing about chunked encodings.  maybe it just picked up the header
accidentally... via some other interal server or whatever, which is what i
suspect is going on here.  so, this patch is perfectly conformant with the HTTP
standards, and it is very low risk IMO.

that said, when i originally wrote this code, i wanted to make mozilla as
flexible as possible.  e.g., as bradley said, why shouldn't mozilla accept a
chunked response from a HTTP/1.0 server... afterall, mozilla did say it supports
chunking, right?  that's a good argument IMO.  so, i'm not really sure at this
point what would be best for mozilla.  maybe we should just shove this off to
evangelism :-/

as far as the useragent string is concerned, maybe they generate junk for any
unknown useragent?!  i don't know... cuz when i just connected to the server via
telnet, and issued a simple request (w/o specifying a User-agent header, i still
got the same junk sent to mozilla).

Comment 23

[Andrew Hagen]

We send an HTTP/1.1 request, and we get back an HTTP/1.0 response with a
Transfer-encoding header. The header would work in a 1.1 response. In 1.0,
though, it is not well formed. The strangeness causes us to not accept the
response. In this case, however, we should try to accept it.

Under the robustness principle, RFC 791, http://www.faqs.org/rfcs/rfc791.html ,
we should be conservative with what we send out, and liberal with what we
receive. See also RFC 2145. The robustness principle would support accepting the
response.

Comment 24

[Darin Fisher]

on that note...

-> Tech Evangelism

Comment 25

[Bradley Baetz (:bbaetz)]

The header works, the problem is that the content is invalid.

If this were a widespread problem, I'd support the patch, but it sounds like
someone at yahoo.com just forgot to print out the size of the first chunk. So
yeah, TE should handle it.

Comment 26

[Andrew Hagen]

Today, with 1.0 RC2 branch Build ID: 2002050106 Windows 98, it works. Don't know
if the site changed or if Mozilla changed. Resolving worksforme.

Comment 27

[Joseph Lindenberg]

Although the problem no longer occurs with "akamait", it still occurs with
"akamaite", "akamaitec", "akamaitech", and so on, up to "akamaitechnologi".  See
what I originally said in comment #14.

I'm reopening the bug.  Sorry.

Comment 28

[Andrew Hagen]

Somewhat ironically, "akamai" is Hawaiian for smart, intelligent.

Comment 29

[Joseph Lindenberg]

Similar problem occurs if you try to click on "Real Estate" (in the "Shop" line)
on Yahoo's main page.  The URL it links to is www.yahoo.com/r/r1, but when you
click it, the same sort of thing happens.

Thanks to Jim Reynolds for pointing this out.  He also says it works with HTTP
1.0 but not HTTP 1.1

Comment 30

[Joseph Lindenberg]

This is a more serious issue, that you can't use the "Real Estate" link on
Yahoo.  If the root cause of this problem is indeed the same as the cause of the
"akamaite" searching problem, Tech Evangelism would find it easier to convince
Yahoo to look into this problem.

Comment 31

[Bob Clary [:bc] (inactive)]

->evang500

Comment 32

[Kai Lahmann (is there, where MNG is)]

don't see it..

Comment 33

[Matthias Versen [:Matti]]

*** Bug 162702 has been marked as a duplicate of this bug. ***

Comment 34

[Malcolm Rowe]

*** Bug 162619 has been marked as a duplicate of this bug. ***

Comment 35

[Kelly Hirano]

i'm a yahoo employee and notified the powers that be about this error. there was
a problem with the search servers and it appears to have been fixed. it also
looks like the real estate link works now. this has been tested on my bulid
2002081603. i have http 1.1 networking on.

Comment 36

[Mike Potter]

I can confirm that this bug is fixed.  Marking as such.

Comment 37

[Andrew Hagen]

Thank you and congratulations to Kelly Hirano and the Yahoo! team for the fix!

Comment 38

[Mike Potter]

I'm reopening this bug, since it also affects the international sites, like
Yahoo.ca, and they haven't been fixed.
Users in Canada are unable to search for text that does not appear in the Yahoo
directory.
Hopefully we can get this fixed so my girlfriend can keep using Mozilla to
search, and I don't have to keep hearing about how IE is better because Mozilla
can't search...
(Adding [e-mail] to the cc: list, since he works for Yahoo)

Comment 39

[Kelly Hirano]

it seems to work for me on my mac using the mozilla 1.1 build (Gecko/20020826).
also seems to work on my win2k box running the same version.

so this http://ca.search.yahoo.com/search/ca?p=akamaite&y=on url works for me.
can someone else confirm?

Comment 40

[Mike Potter]

I can confirm that the URL works, however the word "akamaite" is now in the
directory.
oeone is not in the directory, so the URL: 
http://ca.search.yahoo.com/search/ca?p=oeone&y=on
does not work.

Comment 41

[James Botte]

I had been experiencing this problem with yahoo.ca for some time but chalked it
up to a release 1.0 Mozilla (it worked in Netscape 4.72 so...). When 1.1 came
out and I had the same problem, I searched and found this bug report. I have
mailed (and I *mean* snail-mailed) Yahoo!Canada a letter informing them of the
problem, letting them know it was their server (and that it was fixed already in
the US), pointing them at the URL for bug 138869, and asking them to please let
me know if and when they are going to fix it. Their only contact information was
a physical mailing address in Toronto. Ugh! Bang those rocks together folks!
Please let me know if anyone wants a copy of the letter (PDF) for posterity or
information.

Comment 42

[Mike Potter]

Maybe Kelly Hirano can help evangelise this to the international Yahoo sites. I
can confirm that it is still broken for Yahoo.ca if you search for "oeone".

Comment 43

[Malcolm Rowe]

*** Bug 170286 has been marked as a duplicate of this bug. ***

Comment 44

[Malcolm Rowe]

Still affects kr.yahoo.com (bug 170286) and presumably other international 
sites.

Comment 45

[Susie Wyshak]

removing topembed kw since it's an evangelism issue.

replicated problem on ca.yahoo.com in 11/20 commercial branch.
Doron please see what the problem is. Search button seems to be dead.

Comment 46

[Mike Potter]

I just noticed that I can go to the URL in the bug summary it works fine, but if
I go to the Yahoo.ca page directly and type in "oeone" and then click on the
search button, nothing happens.

Therefor removing http://ca.google.yahoo.com/bin/query_ca?p=oeone URL from the bug.

Comment 47

[Doron Rosenberg (IBM)]

kr and ca are the only ones I could find that are affected.  

Comment 48

[Alan Batie]

With the new yahoo page that seems to have gone up last night with a "Christmas"
theme, Chimera can no longer search anything.  You get a page that says no
search terms were entered.  Mozilla works fine.

Comment 49

[Doron Rosenberg (IBM)]

could someone sum up what yahoo would have to fix so we can communicate to them?

If I understand correctly, our HTTP/1.1 request gets a HTTP/1.0 response with a
malformed Transfer-encoding header (Transfer-Encoding: chunked). What should the
correct one look like?

thanks!

Comment 50

[Katsuhiko Momoi]

HTTP 1.0 does not define transfer-encoding header and so it is illegal to use
chunked transfer encoding header with this version.

So, 

1. They should correct the HTTP version to 1.1 when sending this
   header, and if so, the chunked encoding should follow the correct
   syntax as defined here:

http://www.cis.ohio-state.edu/cgi-bin/rfc/rfc2616.html#sec-3.6.1

or

2. They should not use chunked transfer encoding when using HTTP 1.0.

Comment 51

[Katsuhiko Momoi]

I don't see this problem with yahoo.co.jp. 

Comment 52

[Dave Minogue]

I think I've fixed this for Y! Canada.
Let me know otherwise.

Comment 53

[Andrew Hagen]

I can confirm that. With Phoenix 0.5, I can now search for "oeone" from the
Yahoo! Canada page. 

If there are no known problems anymore, let's close this bug.

Comment 54

[Doron Rosenberg (IBM)]

resolved fixed, go susie.

Comment 55

[Susie Wyshak]

Verified. YAHOOOOOOOOOOOOOOOooo!

Comment 56

[Mike Potter]

Reopening the bug... This is still happening for kr.yahoo.com

Comment 57

[Dave Minogue]

I've sent the answer to one of their engineers.
should be fixed shortly

Comment 58

[Jason Smith]

Well I don't know if this is related to this bug or not... But when I go to
www.yahoo.com initiate a search it completes fine. But when I click on next (to
continue looking at search hits) I get connection refused when attempting to
contact rd.yahoo.com. This is a rather recent occurence.

Comment 59

[JHB]

I just tested 'oeone', 'BitchX' etc., and all results are fine :).
Tested keywords:
 - oeone
 - BitchX, itch, tchx and bitc
 - akamait

I think you could remove 'Yahoo Korea' from the list now.

Thanks,

Comment 60

[JHB]

Oops!

I'm sorry I was confused something...(http 1.0)
Yahoo Korea is still remained trouble site on this bug.

Sorry for confusing you.

Comment 61

[JHB]

Finally, I could search "BitchX" through 'Yahoo Korea' with http 1.1 mode.

I think 'Yahoo Korea' could be removed from this title.

I used Mozilla(Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.3b) Gecko/20030205)
at http 1.1 mode and following keywords:

 - oeone
 - BitchX, itch, tchx and bitc
 - akamait


Thanks even though I don't know who did it!

p.s.)

You could get 'no results' page of yahoo Korea with 'akamait' :-).
http://kr.softwise.yahoo.com/bin/query?p=akamait

Comment 62

[Mike Potter]

Searching for oeone on kr.yahoo.com still does not work for me using Mozilla
1.2.1 on RH 8.

Comment 63

[JHB]

I'm using Mozilla 1.3b(20030205 build) as I mentioned.
I can get searching result with "oeone " in kr.yahoo.com.
---
Then, Mozilla team deserve my thanks for fixing this problem. :-)

Comment 64

[Andrew Hagen]

There are no issues left to resolve here. Yahoo has fixed the problem. Resolving
as fixed.

Thank you to the Yahoo team.