183538 - Blue Coat Proxy server sends two Content-Type HTTP headers and causes us to not feed byte ranges to Acrobat for PDFs

Status: RESOLVED FIXED

(Tech Evangelism Graveyard :: English US, defect, P1)

Description

[lmcquarr]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.2.1) Gecko/20021130
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.2.1) Gecko/20021130

There is a certain class of PDF files that is triggering a very, very serious
issue in Mozilla (can also repo in NS 7).   This class of files (of which the
test file above is just one example) often issue a byte range request very quickly
after the document opens.  (The above example is a file that opens with
Thumbnails ... to get the thumbnails, Acrobat turns around an issues a pretty
quick byte ranges request ... just an example.)

In these circumstances,  a very very terrible thing happens:

1 - Acrobat tells Mozilla to issue the byte range requests.  Mozilla complies
(can see the requests in a Windump log ... which I will put in the attachements). 

2 - The server downloads the requested ranges (which I can also see in the
windump log).

3 - Mozilla does not give us all of the ranges  (I can see this because
I am logging all of the calls to NPP_Write ... saving the offset and length
of the data.)

4 - Because Acrobat requested the data already and it needs it,
it will hang (and hang mozilla) for five minutes until it times
out ... and then usually gives an error, depending on what feature
it was waiting for data for.

YIKES!    
Unfortunately, this is a very, very serious issue. 


Reproducible: Always

Steps to Reproduce:
1. Install Acrobat 5
2. Go to the above URL in mozilla

Actual Results:  
3. HANG in acrobat and mozilla ... wait five minutes or kill the apps


Expected Results:  
Displayed the PDF file.

Two attachments:

1 - The windump log showing the HTTP conversation
2 - My dump of what our Acrobat plug-in NPPDF32.dll gets
passed via NPP_Write  (you can do something similar on your end,
or I can give you my test plug-in to try ... just let me know).

Comment 1

[lmcquarr]

Attachment: Output file from Windump showing byte range request

This log shows the byte range requests going ont ... and being received from
the server

Comment 2

[lmcquarr]

Attachment: NPP_Write calls (offset + length) That Acrobat Gets From Mozilla

This is a log of the NPP_Write calls that Acrobat gets from mozilla.  Notice
that the biggest byte ranges are omitted (e.g. 2,083, 512).  Unfortunately,
those bytes are critical to Acrobat doing its thing (cross reference table). 

I will put a log of NPP_Write calls for Netscape 4.7X in another attachment...
just as a comparison

Comment 3

[lmcquarr]

Attachment: Dump From NPP_Write for Same File With Netscape 4.7X

This is the same NPP_Write dump for the same bug file ... only this one is
generated from Netscape 4.7X, which does not have the bug.

Comment 4

[lmcquarr]

BTW -- unfortunately, have have many more cases that will repo this bug,
but at this point, they are internal files.  If you need any more
cases, let me know and I will scrounge a few up ...

Comment 5

[serge (gone)]

I cannot reproduce the hang on w2k  w/ 
http://acroeng.adobe.com/BrowserBugTestCases/549072/openwiththumbs.pdf
using ns7 (Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.0.1)
Gecko/20020808 Netscape/7.0) or current mozilla debug build w/ acrobat 5.0:(

Comment 6

[lmcquarr]

You need to let it sit for a moment .. it is indeed hung.

What you will see is that the first page draws, and but NO THUMBNAILS
draw  (in the left pane).  The thumbnail code sits there waiting for data
to arrive that does not arrive (at least not to our plug-in). 

Now, try to do something with the document in the window, e.g scroll it.
Or, cover the browser and then uncover it ... the page will be blank
because Acrobat is hung. 

Comment 7

[serge (gone)]

I'm testing nppdf32.dll File version: 5.0.0.327, Size: 103,312 bytes
w/ today's build Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.3a)
Gecko/20021205
No hang, all thumbnails appear right away for me.
Shrir, can you reproduce the hang?

on comment #2
>Notice that the biggest byte ranges are omitted (e.g. 2,083, 512)

hmm, (attachment 108239 [details]) NPP_Write log from mozilla does not contain any
offset==
Range:bytes=1761420-1762965,2003954-2004997,2071272-2073361,1924110-1926824,2083512-2085621,11991-145708
from windump (attachment 108236 [details])
I can assume:
1. NPP_Write log  is not in sync w/ windump log
2. NPP_Write  log is not complete.

Liz, could you attach mozilla log file here, you can get it by setting env vars:
set NSPR_LOG_FILE=c:\tmp\pluginLog.txt
set NSPR_LOG_MODULES=Plugin:9,PluginNPP:9,PluginNPN:9

BTW:  how I can get the log from acrobat5?
I recall the pervious acrobat version prints some output into java console in
ns4.x word, but it should not work w/ mozilla.

Comment 8

[rubydoo123]

reassign

Comment 9

[Andrew Hagen]

Worksforme, 2003022008, Windows 98, Adobe Acrobat Reader 5.1. 

Reporter, are you still seeing this problem on recent nightly builds?

Comment 10

[lmcquarr]

Folks:

Sorry for the silence ... I have been buried. 

We are in the death march to shipping the next version of Acrobat.  And guess 
what? This bug keeps coming up.  I have now many test cases where:

1 - Acrobat requests some ranges via NPN_RequestRead.
2 - Via Windump, I can see the range request go out to the server.
3 - I see the server return the ranges. 

BUT Netscape never gives them to us. 

I am putting together some new test cases for you.  
But I am going to raise the priority of this (also via email to Beth).

I need help figuring out a work around on our side. .. in addition to
a fix in your code in the future.  Exactly what is tickling this problem?
If I know what tickles it, perhaps I can work around it on our
Netscape plug-in.

Since the symptoms of this bug is that the PDF file hangs in the browser
and both apps are essential hung until it times out (five minutes), this
is a BAD bug.

Comment 11

[Peter Lubczynski]

Hey Liz,
Can you generate one of our logs of the failure by setting these environment
variables:
set NSPR_LOG_FILE=c:\pluginLog.txt
set NSPR_LOG_MODULES=nsHTTP:9,Plugin:9,PluginNPP:9,PluginNPN:9
(note: a mozilla trunk build may output more than NS7)

The testcase also WFM (moz debug trunk, acrobat 5.0 3/22/01).

Comment 12

[lmcquarr]

Here is another test case.

1 - Install Acrobat Reader 5.1
2 - Goto to the following URL:

http://acroeng.adobe.com/BrowserTestSuite3/byteserving/opt/fiwbroch.pdf

Result:
blank page display.

Acrobat asked for byte ranges.  mozilla forwarded request to the server.  Server
returned the bytes.  Mozilla never passed them back to us.

Will add dump of read requests and bytes passed back to us and the windump log
showing what bytes were downloaded to this bug. 

Comment 13

[lmcquarr]

Working on the log.  (Trying to make all of the test cases based on 5.1 so that
you folks can work on it without waiting for beta.)

Comment 14

[Andrew Hagen]

That PDF is not blank for me with Acrobat Reader 5.1 plug-in, Mozilla 1.3 branch
nightly build 2003030505 on Windows 98.

Reporter, please try this problem on Mozilla 1.3 beta or later. Do you see the
problem then?

Comment 15

[lmcquarr]

Problem also happens with Mozilla 1.3 beta. :-(   Still working on the logs 
Peter requested (if I run out of time today, I will do it on Sunday). 

Saw something interesting, however, that I am exploring.  On the bottom of the 
mozilla window, I see "Stopped" ... right after the byte range request that we 
never get.

Comment 16

[lmcquarr]

Definitely see "Stopped" on the bottom of the mozilla window when the bug 
occurs.  Have breakpoints set in all of the close/destroy code in the
Acrobat Netscape plug-in but these are never not hit at this point ... only 
when we leave the page.

Running out of steam right now due to illness. On sunday, I will get you the 
log files requested and make more test cases for 5.1 (happens a lot in version 
under development since we changed the model of how we request PDF files some 
what). 

Comment 17

[lmcquarr]

OK folks ... here is something you can try that consistently hangs the app
for me  (because we don't get the bytes we asked for).  Note that the page on 
the right will paint, but the thumbnail pane
on the left will not paint ... because the bytes we asked for for the 
thumbnails do not arrive.   You can not do ANYTHING in either application
until we time out after five minutes.

I have repoed this the Mozilla 1.3 beta (build ID 20030210008).

1. Install Acrobat Reader 5.1 from the Adobe web site.
2 - Start it up and go to the Preferences, under the Edit menu.
3 - Chose "Options" from the left pane and then in the right pane,
disable "Allow Background downloading" by unchecking the checkbox
(default is on ... )
4 - Launch mozilla and go to the above URL.

Results:

PDF page on right paints.  Thumbnails on left do NOT paint and app is hung
for five minutes.  Finally thumbnails paint grey (no data).

Expected:

Thumbnails paint ... app not hung.

Let me know if this DOES not repo the problem for you.  I **am** behind a 
firewall.
This would not be the first bug I have encountered that happens
behind a firewall (I have a machine at home I could try that is not behind
a firewall if you can't repo it). Also note: Mozilla says "Stopped" at the 
bottom of the screen.  (I already verified that Acrobat was not stopping
it by settting breakpoints in our plug-in ... so something in Mozilla
is tickled into stopping.)

Comment 18

[lmcquarr]

A few more comments:

Of course, I would like to have this bug fixed ... I will come up with a few 
more examples.

BUT even more critical, I need some help understanding what is tickling this 
bug to happen, in the event that there is SOMETHING I can do in Adobe code
to prevent it from happening.   My hope is that:

for versions XXX to YYYY -- do some workaround
For > YYYY, skip the work around.

I only have about two weeks left of opportunity to do anything in the
next version of Acrobat ...   Otherwise, I will have to disable byte
range requests for mozilla-based browsers for versions XXXX to YYYY.   It is 
not the
end of the world, but it would be very sad.

Thanks in advance, Liz
(and sorry I didn't push this earlier ... I have been BURIED ... had pneumonia 
at Christmas and have been struggling to catch up ever since. )

Comment 19

[Peter Lubczynski]

Attachment: nspr log

okay Liz, I see the problem with the new testcase: thubnails now showing up. I
see your NPN_RequestReads and the browser closing the channel. Investigating
why....

Comment 20

[lmcquarr]

PPHHEEWWW!  Thanks.
Just in case I don't respond and you need info ... give me a call. 
(OK ... guess I will start up my AIM again too :-)

Comment 21

[lmcquarr]

Here are a couple more URLs that will display the problem (assuming that
you have disabled background downloading in the Acrobat Reader prefs):

http://acroeng.adobe.com/BrowserTestSuite3/testcases/pagelabels/Huck%20Finn.pdf
http://acroeng.adobe.com/BrowserTestSuite3/testcases/pagelabels/Middlemarch.pdf
http://acroeng.adobe.com/BrowserTestSuite3/testcases/opendocinfo/page2.pdf

Comment 22

[lmcquarr]

More URLs (again, make sure that background downloading is disabled).

http://acroeng.adobe.com/BrowserTestSuite3/testcases/namedest/despair.pdf#stupi
dity

(this one has opens to page one and then tries to jump to another page ... the 
#stupidity at the end is the key.  But you will see "Stopped" at the bottom
of the mozilla window and the stupidity page never completely paints)


http://acroeng.adobe.com/BrowserTestSuite3/testcases/structure/PDFSPEC.pdf
(you will get a "file I/O error" .. that is a bug in 5.1. You shouldn't see 
that stuff ... but it is complaining because we didn't get some data
associated with bookmark panel on the left.)

OK ... that is probably enough ways to repo this for now. 

Comment 23

[lmcquarr]

Peter -- I have reproduced this from a machine that is a direct connect to the 
Internet (no firewall or proxy).

I had a thought: perhaps you  need to clear your cache?

Then double check your "background downloading pref".

Here is an easy experiment that I did:

1 - Leave background downloading pref to enabled (default).

2 - Go to
http://acroeng.adobe.com/BrowserTestSuite3/testcases/opendocinfo/page2.pdf

3 - Page comes up right away ... everything is happy.

4 - Clear your cache.

5 - Set your background download pref to NO (disable).

6- Go back to the URL above.   

7- Hang/blank.

Interesting difference from what I see in the office, however.  
I do not see "Stopped" at the bottom of the mozilla screen. 
It just says "Transferring data". 

Comment 24

[Charles Rosendahl]

I can confirm the same problem using acrobat reader 4.0 as well as outlined in
comment 23.

I have trouble reproducing it consistently, but it is consistent enough to note
that it exists.

details: win2k, clean cache, randomly click the links in the defect to open
acroreader in a new window.

results:  most of the time the window opens successfully, occasionally, the
window 'hangs' and the cpu spins at 100% between adobe and netscape.

Comment 25

[Peter Lubczynski]

Okay, here's the problem:

During |nsPluginByteRangeStreamListener::OnStartRequest|,
|nsMultiMixedConv::OnStartRequest| returns error because
|httpChannel->GetResponseHeader| returns 'application/pdf' instead of
'multipart/byteranges; boundary=xxxxx' when there are _two_ 'Content-Type' HTTP
headers.


In Liz's windump capture in attachment 108236 [details], if you scroll down to the first
HTTP 206 repsonse, you'll notice two 'Content-Type' headers. This is what's
causing the problem. Network captures I do on these PDF testcases only show a
single 'Content-Type' header with 'boundry' which is why it works. Our code
seems to prefer the last header which in this case causes our byte range code to
fail. Not sure who's bug this is and I can't really make heads or tails about
what the spec says about double HTTP headers, cc:ing Darin for input.

Comment 26

[Darin Fisher]

if at all possible we should get the website to fix itself.  that said, i
imagine IE manages to work fine some how.  probably it prefers the first
content-type header instead.  in our case, i think we overwrite the first
content-type header with the second.  we could ignore the second to be
consistent with IE.

Comment 27

[Boris Zbarsky [:bzbarsky]]

Well... per _spec_,

Content-Type: foo
Content-Type: bar

is equivalent to:

Content-Type: foo, bar

iirc.  So I think we can safely assume that following the spec will not fix this
bug.. ;)

Some testing of what IE really does would be great; I suspect our behavior is
the way it is for a reason, and changing it should be done carefully...

Comment 28

[Darin Fisher]

bz: right, i seem to recall some modifications to ParseContentType related to
dealing with duplicate content-type headers, but IIRC it was all about
suppressing identical duplicates (text/html, text/html).

Comment 29

[lmcquarr]

Well, I tried "the other browser" and in our configuration here,
it does not get the double content-type header (using the same test files, 
same computer, same network setup).  Go figure!  Exploring what it is in our 
environment here that is tickling this behavior (proxy?firewall?):

HTTP/1.1 206 Partial Content
Connection: Keep-Alive
Date: Tue, 18 Mar 2003 23:46:47 GMT
Server: Apache/1.3.26 (Unix)
Last-Modified: Sat, 11 Nov 2000 03:17:00 GMT
ETag: "50aa-cee02-3a0cba2c"
Accept-Ranges: bytes
Content-Length: 100121
Content-Type: multipart/byteranges; boundary=3e77afe7aca


--3e77afe7aca
Content-type: application/pdf
Content-range: bytes 825448-847361/847362

Comment 30

[lmcquarr]

Folks --
the culprit is a BlueCoat proxy server. Unfortunately, these are pretty wide 
spread.  I will be contacting them via our support contract to get it fixed.
Stay tuned.

And thanks for your hard work, Peter!

Comment 31

[lmcquarr]

I sent a bug report to Blue Coat support.  If you folks feel that you are 
following the spec correctly, you can close this as a "NotOurProduct" bug.
Thanks again, for seeing me through this issue to the gory end. 

Comment 32

[Boris Zbarsky [:bzbarsky]]

Over to evangelism to help convince them to fix the problem..

Comment 33

[Peter Lubczynski]

-->aruner

Comment 34

[lmcquarr]

The support case at blue coat is 
SR#1-4328801 Blue Coat Port 80 Security Appliance hangs with Acrobat and 
Netscape 7.0 



The problem has been resolved!  Here is the info:


Hello Jay,

This has been fixed and will be part of our next release, Security Gateway 
2.1.07 (expected release is early May).

Krista 

Krista Von Mulldorfer 
Senior Technical Support 
Blue Coat Systems, Inc. 
The Web Security Authority 
888.702.3569 USA TOLL FREE 
408.220.2270 DIRECT 
www.bluecoat.com 

APAC Support Center 

Comment 35

[Bob Clary [:bc] (inactive)]

-> evang500

Comment 36

[Bob Clary [:bc] (inactive)]

per comments->fixed.