Ability to observe HTTP responses as they come

NEW
Unassigned

Status

()

Core
XPConnect
8 years ago
7 years ago

People

(Reporter: Honza, Unassigned)

Tracking

Trunk
x86
Windows Vista
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [firebug-p2])

Attachments

(4 attachments, 2 obsolete attachments)

(Reporter)

Description

8 years ago
Created attachment 413090 [details]
Examle extension with nsIOutputStream wrapper (JS)

This bug is created as a fork of bug 515051, see also comment:
https://bugzilla.mozilla.org/show_bug.cgi?id=515051#c83

As it was mentioned in the bug above, Firebug is monitoring all incoming HTTP responses to show them to the user if necessary. This is mainly utilized by the Script panel (to show all the JS source code) and the Net panel (to show all responses).

Since Firebug allows to break on JS error (which can happen during the page load phase), the responses must be caught as they are coming from the network channel (so they are available at the right time). It would be too late to wait for onStopRequest.

Till the bug 515051, Firebug has been using onDataAvailable. Since API has been changed, and now "@mozilla.org/network/stream-listener-tee;1" is used as the request (nsIHttpChannel) stream listener (using nsITraceableChannel), this method isn't available (listener-tee doesn't propagate this to the nsIRequestObserve).

An other way is to use a nsIOutputStream wrapper (implemented in JS) as a sink passed to the listener-tee and use write method (instead of the former onDataAvailable) as the point where Firebug gets the response (but, perhaps there is even another way?). Notice that the wrapper is also use to limit size of the response so, Firebug can safe memory in case of large data.

However, the wrapper doesn't work and there are some exceptions (probably related to xpconnect).

1) Using the wrapper to limit size of the response and there is following exception:

     msvcr90d.dll!_strlen()  + 0x30 bytes    
     js3250.dll!JS_NewStringCopyZ(JSContext * cx=0x00df07d8, const char *
s=0x08ef3fa8)  Line 5266 + 0x9 bytes    C++
>	xpc3250.dll!XPCConvert::NativeData2JS(XPCLazyCallContext & lccx={...}, int * d=0x0012f28c, const void * s=0x0012f498, const nsXPTType & type={...}, const nsID * iid=0x0012f34c, JSObject * scope=0x05533180, unsigned int * pErr=0x00000000)  Line 362 + 0xe bytes	C++
     xpc3250.dll!XPCConvert::NativeData2JS(XPCCallContext & ccx={...}, int *
d=0x0012f28c, const void * s=0x0012f498, const nsXPTType & type={...}, const
nsID * iid=0x0012f34c, JSObject * scope=0x05533180, unsigned int *
pErr=0x00000000)  Line 2974 + 0x24 bytes    C++
     xpc3250.dll!nsXPCWrappedJSClass::CallMethod(nsXPCWrappedJS *
wrapper=0x0658ed30, unsigned short methodIndex=0x0005, const
XPTMethodDescriptor * info=0x01a29ef0, nsXPTCMiniVariant *
nativeParams=0x0012f498)  Line 1570 + 0x31 bytes    C++
     xpc3250.dll!nsXPCWrappedJS::CallMethod(unsigned short methodIndex=0x0005,
const XPTMethodDescriptor * info=0x01a29ef0, nsXPTCMiniVariant *
params=0x0012f498)  Line 571    C++
     xpcom_core.dll!PrepareAndDispatch(nsXPTCStubBase * self=0x08dc8450,
unsigned int methodIndex=0x00000005, unsigned int * args=0x0012f558, unsigned
int * stackBytesToPop=0x0012f548)  Line 114 + 0x21 bytes    C++
     xpcom_core.dll!SharedStub()  Line 142    C++
     xpcom_core.dll!nsInputStreamTee::TeeSegment(const char * buf=0x08ef3fa8,
unsigned int count=0x00000015)  Line 80 + 0x2b bytes    C++
     xpcom_core.dll!nsInputStreamTee::WriteSegmentFun(nsIInputStream *
in=0x065298e8, void * closure=0x07deea28, const char * fromSegment=0x08ef3fa8,
unsigned int offset=0x00000000, unsigned int count=0x00000015, unsigned int *
writeCount=0x0012f5f4)  Line 110    C++
     xpcom_core.dll!nsStringInputStream::ReadSegments(unsigned int
(nsIInputStream *, void *, const char *, unsigned int, unsigned int, unsigned
int *)* writer=0x00488190, void * closure=0x07deea28, unsigned int
aCount=0x00000015, unsigned int * result=0x0012f5f4)  Line 276 + 0x22 bytes   
C++
     xpcom_core.dll!nsInputStreamTee::ReadSegments(unsigned int (nsIInputStream
*, void *, const char *, unsigned int, unsigned int, unsigned int *)*
writer=0x03114310, void * closure=0x093e85d8, unsigned int count=0x00000015,
unsigned int * bytesRead=0x0012f5f4)  Line 157    C++
     gklayout.dll!nsXMLHttpRequest::OnDataAvailable(nsIRequest *
request=0x08bda650, nsISupports * ctxt=0x00000000, nsIInputStream *
inStr=0x07deea28, unsigned int sourceOffset=0x00000000, unsigned int
count=0x00000015)  Line 1886    C++


2) Using the wrapper to store parts of the response as they come
(within the nsIOutputStream.write method). There is following exception:

     msvcr90d.dll!__VEC_memcpy()  + 0x140 bytes    
     msvcr90d.dll!__VEC_memcpy()  + 0x52 bytes    
>	xpcom_core.dll!nsSegmentedBuffer::AppendNewSegment()  Line 108 + 0x12 bytes	C++
     xpcom_core.dll!NS_InvokeByIndex_P(nsISupports * that=0x08322674, unsigned
int methodIndex=0x00000005, unsigned int paramCount=0x00000003, nsXPTCVariant *
params=0x0012d0e8)  Line 103    C++
     xpc3250.dll!XPCWrappedNative::CallMethod(XPCCallContext & ccx={...},
XPCWrappedNative::CallMode mode=CALL_METHOD)  Line 2721 + 0x21 bytes    C++
     xpc3250.dll!XPC_WN_CallMethod(JSContext * cx=0x00ee4ff8, JSObject *
obj=0x06f80920, unsigned int argc=0x00000002, int * argv=0x022ff920, int *
vp=0x0012d3ac)  Line 1740 + 0xe bytes    C++
     js3250.dll!js_Invoke(JSContext * cx=0x00ee4ff8, unsigned int
argc=0x00000002, int * vp=0x022ff918, unsigned int flags=0x00000002)  Line 1360
+ 0x17 bytes    C++
     js3250.dll!js_Interpret(JSContext * cx=0x00ee4ff8)  Line 2240 + 0x16 bytes
   C++
     js3250.dll!js_Invoke(JSContext * cx=0x00ee4ff8, unsigned int
argc=0x00000002, int * vp=0x022ff8f8, unsigned int flags=0x00000000)  Line 1368
+ 0x9 bytes    C++
     xpc3250.dll!nsXPCWrappedJSClass::CallMethod(nsXPCWrappedJS *
wrapper=0x09185bb8, unsigned short methodIndex=0x0005, const
XPTMethodDescriptor * info=0x00f14af8, nsXPTCMiniVariant *
nativeParams=0x0012df6c)  Line 1671 + 0x1b bytes    C++
     xpc3250.dll!nsXPCWrappedJS::CallMethod(unsigned short methodIndex=0x0005,
const XPTMethodDescriptor * info=0x00f14af8, nsXPTCMiniVariant *
params=0x0012df6c)  Line 571    C++
     xpcom_core.dll!PrepareAndDispatch(nsXPTCStubBase * self=0x09157ef8,
unsigned int methodIndex=0x00000005, unsigned int * args=0x0012e02c, unsigned
int * stackBytesToPop=0x0012e01c)  Line 114 + 0x21 bytes    C++
     xpcom_core.dll!SharedStub()  Line 142    C++
     xpcom_core.dll!nsInputStreamTee::TeeSegment(const char * buf=0x0856ff04,
unsigned int count=0x000009d4)  Line 80 + 0x2b bytes    C++
     xpcom_core.dll!nsInputStreamTee::WriteSegmentFun(nsIInputStream *
in=0x08ef3f88, void * closure=0x0831b578, const char * fromSegment=0x0856ff04,
unsigned int offset=0x00000000, unsigned int count=0x000009d4, unsigned int *
writeCount=0x0012e0bc)  Line 110    C++

All can be reproduced using the extension I have attached. You need the final patch from bug 515051 (I have tested in 1.9.2). It's easy to reproduce the #2, just try to load a page e.g. www.google.com #1 seems to be more tricky.

I am attaching an extension with implementation of the wrapper that can be used to reproduce these exceptions.

Honza
(Reporter)

Updated

8 years ago
Whiteboard: [firebug-p1]
When you say exception, do you mean crash (win32 exception), or do you mean a js exception?
(Reporter)

Comment 2

8 years ago
(In reply to comment #1)
> When you say exception, do you mean crash (win32 exception), or do you mean a
> js exception?
Crash (win32 exception)
Honza
which version(s) of Firefox/Gecko does this occur on? Guessing 1.9.1 and up based on bug 515051 but it isn't clear.
(Reporter)

Comment 4

8 years ago
I was testing with 1.9.2
Honza
I can't reproduce the crash on linux. I built mozilla-1.9.2, installed the extension, and loaded google.com. No crash.
(Reporter)

Comment 6

8 years ago
Have you also apply the patch from bug 515051 ?
https://bugzilla.mozilla.org/attachment.cgi?id=412611&action=edit

Honza
That's already checked in on 1.9.2, right?
(Reporter)

Comment 8

8 years ago
Created attachment 413593 [details]
Firebug 1.5
(Reporter)

Comment 9

8 years ago
Created attachment 413594 [details]
FBTest 1.5 (Firebug's test harness)
(Reporter)

Comment 10

8 years ago
(In reply to comment #7)
> That's already checked in on 1.9.2, right?
True

From some reason, I can't found simpler test case than Firebug itself. Of course, it can also be a bug in Firebug, but at least I think Fx shouldn't crash.

I am attaching two extensions (a) Firebug 1.5 and (b) FBTest 1.5 (Firebug's test harness).

1) Install both
2) Firefox Menu : Tools->Firebug->Open Test Console
3) Press Run All in the test console (some tests fail, but don't worry).

For me Fx always crashes during the test.

Honza
I can reproduce the crash with those XPIs. My crash is somewhere inside the jsd APIs though.
#0  0x00007ffff76e6c94 in js_GetOpcode (cx=0x7fffd98a0c00, script=0x7fffe3555a90, pc=0xe3555b7a <Address 0xe3555b7a out of bounds>) at /usr/local/google/home/cbiesinger/mozilla-1.9.2/js/src/jsscript.h:309

There's a thrown exception on the stack too...
So one problem is this, but it doesn't seem to the one causing the crash:

The nsIOutputStream interface looks like:
124     unsigned long write(in string aBuf, in unsigned long aCount);
So if C++ passes binary data to an implementation of this interface, it will be truncated at the first null byte. So the length is shorter than count.

But then you call stream.write(buf, count). This will read uninitialized memory since only the data up the null byte in buf is initialized.
Another problem: basically the same problem just slightly differently: If the string does not contain a null byte then strlen will read uninitialized memory when converting the const char* to a JS string. This is the cause of your first crash, most likely.
(Reporter)

Comment 14

8 years ago
(In reply to comment #13)
> Another problem: basically the same problem just slightly differently: If the
> string does not contain a null byte then strlen will read uninitialized memory
> when converting the const char* to a JS string. This is the cause of your first
> crash, most likely.
I see. So, is there any workaround Firebug could use or do we need a new APIs so, Javascript can safely be used here?
Honza
Well we could change the nsIOutputStream interface to include a size_is option for write(). That'd fix those two problems. Without that, I don't think there's a way to implement nsIOutputStream in JS.

That said, there's still another problem somewhere. I still get a crash. It's really hard to track down though :/
Created attachment 415157 [details] [diff] [review]
nsIOutputStream change

This is the nsIOutputStream change. I'm slightly wary of making this change to a frozen interface (I believe it breaks JS callers of nsIOutputStream.write that pass a length smaller than the string), and this may not be the right bug for this patch anyway. But, here's the patch if you want to look at it :)
(Reporter)

Comment 17

8 years ago
(In reply to comment #16)
> This is the nsIOutputStream change.
Yes, I can confirm, the problem is fixed if the patch is applied.

> I'm slightly wary of making this change to a frozen interface
I am trying the pipe idea yet.

Honza
Interesting. It wasn't enough to fix the problem for me.
(Reporter)

Comment 19

8 years ago
Created attachment 415649 [details]
Firebug 1.5 (tracing responses using pipe)
(Reporter)

Comment 20

8 years ago
Created attachment 415650 [details]
FBTest 1.5 (tracing responses using pipe)
Attachment #413593 - Attachment is obsolete: true
Attachment #413594 - Attachment is obsolete: true
(Reporter)

Updated

8 years ago
Attachment #415650 - Attachment description: (tracing responses using pipe) → FBTest 1.5 (tracing responses using pipe)
(Reporter)

Comment 21

8 years ago
(In reply to comment #18)
> Interesting. It wasn't enough to fix the problem for me.
This is because I have only experienced the problem with the write method.

Anyway, I have implemented a new version using pipe and all Firebug tests now pass for me (except script/debuggerKeyword/testDriver.js, but this is another known problem).

I have attached new Firebug build and FBTest build. 

I would be curious whether it works now for you or not...?
(the steps how to run all FB tests are described in comment 10)

https://bugzilla.mozilla.org/attachment.cgi?id=415649
https://bugzilla.mozilla.org/attachment.cgi?id=415650

Honza
(Reporter)

Comment 22

8 years ago
> Anyway, I have implemented a new version using pipe and all Firebug tests now
> pass for me
Forgot to add, I was using Firefox 3.6b5pre
Honza

Updated

7 years ago
Whiteboard: [firebug-p1] → [firebug-p2]
You need to log in before you can comment on or make changes to this bug.