Closed Bug 542120 Opened 16 years ago Closed 9 years ago

crash uploading through XHR [@memcpy | NS_CopySegmentToBuffer(nsIInputStream*, void*, char const*, unsigned int, unsigned int, unsigned int*)]

Categories

(Core :: Networking, defect)

Product:
Core
Component:
Networking
Version:
1.9.2 Branch
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
RESOLVED WORKSFORME
Tracking Flags:
Tracking Status
firefox49 --- affected
firefox50 --- affected
firefox51 --- affected

People

(Reporter: rcampbell, Unassigned)

References

Details

(Keywords: crash, reproducible, Whiteboard: [firebug-p1][necko-backlog])

Crash Data

Attachments

(1 file)

thread stacks + JS stack
16.01 KB, text/plain
Details
Whiteboard: [firebug-p1]
posted to Firebug issues tracker at: http://code.google.com/p/fbug/issues/detail?id=2755
Test case available here: https://myaddiction.ca/bugs/test.html
Updated test case here: https://www.myaddiction.ca/bugs/test2.html Less code required.
Guys: if you want bugs to be fixed, please don't dump them as 'NEW' in Firefox:General. In fact, if i see further such firebug bugs in this state in this component, i will mark them as INVALID. Signature memcpy | NS_CopySegmentToBuffer(nsIInputStream*, void*, char const*, unsigned int, unsigned int, unsigned int*) UUID 6216071b-18fd-436b-9fd5-83f142100121 Time 2010-01-21 12:06:30.122377 Uptime 3195 Product Firefox Version 3.6 Build ID 20100115144158 Branch 1.9.2 OS Windows NT OS Version 6.1.7600 CPU x86 CPU Info GenuineIntel family 6 model 23 stepping 6 Crash Reason EXCEPTION_ACCESS_VIOLATION Crash Address 0x9300000 User Comments uploaded jquery form with 2 MB image through a jquery ui modal-box. custom script, still in development but should not crash FF. maybe caused by some addons though. Processor Notes Crashing Thread Frame Module Signature [Expand] Source 0 mozcrt19.dll memcpy memcpy.asm:188 1 xul.dll NS_CopySegmentToBuffer xpcom/io/nsStreamUtils.cpp:763 2 xul.dll nsBufferedInputStream::ReadSegments netwerk/base/src/nsBufferedStreams.cpp:331 3 xul.dll nsBufferedInputStream::Read netwerk/base/src/nsBufferedStreams.cpp:314 4 @0x5d58fff 5 xul.dll nsMultiplexInputStream::Read xpcom/io/nsMultiplexInputStream.cpp:207 6 xul.dll nsBufferedInputStream::Fill netwerk/base/src/nsBufferedStreams.cpp:378 7 xul.dll nsBufferedInputStream::ReadSegments netwerk/base/src/nsBufferedStreams.cpp:342 8 xul.dll nsHttpTransaction::ReadSegments netwerk/protocol/http/src/nsHttpTransaction.cpp:458 9 xul.dll nsHttpConnection::OnSocketWritable netwerk/protocol/http/src/nsHttpConnection.cpp:574 10 xul.dll nsHttpConnection::OnOutputStreamReady netwerk/protocol/http/src/nsHttpConnection.cpp:785 11 xul.dll nsSocketOutputStream::OnSocketReady netwerk/base/src/nsSocketTransport2.cpp:515 12 xul.dll nsSocketTransport::OnSocketReady netwerk/base/src/nsSocketTransport2.cpp:1564 13 xul.dll nsSocketTransportService::DoPollIteration netwerk/base/src/nsSocketTransportService2.cpp:674 14 xul.dll nsSocketTransportService::OnProcessNextEvent netwerk/base/src/nsSocketTransportService2.cpp:538 15 xul.dll nsThread::ProcessNextEvent xpcom/threads/nsThread.cpp:508 16 xul.dll NS_ProcessNextEvent_P obj-firefox/xpcom/build/nsThreadUtils.cpp:250
Severity: normal → critical
Component: General → Networking
Keywords: crashreportid
Product: Firefox → Core
QA Contact: general → networking
Version: 3.6 Branch → 1.9.2 Branch
timeless: I filed it under firefox::general to get it in the system and have a place to add more data as it became available. I knew it would get filed or we would be able to file it in the correct location later. Thanks again to Peter and Joel for this excellent test-case!
requesting blocking.
blocking1.9.2: --- → ?
uptick on this crash in last few days, maybe as more people get on 3.6w/firebug 20100119-crashdata 25 NS_CopySegmentToBuffer.nsIInputStream 20100120-crashdata 34 NS_CopySegmentToBuffer.nsIInputStream 20100121-crashdata 42 NS_CopySegmentToBuffer.nsIInputStream 20100122-crashdata 70 NS_CopySegmentToBuffer.nsIInputStream 20100123-crashdata 43 NS_CopySegmentToBuffer.nsIInputStream 20100124-crashdata 45 NS_CopySegmentToBuffer.nsIInputStream 20100125-crashdata 92 NS_CopySegmentToBuffer.nsIInputStream whats up with that drop over the weekend? don't firebug users/web developers work on the weekends? ;-)
checking --- 20100125-crashdata.csv NS_CopySegmentToBuffer.nsIInputStream release total-crashes NS_CopySegmentToBuffer.nsIInputStream crashes pct. all 256970 92 0.000358018 3.0.15 1620 0 3.0.16 1047 1 0.00095511 3.5.5 4599 1 0.000217439 3.5.6 3562 6 0.00168445 3.5.7 133251 9 6.75417e-05 3.6 59319 63 0.00106205 3.6b5 2598 0 3.6b4 1144 4 0.0034965 3.6b3 591 0 3.6b2 615 0 3.6b1 904 0
Summary: crash uploading through XHR [memcpy | NS_CopySegmentToBuffer(nsIInputStream*, void*, char const*, unsigned int, unsigned int, unsigned int*)] → crash uploading through XHR [@memcpy | NS_CopySegmentToBuffer(nsIInputStream*, void*, char const*, unsigned int, unsigned int, unsigned int*)]
So does reproducing this require Firebug?
recent addon correlations for the most part say yes... memcpy | NS_CopySegmentToBuffer(nsIInputStream*, void*, char const*, unsigned int, unsigned int, unsigned int*)|EXCEPTION_ACCESS_VIOLATION (52 crashes) 98% (51/52) vs. 4% (2018/48525) firebug@software.joehewitt.com (Firebug, https://addons.mozilla.org/addon/1843) 0% (0/52) vs. 0% (12/48525) 1.05 0% (0/52) vs. 0% (2/48525) 1.1.0b12 0% (0/52) vs. 0% (1/48525) 1.2.1 0% (0/52) vs. 0% (95/48525) 1.4.5 0% (0/52) vs. 0% (1/48525) 1.4X.5 94% (49/52) vs. 4% (1904/48525) 1.5.0 4% (2/52) vs. 0% (3/48525) 1.6X.0a3
Attached file thread stacks + JS stack —
This looks like a threading bug in Firebug. This assertion triggers just before the crash: "ASSERTION: bad stream state: 'mLength >= mOffset'" The root cause is that we have two threads in nsStringInputStream::ReadSegments for the same stream object. One thread is the main thread executing Firebug JS, the other is a networking thread doing a callback from nsSocketTransport::OnSocketReady. I'm attaching the thread stacks when the error occurs.
Firebug's great fortune includes not having threads, so we don't have threading issues per se. The platform callback we are making needs to lock the stream on our behalf. So we need to figure out what callback or API call we use that triggers the crash. Mats Palmgren, we need the version number of Firebug you used for the JS stack trace. I guess 1.5.0 (If possible in future use one of the tracing versions 1.5X.0 so our line numbers match). If I assume that the JS stack in comment 14 has the arguments all backwards and that the older part of the stack has been elided, then I point to http://code.google.com/p/fbug/source/browse/branches/firebug1.5/content/firebug/lib.js#3959 which starts: this.readFromStream = function(stream, charset, noClose) { var sis = this.CCSV("@mozilla.org/binaryinputstream;1", "nsIBinaryInputStream"); sis.setInputStream(stream); var segments = []; for (var count = stream.available(); count; count = stream.available()) segments.push(sis.readBytes(count)); if (!noClose) sis.close(); ... Is nsStringInputStream::ReadSegments thread safe? I guess not.
xpcom methods don't really support locking the way you're asserting they should, and they can't, because they call out to things and so locking the way you're asserting would result in deadlocks. you're supposed to be sure you own an object before you try mutating it. in this case you clearly don't own it and you were caught racing to mutate it.
Ben, can you help me find the person in charge of xpcom/io/nsStringStream.cpp? I don't see an separate entry for xpcom/io in http://www.mozilla.org/about/owners.html
I am in charge, and I believe timeless is right: the stream in question cannot be "locked" the way you seem to want. I still don't understand exactly what Firebug is doing with the stream, so that might help decide what strategy is correct.
"In charge" is probably not quite right... I am the module owner of the code in question, although biesi and others probably have more experience with it because its network-related.
BTW this code works in FF 3.5. Firebug is reading from the stream, storing it to show a developer the network data input to their web app. As far as I can understand it, the data is being streamed in and copied into two (or more) directions on two or more threads. The underlying streaming code needs to deal with the synchronization. Firebug has no information about threads or the second reader. If there is something we need to do in Firebug, we need to know exactly what it is.
no. firebug needs to create a tee, grab the stream, replace the stream with its tee, and assign the stream as the input to the tee. then firebug needs to consume the other part of the tee.
Honza, do you know what timeless is talking about in comment 21?
As far as I can tell, you're reading the uploadStream from nsIHttpActivityObserver's ACTIVITY_SUBTYPE_REQUEST_HEADER notification. Would it be possible for you to wait until ACTIVITY_SUBTYPE_REQUEST_BODY_SENT with reading it? That should be a simple fix for this problem.
FWIW, looking for an xpcom/io solution is inherently impossible. There is nothing that the stream implementation can do to allow simultaneous reads from multiple threads with anything resembling sane semantics.
(In reply to comment #22) > Honza, do you know what timeless is talking about in comment 21? (In reply to comment #21) > no. firebug needs to create a tee, grab the stream, replace the stream with its > tee, and assign the stream as the input to the tee. then firebug needs to > consume the other part of the tee. Yes, this is what Firebug is doing. Here is the logic of getting all responses coming from the server: 1) Catch http-on-examine-response event (and get the request object) 2) Create stream-listener-tee (nsIStreamListenerTee) 3) Set the tee listener into the request (using setNewListener) 4) Create a pipe 5) Initializing the tee with a) original stream listener from the request (returned from setNewListener), b) the pipe (as the output stream) and c) observer (using initWithObserver) 6) Waiting for onStartRequest 7) Using the pipe & asyncWait & onInputStreamReady to get the response body (firebug can't wait for onStopRequest and need the data as they are coming). Honza
(In reply to comment #23) > As far as I can tell, you're reading the uploadStream from > nsIHttpActivityObserver's ACTIVITY_SUBTYPE_REQUEST_HEADER notification. Would > it be possible for you to wait until ACTIVITY_SUBTYPE_REQUEST_BODY_SENT with > reading it? That should be a simple fix for this problem. I'll try it, thanks for the tip! Honza
Another option would be to read it in onStartRequest, but I guess that's too late for you?
(In reply to comment #26) > (In reply to comment #23) > > As far as I can tell, you're reading the uploadStream from > > nsIHttpActivityObserver's ACTIVITY_SUBTYPE_REQUEST_HEADER notification. Would > > it be possible for you to wait until ACTIVITY_SUBTYPE_REQUEST_BODY_SENT with > > reading it? That should be a simple fix for this problem. > I'll try it, thanks for the tip! > Honza Firebug is now reading the upload stream when ACTIVITY_SUBTYPE_REQUEST_BODY_SENT is received, which fixes the crash. Thanks biesi! Not sure if this workaround is good enough for closing the bug. Honza
I think it is. I don't see a way to do anything about this in Gecko.
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → INVALID
blocking1.9.2: ? → ---
up to 100 crashes per day on this crash since 3.6 was released. I think that might rise to the level of opening the bug back up to make sure the work around is enforced though soft/hard blocking of addons (old versions?), or some more looking at possible defenses in gecko. another inforcement idea would be to add this to the addon screening/review checks. checking --- 20100208-crashdata.csv NS_CopySegmentToBuffer.nsIInputStream release total-crashes NS_CopySegmentToBuffer.nsIInputStream crashes pct. all 249769 109 0.000436403 3.0.15 1331 0 3.0.16 564 0 3.5.5 3777 1 0.00026476 3.5.6 2131 0 3.5.7 121742 6 4.92846e-05 3.6 71433 100 0.00139991 3.6b5 1526 1 0.000655308
Crash Signature: [@memcpy | NS_CopySegmentToBuffer(nsIInputStream*, void*, char const*, unsigned int, unsigned int, unsigned int*)]
Steps to reproduce this: 1. Install Firebug 1.9.0 on Firefox 9.0.1 2. Open Firebug on http://valums.com/ajax-upload/ 3. Enabled all panels 4. Reloaded the page 5. Drag a big file (in my case it was around 300MB) into the page 6. Hover the request inside the Net panel of Firebug Doing the same steps in FF 12.0a1 results in other crash signatures. See http://code.google.com/p/fbug/issues/detail?id=2322 as the related Firebug issue. Sebastian
Status: RESOLVED → REOPENED
Keywords: reproducible
Resolution: INVALID → ---
This crash just happened to me again: https://crash-stats.mozilla.com/report/index/25038c49-a07e-4727-866d-fdf1e2130730 Because the test case of comment 32 is gone, I searched for new steps to reproduce: 1. Install Firebug 1.11.4 from addons.mozilla.org 2. Open Firebug at http://www.dropzonejs.com/ 3. Enable and switch to the Net panel 4. Drag a big file (in my case it was around 200MB) into the first "Drop files to upload" area The crash signature is not always the same. Sometimes I get the signature of bug 896164. Sebastian
Crash Signature: [@memcpy | NS_CopySegmentToBuffer(nsIInputStream*, void*, char const*, unsigned int, unsigned int, unsigned int*)] → [@memcpy | NS_CopySegmentToBuffer(nsIInputStream*, void*, char const*, unsigned int, unsigned int, unsigned int*)] [@memcpy | NS_CopySegmentToBuffer]
Whiteboard: [firebug-p1] → [firebug-p1][necko-would-take]
Whiteboard: [firebug-p1][necko-would-take] → [firebug-p1][necko-backlog]
not in current crash stats
Status: REOPENED → RESOLVED
Closed: 16 years ago → 9 years ago
Resolution: --- → WORKSFORME
Well, there are still crashes in "memmove | NS_CopySegmentToBuffer": bp-aa0b6fd3-7e2a-49de-be58-357b92160509 and "msvcr120.dll@0xf608 | NS_CopySegmentToBuffer": bp-4666c1e5-fdc4-416a-bafd-9b8252160510 which looks like this bug to me.
Flags: needinfo?(mcmanus)
... and "memcpy | NS_CopySegmentToBuffer" bp-aa094147-ab6a-4c06-86a9-568922160504 and "NS_CopySegmentToBuffer" bp-9a277b16-194c-4a8f-a55b-860182160510
those are domFileReader and started in 48.. seems totally different, no?
Flags: needinfo?(mcmanus)
Only one of the crashes I linked to is v48, the other are earlier. But yeah, they all have mozilla::dom::FileReader somewhere in the stack, so I guess they are different if you rule out the bug being in NS_CopySegmentToBuffer itself.
See Also: → 1294533
Crash volume for signature 'memcpy | NS_CopySegmentToBuffer': - nightly (version 51): 3 crashes from 2016-08-01. - aurora (version 50): 12 crashes from 2016-08-01. - beta (version 49): 992 crashes from 2016-08-02. - release (version 48): 0 crashes from 2016-07-25. - esr (version 45): 0 crashes from 2016-05-02. Crash volume on the last weeks (Week N is from 08-22 to 08-28): W. N-1 W. N-2 W. N-3 - nightly 0 0 3 - aurora 3 1 1 - beta 359 360 81 - release 0 0 0 - esr 0 0 0 Affected platform: Windows Crash rank on the last 7 days: Browser Content Plugin - nightly - aurora #251 #160 - beta #50 #41 - release - esr
status-firefox49: --- → affected
status-firefox50: --- → affected
status-firefox51: --- → affected
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: