189689 - intermittent duplicated content on pages, CRC errors in downloads

Status: VERIFIED FIXED

(Core :: Networking: HTTP, defect, P1)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

For the past few days, I've been seeing duplicated content on some pages (in the
manner in which a networking bug would cause duplicated content, i.e., mangled
tables rather than convenient units of layout dupliated).  I've seen it most
frequently (probably 4 times total) on bonsai queries (the time links from
tinderbox), although I saw it twice on tinderbox and I think I saw it once on a
New York Times article.  Two of the times I saw it on bonsai queries were
immediately after each other, and the same for the two times I saw it on tinderbox.

I'm using my optimized Linux build, my own tree, compiled with gcc 3.2.1, on a
dual-CPU PIII-733 running RedHat 8.

I blame the problem on darin's landing (bug 176919), although all I really know
is that I didn't see it before darin's landing.

One interesting thing about these display bugs is that it's possible to save the
page to the hard disk and get the corruption.  This certainly suggests a bug
somewhere on the network side of things.  I'll attach an example of a bonsai
query demonstrating this bug.

Comment 1

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Attachment: bonsai query result as mangled by this bug

Comment 2

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Attachment: what the previous attachment should have looked like

Comment 3

[Darin Fisher]

if you're able to save the exact same corrupted content, then for sure this is a
networking problem... probably not a cache problem.

Comment 4

[Darin Fisher]

i think the bug may be in nsPipe::AdvanceReadCursor.  investigating...

Comment 5

[Asa Dotzler [:asa]]

I've seen this too. We should get this regression repaired before we ship beta. 

Comment 6

[Andreas Kunz]

I thing there might be a VERY bad problem:

since a few days I get CRC errors with downloaded mozilla nightly zips.
After reading this bug I just downloaded the current nightly twice and made a diff:
The to files differ!
There is definitely something very bad going on and this bug should be at least
set to 'blocker' if I'm not the only one who can reproduce it.

See also bug 189677 and http://www.mozillazine.org/forums/viewtopic.php?t=4778
...and stop 1.3b!

Comment 7

[Benjamin Smedberg]

*** Bug 190073 has been marked as a duplicate of this bug. ***

Comment 8

[Ian Nartowicz]

I'm the one who raised the duplicate.  I'm on win98.  I think this should be
OS->All.

Comment 9

[Andreas Kunz]

-> all/all (I also see this on win2k).

On #mozilla some people wondered why tinderbox looked so ugly, the columns all
shuffled around. I think this bug is why.
And tinderbox might be a very good place to reproduce it since I don't remember
it looked ok at any of the one or two dozen times I looked at it with yesterdays
build.
Now I'm using 2003011505 again and tinderbox looks fine.

I just visually diffed the different nightly.zip downloads. Well... I saw the
differences...

Comment 10

[Benjamin Smedberg]

*** Bug 189677 has been marked as a duplicate of this bug. ***

Comment 11

[Jason Bassford]

Adding CRC problem to Summary for Bugzilla queries.

Comment 12

[Alex Vincent [:WeirdAl]]

Is that why I couldn't get OpenOffice to download correctly...

One thing in particular I've noticed when downloading pages (it's most
noticeable in pages with tables, but sometimes I've seen in non-tabled pages
like W3C's new XML Schema Requirements doc) is that reloading the same page
produces different rendering almost every time.  I can give screenshots if
anyone's interested.

Comment 13

[Mark Perris]

I would think so, along with the bug i filed on image corruption and file CRC's,
bug 189677, it seems to be that any networking data coming into mozilla is
affected by very small, rare, perhaps single bit errors. This would produce the
glitches in the HTML, the errors in the images, and the invalid CRC's, without
rendering the whole file/image/page unreadable or undrenderable. The only way to
fix the glitches that sometimes workd is to do a ctrl-F5 to pull a new page from
the server. This suggests the bug resides in the networking code too, rather
than the disk cache or parsing routines.

This seems to occur more often in tables, perhaps there is some kind of bit
pattern that the networking code is taking particular aversion too, however
since the errors seem random every time you refresh a page, (after all,
sometimes the same page will load fine), it makes you think again.

Comment 14

[Juha Luoma]

I looked at one corrupted table source code and the problem was that there were
some data repeating. In that case I got one corrupted table row and couple of
rows were duplicate. So definitely more than single bit error.

Comment 15

[Alex Vincent [:WeirdAl]]

http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=MozillaTinderboxAll&branch=HEAD&branchtype=match&dir=&file=&filetype=match&who=&whotype=match&sortby=Date&hours=2&date=explicit&mindate=01%2F14%2F2003+08%3A00%3A00&maxdate=01%2F18%2F2003+08%3A00%3A00&cvsroot=%2Fcvsroot

Bonsai search for all checkins to Seamonkey Trunk from 01/14/03 @ 8am to
01/18/03  @ 8am.  

MozillaZine thread suggests 01-16 build does not contain this bug, so probably
anything south of 01/16 is not the cause.

Bug 176919's checkins fall on 01/17 @ 17:27 for new files, 18:15 for
modifications to current files, and a (tinderbox?) bustage fix at 20:24.

If I had build strings from working and non-working builds, or a better search
info, I could narrow this down further.  All in all, bug 176919 appears to be
the largest checkin.

Comment 16

[Darin Fisher]

Attachment: v1 patch

this patch should fix the problem.  i found a case where a read overlapped with
a write on a pipe could cause the cursors to be incorrectly modified.

Comment 17

[Doug Turner (:dougt)]

Comment on attachment 112347 [details] [diff] [review]
v1 patch

A for the pretty pictures.

Comment 18

[Boris Zbarsky [:bzbarsky]]

Comment on attachment 112347 [details] [diff] [review]
v1 patch

sr=bzbarsky with the comment and assertion changes we talked about....

Comment 19

[Darin Fisher]

Comment on attachment 112347 [details] [diff] [review]
v1 patch

seeking drivers approval for 1.3b.  this patch corrects a race condition in the
pipe code that would occur only when reading while writing to the pipe.  this
should nail the duplicate content bug, and may even fix some of the mailnews
hangs that have been reported.

Comment 20

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Comment on attachment 112347 [details] [diff] [review]
v1 patch

a=dbaron for trunk (1.3b) checkin

Comment 21

[Darin Fisher]

fixed-on-trunk

Comment 22

[Aleksey Nogin]

This check-in have added a brad TBox warning:

+xpcom/io/nsPipe3.cpp:837
+ `nsresult rv' might be used uninitialized in this function

Indeed, the nsPipeInputStream::Search now has an rv variable that is *never*
initialized, but is still compared to NS_BASE_STREAM_WOULD_BLOCK in certain cases!

P.S. Per bug 179819 using uninitialized variables ought to be considered a
compilation error.

Comment 23

[Darin Fisher]

Aleksey: thanks, i fixed the problem.  the extra code comparing against
NS_BASE_STREAM_WOULD_BLOCK was meant to be removed.

marking FIXED again.

Comment 24

[Jo Hermans]

*** Bug 189953 has been marked as a duplicate of this bug. ***

Comment 25

[Darin Fisher]

*** Bug 189752 has been marked as a duplicate of this bug. ***

Comment 26

[Alex Vincent [:WeirdAl]]

verified Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.3b) Gecko/20030123
tinderbox page looks good
vBulletin page, which in regressed builds always came up strange, looks good
download of Mozilla nightly compared against KMeleon 0.7 download of same file,
files are identical.