353674 - processbuild.pl hangs and eats cpu when the message is compressed with bzip2

Status: RESOLVED FIXED

(Webtools Graveyard :: Tinderbox, defect)

Description

[Dave Miller [:justdave]]

With tinderbox on the tip, processbuild.pl hangs, which consuming as much CPU as it can get, if the message was encoded with bzip2.  Processes got up to 40 minutes old before I finally killed them off.  This seems to be the cause of the queue backups and the high load we were experiencing before (it had nothing to do with running in a VM)

Comment 1

[Dave Miller [:justdave]]

bzip2-libs-1.0.2-13.EL4.3
bzip2-1.0.2-13.EL4.3
perl-Compress-Bzip2-2.09-1.2.el4.rf


This is blocking upgrading Tinderbox.

Comment 2

[cls]

Attachment: v1.0

Here's a swag at a fix.  Assuming that the problem is that the bzip2'd "line" is insanely long and being read into memory, this should limit the amount read to 4k chunks.  I'm not seeing extremely long lines in local bzip2'd files though.
Do you have an example of a message that causes the hangs?

Comment 3

[Dave Miller [:justdave]]

they all have 60-70 character lines, well wrapped. :(

OK, I added the following to the code:

      } elsif ($tbx->{logcompression} eq 'bzip2') {
+       open TESTH, ">>/tmp/tbox-bzip2.pid";
+       print TESTH "$$\n";
+       close TESTH;
+       sleep 60; # give operator time to trigger strace
      my $comp_bz = bzopen($decoded, "rb") or

I watched for tbox-bzip2.pid to show up, snagged, the pid, and straced.

the first couple actually finished.  Then one of them hung.

The trace and the problem message will be attached momentarily.

Comment 4

[Dave Miller [:justdave]]

Attachment: The email that made it hang

Comment 5

[Dave Miller [:justdave]]

Attachment: strace starting at the sleep(60)

Comment 6

[cls]

Hrm. It looks like the message is corrupt.  I stripped the headers and then ran this command on the file:

perl -pe ' use MIME::Base64;  print MIME::Base64::decode($_); ' < ~/Incoming/problem-mail.rfc822 > tblog.txt.bz2

But I get an error when I try to bzcat tblog.txt.bz2 .

Comment 7

[cls]

Nevermind.  The perl script was incorrect.  This works:
 perl -e 'use MIME::Base64; while (<>) { print MIME::Base64::decode($_); } ' problem-mail.rfc822 > tblog.txt.bz2

Comment 8

[cls]

It looks like there's a bug in bzreadline.  Switching the call from bzreadline to bzread in my test case "fixed" the problem.  I let the bzreadline call run for almost 4 hrs now and while sucking up cpu, the memory usage never changed, which means that the file is not being read.  

(Btw, the 60-70 lines of wrapped text is the base64 encoded file.  The lines read from the bzip2 file varied from 1 to 1500+ bytes in size.)

Comment 9

[Dave Miller [:justdave]]

Comment on attachment 239546 [details] [diff] [review]
v1.0

Yeah, I came to the same conclusion with bzreadline vs bzread last night...  I was watching it for a while, and saw several bzip2'd messages come through, and it never hung, with this change applied.

I think the problem is looking for line endings in a file that isn't text, and thus shouldn't have any (and if there are, they should be treated as data, not line endings)

Comment 10

[cls]

Checking in webtools/tinderbox/processbuild.pl;
/cvsroot/mozilla/webtools/tinderbox/processbuild.pl,v  <--  processbuild.pl
new revision: 1.57; previous revision: 1.56
done