54659 - Running out of file descriptors on large runs of transformiix

Status: VERIFIED FIXED

(Core :: XSLT, enhancement, P5)

Description

[Anthony Truhlar]

I have been experiencing some problems load testing transformiix. I have 
modified transformiix.cpp to run several thousand times by placing most
of the code for this in a for loop. I then have a small xml test file (test.xml)
which includes a stylesheet test.xsl to generate some output - the details
of the stylesheet's operation aren't relevant. If I run the normal transformiix
from the command line, using:

	transfrmx -i test.xml

Things behave as one would expect (excutes once, & generates the correct
output). If I use my modified version, the output is fine for values of n (in
the for loop) to 1020. Above 1020 XMLParser complains that it cannot open a
stream. I suspect that this is due to transformiix reaching it's maximum number
of file descriptors (those allowed per process). This was confirmed when I added
2 delete statements at the bottom of the loop (line 147/148 of the file
transformiix.cpp attached below) and re-compiled/re-ran the code - all
iterations (even up to 10000) completed successfully. Currently ifstream objects
(and ofstream objects) are created and never deleted - thus causing the system
to run out of file descriptors (as well as go short on memory). (That is the
case in main/transformiix.cpp).

There may be other areas where ifstream/ofstream objects are not deallocated
that also need fixing.

The situation can be re-created by running my version of transformiix.cpp
supplying just test.xml to transformiix (since test.xml calls xsl:include to
include test.xsl), first with lines 147 and 148 commented out - so ifstream
and ofstream objects are not deleted as is the case in the distributed
version of transformiix.cpp, and then with the lines uncommented. You
should find that with the lines commented out, things go wrong on the
1021st iteration (when we run out of file descriptors), and if the lines
are included things run to completion (I tested with 10000 iterations). The
value of n is currently set to 1024 to make it easier to see the problem.

Hope that helps. (Files will be attached below).

Comment 1

[Anthony Truhlar]

Attachment: Modified version of main/transformiix.cpp to execute multiple times

Comment 2

[Anthony Truhlar]

Attachment: test.xml that forms part of the test case

Comment 3

[Anthony Truhlar]

Attachment: test.xsl that forms part of the test case

Comment 4

[Anthony Truhlar]

That's all the files. 

Best wishes,
Anthony

Comment 5

[Axel Hecht]

The opened files should be closed, but as transformiix isn't created to do
massive repeated transformation (a task really only good for profiling), I 
rate this as coding style.

Keith, if you feel like it, cleaning up transformiix.cpp to do
1) startup info
2) init streams and data structure
3) process
4) cleanup (and closing files ;-))

would be nice.

But anyway, don't delete streams, close 'm. If we talk about style, then ....

Anthony, please attach patches instead of complete files. And reuse testcases
we all allready have, instead of attaching new ones, which are probably in no
way needed to reproduce that behaviour.
It took me more time to find out what you changed than to evaluate those
changes.

Axel

Comment 6

[Anthony Truhlar]

Thanks, sorry for making life more difficult. I'm new to submitting bugs here
and will remember to re use the test cases and supply patches rather than
complete code in the future.

Anthony

Comment 7

[Peter Van der Beken [:peterv]]

Umm, netscape.com work.

Comment 8

[Peter Van der Beken [:peterv]]

Damn, wrong bug. Sorry guys.

Comment 9

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Axel, is this fixed now?

Comment 10

[Axel Hecht]

If I'd only knew if the ifstream destructor closes.
We might wallpaper this in txStandaloneProcessor::parsePath.

Comment 11

[Axel Hecht]

Attachment: close() in txStandaloneProcessor::parsePath

fix for standalone only, not part of build. r=?

Comment 12

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Comment on attachment 103975 [details] [diff] [review]
close() in txStandaloneProcessor::parsePath

r=sicking

Comment 13

[Peter Van der Beken [:peterv]]

Comment on attachment 103975 [details] [diff] [review]
close() in txStandaloneProcessor::parsePath

sr=peterv, but wouldn't it be better to move the close up to just after
xmlParser.parse(...)?

Comment 14

[Christopher Blizzard (:blizzard)]

Comment on attachment 103975 [details] [diff] [review]
close() in txStandaloneProcessor::parsePath

a=blizzard on behalf of drivers for 1.2final

Comment 15

[Axel Hecht]

checked in.

Comment 16

[Axel Hecht]

mass verifying