1197377 - Read callgraph.txt line by line to avoid string size limits

Status: RESOLVED FIXED

(Core :: JavaScript: GC, defect)

Description

[Steve Fink [:sfink] [:s:]]

Bug 1194112 is getting an allocation size error because it snarfs all of callgraph.txt into memory (immediately before splitting it into lines.)

Comment 1

[Steve Fink [:sfink] [:s:]]

Attachment: Read callgraph.txt line by line to avoid string size limits

Ok, this is a little crazy, but we can use libc to read line by line. If I only look at the time to read things into memory:

 snarf the whole file: 1.8 sec
 read a line at a time: 18 sec
 use a generator to read a line at a time: 24 sec

It's kind of painful to take a 10x hit, but adding half a minute to the overall callgraph computation isn't a big deal. I'm not sure how long it takes, but it's a couple of minutes. (I haven't timed the before and after.)

Comment 2

[Terrence Cole [:terrence]]

Comment on attachment 8651239 [details] [diff] [review]
Read callgraph.txt line by line to avoid string size limits

Review of attachment 8651239 [details] [diff] [review]:
-----------------------------------------------------------------

It's neat we can do that at all.

Comment 3

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/6a725147ee24

Comment 4

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/6a725147ee24