497980 - Speed up TraceVis log file processing

Status: RESOLVED WONTFIX

(Core :: JavaScript Engine, enhancement)

Description

[David Mandelin [:dmandelin]]

TraceVis log file processing is very slow for large traces, pretty much anything that comes out of a 10s+ browser run. The main reason is that the algorithm in the Python scripts loads the entire history as a set of tuples, which uses too much memory. I've created a C++ log file processor [1] that prints out the summary from a long trace in <0.1s (compared to binlog.py, which prints out the first part of the summary in about 5s on the same file, and doesn't finish the reason report within a minute). It can also "cook" the log file into a format that vis.py can now process much faster.

[1] http://hg.mozilla.org/users/dmandelin_mozilla.com/tracevis/file/c34684b94044/src/

I can start preparing this for landing once the initial version has landed.

Another idea, suggested to me by danderson, is of using NumPy, which is good for processing large numerical data sets. I don't plan to do that right now, because NumPy isn't part of the standard Python distro and can be a pain to install on Mac, and because the C++ version is pretty simple and I'm pretty happy with it. But if someone else happens to be interested in porting the improved algorithm I used in the C++ version to the Python version or to NumPy, that would also be very cool.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

TraceVis is gone with TM's removal.