1158098 - [meta] Improve debugger editor performance for large JS files

Status: RESOLVED DUPLICATE of bug 1229092

(DevTools :: Debugger, enhancement, P1)

Description

[Jukka Jylänki]

There are existing bug items that discuss large file handling in the JS debugger: 

https://bugzilla.mozilla.org/show_bug.cgi?id=929225: Check if we still need to switch the editor to text mode for large files
https://bugzilla.mozilla.org/show_bug.cgi?id=751901: Slow Source View in Debugger on large files

However those are a bit imprecise about what level of performance exactly constitutes the bug as fixed, so instead of dumping new test cases to those, I would like to propose a new test case in the form of a concrete task that a developer might have to do. Emscripten development can commonly generate large .js files (200-250MB range are the heaviest I've seen), and here is a STR/dev task for a minified medium sized one (41MB) that I recently attempted to do:

1. Download https://dl.dropboxusercontent.com/u/40949268/emcc/bugs/moz_strategyGame_v2.zip
2. Run StrategyGame-HTML5-Shipping.html
3. Open the Debugger in StrategyGame-HTML5-Shipping.js
4. Find all references to the texts "blur" and "focus" in the file. (use Ctrl-F to find). Which JS functions in that file refer to these two strings?

Currently the performance of the debugger is not sufficient to do this task. In comparison, e.g. Notepad++ on Windows is a generic text editor with no specific support for large files, in which this task can be just barely achieved. We should be able to surpass at least its capabilities.

At the moment pressing Ctrl-F in the JS script debugger view to find in the file will crash Firefox 64-bit on an OOM after allocating more than 8GB of RAM. See the following crash reports:

https://crash-stats.mozilla.com/report/index/d524ed17-00ab-4b3a-9d11-ee2462150424
https://crash-stats.mozilla.com/report/index/effbf3f9-9254-4947-b14e-7fe0f2150424

Comment 1

[Jeff Griffiths (:canuckistani) (:⚡︎)]

Adding James, as this would be a good stress-test for future debugger UI improvements.

Comment 2

[James Long (:jlongster)]

We're somewhat limited to what CodeMirror can handle, but this sounds like a case where moving searching to the backend would help. Right now the contents of the source are transferred to the frontend, and then sources, which is... crazy. I bet next year Q1/Q2 we can solve that. (There are bigger UX issues we need to solve first)

Comment 3

[Jukka Jylänki]

bgrins had a hunch on IRC that long lines would cause specific problems, so to investigate that behavior specifically, here is another Emscripten project with unminified content and therefore shorter lines, but lots of lines, 1675253 to be exact. (As a sidenote, this test case suffers from bug 1205604).

https://dl.dropboxusercontent.com/u/40949268/emcc/bugs/2015-08-28-emunittest_0.4-AngryBots-u5.1.3f1_hg-e1.34.6-release-devbuild-exceptions.zip

The large file is Development/2015-08-28-emunittest_0.4-AngryBots-u5.1.3f1_hg-e1.34.6-release-devbuild-exceptions.js, 56.6MB. Looks like tapping Ctrl-F to search on this file hangs the debugger as well.

Comment 4

[James Long (:jlongster)]

Brian, is this not also fixed with bug 1224726?

Comment 5

[Brian Grinstead [:bgrins]]

(In reply to James Long (:jlongster) from comment #4)
> Brian, is this not also fixed with bug 1224726?

Jukka, can you please check the debugger using a nightly build and let us know if it is now working for you?

Comment 6

[Jukka Jylänki]

Very nice, great work!

I am now able to open up the StrategyGame-HTML5-Shipping.js file, after about 25 seconds of loading. After it loads, searches are fast and I'm able to find "blur" and "focus" texts in the file. Smooth!

There are two issues still that I can find here when browsing the script file:
 1. the load time of the file is slow, ~25 seconds, and if one needs to often browse between two script files when debugging, it looks like the 25 second stall reoccurs always whenever coming back to the large file.
 2. another issue is that slowly scrolling through the file StrategyGame-HTML5-Shipping.js hangs the whole browser periodically for ~10 seconds at a time, which creates a stuttering experience.

The situation is definitely much better here now, except for the two above smaller issues. Should we create new items for those, or should we reuse this bug?

Comment 7

[Brian Grinstead [:bgrins]]

Well maybe we should file new issues for these things, but can you tell me - do you see approximately the same level of slowness when pasting that source file into the editor at: http://codemirror.net/?

Comment 8

[Brian Grinstead [:bgrins]]

(In reply to Jukka Jylänki from comment #6)
> There are two issues still that I can find here when browsing the script
> file:
>  1. the load time of the file is slow, ~25 seconds, and if one needs to
> often browse between two script files when debugging, it looks like the 25
> second stall reoccurs always whenever coming back to the large file.

We are going to handle this issue in another bug - there are swapDoc and copy methods that the editor provides where we can store the data structure of a source in memory and then reload it without needing to rebuild the doc state, which is taking a lot of time on these files.

I'm going to separately work on a more reduced testcase that loads big sources in a CM instance in a tab so we can get some profiles outside of our toolbox.

Comment 9

[Brian Grinstead [:bgrins]]

Here's a test page using a plain codemirror instance to load a variety of file sizes: https://bgrins.github.io/devtools-demos/perf/editor.html

Comment 10

[Nick Fitzgerald [:fitzgen] [⏰PST; UTC-8]]

Note as to why this depends on bug 1233857: I'm seeing ~80% of time (!!) in nursery collections due to the cliff described in that bug.

Comment 11

[Nick Fitzgerald [:fitzgen] [⏰PST; UTC-8]]

(In reply to Nick Fitzgerald [:fitzgen] [⏰PST; UTC-8] from comment #10)
> I'm seeing ~80% of time (!!) in nursery collections due to the cliff
> described in that bug.

(When setting the editor to view large sources.)

Comment 12

[Brian Grinstead [:bgrins]]

(In reply to Brian Grinstead [:bgrins] from comment #8)
> (In reply to Jukka Jylänki from comment #6)
> > There are two issues still that I can find here when browsing the script
> > file:
> >  1. the load time of the file is slow, ~25 seconds, and if one needs to
> > often browse between two script files when debugging, it looks like the 25
> > second stall reoccurs always whenever coming back to the large file.
> 
> We are going to handle this issue in another bug - there are swapDoc and
> copy methods that the editor provides where we can store the data structure
> of a source in memory and then reload it without needing to rebuild the doc
> state, which is taking a lot of time on these files.

This is now on file as Bug 1233927

Comment 13

[Jordan Santell [:jsantell] [@jsantell]]

All remaining performance bugs here have been moved to their own bugs (just added bug 1240186 for the scrolling performance), making this a bit redundant with bug 1229092 -- closing this, and moving dependency bugs to there.