Open Bug 1176050 Opened 5 years ago Updated 22 days ago

Firefox liveload with lots of connections slow when netmonitor is open

Categories

(DevTools :: Netmonitor, defect, P2)

38 Branch
defect

Tracking

(Not tracked)

People

(Reporter: petcuandrei, Unassigned)

References

(Blocks 2 open bugs)

Details

(Keywords: DevAdvocacy, Whiteboard: [polish-backlog][difficulty=medium][DevRel:P2][qf:p2:pageload])

Attachments

(2 files)

User Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0
Build ID: 20150605133456

Steps to reproduce:

1) install this js app from web
git clone git@github.com:andreicristianpetcu/foobar.git
cd foobar
git checkout ef75f84
npm install -g grunt
npm install
bower install
grunt server

go to http://localhost:8000 and either run in the cli "touch app/index.html" or click the first link in the page several times.


Actual results:

The DevTools network tab shows the connection is really slow, it loads the whole page in 4 seconds while Chromium loads the page in 532 milliseconds. Firefox Nightly loads it a little faster than the stable version in around 3.5 seconds (probably due to bug 1143224 being fixed). I think this issue is more related to the large number of connections, but this is a normal JavaScript project these days, it is not a huge project.

Developers rely on liveload a lot these days and a slow reloading of the page really breaks the workflow.

Here is a video (watch in fullhd) https://youtu.be/zbp2syKiyj0
I made a demo page http://andreicristianpetcu.github.io/foobar/app/ but it does reproduce the bug as well as the installed version :(


Expected results:

The page should load in under a second. Please do not mark this bug as duplicate of 1143224 :P since it is not. I tested with a Nightly that already has the fix for that bug.
If anybody is interested I plan to put a small bounty on this bug. I know I cannot cover the cost of this fix but I hope more and more people start putting bounties on features and bugs and make Firefox more independent.
I can confirm that it's slower in FF than chrome.  I'm seeing something like ~2 seconds in FF (on an fx-team build with a clean profile) and ~1 second on Chrome canary in a clean profile.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Summary: Firefox liveload slow with lots of connections → Firefox liveload with lots of connections slow when netmonitor is open
Awesome! Hope it's fixable :D
Component: Untriaged → Developer Tools: Netmonitor
(In reply to andrei from comment #0)
> User Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:38.0) Gecko/20100101
> Firefox/38.0
> Build ID: 20150605133456
> 
> Steps to reproduce:
> ...

This is an amazing bug report - thanks so much for taking the time!
Do you have an estimation when this bug will be fixed or if it can be fixed?

If this issue gets fixed and lands in Nightly or Aurora before #1143224 lands in stable I can make a video with the performance gain of both bugs :D

I think lots of JS developers switched to other browsers because of this. I know how annoyed my work colleagues are when they work on my computer and brag on how fast "their" browser is :P

Thank you!
I've just put another 30$ on this bounty. It is a total of 45$ now.

I guess this is peanuts for a software developer (especially if he or she is living in the US) but if more people donated small amounts as I do, maybe Firefox can become more independent.

I really want this bug fixed, and I do not have the skills to fix it myself. This really breaks my flow since I loose a few seconds each time I save a file in my web project.

I hope some Mozilla developers will have time for this bug soon :)
I ran this with a fresh profile, no addons, no AdBlock.
Adding DevAdvocacy keyword since there's some social proof of this being a real issue for folks on Reddit: https://www.reddit.com/r/firefox/comments/3ghmba/firefox_is_really_slow_with_liveload_i_loose_a/

Andrei: Thank you so much for the detailed report and test case. I'll take that over a bounty any day. :)
Keywords: DevAdvocacy
Hi, 
I was also able to replicate this. Hope it gets some attention :)

Cheers,
Petre
Priority: -- → P1
Brian you were the last person in the netmonitor IIRC - can you estimate difficulty for this?
Flags: needinfo?(bgrinstead)
Whiteboard: [polish-backlog]
(In reply to Jeff Griffiths (:canuckistani) from comment #11)
> Brian you were the last person in the netmonitor IIRC - can you estimate
> difficulty for this?

Running in e10s does seem to help a bit (maybe half a second faster), which makes me think there's still some frontend jank but it isn't fixing the bulk of the problem.  And there probably isn't a ton more low hanging fruit for that after Bug 1143224.

Getting this to have (near)zero impact would be hard, we are always going to have some overhead for platform level monitoring and RDP traffic.  But I think if we put some effort towards profiling this we should be able to get better than today with less work than that.

I'm seeing something weird in the waterfall on the demo where there is a long delay before the recaptcha__en.js request happens.  It looks like nearly all of the requests are done in < .5 seconds, but that one doesn't start until 2.5 seconds in.  I see a similar thing on a full reload, but it's not as pronounced.  I'll upload a HAR file with more info.
Depends on: 1143224
Flags: needinfo?(bgrinstead)
Whiteboard: [polish-backlog] → [polish-backlog][difficulty=medium]
Attached file livereload.har
HAR export - this can viewed by dropping it onto http://www.softwareishard.com/har/viewer/
Not sure if this was already the case during your last profile, but I noticed that desktop APZ (which defaults on in Nightly for Mac) makes scrolling the netmonitor much jankier and slower.  Disabling APZ makes this better.

Not sure if there's anything the tool itself can do for this piece of it.
(In reply to J. Ryan Stinnett [:jryans] (use ni?) from comment #14)
> Not sure if this was already the case during your last profile, but I
> noticed that desktop APZ (which defaults on in Nightly for Mac) makes
> scrolling the netmonitor much jankier and slower.  Disabling APZ makes this
> better.

I filed bug 1201192 about the APZ issues with net monitor.
Is there any update on this bug? I see it's a P1, does this mean it has high priority or low priority? Thank you!
I have been testing this and I can also reproduce the problem.

I tried two scenarios:

1) Loading the page *with* the toolbox open (having the Net panel selected or unselected didn't make visible impact). The loading time in this scenario was ~3-4 sec, which corresponds to comment #0.

2) Loading the page *without* the toolbox open, but having HARTriggerExport extension [1] installed so, I can collect HAR timing info. The loading time in this scenarion was ~1.3 sec, which is equal to what I am seeing in Chromium.

[1] http://www.softwareishard.com/blog/har-export-trigger/

These results clearly points to the Net panel UI. It looks like the Network panel logic or rather the Network panel rendering is too expensive and affects the overall page load timing - slowing it significantly down.

I'd really like to work on this, but I have too much other priorities on my plate at the moment and so, I am lowering the priority to #2 - just because I don't know about anyone who could work on this now. 

Also adopting React in the Net panel, which is already planned, could help to solve this problem. React is fast and better equipped for rendering optimizations (comparing to XUL).

---

Just a note for the scenario #2, you need to:
devtools.netmonitor.har.enableAutoExportToFile = true
extensions.netmonitor.har.autoConnect = true
+ restart the browser

Honza
Whiteboard: [polish-backlog][difficulty=medium] → [polish-backlog][difficulty=medium][DevRel:P2]
When is the switch to React planned? Can it be done incrementally (only some dev tools parts at first and then the rest)?
We are already switching to react (see bug 1263741 as one of the meta bugs), focusing on the Inspector panel atm. The Net panel will follow.

Honza
Can you point me to the specific bugs that target the migration of the network panel to React? I want to put a bounty on their heads :D I did not manage to find them myself :P
I don't think there are reports specifically for the Net panel.
So far, we have meta bug 1263750

Honza
When the Netmonitor frontend receives an event like "responseContent", it immediately fetches the response data with webConsoleClient.getResponseContent(), although this data is not needed to display the request list. It's only needed to show the request detail. So, there is a lot of useless RDP traffic.

Making this fetching lazy immediately cuts the load time to half, from 4s to 2s. Chrome is still far better at 0.5s, so I hope there are more low-hanging optimizations like this.
Assignee: nobody → jsnajdr
This is really nice! Is this lazy loading in Nightly? Can I test it out? :D
(In reply to andrei from comment #23)
> This is really nice! Is this lazy loading in Nightly? Can I test it out? :D

Right now, it's just an unfinished patch on my machine. You'll be able to test a try build when I start testing this, and it'll get to Nightly only when it lands and gets RESOLVED FIXED.
cool. I can test the try build when you make it.
(In reply to Jarda Snajdr [:jsnajdr] from comment #22)
> Making this fetching lazy immediately cuts the load time to half, from 4s to
> 2s. Chrome is still far better at 0.5s, so I hope there are more low-hanging
> optimizations like this.

Great find, looking forward to this improvement!
First working version of the lazy-loading patch. In my testing, it cuts the loading time to half compared to the non-patched version.

Almost all mochitests are passing, except HAR and filtering ones.

It will require a lot of cleanup and dividing into multiple reviewable patches, but is ready to be tested.
Andrei, when the try build in comment 28 finishes, you can download the linux64 binaries and try out how the fix works for you.
Flags: needinfo?(andrei)
(In reply to andrei from comment #30)
> So I need to download from here
> https://archive.mozilla.org/pub/firefox/try-builds/jsnajdr@gmail.com-
> 34b97e789e5b1e9999c87a826f745caff080e0c9/try-linux64/ this
> https://archive.mozilla.org/pub/firefox/try-builds/jsnajdr@gmail.com-
> 34b97e789e5b1e9999c87a826f745caff080e0c9/try-linux64/firefox-51.0a1.en-US.
> linux-x86_64.tar.bz2 right?

Yes
Flags: needinfo?(andrei)
Are you sure? I get the same performance on both Nightly and the try build. Each one has a brand new profile so there are no addons messing around with my data.

The page in the project above (foobar) is ~1.7s for both builds as an average and I get 1.5 best and 2s worst but the same data is for both of them. I tried both manually loading each page with F5 so that the two builds don't mess with each others data and network and I also used the "touch app/index.html" command.

Am I doing something wrong? The try build did not update itself, I did not restart it.
There has been some other things improving the performance besides your work. My initial bug report showed 3.7s on that nightly but today's nightly is 1.7s.
I run my FF instances with:

~/Downloads/firefox_liveload/firefox -P "vanilla_fast_connections" -new-window

~/firefox_nightly/firefox -P "vanilla" -new-window
(In reply to andrei from comment #32)
> Are you sure? I get the same performance on both Nightly and the try build.
> Each one has a brand new profile so there are no addons messing around with
> my data.

Hmm, do you have e10s enabled? I now realize that the patch only improves performance on e10s. It avoids unnecessary transfers of large data, and that has impact only if the transfer has significant cost. I.e., moving the data across process boundaries. On non-e10s, the transfer costs almost nothing.

By the way, my latest unpatched Nightly is also significantly faster than few days ago. I'm wondering what caused that.
e10s is default in Nightly so yes.... I use it.
(In reply to andrei from comment #36)
> e10s is default in Nightly so yes.... I use it.

The try build has a lot of test failures that shouldn't be there - these tests succeeded locally. So there's something wrong with the pushed patch. I'll need to fix that and then try again.
(In reply to andrei from comment #34)
> I run my FF instances with:
> 
> ~/Downloads/firefox_liveload/firefox -P "vanilla_fast_connections"
> -new-window
> 
> ~/firefox_nightly/firefox -P "vanilla" -new-window

You might need to do -no-remote (or close out any other running process) to make sure it's actually loading the try build and not using an existing process.
I have been also testing this using STR from comment #0 (on my own build, Win10)

Unfortunately, I couldn't see any differences in page-load-speed when testing with/without the patch applied. Simple re-load of the page (pressing F5 or modifying foobar/app/index.html) takes about 2.5 s on my machine. I tried to switch on/off e10s, but didn't observe any changes...

Honza
(In reply to Jan Honza Odvarko [:Honza] from comment #39)
> Unfortunately, I couldn't see any differences in page-load-speed when
> testing with/without the patch applied. Simple re-load of the page (pressing
> F5 or modifying foobar/app/index.html) takes about 2.5 s on my machine. I
> tried to switch on/off e10s, but didn't observe any changes...

Very strange, I stopped seeing any improvements, too. :(
See Also: → 1288493
Firefox gets a bad rep due to this DevTools bug when people do benchmarks :(

I seen it several times, not just in this video.

https://youtu.be/ULnd_PoW-kY?t=140
Assignee: jsnajdr → nobody
Whiteboard: [polish-backlog][difficulty=medium][DevRel:P2] → [polish-backlog][difficulty=medium][DevRel:P2][qf]
Whiteboard: [polish-backlog][difficulty=medium][DevRel:P2][qf] → [polish-backlog][difficulty=medium][DevRel:P2][qf:f64][qf:p1]
Whiteboard: [polish-backlog][difficulty=medium][DevRel:P2][qf:f64][qf:p1] → [polish-backlog][difficulty=medium][DevRel:P2][qf:p1:f64]
Product: Firefox → DevTools
Whiteboard: [polish-backlog][difficulty=medium][DevRel:P2][qf:p1:f64] → [polish-backlog][difficulty=medium][DevRel:P2][qf:p2:pageload]

The touch app/index.html thing stopped working for some reason.
I ran 10 refresh by clocking the app "Reload page (old school)" on the page in the foobar repository I linked above. Here is the perf data I got.

https://perfht.ml/2pfWgkP

FYI
Chrome:
Load: 217ms
Dom Content Load: 203ms
Finish: 197ms
Firefox:
Load: 1.10s
Dom Content Load: 597ms
Finish: 1.09s

My initial report was with 3.7s, my last comment was 1.7s and currently it is 1.10s. That's a pretty nice improvement but it's not yet as fast as Chrome. Hope my extra data helps.

You need to log in before you can comment on or make changes to this bug.