1442556 - treeherder feels slower

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P1)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Andreea confirmed this, so filing this as a bug.

Starting this week, treeherder became slower. On my machine, that means sometimes getting shown a slow script warning which else was only shown if I modified the job status filters with many pushes visible.

Started ~2 days ago?

Comment 1

[Andreea Pavel [:apavel]]

(In reply to Sebastian Hengst [:aryx][:archaeopteryx] (needinfo on intermittent or backout) from comment #0)

No, this actually started a while ago (~ 2 weeks), however at first we thought it was our internet connection. 
My shift colleagues are experiencing this issue too.

Comment 2

[Ed Morley [:emorley]]

I've just done a prod push, which includes a few fixes.

Could you try capturing some profiles using the devtools profiler and/or list a few specific actions which seem slower, so we can try to reproduce?

Comment 3

[Andreea Pavel [:apavel]]

:emorley After loading more pushes, when clicking to see the details of a certain failure it takes ~20/30 seconds for the info to load. Also, when using the load more pushes options, we receive very frequently that a page is slowing down the browser. I think you can easily repro this if you go to autoland and load more pushes.

Comment 4

[Ed Morley [:emorley]]

Those actions are pretty speedy for me.

Could you list full standard STR? (Which browser and version, does it repro in another browser too, ...)

Comment 5

[Andreea Pavel [:apavel]]

Firefox Quantum 58.0.2 (64 bit), it also reproduces on Google Chrome 64.0.3282.186 (Official Build) (64-bit) and on Opera 51.0.2830.40 64-bit seems to work faster.

Comment 6

[Bogdan Tara[:bogdan_tara | bogdant]]

Attachment: treeherder_slowness_preformance_recording

Comment 7

[Cameron Dawson [:camd]]

Hmm, I must say those seem pretty speedy for me, too on the autoland branch.  I tried loading 50 extra pushes, then did it again and it seemed quite responsive while loading.  But I'll try to work with this more to reproduce it.  I'd be curious to see some profiles as well.  Also if you could not the speed of the XHR requests.  For each push, it generally takes 3-4 seconds to download the data (my internet speed is about 130mb, though, so pretty fast)

One thing I HAVE noticed now and then is that a particular failure is slow to load.  I don't have the link to the particular one, but some have more data than others to load in the bottom panel.  But it sounds like your issue is with most or all of the failures taking 30-ish seconds.

Comment 8

[Cameron Dawson [:camd]]

Here's an example of a job that loads the details very slowly:

http://localhost:5000/#/jobs?repo=mozilla-inbound&selectedJob=165612523

appears to be the /text_log_errors/ endpoint.  There may be lots of calculations happening there.  Or perhaps there's just a lot of data.

Comment 9

[Raul Gurzau (:RaulG)]

Hi, i've encountered a slowness when starring jobs. Can't say how far back this occurred, just got back to work today. After choosing it's corresponding bug i ussualy hit Ctrl+Enter and it takes about 3-4 seconds until it's actually starred. Can't click on anything else or do any other action in that window. I've used FF 58.0.2 (64-bit) and Chrome Version 64.0.3282.186 (Official Build) (64-bit) on Windows 10 Pro (i5,16GB RAM)and a very fast internet connection, ~500mbps after doing a speedtest.

Comment 10

[Andreea Pavel [:apavel]]

Attachment: mozz_performance

Hi. I just opened the trees and taskcluster and got the error that the page is slowing down. I also checked to see all successful jobs on treeherder, however the data overlapped, got again the error message regarding slow down. Please see if attachment helps.

Comment 11

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/3307

Comment 12

[Cameron Dawson [:camd]]

Comment on attachment 8956521 [details] [review]
Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/3307

I have a couple more ideas that may help speed up these issues.  But taking one at a time won't hurt.

Comment 13

[Ed Morley [:emorley]]

Comment on attachment 8956521 [details] [review]
Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/3307

Ah nice! :-)

Comment 14

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/c403895cafe30fc9f59b22404cf8ad1da5924c16
Bug 1442556 - Improve rendering speed in treeherder by removing refs (#3307)

These refs were not actually used, so can be removed.  It does appear to
improve rendering speed, especially when changin filtering.  I had noticed
that the JobGroup was re-rendering more times than it should, and this
change stops that.  I believe more work is to be done here, but this is one
step.

This also includes a couple other small tweaks that may help speed.

Comment 15

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/3309

Comment 16

[Ed Morley [:emorley]]

Comment on attachment 8956675 [details] [review]
Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/3309

Good spot :-)

Comment 17

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/cffdb1b0f25239aff169f21a54c7dfbb2ef390dc
Bug 1442556 - Fix slowness on filtering (#3309)

I believe this was the main bug causing the slowdown.  It was setting state
in ``filterPlatform`` which is called once per platform when a filter change is
made.  But that ``setState`` was updating ALL the platforms for that push.

* Remove unnecessary clonejobs artifact
* Fix the regression with expand/collapse counts
* Very minor optimizations and cleanup
* Add some unit tests for groups

Comment 18

[Cameron Dawson [:camd]]

Aryx and Andrea: if either of you would like to check on the perf fixes, please feel free to test-drive stage: treeherder.allizom.org and provide feedback.  :)

Comment 19

[Cameron Dawson [:camd]]

I believe with the latest PR, when it goes to production, this bug will be fixed.  Please reopen or open a new one if you find more issues with this.  Thanks!

Comment 20

[Ed Morley [:emorley]]

If you see any further issues please open a new bug with detailed steps to reproduce (exact steps and URLs listed using bullets etc). Many thanks :-)

Comment 21

[Ed Morley [:emorley]]

The fixes here have just been deployed to production. Refresh the UI to pick up the changes :-)

Comment 22

[Andreea Pavel [:apavel]]

Hi, sorry for the delayed response. Tested via treeherder.allizom.org and it worked faster, did not encounter issue when i tested last shift. 

Thanks.