839651 - Frequent unresponsive script dialogs triggered by slavealloc

Status: RESOLVED WONTFIX

(mozilla.org Graveyard :: Webdev, task, P3)

Description

[Chris Cooper [:coop] (he/him)]

As the number of machines managed has increased (now over 3000), we see the unresponsive script dialog pop up more and more often when slavealloc tries to load the entire table of slaves at the outset.

Barring a more invasive fix, there are a few things we can do to try to improve this:

1) Don't display decommissioned slaves (I'm apparently the only one who wants to see them anyway.
2) Update the included JS libs (backbone, underscore, jquery(-ui), Datatables) to current release versions for the performance improvements inherent in each.

I've already made these changes to the staging install and the results are promising so far:

http://slavealloc-staging.build.mozilla.org/ui/#slaves

I encourage people to reload slavealloc-staging a bunch and give it a good try.

I'll post my patches shortly.

Comment 1

[Chris Cooper [:coop] (he/him)]

I tried measuring page load times (well, more accurately, the time until you can actually interact with the table of slaves) for production and staging, but neither the Web Console nor Firebug seem to be tracking the critical time spent beachballing while the slave table is being created. The profiler tells me we're spending over 50% of our load time creating that table, but I can't be sure whether that includes that beachballing time that doesn't seem to be counted elsewhere.

Resorting to more brute force timing methods, a stopwatch reveals that the staging site takes just over 12s to become usable, whereas the production site currently takes just over 15s (avg of 10 loads each).

Not outstanding, but improvement.

Comment 2

[Chris Cooper [:coop] (he/him)]

Attachment: Exclude decommissioned slaves for slavealloc

This patch:

* excludes decommissioned slaves from the display. You can still mark slaves as decommissioned and they will appear in the interface until you reload, i.e. they don't vanish immediately.
* some styling changes
* call the updated versions of the various js libs

I refrained from attaching the updated js libs themselves because it's just 630K of minified js. Let me know if you really want to review that. ;)

Comment 3

[John Hopkins (:jhopkins)]

Attachment: set bDeferRender:true to speed up table load

This small patch is deployed to staging slavealloc.  The load time is down to about 2-3 seconds.
coop: you may be able to keep the decomm'd slaves visible with this patch.  I'll leave that up to you.

Comment 4

[John Hopkins (:jhopkins)]

Comment on attachment 713266 [details] [diff] [review]
set bDeferRender:true to speed up table load

Unfortunately, Callek found an issue with my patch where the edit buttons don't draw properly when you hit the 'next' button to view another page of results.  Not sure if that's a showstopper (personally, I use the "all" results drop down instead).

Comment 5

[Chris Cooper [:coop] (he/him)]

Comment on attachment 712934 [details] [diff] [review]
Exclude decommissioned slaves for slavealloc

https://hg.mozilla.org/build/tools/rev/cc9d508b39bd

This is running in production now.

Comment 6

[Chris Cooper [:coop] (he/him)]

Comment on attachment 713266 [details] [diff] [review]
set bDeferRender:true to speed up table load

Review of attachment 713266 [details] [diff] [review]:
-----------------------------------------------------------------

It's obviously part of what we want (god, it's so fast), but we need to find a way to trigger the render of that one form control when we change the # of entries, search, or use the next/back buttons.

Comment 7

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #6)
> It's obviously part of what we want (god, it's so fast), but we need to find
> a way to trigger the render of that one form control when we change the # of
> entries, search, or use the next/back buttons.

Dustin: any idea how to force a refresh/render of the edit button when we change the set of visible slaves?

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

I'm not sure how to hook into visibility changes, but you'd basically need to do that and then call the re-render function on the DataTable.

Comment 9

[Chris Cooper [:coop] (he/him)]

I'm out of my depth here, so I started a discussion over on the datatables forums:

http://datatables.net/forums/discussion/14126/button-and-associated-event-only-present-on-first-page-of-results#Item_1

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

I'll ping you later today and see if I can take a deeper look.

Comment 11

[Chris Cooper [:coop] (he/him)]

(In reply to Dustin J. Mitchell [:dustin] from comment #10)
> I'll ping you later today and see if I can take a deeper look.

Dustin: any chance you could look at this in the near future? 

There have been no replies to the datatables forum thread, but I haven't done any further debugging either.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

I think the Mozpool UI has the same problem, and that was written ground-up, although still using backbone.js.  So either backbone.js itself leaks or, more likely, I'm doing something boneheaded.  But given that I've made the same mistake twice and been unable to find it, I doubt I can be much help.

Could we seek out some JS skill from webdev?

Comment 13

[Chris Cooper [:coop] (he/him)]

Hello webdev,

We're looking for help with an existing apps: slavealloc. App performance was fine initially, but the number of rows the app has been forced to display has grown over time (# of machines), and the performance has taken a nosedive. I now frequently see 20+ sec pauses as the slavealloc slave page loads: my browser beachballs, and the slow script dialog in Firefox is often triggered. 

We did some basic debugging and found that the app is spending most of that time rendering *all* ~5000 rows despite only 10 rows being presented by the default interface. jhopkins did some digging and found an option for datatables, bDeferRender:true, that cuts down our load time substantially. Unfortunately, only the initial default set of rows (10) is properly rendered. When we navigate away from that initial default set via next/back button or by searching, all other rows are missing the buttons that allow us to edit the contents of the row. I'll attach a screenshot of the observed behavior shortly.

The technologies in use are:
* backbone-0.9.10.min.js
* jquery-1.9.1.min.js
* jquery.dataTables-1.9.4.min.js
* jquery-ui-1.10.0.js
* load-min.js
* underscore-1.4.4.min.js

Dustin wrote the app initially, so he may be able to answer design questions. He should also be able to grant access to the slavealloc staging instance to whoever is able to take this on

The staging instance currently has bDeferRender set, and can be accessed here:

http://slavealloc-staging.build.mozilla.org/ui/#slaves

Comment 14

[Chris Cooper [:coop] (he/him)]

Attachment: Screenshot of editing button not being rendered as expected.

In the image you can see two slaves that are part of the initial 10 that are displayed, linux-ix-slave02 and linux64-ix-slave02. These two have the editing button properly rendered.

I've used the search box with the substring 'slave02' which displays 3 other slaves not in the default. These slaves have smaller, placeholder editing button that can still be clicked, but do not toggle the editing status of the row.

Comment 15

[Chris Cooper [:coop] (he/him)]

I should note that simply searching for bDeferRender on the datatables site (http://www.datatables.net/index) turns up lots of potential solutions, but I'm not well-versed in jquery, so I don't really know where to start.

Comment 16

[Chris Cooper [:coop] (he/him)]

Didn't realize this was still assigned to me.

Still actively looking for help from webdev on this (see comment #13).

Comment 17

[Dustin J. Mitchell [:dustin] (he/him)]

This seems even slower than it did before :(

I think the main problem is DataTables.  It's meant for displaying relatively simple data, not table rows full of HTML and controls.  There's a lot of rendering things with the DOM, flattening to HTML, and passing them to DataTables, which then re-creates the DOM objects.  That's slow, and I suspect it happens more often than it should.

I wonder if there's a better tool out there that would still get the pagination, sorting, and searching functionality we want, but in a more DOM-friendly fashion?

Comment 18

[Scott Bryner]

Looks like this has been open for a while, so I'll throw my quick 2 cents in (at the risk that I've missed something vital.)

On a cursory glance, it seems like the issue is in your query returning too many rows.  

A general rule I follow is for the DB layer to only ever return to the client layer the data to be displayed, never more than that.  Since JS is interpreted, it's usually too slow to do any data processing on large data sets.  

In this case, having the client process 5000 rows when you only need 10 means that 99.8% of that data and the client resources (memory and processor cycles) that go into handling it are wasted resources.

So, craft your queries so they only return 1 page's worth of data at a time.  That means you'll be doing the pagination of data in the DB layer.

Hope that helps!

Comment 19

[Ed Morley [:emorley]]

This combined with not having bug 911136 means a pretty painful experience.

Comment 20

[Chris Cooper [:coop] (he/him)]

WONTFIX in favor of bug 1135818.