932787 - incremental loading of rules/releases data

Status: RESOLVED WONTFIX

(Release Engineering Graveyard :: Applications: Balrog (frontend), defect, P3)

Description

[[e-mail] (:bhearsum)]

Both the rules and releases pages slow down tremendously when you have lots of releases. For releases, this is because we return every release once. For rules, we actually return every release _for every rule_ (in the "Mapping" dropdown).

We should do this more smartly. For releases.html we should probably load an initial set of data, and then have the "next" button request more from the server.

For rules.html we should probably only return the current mapping for each rule and then load the full set when someone clicks a Mapping dropdown or starts typing. It might be good to cache the result on the client too (probably with a TTL, because otherwise you'll miss newer data if you leave the page open for too long).

Comment 1

[[e-mail] (:bhearsum)]

mass priority change after component move

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Add Flask-Compress

As a stopgap we could enable gzip compression in the app, which this patch does. I did 
  pip install -t vendor/lib/python/ --no-deps Flask-Compress
and updated requirements/prod.txt by hand (we meet the requrements for Flask-Compress already). This may not be entirely correct.

I've verified the admin app still works using vagrant, and we don't lose two other the headers we set in base.py. Didn't have production quantities of data on hand to verify an improvement, but our current 10M of rules.html shrinks to 660K with a regular gzip -6 (Flask-Compress defaults to this level). Using -9 isn't worth it.

Comment 3

[[e-mail] (:bhearsum)]

Comment on attachment 8441120 [details] [diff] [review]
Add Flask-Compress

Review of attachment 8441120 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Nick Thomas [:nthomas] from comment #2)
> Created attachment 8441120 [details] [diff] [review]
> Add Flask-Compress
> 
> As a stopgap we could enable gzip compression in the app, which this patch
> does. I did 
>   pip install -t vendor/lib/python/ --no-deps Flask-Compress
> and updated requirements/prod.txt by hand (we meet the requrements for
> Flask-Compress already). This may not be entirely correct.

The installation looks fine to me.

> I've verified the admin app still works using vagrant, and we don't lose two
> other the headers we set in base.py. Didn't have production quantities of
> data on hand to verify an improvement, but our current 10M of rules.html
> shrinks to 660K with a regular gzip -6 (Flask-Compress defaults to this
> level). Using -9 isn't worth it.

I tried this with my big database locally and it (somewhat surprisingly) improved page load time by ~1.5s on rules.html. Render time seemed about the same, unsuprisingly. In any case, I think this is worth a try in production.

Thanks!

Comment 4

[[github robot]]

Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/b6164a97369f04fbdd6f5b4dedeb1051ee375bbc
Bug 932787, gzip responses from admin app, r=bhearsum

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Hit a bug in dev:
[Tue Jun 17 21:00:10 2014] [error] [client 10.8.81.217] mod_wsgi (pid=8958): Exception occurred p
rocessing WSGI script '/data/www/aus4-admin-dev.allizom.org/app/admin.wsgi'.
[Tue Jun 17 21:00:10 2014] [error] [client 10.8.81.217] Traceback (most recent call last):
[Tue Jun 17 21:00:10 2014] [error] [client 10.8.81.217]   File "/data/www/aus4-admin-dev.allizom.
org/app/admin.wsgi", line 10, in <module>
[Tue Jun 17 21:00:10 2014] [error] [client 10.8.81.217]     from auslib.admin.base import sentry
[Tue Jun 17 21:00:10 2014] [error] [client 10.8.81.217]   File "/data/www/aus4-admin-dev.allizom.
org/app/auslib/admin/base.py", line 2, in <module>
[Tue Jun 17 21:00:10 2014] [error] [client 10.8.81.217]     from flask.ext.compress import Compre
ss
[Tue Jun 17 21:00:10 2014] [error] [client 10.8.81.217] ImportError: No module named ext.compress
[Tue Jun 17 21:00:10 2014] [debug] mod_headers.c(768): headers: ap_headers_error_filter()

Comment 6

[[github robot]]

Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/fdddea20931f63896a10aaab272fb6bd209a9b16
Bug 932787, followup to fix import of compress module

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Not sure what was happening there. The flask.ext from flask-wtf wasn't picking up flask-compress on the dev server, but was in my vagrant, so I just worked around it. 

Anyway, for me this drops the data transferred for rules.html to 600KB, reducing the data transfer time from ~7200s to ~1600s. It increases the waiting time by ~500ms, but overall a nice win.

Comment 8

[[e-mail] (:bhearsum)]

Nick, your patch is in production now.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Thanks, it brings the page load time down to 16 seconds for me, so we still need a general fix.

Comment 10

[[e-mail] (:bhearsum)]

The new Balrog UI is super fast.