The default bug view has changed. See this FAQ.

incremental loading of rules/releases data

RESOLVED WONTFIX

Status

Release Engineering
Balrog: Frontend
P3
normal
RESOLVED WONTFIX
3 years ago
2 years ago

People

(Reporter: bhearsum, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2947] )

Attachments

(1 attachment)

(Reporter)

Description

3 years ago
Both the rules and releases pages slow down tremendously when you have lots of releases. For releases, this is because we return every release once. For rules, we actually return every release _for every rule_ (in the "Mapping" dropdown).

We should do this more smartly. For releases.html we should probably load an initial set of data, and then have the "next" button request more from the server.

For rules.html we should probably only return the current mapping for each rule and then load the full set when someone clicks a Mapping dropdown or starts typing. It might be good to cache the result on the client too (probably with a TTL, because otherwise you'll miss newer data if you leave the page open for too long).
(Reporter)

Comment 1

3 years ago
mass priority change after component move
Priority: -- → P3
Created attachment 8441120 [details] [diff] [review]
Add Flask-Compress

As a stopgap we could enable gzip compression in the app, which this patch does. I did 
  pip install -t vendor/lib/python/ --no-deps Flask-Compress
and updated requirements/prod.txt by hand (we meet the requrements for Flask-Compress already). This may not be entirely correct.

I've verified the admin app still works using vagrant, and we don't lose two other the headers we set in base.py. Didn't have production quantities of data on hand to verify an improvement, but our current 10M of rules.html shrinks to 660K with a regular gzip -6 (Flask-Compress defaults to this level). Using -9 isn't worth it.
Attachment #8441120 - Flags: review?(bhearsum)
(Reporter)

Comment 3

3 years ago
Comment on attachment 8441120 [details] [diff] [review]
Add Flask-Compress

Review of attachment 8441120 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Nick Thomas [:nthomas] from comment #2)
> Created attachment 8441120 [details] [diff] [review]
> Add Flask-Compress
> 
> As a stopgap we could enable gzip compression in the app, which this patch
> does. I did 
>   pip install -t vendor/lib/python/ --no-deps Flask-Compress
> and updated requirements/prod.txt by hand (we meet the requrements for
> Flask-Compress already). This may not be entirely correct.

The installation looks fine to me.

> I've verified the admin app still works using vagrant, and we don't lose two
> other the headers we set in base.py. Didn't have production quantities of
> data on hand to verify an improvement, but our current 10M of rules.html
> shrinks to 660K with a regular gzip -6 (Flask-Compress defaults to this
> level). Using -9 isn't worth it.

I tried this with my big database locally and it (somewhat surprisingly) improved page load time by ~1.5s on rules.html. Render time seemed about the same, unsuprisingly. In any case, I think this is worth a try in production.

Thanks!
Attachment #8441120 - Flags: review?(bhearsum)
Attachment #8441120 - Flags: review+
Attachment #8441120 - Flags: checked-in+

Comment 4

3 years ago
Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/b6164a97369f04fbdd6f5b4dedeb1051ee375bbc
Bug 932787, gzip responses from admin app, r=bhearsum
Hit a bug in dev:
[Tue Jun 17 21:00:10 2014] [error] [client 10.8.81.217] mod_wsgi (pid=8958): Exception occurred p
rocessing WSGI script '/data/www/aus4-admin-dev.allizom.org/app/admin.wsgi'.
[Tue Jun 17 21:00:10 2014] [error] [client 10.8.81.217] Traceback (most recent call last):
[Tue Jun 17 21:00:10 2014] [error] [client 10.8.81.217]   File "/data/www/aus4-admin-dev.allizom.
org/app/admin.wsgi", line 10, in <module>
[Tue Jun 17 21:00:10 2014] [error] [client 10.8.81.217]     from auslib.admin.base import sentry
[Tue Jun 17 21:00:10 2014] [error] [client 10.8.81.217]   File "/data/www/aus4-admin-dev.allizom.
org/app/auslib/admin/base.py", line 2, in <module>
[Tue Jun 17 21:00:10 2014] [error] [client 10.8.81.217]     from flask.ext.compress import Compre
ss
[Tue Jun 17 21:00:10 2014] [error] [client 10.8.81.217] ImportError: No module named ext.compress
[Tue Jun 17 21:00:10 2014] [debug] mod_headers.c(768): headers: ap_headers_error_filter()

Comment 6

3 years ago
Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/fdddea20931f63896a10aaab272fb6bd209a9b16
Bug 932787, followup to fix import of compress module
Not sure what was happening there. The flask.ext from flask-wtf wasn't picking up flask-compress on the dev server, but was in my vagrant, so I just worked around it. 

Anyway, for me this drops the data transferred for rules.html to 600KB, reducing the data transfer time from ~7200s to ~1600s. It increases the waiting time by ~500ms, but overall a nice win.
(Reporter)

Updated

3 years ago
Depends on: 1037440
(Reporter)

Comment 8

3 years ago
Nick, your patch is in production now.
Thanks, it brings the page load time down to 16 seconds for me, so we still need a general fix.

Updated

2 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2938]

Updated

2 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2938] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2942]

Updated

2 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2942] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2947]
(Reporter)

Comment 10

2 years ago
The new Balrog UI is super fast.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.