251416 - Preload Products and Components information

Status: NEW

(Bugzilla :: User Accounts, enhancement)

Description

[Joel Peshkin]

When a huge number of products and components are defined the UI can become very
clunky.  This could be avoided if we did the following....

1) If we don't get a cookie HAS_JS from the client, include a chunk of
javascript that sets a HAS_JS cookie. (or else that sets a form variable that
causes us to set HAS_JS on the next page -- in case scripts are not allowed to
set cookies)

2) If we don;t get a HAS_JS cookie, behave exactly as we do today.

3) If we get a HAS_JS cookie, omit contents of long option lists (Selectable
products, selectable components, enterable products (with their descriptions),
etc...) and use javascript to populate those lists from arrays in cachable
javascript.

4) The cachable javascript would be generated by a cgi, taking the user's login
credentials (cookie) into account.

5) Change the GoAgeadAndLogIn behavior after the initial welcome page to take
the user back to index.cgi

6) Build code into the end of the page footer that triggers the javascript to
load if it has not already loaded.

7) Build some session token or hash of the login cookie into the url of the
javascript so that the browser will consider it a new thing to be loaded if the
user logs out or logs in as a differrent user.

Comment 1

[Gervase Markham [:gerv]]

How would you force clearing of the cache from the server side if, for example,
a new product or component was added?

> When a huge number of products and components are defined the UI can become very
> clunky.

Could you define 'clunky' a bit more? Do you mean slow to load? Or slow to
manipulate? Or hard to use?

Gerv

Comment 2

[Christian Reis]

Hmmm. I don't see how this would reduce clunkiness (but maybe I'm just too
sleepy), and at any rate, I don't see what specific advantage the cookie gives us.
Wouldn't the proper solution be having a separate JS file that contained these
items? Oh, but then, we already do: Bug 72837 -- isn't it a matter of starting
to use those there?

Comment 3

[Joel Peshkin]

If you look at a site with a huge number of products and.or components, loading
time for entry or query pages is terrible.

javascript can leave an array on the client machine that is cachable.  However,
we do not want to be javascript-dependant, so we need to always know if we
should rely on it or not... hence the cookie.

When a new user visits a site, he usually stares at index.cgi for a few moments.
Ideally, we could load the arrays into cache then.  When the user subsequently
visits query.cgi (probably many times), the javascript would already be cached
in the browser and presumably the total page load time would be much smaller.

If the javascript for ny session has a hash of my login token built into its
name, there is no need to worry about purging the cache. As far as the browser
is concerned, it it a distinct file every time the token - and hence the name --
changes.

Comment 4

[Gervase Markham [:gerv]]

> If you look at a site with a huge number of products and.or components, 
> loading time for entry or query pages is terrible.

bugzilla.gnome.org has the most products and components of any Bugzilla I know
of. Their query.cgi page is 164KB. (KDE's is 106KB.) 164KB is 41 seconds over a
modem, but about 4 seconds over ADSL/cable. So this is a modem-only problem,
really. Do many software developers still use modems? Perhaps there are a few.
I'd like to see some stick their hands in the air, though :-)

Anyway, what can we do to reduce this? How about a simpler solution - we offer
the option of opening query results in a new window (as a per-user toggle in
prefs.) That way, they can load the query page once and do as many searches as
they like. No JS, no cookies, very simple.

In fact, we could use this "slow connection" toggle for various other things. We
could do the same "new window" trick for reports, page buglists at 100 bugs per
page, that sort of thing.

> If the javascript for ny session has a hash of my login token built into its
> name, there is no need to worry about purging the cache.

What about if a new product or component gets added?

Gerv

Comment 5

[Christian Reis]

Can someone explain to me why using config.js won't solve the problem here? Is
it because it's not cacheable? What if we *did* generate a cacheable version
(like versioncache)?

Comment 6

[Joel Peshkin]

WRT Comment 4:
Try Mandrake's Bugzilla.

I loaded it on a site with a >10MBPS corporate connection and it took long
enough to load pages that I said "what a piece of junk."  I don't think that is
what we want people to think when they see Bugzilla.

On my home ADSL line, it took 18 seconds to load
http://qa.mandrakesoft.com/query.cgi?format=advanced of which 9 seconds were
serious data transfer and 9 seconds was just a pause before starting.

Forcing the browser to go onto a freshly loaded javascript file whenever
something changes (login, logout, new products, etc..) is a very simple matter.
 All we have to do is change the name of the "file" (dynamically generated by cgi)

WRT Comment 5:
In order to help, we need to provide the browser with cachable javascript that
makes it unnecessary to send hard-coded HTML.  Therefore, we need to know if we
should send an HTML form that has every possible option coded into it or if we
should send a form that has empty pick-lists and relies on javascript sent with
the form to read tables in cached javascript to populate the form.

To keep this on track, don't forget that the contents of the javascript tables
are only those items to which you have access.

Comment 7

[Gervase Markham [:gerv]]

> Try Mandrake's Bugzilla.

627KB. Hmm.

> 9 seconds was just a pause before starting.

Yeah - I saw that on the others. What's with that? Is Mozilla lying to us and
actually downloading data during this time?

Even after all this, I'm still concerned about the complexity of the proposed
solution, and the potential for hard-to-track-down bugs. What do you think of my
other idea for having a per-user "low bandwidth" option which opens query
results in a new window?

Another way of attacking the problem might be by making it possible to enable
mod_gzip on Bugzilla servers. Is that possible today? A gzip of Mandrake's
query.cgi output is 90K.

Another way of cutting it in half which is less complex would be to only
populate the option name, and get JS to go through populating the contents (as
it's the same text string). Poor man's compression in only a few lines.

The last option is to throw up our hands and say that Bugzilla really wasn't
designed for 11,356 (no, that's not a typo) products in a single installation.

Gerv

Comment 8

[Erik Stambaugh]

(In reply to comment #7)
> > 9 seconds was just a pause before starting.
> 
> Yeah - I saw that on the others. What's with that? Is Mozilla lying to us and
> actually downloading data during this time?

I'm no browser expert, but we have plenty of those around here to tell me if I'm
wrong.  I'm pretty sure that amount of time is spent by the renderer trying to
lay the page out.  If that's true, there should be a few ways to mitigate that
time in addition to the time spent in data transfer.

> Another way of cutting it in half which is less complex would be to only
> populate the option name, and get JS to go through populating the contents (as
> it's the same text string). Poor man's compression in only a few lines.

That sounds great.  Then, when you do that, cache the data portion of that JS!

Comment 9

[Gervase Markham [:gerv]]

> I'm pretty sure that amount of time is spent by the renderer trying to
> lay the page out.

That can't be right. The pause is the first thing that happens. Layout out the
page is the _last_ thing that happens :-)

> That sounds great.  Then, when you do that, cache the data portion of that JS!

You've misunderstood me. I'm suggesting JS which turns:

<option value="foo"></option>
<option value="somelongoption"></option>

into

<option value="foo">foo</option>
<option value="somelongoption">somelongoption</option>

i.e. using it to allow us to remove the redundancy.

Gerv

Comment 10

[Christian Reis]

I can work on using config.js and extra perf hacks if somebody's available to
guide me around the problem (I don't have it here), let me just clean up the JS
so we can start on this.