1014133 - phonedash - improve performance

Status: RESOLVED FIXED

(Testing Graveyard :: Autophone, enhancement)

Description

[Bob Clary [:bc] (inactive)]

As discussed in mobile/testing today, we should change phonedash's ui to allow selectable branches and default to mozilla-central to reduce the machine load on initially loading the phonedash ui. We could also change the default date range to less than a week if needed.

Comment 1

[Bob Clary [:bc] (inactive)]

"Premature optimization is the enemy"

http://phonedash.mozilla.org/api/s1s2/info/ is the bottle neck. It
takes over 10 seconds to return a short list of testnames and device
names.

is handled in phonedash server/handler.py

    '/s1s2/info/?', 'S1S2RawFennecParameters',

queries the entire database for all phones, tests and products ever
recorded!

class S1S2RawFennecParameters(object):

    @templeton.handlers.json_response
    def GET(self):
        phones = [x['phoneid'] for x in autophonedb.db.query(
            'select distinct phoneid from %s' % autophonedb.SQL_TABLE)]
        tests = [x['testname'] for x in autophonedb.db.query(
            'select distinct testname from %s' % autophonedb.SQL_TABLE)]
        products = [x['productname'] for x in autophonedb.db.query(
            'select distinct productname from %s' % autophonedb.SQL_TABLE)]
        return {'phone': phones, 'test': tests, 'product': products}

this is only called when main is called. which is why we have the big
delay on first load but not when we toggle checkboxes or change tests.

I wonder if we should just eliminate the call to /s1s2/info
S1S2RawFennecParameters and use '/s1s2/data/?', 'S1S2RawFennecData' to
return the phones, tests and products for the date range. This will be
much faster.

S1S2RawFennecData returns the results in json of the form

results[phone][test][metric][blddate] = value

I think if I add the list of phones, tests and produces to the results
as in results['parameters']['phone'|'test'|'product'] I can get the
required data for the specified date range rather than for all of the
data in the database. I'll need to rebuild the html after every data
load that changes the date range, but that will be a welcome tradeoff
I think.

Comment 2

[Bob Clary [:bc] (inactive)]

Just adding indexes has improved things considerably. info is no longer the bottleneck and loading the data for a week (750KB) is now the largest contributor. A week will now load for me in 10 seconds or so.

alter table rawfennecstart add index(phoneid);
alter table rawfennecstart add index(testname);
alter table rawfennecstart add index(productname);
analyze table rawfennecstart;

Not sure anything else is worth doing since phonedash is on the way out.


blassey: how does this compare to previously?

mcote?

Comment 3

[Brad Lassey [:blassey] (use needinfo?)]

much much better

Comment 4

[Bob Clary [:bc] (inactive)]

blassey, Could you let us know which of the following you would prefer to improve performance further?

1. Leave the default date range to 7 days, change the default to only display mozilla-central but allow the selection of multiple branches from mozilla-central, fx-team and mozilla-inbound.

2. Change the default date range to a lower value, something like 2-3 days and continue to display all branches all the time?

3. Leave alone. It's good enough for now.

Comment 5

[Brad Lassey [:blassey] (use needinfo?)]

rank order of preferenc is 3, 1 then 2.

Comment 6

[Bob Clary [:bc] (inactive)]

Attachment: bug-1014133.patch

I updated the running instances of the databases but didn't change the sql that creates them if they don't already exist. I tested this locally and it works ok.

Comment 7

[Bob Clary [:bc] (inactive)]

https://github.com/markrcote/phonedash/commit/7e054bc12c49213bcc0b009ddb41d959da97763a