631345 - Run weekly performance tests of top new add-on versions

Status: RESOLVED WONTFIX

(Testing :: General, defect)

Description

[Wil Clouser [:clouserw]]

Last time I talked with Alice our next step was going to be running weekly performance tests of the top 100 (by usage, I assume) add-on's latest versions.

Now that the testing infra can talk to AMO's db we should do this.  Let me know how I can help.

Comment 1

[Justin Scott [:fligtar]]

I've thought about this recently and while we should definitely run tests on the top 100 add-ons at least once, we don't need to do it every week because most of the tests will just be re-testing what we tested the previous week.

Instead, we should test new versions of add-ons submitted in the last week, ordered by add-on popularity. So new versions of top add-ons will be tested first but we'll also get the mildly popular, and we won't be duplicating any effort.

Comment 2

[Wil Clouser [:clouserw]]

The infrastructure is in place (save bug 639573) and results are flowing.  bhearsum is going to trigger the top 100 job to get us some initial results.

Heather: are there any updates on the weekly script?  Blockers? ETA?

Comment 3

[alice nodelman [:alice] [:anode]]

Is it worth pursing building a custom script to generate different lists of addons once a week, or continue the effort to set up a separate addon testing pool that we can submit tests requests to on an as-needed basis (ie, so that a given user can just press a 'get perf results' button and a test request is pushed into the queue)?

Comment 4

[Wil Clouser [:clouserw]]

(In reply to comment #3)
> Is it worth pursing building a custom script to generate different lists of
> addons once a week, or continue the effort to set up a separate addon testing
> pool that we can submit tests requests to on an as-needed basis (ie, so that a
> given user can just press a 'get perf results' button and a test request is
> pushed into the queue)?

The script is just changing the current one from sorting by usage to sorting by date.  I don't think that is the blocker here.  Last I heard the blocker was running untrusted code on your clusters.  If that is fixed, changing the sort should take a couple minutes, so yes, I think it would be worth it.

Comment 5

[alice nodelman [:alice] [:anode]]

Currently, we use a text file that contains a list of the top 100 addons.  To
get this going we'll have to switch to generating a list - is that list posted
somewhere easily web scrapable?  

It would be simple enough to add another cron job to update the list before
spawning the weeks tests.

Comment 6

[Wil Clouser [:clouserw]]

Here is an rss feed you could use to build a list: https://addons.mozilla.org/en-US/firefox/extensions/format:rss?sort=updated

Comment 7

[Jorge Villalobos [:jorgev] (he/him)]

I don't think we want this anymore.