Closed Bug 717980 Opened 13 years ago Closed 12 years ago

Get a list of source addons for JetPerf testing

Categories

(Testing :: Talos, defect)

Product:
Testing
Component:
Talos
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: k0scist, Unassigned)

References

Details

(Whiteboard: [jetpack+talos]) 

WRT JetPerf testing (bug 717036), we need a list of source addons to
be generated with the SDK.  The first should be a "hello world" app
that doesn't do much of anything, then we can expand from there.
Whiteboard: [jetpack+talos]
Blocks: 717036
Blocks: 729205
Here is a list of potential addons to test:
  https://github.com/ochameau/jetperf-test-addons

These addons just prints some messages like this:

  [PERF][1334758986946] Addon running
  [PERF][1334758987115] Content script loaded
  [PERF][1334758987141] DOM Updated

The big number being the result of (new Date().getTime()). 
You may adapt them to use something else, but that gives an idea on what kind of simple addon we may want to test.

I think we would like to test two thing:
 - Do we hit a perf regression on Jetpack? i.e. is Jetpack becoming slower with one particular revision? Like addons are slower to start, or Widget takes longer to load, or content script are executing slower.
 - Do we hit a perf regression against Firefox? i.e. is Jetpack slowing down Firefox with one particular revision? Like web content takes more time to be loaded/display when there is a page-mod running.

Here, with these examples, I'm only trying to verify first case: "Is Jetpack itself running slower".
It would be interesting to run tests against the second case in order to see impact on Firefox that Jetpack has.
See also https://bugzilla.mozilla.org/show_bug.cgi?id=729205#c16

As I understand it, using the addons from comment 1 will prevent us from installing an addon multiple times.  Do we care about this?
We should also figure out where these should live in relationship to bug 729205 . If we're running jetperf in production, AFAIK it is not kosher to touch github (or similar) in production runs.  Should we put these addons in the mozharness repository?  Should we put them somewhere on a Mozilla site and download them?  I'm more inclined to include them directly in mozharness where you have a notion of directory.
(In reply to Jeff Hammel [:jhammel] from comment #2)
> See also https://bugzilla.mozilla.org/show_bug.cgi?id=729205#c16
> 
> As I understand it, using the addons from comment 1 will prevent us from
> installing an addon multiple times.  Do we care about this?

You are right, but I don't think we care that much about such usecase, do we?
But I'm just jumping into your project so I may have different/new ideas compared to what you have in mind.
If you think that what I'm suggesting doesn't go in your direction, please tell me what you have in mind for the long run. So that I can help you driving in this particular direction.

(In reply to Jeff Hammel [:jhammel] from comment #3)
> We should also figure out where these should live in relationship to bug
> 729205 . If we're running jetperf in production, AFAIK it is not kosher to
> touch github (or similar) in production runs.  Should we put these addons in
> the mozharness repository?  Should we put them somewhere on a Mozilla site
> and download them?  I'm more inclined to include them directly in mozharness
> where you have a notion of directory.

I used github as a way to show/share these examples with you.
You can consider them as being in public domain and do whatever you think is relevant with them. Host them on some other repo, modify them ...
It is up to you to decided where/how host such files in "Testing universe".
:aki, what would you like to do with these directories?
Would these fit in http://hg.mozilla.org/projects/addon-sdk ?
That would allow addon sdk developers to update the tests if wanted/needed.
(In reply to Aki Sasaki [:aki] from comment #6)
> Would these fit in http://hg.mozilla.org/projects/addon-sdk ?
> That would allow addon sdk developers to update the tests if wanted/needed.

It works for me, but I have no jurisdiction over this repo.  :ochameau, does this work for you and the jetpack team or should we seek another solution?
I'm not sure that's the best place to land them.
Any file added to this repository is going to be included in the SDK downloaded by developers, whereas I don't see why they would need them?
I'll try to bring up this question to the Jetpack team.
So this decision is blocking any further jetperf. It is clear, moving from text addons to directories of addons that most of the jetperf script will need a rewrite and the form of this rewrite is contingent upon where these testing addons live.  I can't really do much here until we figure out a production home for these (that is, presumedly, developer accessible)
I'm fine with a subdirectory in addon-sdk, or a separate sibling repository if that speeds things up.
Why can't we just put these things somewhere to unblock ourselves and make that location a parameter to the test so that it is easy to change if/when the jetpack devs change their minds about where to put them?  I'm not clear why we're letting the location of some files block us here.
:aki, :ochameau : what about a repository that lives at http://hg.mozilla.org/projects/addon-sdk-jetperf for these addons? This would have the advantage that:

- it is accessible to devs so that they can make changes or add addons as they desire
- it is a mozilla repository so that the buildsystem can be happy
wfm.
WFM as weel.

As I said in comment 4 you can do whatever you like with these addons.
Host them however/whereever is best for you.
Landing them in addon-sdk repository is another question.
Depends on: 749681
Depends on: 755058
Commited: http://hg.mozilla.org/projects/addon-sdk-jetperf-tests/rev/d7e90684309c
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.