650778 - Document how to test ts (warm) yourself

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Developer Pages, defect, P2)

Description

[Nils Maier [:nmaier]]

Right now the documentation on ts tests/Talos is sparse and scattered. The official documentation is not easily accessible and does not address the particular needs of add-on authors.

Allowing interested authors to use their own infrastructure as opposed to future web-based testing has a lot of upsides, such as easier use and scripting (incl. easier continuous integration), less latencies to obtain results and verifying the results or result trends of AMO.

There need to be documentation:
- What is tested (warm ts right now)
- How it is tested
- How are measurements interpreted/How are results calculated
- The meaning/importance of the test(s)

And HowTos on:
- How to set up the testing infrastructure, i.e. detailed prerequisites and installation for all 1-tier platforms
- How to configure and run tests
- How to analyze measurements and compute results, preferably incl. information on stddev and its meaning and how to check for unstable runs.
- How to interpret the results, especially regarding different local baseline values compared to mozilla Talos boxen.
- What about nsIAppStartup(_MOZILlA_2_0)?.getStartupInfo() ?

The documents should be accessible (linked) from Developer Pages, and either be hosted on AMO or MDC.

Existing documentation:
https://wiki.mozilla.org/Auto-tools/Tools/Talos
https://wiki.mozilla.org/StandaloneTalos
https://wiki.mozilla.org/Buildbot/Talos
https://developer.mozilla.org/en/Running_automated_tests
http://xulforge.com/blog/2011/04/testing-add-on-startup-performance/
Taras announced an article in: (hence CC'ing)
http://blog.mozilla.com/tglek/2011/04/14/arguing-about-startup-communicating-performance/

Comment 1

[cmtalbert]

Hi Nils,

I don't understand what you want. I did document how to run the talos ts test (which by running it by hand you are by default doing a warm ts) on http://wiki.mozilla.org/Auto-tools/Tools/Talos.  In that document I tried to pull all of the other "how to run Talos" pages together, since there are many ways to run Talos depending on what you want to do.  I added it to the http;//developer.mozilla.org/en/Running_automated_tests page so that Talos would be discoverable from MDC.  I can host that wiki page on MDC if you prefer, I put it on wiki as that was easier to get some documentation in place because before that page we truly had a lot of pages that talked around talos and nothing that talked about it.

I have more specific questions below:

(In reply to comment #0)
> Right now the documentation on ts tests/Talos is sparse and scattered. The
> official documentation is not easily accessible and does not address the
> particular needs of add-on authors.
>
No it isn't.  The official documentation is http://wiki.mozilla.org/Auto-tools/Tools/Talos.  (per bug 648113). That said, docs can always be made better, so the fact that you're confused says that we need to make the docs better. 
> Allowing interested authors to use their own infrastructure as opposed to
> future web-based testing has a lot of upsides, such as easier use and scripting
> (incl. easier continuous integration), less latencies to obtain results and
> verifying the results or result trends of AMO.
> 
There's nothing stopping you from running Ts now on your own hardware. That said, there is still value in doing centralized testing as well.  By doing centralized testing, we can generate comparable numbers for various addons based on known hardware configurations.

So sounds like below you're looking for addon-talos specific testing. Is that right?  Maybe I need to write another document covering that?

> There need to be documentation:
> - What is tested (warm ts right now)
Yes.
> - How it is tested
What are you looking for here?  It tests by taking a time stamp, then starting firefox pointed at a webpage that takes a timestamp during the onload handler.  The value reported is the difference between the two timestamps.  This method has numerous issues (see Taras's blog), but it has been used since Firefox 2.0 days, and therefore, it continues to be an interesting metric if only to demonstrate changes over time. 
> - How are measurements interpreted/How are results calculated
Do you mean how are the results calculated before they go onto the AMO graph?  I know there is some throwing out the first run, and the slowest run and then the rest of the runs are averaged.  That could be documented.  The graphing piece is less clear to me as well.

> - The meaning/importance of the test(s)
What are you looking for here?  I think this is what Fligtar is excellently writing about on his blog.  And I think this is out of scope for documentation of the Talos tool itself.  The Talos results have various meanings in various contexts, like any set of automated tests.  For example, for AMO we envisioned that these results would help add on authors be able to find and fix problems in their addons.  For patches to Mozilla-Central, Talos results are used as a judge of whether or not a patch should be backed out.  So, it's a very different standard depending on context, and has nothing at all to do with the tool or the testing framework.
> 
> And HowTos on:
> - How to set up the testing infrastructure, i.e. detailed prerequisites and
> installation for all 1-tier platforms
I believe that's done on my wiki page.  Are there things missing?

> - How to configure and run tests
Also on the wiki.  Did you find things missing?

> - How to analyze measurements and compute results, preferably incl. information
> on stddev and its meaning and how to check for unstable runs.
This would indeed be a good addition.

> - How to interpret the results, especially regarding different local baseline
> values compared to mozilla Talos boxen.
Running on different machines is quite obviously going to give you different results.  The best you can do comparing machine to machine is to do a run of "vanilla" firefox, then firefox with the changes you want to test and compute a percentage of slowness from them.  Then you can sorta squint and compare the two different machines.  It's not a perfect system by any means, and I don't think it can be made into one.  Is that the kind of doc you're looking for?

> - What about nsIAppStartup(_MOZILlA_2_0)?.getStartupInfo() ?
This is one of Taras's projects which just landed this January.  It landed as part of bug 522375, and is being used (I believe) by the new Telemetry (https://wiki.mozilla.org/Platform/Features/Telemetry) project.  It is quite new.  Talos has not been updated to use that information and if Talos is updated to use that, then it will be as part of a new test because the Ts test needs to run against all released versions of Firefox, not just Firefox 4/5+.

> The documents should be accessible (linked) from Developer Pages, and either be
> hosted on AMO or MDC.
I can quite easily host the wiki page I wrote on MDC if that helps.  I'm not sure what you mean by "Developer Pages" though.  What are those?

Comment 2

[Jorge Villalobos [:jorgev] (he/him)]

I'm actually working on a draft document for this bug, as you can read here: https://developer.mozilla.org/User:Jorge.villalobos/Testing_Add-on_Startup_Performance. My intention is to eventually move this document somewhere on MDC. It's valuable for add-on developers to have a doc aimed specifically at them, since they shouldn't bother learning all the details about how Talos works.

I'm still editing it today, but it's fairly complete right now IMO, so I'm happy to hear feedback. You're also free to edit the document yourselves.

(In reply to comment #1)
> > The documents should be accessible (linked) from Developer Pages, and either be
> > hosted on AMO or MDC.
> I can quite easily host the wiki page I wrote on MDC if that helps.  I'm not
> sure what you mean by "Developer Pages" though.  What are those?

I think he's referring to the AMO Developer Hub pages.

Comment 3

[Nils Maier [:nmaier]]

The intension of this bug is to create a guide for add-on developers that describes not only the methodology and meaning/significance of the tests, but also how to install and set up the required tools, run tests and interpret the results, while avoiding any aux information (e.g. other Talos tests).
That guide has to be readily accessible from a place were add-on authors would actually look for it, i.e. AMO Dev Hub or at least MDC, but not wiki.mo

While this is not within the scope of Talos, it is well within the scope of AMO/DevPages.

Basically Jorge's draft is what I'm looking for
+ some in-document installation instructions for Talos (ts), incl. what easy_install/pip commands are required.
+ platform specific documentation where it makes sense (e.g. Windows)
+ a central and discoverable place to access that document

I'm also looking for additional documentation describing how AMO performs the tests in particular, e.g. mentioning what add-ons are currently tested, how often and on what kind of boxen, so that add-on authors can better understand what's happening.
(A blog is not "documentation")

(In reply to comment #1)
> (In reply to comment #0)
> > Right now the documentation on ts tests/Talos is sparse and scattered. The
> > official documentation is not easily accessible and does not address the
> > particular needs of add-on authors.
> >
> No it isn't.  The official documentation is
> http://wiki.mozilla.org/Auto-tools/Tools/Talos.  (per bug 648113). That said,
> docs can always be made better, so the fact that you're confused says that we
> need to make the docs better.

I'm not talking about Talos, but AMO Slow Addons tests. It took a while until I had the various things pieced together (Talos, BuildBot, Tinderbox, Zamboni) to get the whole picture and from there determine how to perform my own tests the same way AMO does.
This bug is about creating documentation for add-on authors wanting to perform the tests themselves, but who do not even know about the existence of Talos to begin with.

> > - The meaning/importance of the test(s)
> What are you looking for here?  I think this is what Fligtar is excellently
> writing about on his blog.
Again, a blog is not "documentation". Documentation is (or should be) kept up to date, while a blog entry usually isn't. Wladimir and Taras wrote some very good things about warm/cold startup. If an author is not subscribed to planet, however, then he never saw the information.
I see that Jorge already included a paragraph in his draft...

> > - How to interpret the results, especially regarding different local baseline
> > values compared to mozilla Talos boxen.
> Running on different machines is quite obviously going to give you different
> results.  The best you can do comparing machine to machine is to do a run of
> "vanilla" firefox, then firefox with the changes you want to test and compute a
> percentage of slowness from them.  Then you can sorta squint and compare the
> two different machines.  It's not a perfect system by any means, and I don't
> think it can be made into one.  Is that the kind of doc you're looking for?

Yes, that kind of documentation.
But you cannot really "squint" here. Experimentation showed that add-on impact does not grow proportionally to baseline times, and especially that observation should be mentioned. Wladimir confirmed those findings and also wrote a couple of times about it.
Actually, for many add-ons the absolute startup time impact is nearly the same no matter what box you use to test. As there is a high probability that developers test on slower machines the resulting relative impact values would be far less than what AMO measures (e.g. for one my add-on I measure up to 13 percent points less than compare to AMO/tinderbox results).
Omitting such information would get add-on authors pretty confused when their local tests come out as 10% impact, but AMO slaps them a 25% warning later.

Comment 4

[Jorge Villalobos [:jorgev] (he/him)]

(In reply to comment #3)
> + some in-document installation instructions for Talos (ts), incl. what
> easy_install/pip commands are required.
> + platform specific documentation where it makes sense (e.g. Windows)

I'm not sure if this is appropriate for the document, given that it will be redundant with available documentation and will need to be kept up to date. I prefer a guide that says "1. install python, 2. install pyyaml, 3. install windows extensions" with appropriate links, because new versions of those packages may require different installation instructions.

Maybe it's because I'm on Mac, but I didn't have any problems getting everything set up in a few minutes following the links.

> + a central and discoverable place to access that document

I think it should be at https://developer.mozilla.org/en/Measuring_Add-on_Startup_Performance, and there should be links from main pages, like the Extensions page, the Developer Hub and the Add-ons Blog. I'll move the doc there if there are no major objections to the draft.

> I'm also looking for additional documentation describing how AMO performs the
> tests in particular, e.g. mentioning what add-ons are currently tested, how
> often and on what kind of boxen, so that add-on authors can better understand
> what's happening.
> (A blog is not "documentation")

This I think we should have elsewhere. It should either be an AMO-hosted doc or a section in the AMO Developer FAQ page. I don't think it belongs on MDC, but like you say, it's probably best to have it in a more permanent location than the blog.

Comment 5

[cmtalbert]

(In reply to comment #2)
> I'm actually working on a draft document for this bug, as you can read here:
> https://developer.mozilla.org/User:Jorge.villalobos/Testing_Add-on_Startup_Performance.
> My intention is to eventually move this document somewhere on MDC. It's
> valuable for add-on developers to have a doc aimed specifically at them, since
> they shouldn't bother learning all the details about how Talos works.
> 
Ah excellent. Nils and Jorge, thanks for the clarification. Jorge, I'll move the detailed talos tool-specific stuff I wrote to MDC so you can easily link to it from your guide and it will be more googlable.

Comment 6

[Jorge Villalobos [:jorgev] (he/him)]

The document is now up at https://developer.mozilla.org/en/Measuring_Add-on_Startup_Performance, so I'm resolving this as fixed. We can improve the document through the wiki instead of in this bug. I also linked to it from the Performance best practices document and the from the general Performance home at MDC, so this document should be easy to find for those interested in add-on performance. I will also mention it on my next review update, and may blog separately about it, since it is fairly important.

Pending issues:
* Mention which add-ons are being tested: bug 647398.
* Mention which kind of hardware is being used. There's no bug for this, and I'm not sure how useful this information would be for add-on developers in general. Nils, you can file it if you think it's necessary.

Clint, if you move any content to MDC, let me know so I can update this doc. Or you can do this yourself as well.