write up consolidated SfN findings

RESOLVED FIXED

Status

Testing
Talos
RESOLVED FIXED
6 years ago
5 years ago

People

(Reporter: Jeff Hammel, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [SfN])

Attachments

(1 attachment)

(Reporter)

Description

6 years ago
We've discovered a lot of things we're doing wrong in the Signal from
Noise project.  However, these are scattered across the universe.  We
should consolidate them and make a single report for 2012Q4 that
documents all the things
(Reporter)

Updated

6 years ago
Whiteboard: [SfN]
(Reporter)

Comment 1

6 years ago
We should also probably document the end-to-end process of what we're doing now as a part of this in an easy and understandable way.
(Reporter)

Comment 2

6 years ago
Some source material: 

Here are a few blog posts I've written on how Talos works.  The first
is probably the most enlightening:

* http://www.k0s.org/mozilla/blog/20120829151007 : How Talos Works and
Why SfN is Hard

* http://www.k0s.org/mozilla/blog/20120425093346 : Considering a
Page-Centric Talos

* http://www.k0s.org/mozilla/blog/20120131164249 : analyzing the data

Relatedly, several things are a bit challenging because talos suites
and test names vary across buildbot, TBPL, talos, and graphserver.
More about this is documented at
http://k0s.org/mozilla/blog/20120724135349 and http://k0s.org:8080/

The links on the Signal from Noise page may also be worth reading for
more depth: https://wiki.mozilla.org/Auto-tools/Projects/Signal_From_Noise#Links

Comment 3

6 years ago
What's the ETA on this?
(Reporter)

Comment 4

6 years ago
Just getting back to forward-facing work after 2 offsites and being sick.  I'll take a stab at an outline with ETAs today or tomorrow. (And yes, that is an ETA on an ETA ;)
(Reporter)

Comment 5

6 years ago
Created attachment 680882 [details]
first stab at an outline

This is just a few notes and a bit of an outline.  I'd say to finish a report that I would want to read would be 8-16 hours more work.  :jmaher, :ctalbert, maybe we can talk about this tommorow
Comment on attachment 680882 [details]
first stab at an outline

A few areas here take a [implied] negative tone about problems, frustrations and issues with how talos is used or how it is written.

Specifically in the "Review of Performance Testing Workflow", and somewhat in the "State of Talos".  I think to keep this 'year in review' summary as something that inhibits curiosity and willingness to look at the changes, we should focus on:

* goals
* limitations (minimal)
* changes
** graph server
** buildbot
** talos
** tests
** other

Also leaving off the trailing bits in the roadmap about a dedicated stats person would keep this focused on what consumers of Talos can do, instead of what somebody else could do.  

Roughly speaking this hits the core points and puts a good outline of what is going on.
(Reporter)

Comment 7

6 years ago
:jmaher, do you want to take a stab at this?  Or maybe we should post this to an etherpad?
(Reporter)

Comment 8

6 years ago
posted to https://etherpad.mozilla.org/GxTsTCvpAo for flushing out
(Reporter)

Updated

6 years ago
Assignee: nobody → jhammel
(Reporter)

Comment 9

6 years ago
I'm done with the first draft at https://etherpad.mozilla.org/GxTsTCvpAo . Please feel free to add your own edits!
(Reporter)

Comment 10

6 years ago
Joel, let me know what you think of this please.  Also, if you have any ideas on where to put this, I'm happy to move it to a better medium.  (Likewise, Clint.)
Flags: needinfo?(jmaher)
I have added a series of comments in the document.  This has a wide collection of great information.  Who is the target audience for this?  Is this intended to be reference material?  

My general take is we need to build a document that outlines the work we have done and shows how things have changed and will be changing with an intended audience of developers who might be affected by changes.
Flags: needinfo?(jmaher)
(Reporter)

Comment 12

6 years ago
(In reply to Joel Maher (:jmaher) from comment #11)
> I have added a series of comments in the document.  This has a wide
> collection of great information.  Who is the target audience for this? 

It is a general audience, though particularly Firefox developers.

> Is this intended to be reference material?

For some definition, sure.
 
> My general take is we need to build a document that outlines the work we
> have done and shows how things have changed and will be changing with an
> intended audience of developers who might be affected by changes.

I have tried to do this.  If its going to fall to me to do this, we might as well post what we have on some wiki page and send a note out to a newsgroup now, because I don't have time to make it much better.  If this material is actually potentially beneficial, I would welcome edits.  In particular, the datazilla material is not well documented.  It would be nice if :jeads could fill this in.

I'd like to get this out this week.  I would encourage anyone who is interested to edit, edit, edit.  If no one has the time/inclination, I say we take what we have and put it up somewhere, e.g. https://wiki.mozilla.org/Auto-tools/Projects/Signal_From_Noise/Status-Q42012
(Reporter)

Comment 13

6 years ago
I've taken another pass at this, as has Clint, Christina, and :jeads : https://etherpad.mozilla.org/GxTsTCvpAo . While there are a few things that need touching-up, I think the main task is finding a place to put this.

I'll suggest, knowing that my suggestion will be hated, to put in

https://wiki.mozilla.org/Buildbot/Talos/Signal_From_Noise/2012Report
  - /PerformanceTesting2011
  - /Investigation
  - /PerformanceTesting2012
:jhammel, can you post the sanitized report in a few pieces here:
https://wiki.mozilla.org/Auto-tools/Projects/Signal_From_Noise#Status
(Reporter)

Comment 15

6 years ago
(In reply to Joel Maher (:jmaher) from comment #14)
> :jhammel, can you post the sanitized report in a few pieces here:
> https://wiki.mozilla.org/Auto-tools/Projects/Signal_From_Noise#Status

Sure, I'll post the sections in:

/StatusNovember2011
/StatusNovember2012
/Execution2012

We can move this information around later as we choose (with the caveat that we can't actually delete wiki pages, as usual)
(Reporter)

Comment 16

6 years ago
Posted in:

https://wiki.mozilla.org/Auto-tools/Projects/Signal_From_Noise/StatusNovember2011
https://wiki.mozilla.org/Auto-tools/Projects/Signal_From_Noise/StatusNovember2012
https://wiki.mozilla.org/Auto-tools/Projects/Signal_From_Noise/Execution2012

Please look these over.  There are a few ''comments'' that should be fixed up.  Also, we may want these informations in different places. We should figure that out.

Then we should:
- blog about this
- send notification to dev-${Something} mailing list
(Reporter)

Comment 17

6 years ago
I've taken another pass at these, correcting grammar and loose ends and adding bug #s where applicable.  Again, please take a look.

I'm thinking of combining https://wiki.mozilla.org/Auto-tools/Projects/Signal_From_Noise/Execution2012 and https://wiki.mozilla.org/Auto-tools/Projects/Signal_From_Noise/StatusNovember2012 ; specifically, putting the former into the latter.  The latter has a weak metrics section without real conclusions.  Any opposition?  I'll plan on doing this tomorrow if there is no opposition, with the standard "This page is a stub because we can't delete mediawiki pages" on the "deleted" page.

The conclusion is a little weak.  That probably needs doin' too.

Other than that, after this I think we can blog about this and post to dev-$WHATEVER and close this bug.
(Reporter)

Comment 18

6 years ago
(In reply to Jeff Hammel [:jhammel] from comment #17)
<snip/>
> I'm thinking of combining
> https://wiki.mozilla.org/Auto-tools/Projects/Signal_From_Noise/Execution2012
> and
> https://wiki.mozilla.org/Auto-tools/Projects/Signal_From_Noise/
> StatusNovember2012 ; specifically, putting the former into the latter.  The
> latter has a weak metrics section without real conclusions.  Any opposition?
> I'll plan on doing this tomorrow if there is no opposition, with the
> standard "This page is a stub because we can't delete mediawiki pages" on
> the "deleted" page.

Gonna do this thing
(Reporter)

Comment 19

6 years ago
So I've taken yet another pass here.

I ended up mostly punting wrt comment 18 for the time being; I decided it is probably better, given the desire to appeal to a wider audience that it would be better to confer with :jmaher, :jeads, and whoever else is interested before pulling the trigger.  I did move metrics over as A. given the state of the write up, it didn't really belong where it was; B. there was a bullet point about it at some point.

I still tend to think that putting the rest of https://wiki.mozilla.org/Auto-tools/Projects/Signal_From_Noise/Execution2012 to https://wiki.mozilla.org/Auto-tools/Projects/Signal_From_Noise/StatusNovember2012 . This is basically two parts at this point:

- Signal From Noise summary, Goals of Signal from Noise, Execution of Signal from Noise : these are short sections that could be put together.  Alternatively, some/all of this could go in https://wiki.mozilla.org/Auto-tools/Projects/Signal_From_Noise , which badly needs a rewrite anyway.  TBH, I would rather not do it myself as I feel if I do it will likely take longer than its worth with my comparatively stringent level of expectations. (I would also argue that much of the rest of the information should go with the talos documentation, currently at the poorly-chosen URL https://wiki.mozilla.org/Buildbot/Talos , but again, TBH, I'd rather get this bug done and do more coding and less wiki-editing, and not just because I dislike editing wiki.m.o).

- Problems We Aimed to Solve with Datazilla : https://wiki.mozilla.org/Auto-tools/Projects/Signal_From_Noise/StatusNovember2012#State_of_Datazilla:_November_2012 doesn't really give much context or an executive summary or links to where you could figure out such things, so I am inclined to port this section and put it in the past tense, since we did achieve these goals (or, more accurately, mostly :jeads did).

One reason I didn't do this today is, thinking in these terms, I would also be inclined to go ahead and put in https://wiki.mozilla.org/Auto-tools/Projects/Signal_From_Noise/StatusNovember2011 too and have a fairly complete document.  As I know that there is concern that long documents are hard to navigate and use, I decide to step back and make sure we're on the same page here and converge on what we want to do. I'd advise porting the datazilla schtuff over and reworking it a bit (volunteers?).  The other parts i could port to Status2012, to the SfN main page, or leaving where they are.  If this is annoying to work out via bug comments, i'd be happy to have a 5-10 minute meeting where we figure this out or wait for the SfN meeting.

Outside of that, this seems basically done (given sufficiently tight scope) save for some nits:

- there are two requests for more info:

"""
Test definitions are no longer duplicated throughout the Talos codebase. These are now in a python file: http://hg.mozilla.org/talos/file/tip/talos/test.py. This allows the test definitions to take advantage of inheritance and to stop duplicating repeating code and have made Talos far easier to configure, change, maintain, and expand. There is more we want to do here, this is merely a start. For more details see https://bugzilla.mozilla.org/show_bug.cgi?id=814228. jmaher - we should expand on the usage of this, the audience is developers
"""

Well, I do agree we need more documentation.  OTOH, I would rather figure out what we actually want (and maybe even implement it) before devoting time+energy to documenting something that is slated for obsolescence (for two reasons: the first is time+energy, the second is that if we do get people to read it and then we change it, IMHO we've done more harm than good). Is there something in particular we should add here? MHO is "no", but feel free to add stuff if we're thinking about something specific.

"""
Talos testing on try server with talos.json: Release engineering put forth considerable effort to allow talos changes to be tested with try server. The results of this effort include having the URL of a talos.zip file listed in https://mxr.mozilla.org/mozilla-central/source/testing/talos/talos.json . This change alone probably saved hundreds of man-hours. ctalbert - expand this so that we can detail a bit about how to actually go about making use of the functionality.
"""

IMHO, this documentation goes with talos, not SfN.  This document is basically an executive summary for SfN.  I haven't had a chance to document try for Talos....nor has anyone else, for that matter. I can prioritize this, if desired.  I have been waiting to finish a script that basically works to make this easier: http://k0s.org/mozilla/hg/TalosTry .

I'm fine doing whatever with these items.

The conclusion also needs work.
I thought we were mostly done with this a month ago.  If you are not happy with it, please rearrange and edit to make it more satisfactory.

In my opinion we should have all documentation written to an audience of a new contributor but as concise as possible.  While it might be overkill for many people reading the documentation, it will help new developers or mozillians get up to speed.  As I am one voice, we can find an audience/writing style which is agreeable amongst interested parties.  Linking to countless bugs reads more like a status report which doesn't help a new contributor get up to speed on talos and what we are doing with SfN.
(Reporter)

Comment 21

6 years ago
So I've done about all I think makes sense here from my point of view.  If anyone else has edits they want, please get them in promptly.

We still need to communicate this.  I advise a cross-post to dev-platform and dev-tree-management.  I also intend to blog this and would encourage others to do so as well.

In the spirit of the fiscal cliff, I'll intend to do this Friday if we can't figure out consensus or do it before.  I would also encourage others to do the email to dev-* vs myself as they may communicate better.
Assignee: jhammel → nobody
(Reporter)

Comment 22

5 years ago
Mailed to dev-platform and blogged: http://k0s.org/mozilla/blog/20130108154813

Calling this done
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.