343653 - Please update Mozilla Triage Guide

Status: RESOLVED WORKSFORME

(Developer Documentation Graveyard :: Mozilla Platform, enhancement)

Description

[Aaron Leventhal]

There is a new triage guide which was quickly thrown together, but if given some better content, could be useful for new QA folks getting involved in the project:

http://wiki.mozilla.org/MozillaQualityAssurance:Triage

We need a page like this -- I need something that I can show IBM QA developers, but it's generally useful. It tells how to deal with community bugs, how you can really help developers (finding minimal testcases & regression windows), and how to make sure important bugs and patches aren't ignored.

I also tried to describe what kinds of problems crop up in bugs, how to recognize them, and how to solve them.

Comment 1

[Aaron Leventhal]

Martijn had a lot of feedback and said he would own the document -- was planning on writing something like it anyway.

Comment 2

[Martijn Wargers (dead)]

(In reply to comment #1)
> Martijn had a lot of feedback and said he would own the document -- was
> planning on writing something like it anyway.

I meant this more in the sense of "planning on doing it, but never actually doing it kind of thing" ;)

I'll update that wiki page with the feedback I have.
Also, maybe this is a nice topic for one of the qa blogs, mentioning this page?

Comment 3

[Mats Palmgren (inactive)]

Nice triage guide Aaron!

I have a few suggestions below.

In "How to Clean Incoming Bugs" 

in 3, maybe we should add a few typical requests to the Reporter:
Does the bug occur in Safe Mode?  http://kb.mozillazine.org/Safe_mode
with a clean profile? http://kb.mozillazine.org/Profile_Manager
If it's a crash - ask for Talkback data: http://kb.mozillazine.org/Talkback
If the reporter still can reproduce but you can't - ask him nicely if he
can test a nightly build:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/
http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/latest-trunk/

in 4, remove WONTFIX, add WORKSFORME. As I understand it, INVALID is for bug
reports on things that are actually the correct behaviour or the bug report
lacks information to be useful or the problem is not with our code at all.
WORKSFORME is for bugs that aren't reproducable.
We should probably spell out a rule of thumb regarding when to mark a bug
WORKSFORME - should we require 1 WFM on the same platform as the reporter +
2 other on independent platforms?
(WONTFIX means "The problem described is a bug which will never be fixed."
according to https://bugzilla.mozilla.org/page.cgi?id=fields.html#resolution
I think it should rarely be used by bug triagers that are not developers.)


In "Minimal Testcases". I assume you mean "save as complete"?
You should probably say so explicitly. Personally, I save HTML-only and add
a <base href="original-URL"> at the top instead, that preserves the original
markup better. I usually start by minimizing the HTML first and only then
start to reduce the CSS and script stuff (inlining it as I go). At the end,
remove the <base> tag.

A couple of things you should consider adding to your steps:

6. For any remaining images, try changing the SRC attribute to an URI
   that isn't likely to expire, such as http://www.mozilla.org/images/hack.gif
   or http://www.google.com/intl/en/images/logo.gif
7. If it doesn't affect the test - add a title so it can be referred to, e.g.
   <title>Testcase #1 for bug 123456</title>

Also, consider moving the last paragraph "Keep in mind..." directly after
the list of steps.

In "Finding Regression Windows"

Point 3 - we could recommend that they use a separate dedicated profile when
they do this and start with -profilemanager
(which I think makes the "Firefox already running" and MOZ_NO_REMOTE
comments unnecessary?)

Comment 4

[Aaron Leventhal]

That's all good stuff, and Martijn had other great comments which I hope to get lost to newsgroup history. Will someone from the Mozilla Corporation take on ownership of this?

I'd like to see a section added on something like "Defining Bug Boundaries", where you test similar conditions to see if the bug can be made more general or described more exactly.

Comment 5

[Martijn Wargers (dead)]

Sorry, I've now updated my the triage guide with my comments.
I've removed WONTFIX, since only peers are allowed to set a bug to WONTFIX (which I happened to learn by mistake).

I think I did most of what Mats wanted in comment 3, except:
> In "Minimal Testcases". I assume you mean "save as complete"?
> You should probably say so explicitly. Personally, I save HTML-only and add
> a <base href="original-URL"> at the top instead, that preserves the original
> markup better. I usually start by minimizing the HTML first and only then
> start to reduce the CSS and script stuff (inlining it as I go). At the end,
> remove the <base> tag.

I usually use "save as complete". If I can't reproduce, I turn scripting off and do "save as complete" again. If that doesn't work, then I do "Save link as" and try to collect the relevant scripts/css/images/etc.

> Point 3 - we could recommend that they use a separate dedicated profile when
> they do this and start with -profilemanager
> (which I think makes the "Firefox already running" and MOZ_NO_REMOTE
> comments unnecessary?)

I've made that change, but to be honest, I almost never use a seperate profile, most of the time, I don't find it necessary.

Aaron, I'm not really sure what text you would like to have regarding the "Defining Bug Boundaries" section you would like to see. (I remember you mail about 'testing related cases for more data points'. I hope you could add that text yourself to the wiki.

In general, I hope that people with good suggestions will add/change the text themselves in the wiki page (if I do it, there is a good change, I do it in an unsatisfactory way for the person with the suggestion).

There is something terribly wrong with the wiki page, by the way:
http://wiki.mozilla.org/MozillaQualityAssurance:Triage
I can't edit that part, instead, I get the text above (which btw, should also be in that section). Not sure how to solve that problem.

Comment 6

[Martijn Wargers (dead)]

(In reply to comment #5)
> There is something terribly wrong with the wiki page, by the way:
> http://wiki.mozilla.org/MozillaQualityAssurance:Triage
> I can't edit that part, instead, I get the text above (which btw, should also
> be in that section). Not sure how to solve that problem.

Fixed now. The solution was actually very simple.

Comment 7

[Aaron Leventhal]

Yeah, by defining bug boundaries I meant the the same thing as testing related cases for more data points.

Wouldn't we get better results to have more of a QA pro write that than myself? The point is just to tell testers how they can find what the bug actually is by investigating a bit more. Does it break if I do this? etc.

Comment 8

[Mats Palmgren (inactive)]

CC:ing a few more people for possible input.

Comment 9

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Thanks for getting this started, Aaron and Martijn; this was on our list of things to do for Camino at some point, but it's so much nicer to point to a complete common guide and then only deal with the few Cm-specific exceptions ourselves ;)  

I've added a new section at the end, "See Also", linking to the disparate existing documents on this that I'm aware of and which probably should be replaced/refactored when this is all done (and cc'ing Gerv as he's the author of a number of them).

Gerv has some nice language in some of his documents that we should mimic when prefacing the "ignored bugs" section so that people don't end up being counterproductive when try things to get those fixed....

Comment 10

[Adam Guthrie]

I think it looks pretty good overall. Thanks for starting this Aaron!

A couple of things... 

I think we need to somehow break this up into a more-readable format. The first time I looked at this page I just closed the tab because it just looked like a big blob of text. We need to split it up into more-manageable subsections somehow.

How to choose the right component for a bug took me a while to learn. Maybe we could have a subsection for that that gave examples of bugs for specific components? e.g. Core: DOM or Core: XP Widgets or Core: Layout.

I think we ought to add more info. on how to triage crash bugs. I have a certain methodology that I've developed for doing this; I'd be glad to write it up.

Giving a concrete example in the "Finding a regression" section would be good, too, I think.

I think we need to clarify in "How to deal with ignored bugs" that there's a difference between wanting a certain bug to get fixed because YOU want it to get fixed vs. a bug being ignored when it's really something that needs to get fixed. I find that many topcrasher bugs (that haven't already been flagged) are a good example of a valid "ignored" bug.

Comment 11

[Gervase Markham [:gerv]]

This looks great; I have no problem with it obsoleting my earlier guide. Feel free to set up a redirect when it's finished. It could do with some editing to make it a bit more concise, particularly the intro.

> Will someone from the Mozilla Corporation take on
> ownership of this?

I hope the current authors and maintainers will continue to keep ownership!

Gerv

Comment 12

[Martijn Wargers (dead)]

(In reply to comment #11)
> make it a bit more concise, particularly the intro.

Yeah, I agree with that, so I just removed most of the text of the intro (and put one sentence at the beginning of the next chapter.

Comment 13

[Mats Palmgren (inactive)]

In "How to Clean Incoming Bugs" bullet 4;  I've updated the text about
INVALID saying that bugs lacking useful info should be resolved as
INCOMPLETE rather than INVALID.

Maybe we should add a description of how and when to use [CLOSEME <date>]
in the Whiteboard somewhere too? (bullet 3/4?)

Comment 14

[John Karahalis [:openjck]]

Automatically closing all bugs that have not been updated in a while. Please reopen if this is still important to you and has not yet been corrected.

Comment 15

[John Karahalis [:openjck]]

Reopening for review by Sheppy.

Comment 16

[Florian Scholz (Open Web Docs)]

If I see it correctly the current triage portal is this one and most of the docs discussed here from 2007 are obsolete. Closing.

https://wiki.mozilla.org/Bugmasters