1050235 - Ask web developers who file Core bugs different questions in the last step of filing a bug

Status: RESOLVED INCOMPLETE

(bugzilla.mozilla.org :: Extensions, defect)

Description

[:Gijs (he/him)]

Instead of the STR questions, Zack suggested:

> 0) Is this, in your opinion, an outright *bug*, or is it a Web feature
> we haven't gotten around to implementing yet, or is it a missing
> feature that hasn't even been specified yet but really should be?

I wonder if we should add keywords for this, and/or if we don't, if it is valuable to ask this question...

> 1) Can you summarize the problem in a single sentence?

I think this should be the summary of the bug, so I don't know if we need to keep this as a separate question...


> 2) Please construct a self-contained test case.  This is, ideally, a
> single HTML page that shows the problem you are having, with as little
> as possible that isn't relevant, with all CSS, Javascript, and images
> embedded in the page, and with all proprietary or confidential
> information stripped out (replace with lorem ipsum if necessary).
> Loading common third-party libraries like jQuery from public CDNs is okay.
>
> If the problem is with Firefox's implementation of HTTP, WebSockets,
> XHR, or like that, or it's just not possible to demonstrate the
> problem without talking to the network (for whatever reason), you can
> either set up a public *site* that shows the problem, or you can give
> us all the files we would need to do that for ourselves.  If you set
> up a site, it may need to stick around for several years, so please
> keep that in mind.  jsFiddle, codepen, gists, and so on are all
> reasonable.

Ideally I think we should make this a separate section that links into the URL and attachment fields. Perhaps this is also where we could let them tick a box that says "this is a request for a new feature, so there is no testcase" (per question 0)?

> 3) Please describe what is wrong: that is, what does Firefox do with
> the test case, that it shouldn't have?  Please be very specific.  If
> the problem should be obvious when the test case is loaded, that is
> best, but we still need to know what to look for.  If we need to
> interact with the test case after it's loaded to see the problem,
> please give step-by-step instructions for what to do and what we
> should notice at each stage.  Describe both what the test case
> actually does, and what it *should* be doing, if there were no bug.

We should definitely have something along these lines, but we already kind of do with the STR/expected results/actual results... :-)

> 
> If this is a missing feature, rather than a bug, the test case should
> demonstrate the best workaround you have at the moment, and you should
> explain why this is not good enough and/or too much trouble and/or
> interferes with other things you need to do.  It is also helpful to
> describe your use cases for the feature.

I'm not sure it's useful to make web devs put in work to do a testcase for a missing web feature... Zack, can you elaborate on why you added this? :-)

> 4a) Did this work in an older version of Firefox?  If so, do you know
> roughly when it broke?

Again, it would be awesome if this was a tick box that added the "regression, regressionwindow-wanted" keywords, with a text field to describe when it worked/broke.

> 4b) Have you tested this in other browsers (e.g. Chrome, IE, Safari,
> Opera)? Which versions of each, on what operating system?  Is their
> behavior better, worse, the same, or just as broken but in a different
> way?
> 
> 4c) Do you know what the official Web standards¹ say about this?  If
> you don't, that's fine, but if you do, it's helpful.

I think both of these questions are useful and we should add them. 



(I think this might need to be split up into different bits for implementation, but I've left it as a single bug for now)

Comment 1

[Zack Weinberg (:zwol)]

(In reply to :Gijs Kruitbosch from comment #0)
> Instead of the STR questions, Zack suggested:
> > 0) Is this, in your opinion, an outright *bug*, or is it a Web
> > feature we haven't gotten around to implementing yet, or is it a
> > missing feature that hasn't even been specified yet but really
> > should be?
>
> I wonder if we should add keywords for this, and/or if we don't, if
> it is valuable to ask this question...

You would know better than me whether keywords would be useful.
I found this question useful primarily in deciding how to start the
conversation, and *where* to start the conversation.  For instance, a
not-yet-specified CSS feature is maybe better brought up on www-style
than filed as a feature request specifically against Gecko.

> > 1) Can you summarize the problem in a single sentence?
>
> I think this should be the summary of the bug, so I don't know if we need to
> keep this as a separate question...

This is a recommended reword *of* the prompt for the bug-summary
field.  I see that ?format=guided#h=dupes|Core| already does use
essentially this wording, so maybe there's no action needed; however,
it might be appropriate to show this on the same page as some other
prompts, because a screen with "Please summarise your issue or request
in one sentence", a single text-entry field, and nothing else seems
like an invitation to writers' block.

> > 2) Please construct a self-contained test case. [...]
>
> Ideally I think we should make this a separate section that links
> into the URL and attachment fields. Perhaps this is also where we
> could let them tick a box that says "this is a request for a new
> feature, so there is no testcase" (per question 0)?

Yah.

> > 3) Please describe what is wrong: that is, what does Firefox do with
> > the test case, that it shouldn't have?  Please be very specific.  If
> > the problem should be obvious when the test case is loaded, that is
> > best, but we still need to know what to look for.  If we need to
> > interact with the test case after it's loaded to see the problem,
> > please give step-by-step instructions for what to do and what we
> > should notice at each stage.  Describe both what the test case
> > actually does, and what it *should* be doing, if there were no bug.
>
> We should definitely have something along these lines, but we already
> kind of do with the STR/expected results/actual results... :-)

My point here is that the STR/expected/actual pattern isn't freeform
enough!  It really only fits a bug in Firefox's UI.  By asking for a
test case first, and then asking "what does Firefox do with this test
case, that it shouldn't have? Be specific." we allow the reporter to
describe the problem in their own terms, while still being clear about
what we need.  I have been told, in so many words, that the
STR/expected/actual boxes make it look like we don't want to hear
about bugs that don't fit into that straightjacket.

(I don't think I can overstate how scared people can be of reporting
bugs.  This is potentially their very first interaction with Mozilla
as a community, and they may treat the *slightest* speedbump as a big
neon sign saying GO AWAY, WE DON'T CARE and/or SCREW THIS UP EVEN
SLIGHTLY AND WE WILL HOLD YOU IN CONTEMPT FOREVER.  Especially if (as
is depressingly common) they've had bad experiences trying to report
bugs in other open source software.)

> > If this is a missing feature, rather than a bug, the test case
> > should demonstrate the best workaround you have at the moment, and
> > you should explain why this is not good enough and/or too much
> > trouble and/or interferes with other things you need to do.  It is
> > also helpful to describe your use cases for the feature.
>
> I'm not sure it's useful to make web devs put in work to do a
> testcase for a missing web feature... Zack, can you elaborate on why
> you added this? :-)

By the time they get around to filing a bug report on a missing
web-platform feature, they probably already *have* a mockup of some
sort demonstrating the best they can do with currently available tech.
We could adjust the wording in this case, to indicate that this is
nice to have but not required.  Maybe put more emphasis on use cases
(and perhaps ask for mockups of *those*) instead.

> > 4a) Did this work in an older version of Firefox?  If so, do you know
> > roughly when it broke?
>
> Again, it would be awesome if this was a tick box that added the
> "regression, regressionwindow-wanted" keywords, with a text field to
> describe when it worked/broke.

Yah, but be careful not to make it sound like we are asking them to do
the entire regression-hunt process themselves.

Comment 2

[Dylan Hardison [:dylan] (he/him)]

This one dropped from radar, it's pretty trivial. Adding the patch now.

Comment 3

[Dylan Hardison [:dylan] (he/him)]

Extending the due date on this bug out as I'm in the midst of fixing a higher-priority bug.

Comment 4

[Dylan Hardison [:dylan] (he/him)]

I believe it's better to overestimate than repeatedly fail to meet deadlines, so let's push this way the heck out while I sort the styling issues in bug 1050232.

Comment 5

[:glob ✱]

using the text exactly as presented in the comments here will result in a wall of text, which is something that we need to avoid.

i've thrown together a layout of the page which i believe encompasses the requirements without an overwhelming amount of text.  we can additionally put more verbose information behind the "?" help icon.

https://etherpad.mozilla.org/bug-1050235-edited


thoughts?

Comment 6

[:glob ✱]

gijs - any thoughts on proposal in https://etherpad.mozilla.org/bug-1050235-edited ?

Comment 7

[:Gijs (he/him)]

(In reply to Byron Jones ‹:glob› (limited availability until 19th jan) from comment #6)
> gijs - any thoughts on proposal in
> https://etherpad.mozilla.org/bug-1050235-edited ?

> ( o ) Bug: An outright bug in Firefox

I'd suggest "in Firefox's implementation of a Web feature" (somewhat in parallel to the next option, and to be clear that this means it's not related to e.g. bookmarks/downloads)


> Attach or link to a self-contained test case:

Perhaps we can suggest sites like jsbin, jsfiddle or codepen here?

> Do you know roughly when it broke (Firefox version or date)?

I wonder if it makes sense to plug mozregression here. Maybe not yet, because it might seem like we're asking for everything and their social security number otherwise? :-)



> With as much detail as possible describe the feature.  A good report comprises of:
>   an detailed explanation of the feature
>   an explanation of why the workaround is not good enough
>   descriptions of use-cases for the feature

I'm not sure about this. In principle I don't think our bugzilla instance is the right place for requesting features without specs. Boris, can you comment as to whether you agree and/or what would be a good place people should go instead?

If the feature /is/ specced and we've not implemented it (yet), it would be helpful if they list which browser(s) have and/or are planning to. It might still be better if they asked in e.g. m.d.platform (where we announce/discuss enabling newly implemented features, too, AFAICT)

Comment 8

[Boris Zbarsky [:bzbarsky]]

> Perhaps we can suggest sites like jsbin, jsfiddle or codepen here?

There's a tradeoff here.  On the one hand, they make it easy to create a testcase.  On the other, they make it really hard to create a _self-contained_ testcase because of all the scaffolding they load.  We've had multiple cases of someone reporting a bug that could only be reproduced in jsfiddle or whatnot, because it was due to something weird the framework itself was doing.  :(

So if someone has an existing self-contained testcase, it's much better to attach that than to link to jsfiddle and company.

> Boris, can you comment as to whether you agree and/or what would be a good place people
> should go instead?

I don't see a problem with filing a bug for this sort of thing, though you're right that the spec work won't happen in the bug.  Where _that_ should go depends very strongly on the feature involved and the politics involved; I think we should leave it up to the module owner or other domain expert who ends up looking at the bug to suggest a better venue as needed.

Comment 9

[:glob ✱]

Attachment: specifications

copy of the updated specs from the etherpad.  let's start with this and iteration on the design if required on bugzilla-dev.