1563611 - Dealing with regression bugs in Bugzilla is way too difficult and error prone

Status: NEW

(bugzilla.mozilla.org :: General, defect)

Description

[(no longer active)]

For an example please look at the change I just made right below bug 1562837 comment 4. Here are some issues:

• Comment 3 on the bug added a note that bug 1535697 is the regressor, but the commentor forgot to set the "regressed by" field. I had to come and clean up after them.
• The firefox68/69 fields had their status set to affected but their blocking field was set to "---". I had to clean it up after the fact and set them to ?, otherwise release managers would miss the bug during their triage.
• Comment 0 mentions a regression range very clearly, still the "Has Regression Range" field was set to "---". I had to clean it up after the fact.
• Comment 0 very clearly has a STR. Still the "Has STR" field was set to "---". I had to clean it up after the fact.

The common theme here is: Bugzilla could be smarter about detecting regression bugs and offer assistance to users, for example: "Did you mean to set Has STR to Yes since you filed the bug with steps to reproduce?", "Did you mean to mark the bug as having a regression range since you included a regression range when filing the bug?" and so on.

The lack of this automation would cause us to rely on fallible humans to catch these mistakes, and when that fails, we'll ship regressions. :-( Sadly as bugzilla has become more and more complex around handling regressions (e.g. you now have to set 7 or 8 fields when dealing with a beta regression) we're on the verge of having the usability of this system impose a cost on the quality of Firefox.

Comment 1

[(no longer active)]

For another similar cleanup I did earlier today please see bug 1561036 comment 1.

Comment 2

[Kohei Yoshino [:kohei]]

This is what I mentioned in my bug field analysis, and the solution is creating the new “Is Regression” field with 5 values and implementing some automation.

Comment 3

[Marco Castelluccio [:marco]]

(In reply to :Ehsan Akhgari from comment #0)

For an example please look at the change I just made right below bug 1562837 comment 4. Here are some issues:

• Comment 3 on the bug added a note that bug 1535697 is the regressor, but the commentor forgot to set the "regressed by" field. I had to come and clean up after them.

For now, our bot is suggesting setting the "regressed_by" field in some specific cases (if a bug has "regression" keyword, and there is only one bug in depends_on/blocks after filtering out meta bugs and bugs that aren't closed yet). We could possibly extend it to detect regressors from comments too instead of just depends_on/blocks. It wouldn't be perfect since there are a lot of different ways in which people can say "this was caused by bug XXX", but at least we should be able to catch the most common cases.

• The firefox68/69 fields had their status set to affected but their blocking field was set to "---". I had to clean it up after the fact and set them to ?, otherwise release managers would miss the bug during their triage.

If I'm not mistaken, they'd see it anyway if the 'regression' field is set correctly.

• Comment 0 mentions a regression range very clearly, still the "Has Regression Range" field was set to "---". I had to clean it up after the fact.
• Comment 0 very clearly has a STR. Still the "Has STR" field was set to "---". I had to clean it up after the fact.

We are working on doing these with our bot (no ETA though). Possibly, when we detect STRs/regression ranges are there in the bug, we'd add a comment asking to set the fields. When we detect STRs/regression ranges are not there, we'd add a comment asking if they are needed.

We are also working on additional checks to prevent inconsistent data in the fields. E.g. if "Has Regression Range" is "yes" and "regressed_by" is empty, we will add a comment asking if it could be set.

Comment 4

[(no longer active)]

(In reply to Marco Castelluccio [:marco] from comment #3)

(In reply to :Ehsan Akhgari from comment #0)

For an example please look at the change I just made right below bug 1562837 comment 4. Here are some issues:

• Comment 3 on the bug added a note that bug 1535697 is the regressor, but the commentor forgot to set the "regressed by" field. I had to come and clean up after them.

For now, our bot is suggesting setting the "regressed_by" field in some specific cases (if a bug has "regression" keyword, and there is only one bug in depends_on/blocks after filtering out meta bugs and bugs that aren't closed yet).

I have noticed, thanks! :-) I used to set that obsessively too, but now that seems to no longer be an issue in most cases (unless the regression keyword is missing, in which case usually nobody has noticed the bug is a regression yet.)

We could possibly extend it to detect regressors from comments too instead of just depends_on/blocks. It wouldn't be perfect since there are a lot of different ways in which people can say "this was caused by bug XXX", but at least we should be able to catch the most common cases.

Of course, understood. Just to be clear, I was asking for a best effort thing here, not a perfect solution, since I don't think that's an attainable goal!

• The firefox68/69 fields had their status set to affected but their blocking field was set to "---". I had to clean it up after the fact and set them to ?, otherwise release managers would miss the bug during their triage.

If I'm not mistaken, they'd see it anyway if the 'regression' field is set correctly.

My experience has been that release managers don't look at bugs that don't have the blocking field set to something other than "---". Basically, "---" means "Don't Care" for release managers. (I could be wrong here, of course, but the "---" set is the vast majority of them...)

• Comment 0 mentions a regression range very clearly, still the "Has Regression Range" field was set to "---". I had to clean it up after the fact.
• Comment 0 very clearly has a STR. Still the "Has STR" field was set to "---". I had to clean it up after the fact.

We are working on doing these with our bot (no ETA though). Possibly, when we detect STRs/regression ranges are there in the bug, we'd add a comment asking to set the fields. When we detect STRs/regression ranges are not there, we'd add a comment asking if they are needed.

Sounds great! Thanks.

We are also working on additional checks to prevent inconsistent data in the fields. E.g. if "Has Regression Range" is "yes" and "regressed_by" is empty, we will add a comment asking if it could be set.

Great, thanks!

BTW, have you considered just setting the fields instead of adding a comment? That may give people more of an incentive to react if the bot is doing something wrong, and is more automated in the majority of the cases. Is the accuracy too low to be able to do that?

Comment 5

[Marco Castelluccio [:marco]]

(In reply to :Ehsan Akhgari from comment #4)

We are also working on additional checks to prevent inconsistent data in the fields. E.g. if "Has Regression Range" is "yes" and "regressed_by" is empty, we will add a comment asking if it could be set.

Great, thanks!

BTW, have you considered just setting the fields instead of adding a comment? That may give people more of an incentive to react if the bot is doing something wrong, and is more automated in the majority of the cases. Is the accuracy too low to be able to do that?

We are doing that in many cases (the bot performs the action and adds a comment saying "Please revert if I was wrong").
For STR and Has Regression Range we don't have a lot of labelled data (we have a lot of examples with but few examples without) so the accuracy is probably lower. We can set a confidence threshold so that we perform the action when we're sure enough and we only add the comment when we are not sure enough.
For preventing inconsistent data, the accuracy should be very high (they are mostly simple rules), so I'd say we can just perform the action in those cases.

Comment 6

[Marco Castelluccio [:marco]]

(In reply to :Ehsan Akhgari from comment #4)

• The firefox68/69 fields had their status set to affected but their blocking field was set to "---". I had to clean it up after the fact and set them to ?, otherwise release managers would miss the bug during their triage.

If I'm not mistaken, they'd see it anyway if the 'regression' field is set correctly.

My experience has been that release managers don't look at bugs that don't have the blocking field set to something other than "---". Basically, "---" means "Don't Care" for release managers. (I could be wrong here, of course, but the "---" set is the vast majority of them...)

I asked Sylvestre, there is a periodic regression triage which release managers attend too, so they should see regression bugs with the field set to "---" too.

Comment 7

[(no longer active)]

(In reply to Marco Castelluccio [:marco] from comment #6)

(In reply to :Ehsan Akhgari from comment #4)

• The firefox68/69 fields had their status set to affected but their blocking field was set to "---". I had to clean it up after the fact and set them to ?, otherwise release managers would miss the bug during their triage.

If I'm not mistaken, they'd see it anyway if the 'regression' field is set correctly.

My experience has been that release managers don't look at bugs that don't have the blocking field set to something other than "---". Basically, "---" means "Don't Care" for release managers. (I could be wrong here, of course, but the "---" set is the vast majority of them...)

I asked Sylvestre, there is a periodic regression triage which release managers attend too, so they should see regression bugs with the field set to "---" too.

Thank you, glad I was wrong there!