1522348 - Bulk assign open bugs to task, enhancement, defect field

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Bulk Bug Edit Requests, task, P2)

Comment 1

[Kohei Yoshino [:kohei]]

Attachment: GitHub Pull Request

This is a script to do the migration task 1, 2, 3 in Bug 1524738 Comment 6.

Comment 2

[Jaroslavaskar]

Comment on attachment 9043712 [details] [review]
GitHub Pull Request

>diff --git a/scripts/migrate-bug-type.pl b/scripts/migrate-bug-type.pl
>new file mode 100644
>index 0000000000..f3a4b7235a
>--- /dev/null
>+++ b/scripts/migrate-bug-type.pl
>@@ -0,0 +1,145 @@
>+#!/usr/bin/env perl
>+
>+# This Source Code Form is subject to the terms of the Mozilla Public
>+# License, v. 2.0. If a copy of the MPL was not distributed with this
>+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
>+#
>+# This Source Code Form is "Incompatible With Secondary Licenses", as
>+# defined by the Mozilla Public License, v. 2.0.
>+
>+use strict;
>+use warnings;
>+use lib qw(. lib local/lib/perl5);
>+
>+use Bugzilla;
>+
>+# List of products and components that use a bug type other than "defect"
>+use constant MIGRATION_MAP => (
>+  ['Air Mozilla', 'Events', 'task'],
>+  ['AUS Graveyard', 'Administration', 'task'],
>+  ['Bugzilla', 'Administration', 'task'],
>+  ['bugzilla.mozilla.org', 'Administration', 'task'],
>+  ['bugzilla.mozilla.org', 'Bulk Bug Edit Requests', 'task'],
>+  ['bugzilla.mozilla.org', 'Graveyard Tasks', 'task'],
>+  ['Cloud Services', 'Operations', 'task'],
>+  ['Cloud Services', 'Operations: Activedata', 'task'],
>+  ['Cloud Services', 'Operations: AMO', 'task'],
>+  ['Cloud Services', 'Operations: Antenna', 'task'],
>+  ['Cloud Services', 'Operations: Autopush', 'task'],
>+  ['Cloud Services', 'Operations: AWS Account Request', 'task'],
>+  ['Cloud Services', 'Operations: Bzetl', 'task'],
>+  ['Cloud Services', 'Operations: Delivery Console', 'task'],
>+  ['Cloud Services', 'Operations: Deployment Requests', 'task'],
>+  ['Cloud Services', 'Operations: LandoAPI', 'task'],
>+  ['Cloud Services', 'Operations: LandoUI', 'task'],
>+  ['Cloud Services', 'Operations: Marketplace', 'task'],
>+  ['Cloud Services', 'Operations: Metrics/Monitoring', 'task'],
>+  ['Cloud Services', 'Operations: Normandy', 'task'],
>+  ['Cloud Services', 'Operations: Pageshot', 'task'],
>+  ['Cloud Services', 'Operations: Phabricator', 'task'],
>+  ['Cloud Services', 'Operations: Product Delivery', 'task'],
>+  ['Cloud Services', 'Operations: Sentry', 'task'],
>+  ['Cloud Services', 'Operations: Shavar', 'task'],
>+  ['Cloud Services', 'Operations: Storage', 'task'],
>+  ['Community Building', '', 'task'],
>+  ['Conduit', 'Administration', 'task'],
>+  ['Data & BI Services Team', '', 'task'],
>+  ['Data & BI Services Team Graveyard', '', 'task'],
>+  ['Data Compliance', '', 'task'],
>+  ['Data Science', '', 'task'],
>+  ['Developer Engagement', '', 'task'],
>+  ['Developer Services', 'General', 'task'],
>+  ['developer.mozilla.org', 'Account Help', 'task'],
>+  ['developer.mozilla.org', 'Administration', 'task'],
>+  ['developer.mozilla.org', 'Events', 'task'],
>+  ['developer.mozilla.org', 'Marketing', 'task'],
>+  ['developer.mozilla.org', 'User management', 'task'],
>+  ['Enterprise Information Security', '', 'task'],
>+  ['Enterprise Information Security Graveyard', '', 'task'],
>+  ['Finance', '', 'task'],
>+  ['Firefox Build System', 'Task Configuration', 'task'],
>+  ['FSA Graveyard', '', 'task'],
>+  ['Infrastructure & Operations', '', 'task'],
>+  ['Infrastructure & Operations Graveyard', '', 'task'],
>+  ['Internet Public Policy', '', 'task'],
>+  ['Legal Graveyard', '', 'task'],
>+  ['Localization Infrastructure and Tools', 'Administration / Setup', 'task'],
>+  ['Marketing', '', 'task'],
>+  ['Mozilla Foundation', '', 'task'],
>+  ['Mozilla Foundation Communications', '', 'task'],
>+  ['Mozilla Foundation Operations', '', 'task'],
>+  ['Mozilla Grants', '', 'task'],
>+  ['Mozilla Metrics', 'Metrics Operations', 'task'],
>+  ['Mozilla Reps', '', 'task'],
>+  ['Mozilla Reps Graveyard', 'Community IT Requests', 'task'],
>+  ['Mozilla Reps Graveyard', 'Planning', 'task'],
>+  ['mozilla.org', '', 'task'],
>+  ['mozilla.org Graveyard', '', 'task'],
>+  ['Participation Infrastructure', 'Account Help', 'task'],
>+  ['Participation Infrastructure', 'API Requests', 'task'],
>+  ['Participation Infrastructure', 'Community Ops', 'task'],
>+  ['Participation Infrastructure', 'Data Complaints', 'task'],
>+  ['Privacy Graveyard', '', 'task'],
>+  ['Recruiting', '', 'task'],
>+  ['Snippets', 'Campaign', 'task'],
>+  ['Snippets', 'Surveys', 'task'],
>+  ['Socorro', '', 'task'],
>+  ['support.mozilla.org', 'Army of Awesome', 'task'],
>+  ['support.mozilla.org', 'Code Quality', 'task'],
>+  ['support.mozilla.org', 'Forum', 'task'],
>+  ['support.mozilla.org', 'Knowledge Base Articles', 'task'],
>+  ['support.mozilla.org', 'Knowledge Base Content', 'task'],
>+  ['support.mozilla.org', 'Knowledge Base Software', 'task'],
>+  ['support.mozilla.org', 'Lithium Migration', 'task'],
>+  ['support.mozilla.org', 'Localization', 'task'],
>+  ['support.mozilla.org', 'Mobile', 'task'],
>+  ['support.mozilla.org', 'Questions', 'task'],
>+  ['support.mozilla.org', 'Users and Groups', 'task'],
>+  ['Taskcluster', 'Operations and Service Requests', 'task'],
>+  ['Websites', 'Web Analytics', 'task'],
>+  ['Firefox Build System', 'Mach Core', 'enhancement'],
>+  ['support.mozilla.org - Lithium', 'Feature request', 'enhancement'],
>+);
>+
>+my $dbh = Bugzilla->dbh;
>+
>+foreach my $target (@MIGRATION_MAP) {
>+  my ($product, $component, $type) = @$target;
>+
>+  $dbh->bz_start_transaction;
>+
>+  # Select bugs in the product (and component)
>+  my $bug_ids = $dbh->selectcol_arrayref(
>+    'SELECT bug_id FROM bugs AS bug
>+      JOIN products AS product ON bug.product_id = product.id
>+      JOIN components AS component ON bug.component_id = component.id
>+      WHERE product.name = ?' . ($component ? ' AND component.name = ?' : ''),
>+    undef, ($component ? ($product, $component) : ($product)));
>+
>+  # Set type on these bugs
>+  # Since it's a silent migration, we don't update the timestamp
>+  $dbh->do('UPDATE bugs SET type = ?
>+    WHERE ' . $dbh->sql_in('bug_id', $bug_ids), undef, ($type));
>+
>+  # Select components
>+  my $comp_ids = $dbh->selectcol_arrayref(
>+    'SELECT component.id FROM components as component
>+      JOIN products AS product ON component.id = product.id
>+      WHERE product.name = ?' . ($component ? ' AND component.name = ?' : ''),
>+    undef, ($component ? ($product, $component) : ($product)));
>+
>+  # Set default bug type on these components
>+  $dbh->do('UPDATE components SET default_bug_type = ?
>+    WHERE ' . $dbh->sql_in('id', $comp_ids), undef, ($type));
>+
>+  $dbh->bz_commit_transaction;
>+}
>+
>+$dbh->bz_start_transaction;
>+
>+# Finally, disable the "enhancement" severity
>+$dbh->do('UPDATE bug_severity SET isactive = 0 WHERE value = "enhancement"');
>+
>+$dbh->bz_commit_transaction;
>+
>+Bugzilla->memcached->clear_all();

Comment 3

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

(In reply to Jaroslavaskar from comment #2)

Comment on attachment 9043712 [details] [review]
GitHub Pull Request

Was there a question about this script?

Comment 4

[Jaroslavaskar]

Comment on attachment 9043712 [details] [review]
GitHub Pull Request

>diff --git a/scripts/migrate-bug-type.pl b/scripts/migrate-bug-type.pl
>new file mode 100644
>index 0000000000..f3a4b7235a
>--- /dev/null
>+++ b/scripts/migrate-bug-type.pl
>@@ -0,0 +1,145 @@
>+#!/usr/bin/env perl
>+
>+# This Source Code Form is subject to the terms of the Mozilla Public
>+# License, v. 2.0. If a copy of the MPL was not distributed with this
>+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
>+#
>+# This Source Code Form is "Incompatible With Secondary Licenses", as
>+# defined by the Mozilla Public License, v. 2.0.
>+
>+use strict;
>+use warnings;
>+use lib qw(. lib local/lib/perl5);
>+
>+use Bugzilla;
>+
>+# List of products and components that use a bug type other than "defect"
>+use constant MIGRATION_MAP => (
>+  ['Air Mozilla', 'Events', 'task'],
>+  ['AUS Graveyard', 'Administration', 'task'],
>+  ['Bugzilla', 'Administration', 'task'],
>+  ['bugzilla.mozilla.org', 'Administration', 'task'],
>+  ['bugzilla.mozilla.org', 'Bulk Bug Edit Requests', 'task'],
>+  ['bugzilla.mozilla.org', 'Graveyard Tasks', 'task'],
>+  ['Cloud Services', 'Operations', 'task'],
>+  ['Cloud Services', 'Operations: Activedata', 'task'],
>+  ['Cloud Services', 'Operations: AMO', 'task'],
>+  ['Cloud Services', 'Operations: Antenna', 'task'],
>+  ['Cloud Services', 'Operations: Autopush', 'task'],
>+  ['Cloud Services', 'Operations: AWS Account Request', 'task'],
>+  ['Cloud Services', 'Operations: Bzetl', 'task'],
>+  ['Cloud Services', 'Operations: Delivery Console', 'task'],
>+  ['Cloud Services', 'Operations: Deployment Requests', 'task'],
>+  ['Cloud Services', 'Operations: LandoAPI', 'task'],
>+  ['Cloud Services', 'Operations: LandoUI', 'task'],
>+  ['Cloud Services', 'Operations: Marketplace', 'task'],
>+  ['Cloud Services', 'Operations: Metrics/Monitoring', 'task'],
>+  ['Cloud Services', 'Operations: Normandy', 'task'],
>+  ['Cloud Services', 'Operations: Pageshot', 'task'],
>+  ['Cloud Services', 'Operations: Phabricator', 'task'],
>+  ['Cloud Services', 'Operations: Product Delivery', 'task'],
>+  ['Cloud Services', 'Operations: Sentry', 'task'],
>+  ['Cloud Services', 'Operations: Shavar', 'task'],
>+  ['Cloud Services', 'Operations: Storage', 'task'],
>+  ['Community Building', '', 'task'],
>+  ['Conduit', 'Administration', 'task'],
>+  ['Data & BI Services Team', '', 'task'],
>+  ['Data & BI Services Team Graveyard', '', 'task'],
>+  ['Data Compliance', '', 'task'],
>+  ['Data Science', '', 'task'],
>+  ['Developer Engagement', '', 'task'],
>+  ['Developer Services', 'General', 'task'],
>+  ['developer.mozilla.org', 'Account Help', 'task'],
>+  ['developer.mozilla.org', 'Administration', 'task'],
>+  ['developer.mozilla.org', 'Events', 'task'],
>+  ['developer.mozilla.org', 'Marketing', 'task'],
>+  ['developer.mozilla.org', 'User management', 'task'],
>+  ['Enterprise Information Security', '', 'task'],
>+  ['Enterprise Information Security Graveyard', '', 'task'],
>+  ['Finance', '', 'task'],
>+  ['Firefox Build System', 'Task Configuration', 'task'],
>+  ['FSA Graveyard', '', 'task'],
>+  ['Infrastructure & Operations', '', 'task'],
>+  ['Infrastructure & Operations Graveyard', '', 'task'],
>+  ['Internet Public Policy', '', 'task'],
>+  ['Legal Graveyard', '', 'task'],
>+  ['Localization Infrastructure and Tools', 'Administration / Setup', 'task'],
>+  ['Marketing', '', 'task'],
>+  ['Mozilla Foundation', '', 'task'],
>+  ['Mozilla Foundation Communications', '', 'task'],
>+  ['Mozilla Foundation Operations', '', 'task'],
>+  ['Mozilla Grants', '', 'task'],
>+  ['Mozilla Metrics', 'Metrics Operations', 'task'],
>+  ['Mozilla Reps', '', 'task'],
>+  ['Mozilla Reps Graveyard', 'Community IT Requests', 'task'],
>+  ['Mozilla Reps Graveyard', 'Planning', 'task'],
>+  ['mozilla.org', '', 'task'],
>+  ['mozilla.org Graveyard', '', 'task'],
>+  ['Participation Infrastructure', 'Account Help', 'task'],
>+  ['Participation Infrastructure', 'API Requests', 'task'],
>+  ['Participation Infrastructure', 'Community Ops', 'task'],
>+  ['Participation Infrastructure', 'Data Complaints', 'task'],
>+  ['Privacy Graveyard', '', 'task'],
>+  ['Recruiting', '', 'task'],
>+  ['Snippets', 'Campaign', 'task'],
>+  ['Snippets', 'Surveys', 'task'],
>+  ['Socorro', '', 'task'],
>+  ['support.mozilla.org', 'Army of Awesome', 'task'],
>+  ['support.mozilla.org', 'Code Quality', 'task'],
>+  ['support.mozilla.org', 'Forum', 'task'],
>+  ['support.mozilla.org', 'Knowledge Base Articles', 'task'],
>+  ['support.mozilla.org', 'Knowledge Base Content', 'task'],
>+  ['support.mozilla.org', 'Knowledge Base Software', 'task'],
>+  ['support.mozilla.org', 'Lithium Migration', 'task'],
>+  ['support.mozilla.org', 'Localization', 'task'],
>+  ['support.mozilla.org', 'Mobile', 'task'],
>+  ['support.mozilla.org', 'Questions', 'task'],
>+  ['support.mozilla.org', 'Users and Groups', 'task'],
>+  ['Taskcluster', 'Operations and Service Requests', 'task'],
>+  ['Websites', 'Web Analytics', 'task'],
>+  ['Firefox Build System', 'Mach Core', 'enhancement'],
>+  ['support.mozilla.org - Lithium', 'Feature request', 'enhancement'],
>+);
>+
>+my $dbh = Bugzilla->dbh;
>+
>+foreach my $target (@MIGRATION_MAP) {
>+  my ($product, $component, $type) = @$target;
>+
>+  $dbh->bz_start_transaction;
>+
>+  # Select bugs in the product (and component)
>+  my $bug_ids = $dbh->selectcol_arrayref(
>+    'SELECT bug_id FROM bugs AS bug
>+      JOIN products AS product ON bug.product_id = product.id
>+      JOIN components AS component ON bug.component_id = component.id
>+      WHERE product.name = ?' . ($component ? ' AND component.name = ?' : ''),
>+    undef, ($component ? ($product, $component) : ($product)));
>+
>+  # Set type on these bugs
>+  # Since it's a silent migration, we don't update the timestamp
>+  $dbh->do('UPDATE bugs SET type = ?
>+    WHERE ' . $dbh->sql_in('bug_id', $bug_ids), undef, ($type));
>+
>+  # Select components
>+  my $comp_ids = $dbh->selectcol_arrayref(
>+    'SELECT component.id FROM components as component
>+      JOIN products AS product ON component.id = product.id
>+      WHERE product.name = ?' . ($component ? ' AND component.name = ?' : ''),
>+    undef, ($component ? ($product, $component) : ($product)));
>+
>+  # Set default bug type on these components
>+  $dbh->do('UPDATE components SET default_bug_type = ?
>+    WHERE ' . $dbh->sql_in('id', $comp_ids), undef, ($type));
>+
>+  $dbh->bz_commit_transaction;
>+}
>+
>+$dbh->bz_start_transaction;
>+
>+# Finally, disable the "enhancement" severity
>+$dbh->do('UPDATE bug_severity SET isactive = 0 WHERE value = "enhancement"');
>+
>+$dbh->bz_commit_transaction;
>+
>+Bugzilla->memcached->clear_all();

Comment 5

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

:marco, can you send me results from a sample run of the bugbug classifier to review?

Comment 6

[Marco Castelluccio [:marco]]

(In reply to Emma Humphries, Bugmaster ☕️🎸🧞‍♀️✨ (she/her) [:emceeaich] (UTC-8) needinfo? me from comment #5)

:marco, can you send me results from a sample run of the bugbug classifier to review?

The classifier between defect vs non-defect, or the one between defect vs feature vs task?
N.B.: For the second we don't have enough training labels for feature and task (especially for task), so the classifier is really good at detecting defects, but not at differentiating between features and tasks.

Comment 7

[Kohei Yoshino [:kohei]]

The staging server gets updated with a recent production DB snapshot and new bug fields, and my component-based migration script was run successfully. You can see non-defect bugs like this:

https://bugzilla.allizom.org/buglist.cgi?quicksearch=type:enhancement
https://bugzilla.allizom.org/buglist.cgi?quicksearch=type:task

It’s time to utilize bugbug 🥳

Comment 8

[Marco Castelluccio [:marco]]

(In reply to Marco Castelluccio [:marco] from comment #6)

(In reply to Emma Humphries, Bugmaster ☕️🎸🧞‍♀️✨ (she/her) [:emceeaich] (UTC-8) needinfo? me from comment #5)

:marco, can you send me results from a sample run of the bugbug classifier to review?

The classifier between defect vs non-defect, or the one between defect vs feature vs task?
N.B.: For the second we don't have enough training labels for feature and task (especially for task), so the classifier is really good at detecting defects, but not at differentiating between features and tasks.

We have decided about option 1 (make bugbug detect defects and put everything else in the 'enhancement' bucket). I'll produce a sample run for a few people to review.

Comment 10

[Kohei Yoshino [:kohei]]

These are product/component changes since Feb 13 when my pull request was sent. Looks like we don’t have to update the migration script. Emma, can you please double check that?

New/Renamed

Bug 1527140 - Merge the Web Compatibility and Tech Evangelism products
Bug 1529703 - Request new component: "Web Crypto" in DOM
Bug 1533440 - reorganize and update descriptions of some DOM-related components
Bug 1533878 - Rename bugzilla queue
Bug 1534559 - Create l10n Bugzilla component for Bengali (bn), add value for Locale field
Bug 1534970 - Request for further components under Remote Protocol product
Bug 1538960 - Rename Core::Geolocation to Core::DOM: Geolocation
Bug 1539201 - Add a Cloud Services :: Iodide component

Moved to Graveyard

Bug 1364508 - The Tech Evangelism::Preinstalled B2G Apps component should probably be graveyarded
Bug 1497358 - Close Core::Identity to new bugs, close existing open bugs, move to core graveyard
Bug 1509946 - Deprecate the distribution viewer component
Bug 1522419 - Move Social Api components to the graveyard
Bug 1522454 - Move the IP Reputation Service component to the Graveyard
Bug 1522463 - Move "core:Widget: WinRT" to the Graveyard?
Bug 1527449 - bespinplugins.mozillalabs.com component should be moved to graveyard
Bug 1530337 - please graveyard the Webtools :: Mozbot component
Bug 1533080 - Remove 2 queues
Bug 1535591 - Remove DevTools : Canvas Debugger Component
Bug 1535594 - Remove DevTools : WebGL Shader Editor Component
Bug 1535595 - Remove DevTools : Web Audio Editor Component
Bug 1536886 - Remove the getfirebug.com component from the Websites product in Bugzilla
Bug 1538339 - Remove the gear.mozilla.org component from the Websites product in Bugzilla
Bug 1538959 - Graveyard Core::Embedding: APIs

Comment 11

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

I'm okay with defaulting the new items to defect except for Cloud Services::Iodide which would be task.

Comment 12

[Kohei Yoshino [:kohei]]

Thanks! Updated the script.

Comment 13

[Kohei Yoshino [:kohei]]

Marco says bugbug itself doesn’t do a migration. I’ll update my script to do it against bugs on his latest list without updating last_changed_time.

Comment 14

[Kohei Yoshino [:kohei]]

Attachment: GitHub Pull Request - Add support for bugbug-generated bug list

Comment 15

[Kohei Yoshino [:kohei]]

Merged to master.