791758 - Mass change from whiteboard '[orange]' to keyword 'intermittent-failure'

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[Ed Morley [:emorley]]

[I'm filing this bug now so I can get all the dependencies straight, but please hold off any work for a couple of days, until feedback on the dev.platform post (linked in bug 790571) has been obtained.]

In bug 790571, we would like to switch from using whiteboard '[orange]' to using a new keyword 'intermitent-failure', for our intermittent failure bugs.

As such we would like:
1) A new 'intermitent-failure' keyword added to b.m.o
2) A script run to add this new keyword to all bugs on b.m.o whose whiteboard contain '[orange]', to allow testing to begin in bug 790571. (Without causing bugmail).
3) Once the patches in bug 790571 have been staged/tested/deployed:
  a) the script in step #2 run again to catch any stragglers.
  b) a script run to mass remove '[orange]' from the whiteboard of any bug that has the keyword intermitent-failure - again without causing bugmail.

Comment 1

[Ed Morley [:emorley]]

The other bugs are shaping up, so I don't suppose we can now:

1) Add a new "intermittent-failure" keyword to b.m.o
2) Create the script mentioned in comment 0, so we have it ready to be run on production.

Comment 2

[David Lawrence [:dkl]]

(In reply to Ed Morley [:edmorley UTC+0] from comment #1)
> The other bugs are shaping up, so I don't suppose we can now:
> 
> 1) Add a new "intermittent-failure" keyword to b.m.o

Sure. Can I get a description for the new keyword and I will go ahead and get that created.

> 2) Create the script mentioned in comment 0, so we have it ready to be run
> on production.

Will start working on a script.

dkl

Comment 3

[Ed Morley [:emorley]]

(In reply to David Lawrence [:dkl] from comment #2)
> Sure. Can I get a description for the new keyword and I will go ahead and
> get that created.

"For the tracking of intermittent test failures within the test-suites of Gecko related products."

> Will start working on a script.

Many thanks! :-)

Comment 4

[David Lawrence [:dkl]]

keyword created

Comment 5

[David Lawrence [:dkl]]

Attachment: Script to add intermittent-failure keyword to orange bugs (v1)

Comment 6

[David Lawrence [:dkl]]

Attachment: Script to remove orange from whiteboard for intermittent-failure bugs (v1)

Comment 7

[:glob ✱]

these two scripts are almost identical .. any reason why they aren't combined into a single script?

Comment 8

[David Lawrence [:dkl]]

(In reply to Byron Jones ‹:glob› from comment #7)
> these two scripts are almost identical .. any reason why they aren't
> combined into a single script.

I could do it as a single script but I was trying to keep them both as simple as possible. According to Ed's instructions, he wanted to do the migration as a two step process. 1) Just add the new keyword but yet leave the whiteboard alone. 2) Later on once everything seems to be going ok, then run another script to go back and remove the orange whiteboard. No reason technically that they can't be in one script but figured it would be easier to review this way.

dkl

Comment 9

[Ed Morley [:emorley]]

Since comment 0 I've realised we need to match against both [orange] and [tb-orange], sorry!

(In reply to David Lawrence [:dkl] from comment #8)
> (In reply to Byron Jones ‹:glob› from comment #7)
> > these two scripts are almost identical .. any reason why they aren't
> > combined into a single script.
> 
> I could do it as a single script but I was trying to keep them both as
> simple as possible. According to Ed's instructions, he wanted to do the
> migration as a two step process. 1) Just add the new keyword but yet leave
> the whiteboard alone. 2) Later on once everything seems to be going ok, then
> run another script to go back and remove the orange whiteboard. No reason
> technically that they can't be in one script but figured it would be easier
> to review this way.

I guess the only other consideration, is that for (2) above, we'll have to re-run the part 1 script anyway, to catch bugs filed/changed since. Whilst unlikely, someone could then still file additional bugs in-between the first and second script being run, unless we add the keyword at the same time as removing the whiteboard annotation.

As such, how about a unified script along the lines of:

* Search for all bugs with [orange] or [tb-orange] in the whiteboard
* For each bug:
  If "--remove" parameter specified:
    ** Remove [orange] / [tb-orange] from whiteboard
    ** Add keyword intermittent-failure
  else
    ** Just add keyword intermittent failure
* Print vs print+updateDB would depend on an additional parameter, similar to the current script.

Comment 10

[David Lawrence [:dkl]]

(In reply to Ed Morley [:edmorley UTC+0] from comment #9)
> I guess the only other consideration, is that for (2) above, we'll have to
> re-run the part 1 script anyway, to catch bugs filed/changed since. Whilst
> unlikely, someone could then still file additional bugs in-between the first
> and second script being run, unless we add the keyword at the same time as
> removing the whiteboard annotation.
> 
> As such, how about a unified script along the lines of:
> 
> * Search for all bugs with [orange] or [tb-orange] in the whiteboard
> * For each bug:
>   If "--remove" parameter specified:
>     ** Remove [orange] / [tb-orange] from whiteboard
>     ** Add keyword intermittent-failure
>   else
>     ** Just add keyword intermittent failure
> * Print vs print+updateDB would depend on an additional parameter, similar
> to the current script.

Sure I could do that. Will work on another version of the script here shortly.

dkl

Comment 11

[:glob ✱]

Comment on attachment 676902 [details] [diff] [review]
Script to add intermittent-failure keyword to orange bugs (v1)

Review of attachment 676902 [details] [diff] [review]:
-----------------------------------------------------------------

since the script will be updated, i'll dump two minor nits...

::: contrib/reorg-tools/add_orange_keyword.pl
@@ +51,5 @@
> +
> +# intermittent-keyword id (assumes already created)
> +my $keyword_id = $dbh->selectrow_array(
> +    "SELECT id FROM keyworddefs WHERE name = 'intermittent-failure'");
> +$keyword_id or die "Can't find keyword id for 'intermittent-failure'\n";

nit: get_field_id() may be clearer for both of these.

@@ +61,5 @@
> +my $bugs = $dbh->selectall_arrayref("
> +    SELECT DISTINCT bugs.bug_id
> +      FROM bugs LEFT JOIN keywords ON bugs.bug_id = keywords.bug_id
> +     WHERE keywords.keywordid != ?
> +           AND bugs.status_whiteboard REGEXP '\\\\[orange\\\\]'

there's no reason for this to be a regex; use LIKE '%[orange]%'

Comment 12

[David Lawrence [:dkl]]

Attachment: Script to add intermittent-failure to orange bugs and optionally update whiteboard (v1)

New single script that handles both adding the keyword and optionally cleaning the whiteboard.

dkl

Comment 13

[:glob ✱]

Comment on attachment 679748 [details] [diff] [review]
Script to add intermittent-failure to orange bugs and optionally update whiteboard (v1)

Review of attachment 679748 [details] [diff] [review]:
-----------------------------------------------------------------

r=glob

::: contrib/reorg-tools/migrate_orange_bugs.pl
@@ +73,5 @@
> +$field_id or usage("Can't find field ID for 'keywords' field\n");
> +
> +# intermittent-keyword id (assumes already created)
> +my $keyword_id = Bugzilla::Keyword->new({ name => 'intermittent-failure' })->id;
> +$keyword_id or usage("Can't find keyword id for 'intermittent-failure'\n");

this throws:
Can't call method "id" on an undefined value at ./contrib/reorg-tools/migrate_orange_bugs.pl line 76.
when the keyword doesn't exist.

@@ +135,5 @@
> +
> +    print "done.\n";
> +}
> +
> +$dbh->bz_commit_transaction() if $doit;

i'd like to see a message at the end of the run if $doit is false so the user knows that nothing actually happened.

Comment 14

[Ed Morley [:emorley]]

Comment on attachment 679748 [details] [diff] [review]
Script to add intermittent-failure to orange bugs and optionally update whiteboard (v1)

>+    FROM bugs WHERE bugs.status_whiteboard LIKE '%[orange]%'",
...
>+            $whiteboard =~ s/\[orange\]//ig;

Sorry to be a pain, but we need to match against [orange] and also [tb-orange] bugs (comment 9).

Thank you so much for doing this! :-D

Comment 15

[David Lawrence [:dkl]]

Thanks glob. Committed.

Committing to: bzr+ssh://[e-mail]/bmo/4.0
added contrib/reorg-tools/migrate_orange_bugs.pl
Committed revision 8377.

Committing to: bzr+ssh://[e-mail]/bmo/4.2             
added contrib/reorg-tools/migrate_orange_bugs.pl
Committed revision 8400.

Ed, please advise when we should schedule to have the 1st phase of the script ran against production.

dkl

Comment 16

[Ed Morley [:emorley]]

(In reply to David Lawrence [:dkl] from comment #15)
> Ed, please advise when we should schedule to have the 1st phase of the
> script ran against production.

At whatever point you have a spare moment would be great (no immediate rush), thank you :-)
(1st phase, so running without the --remove option)

Comment 17

[David Lawrence [:dkl]]

(In reply to Ed Morley [:edmorley UTC+0] from comment #16)
> At whatever point you have a spare moment would be great (no immediate
> rush), thank you :-)
> (1st phase, so running without the --remove option)

Thanks. Will do now then :)

Admin, please execute the following script on the production BMO web head of your choice:

# cd /path/to/bugzilla/code
# perl -wT -I. contrib/reorg-tools/migrate_orange_bugs.pl --doit

Thanks
dkl

Comment 18

[Ed Morley [:emorley]]

Ben/Shyam, let me know if you need any more info for this :-)

Comment 19

[David Lawrence [:dkl]]

(In reply to Ed Morley [:edmorley UTC+0] from comment #18)
> Ben/Shyam, let me know if you need any more info for this :-)

I know fox2mike is traveling at the moment but there are other IT admins that can run scripts for us on production. Maybe Ashish?

dkl

Comment 20

[Shyam Mani [:fox2mike]]

I think we haven't pushed bugzilla since the script was committed to the tree and therefore the script isn't available to be run from the webheads...

Comment 21

[Shyam Mani [:fox2mike]]

Attachment: Command output

And as soon as I said that... heh.

All good!

Comment 22

[David Lawrence [:dkl]]

(In reply to Shyam Mani [:fox2mike] from comment #21)
> Created attachment 683320 [details]
> Command output
> 
> And as soon as I said that... heh.
> 
> All good!

Thanks Shyam. 

Ed, when you are ready to remove the whiteboard entries with --remove, please clone this bug and create a new one so we can have IT run the script again then.

Thanks
dkl

Comment 23

[Ed Morley [:emorley]]

Will do; thank you Shyam! :-)