69621 - Keyword cache not updated on keyword rename/delete (remove the keyword cache)

Status: RESOLVED FIXED

(Bugzilla :: Administration, task, P1)

Description

[Matthew Tuck [:CodeMachine]]

I noticed this morning this from mozilla.org sanitycheck.cgi:

Checking cached keywords 

Bug(s) found with incorrect keyword cache: 46700, 47682, 48303, 50670, 50810,
50875, 51067, 51316, 51574, 51748,
51768, 52134, 52141, 52451, 52457, 52527, 52528, 52530, 52558, 52654, 52887,
52971, 53085, 53087, 53482, 54327,
54328, 55091, 55502, 55667, 56114, 56218, 56379, 56568, 57676, 58951, 58954,
59122, 59480, 59613, 59928, 59964,
61830, 63764, 64139, 64151, 65076, 8388607 

At least the first one, probably most of these were due to a keyword rename.  It
turns out editkeywords.cgi does not update the version cache of bugs when it
renames a keyword.

As far as I can tell, show_bug doesn't use the version cache, so it's not
terrible, but I presume queries use it.

I'm still looking at the last, illegal bug number.

Comment 1

[Matthew Tuck [:CodeMachine]]

Err, keywords cache, not version cache.

Comment 2

[Matthew Tuck [:CodeMachine]]

All of these bug numbers except for the last one are due to a rename of newmod
to modern.  The last one is not, and is covered at bug #69629.

Comment 3

[Jacob Steenhagen]

I think this is most likely a direct effect of bug 64760.

Comment 4

[Matthew Tuck [:CodeMachine]]

It looks like it's similar if not the same.  It's not suprising that if the
keyword cache got out of date process_bug might try to fix it and give
permission errors.

IMO this is major problem since I think it can result in bugs not being on
queries.

Comment 5

[Matthew Tuck [:CodeMachine]]

OK, neither the rename nor the delete code path update the keywords cache of
affected bugs.  This could probably be resolved best with code that applies to
both, so one should probably be marked as dupe.  I'd be more inclined to leave
this here since I think it's more on the money.

I am certainly disheartened to learn the seriousness of this was known as far
back as December (bug #62893).  Your report was the only Bugzilla bug report
filed on it that I can see, but it didn't really explain the true seriousness of
the problem.

Comment 6

[Matthew Tuck [:CodeMachine]]

There seems to be a consensus we should try to get this done for 2.12 due to 
its serious nature.

Comment 7

[Chris Yeh]

How long does it take to run sanitycheck.cgi on mozilla.org's installation?

I recommend fixing this in two parts:

1) The stupid way. After editing or removing a keyword, notify the administrator 
that they have to run sanitycheck.cgi and rebuild the keyword cache.

2) Later on, the right way. Break out keyword cache rebuilding from 
sanitycheck.cgi and call that routine anytime an edit or delete operation is 
performed.

Comment 8

[Matthew Tuck [:CodeMachine]]

sanitycheck.cgi takes about 20 minutes on mozilla.org, and I don't know how much
it loads the machine down.

I'm not sure, are you're implying here that we should rebuild the keyword caches
of all bugs, as opposed to just the ones with the keyword on it?

Comment 9

[Chris Yeh]

No, we should just be rebuilding the cache's on those bugs that are effected by 
a keyword remove or rename.

Comment 10

[Chris Yeh]

Okay, I have implemented the stupid fix, namely, that editing or deleting a 
keyword pops up extra html that reminds the user to rebuild the keyword cache 
using sanitycheck.cgi.  I am checking this in, and then moving this bug to 2.16 
for a more 'proper' fix.

Comment 11

[Dave Miller [:justdave]]

moving to real milestones...

Comment 12

[Jacob Steenhagen]

*** Bug 64760 has been marked as a duplicate of this bug. ***

Comment 13

[Dave Miller [:justdave]]

*** Bug 75222 has been marked as a duplicate of this bug. ***

Comment 14

[Matthew Tuck [:CodeMachine]]

This happened again the other day when someone forgot to update the caches
despite the message.

I very strongly suggest that the fix to this should write changes on
bugs_activity and send out notifications.

Comment 15

[Dave Miller [:justdave]]

*** Bug 52993 has been marked as a duplicate of this bug. ***

Comment 16

[Jacob Steenhagen]

-> New Bugzilla Product

Comment 17

[Jacob Steenhagen]

Oops... forgot to click "Reassign"

Comment 18

[Dave Miller [:justdave]]

This is now on the "we really want this for 2.16, but won't hold the release for
it if it's not done by then" list.

Comment 19

[Dave Miller [:justdave]]

No patch yet, not a release blocker, 2.16 is now it freeze mode.

-> 2.18

Comment 20

[A. Craig West]

So is there a workaround for this? At the moment, I am unable to add a comment
to Bug 68985. Is there a way to get sanitycheck.cgi run?

Comment 21

[A. Craig West]

So why isn't this a release blocker? This makes affected bugs un-editable by us
peons... Also, there doesn't seem to be any way to get someone to fix a bug that
is exhibiting symptoms of this. For example, when I try to add myself to the CC
list for Bug 68985, I get:
Only the owner or submitter of the bug, or a sufficiently empowered user, may
make that change to the keywords field.
Old value: helpwanted, nsbranch, vtrunk
New value: helpwanted, nsbranch

I've been looking for somebody to help me fix that for a month, now.

Comment 22

[Matthew Tuck [:CodeMachine]]

File bugs on product mozilla.org component bugzilla: other moz.org issues

Comment 23

[Bradley Baetz (:bbaetz)]

I've 'fixed' bug 68985.

Comment 24

[Bradley Baetz (:bbaetz)]

*** Bug 159948 has been marked as a duplicate of this bug. ***

Comment 25

[GavinS]

I don't think this should depend on a bug which was marked as a duplicate of
this one, so removing dependency on bug#64760

Comment 26

[Joel Peshkin]

Unloved bugs targetted for 2.18 but untouched since 9-15-2003 are being
retargeted to 2.20
If you plan to act on one immediately, go ahead and pull it back to 2.18.

Comment 27

[Joel Peshkin]

Can anyone see a reason the keyword cache exists at all?   Why are we not
killing it?

Comment 28

[Matthew Tuck [:CodeMachine]]

Joel the caches exist so they can be output on buglist.cgi.

Comment 29

[Max Kanat-Alexander]

This bug has not been touched by its owner in over six months, even though it is
targeted to 2.20, for which the freeze is 10 days away. Unsetting the target
milestone, on the assumption that nobody is actually working on it or has any
plans to soon.

If you are the owner, and you plan to work on the bug, please give it a real
target milestone. If you are the owner, and you do *not* plan to work on it,
please reassign it to [e-mail] or a .bugs component owner. If you are
*anybody*, and you get this comment, and *you* plan to work on the bug, please
reassign it to yourself if you have the ability.

Comment 30

[Dave Miller [:justdave]]

Reassigning bugs that I'm not actively working on to the default component owner
in order to try to make some sanity out of my personal buglist.  This doesn't
mean the bug isn't being dealt with, just that I'm not the one doing it.  If you
are dealing with this bug, please assign it to yourself.

Comment 31

[Frédéric Buclin]

We no longer accept new features for Bugzilla 3.6. Retargetting to 3.8.

Comment 32

[Frédéric Buclin]

It shouldn't be hard to do now that we have sql_group_concat().

Comment 33

[Frédéric Buclin]

Attachment: patch, v1

Tested on PostgreSQL, and works like a charm.

Comment 34

[Max Kanat-Alexander]

Comment on attachment 437610 [details] [diff] [review]
patch, v1

  This is awesome.

Comment 35

[Frédéric Buclin]

Committing to: bzr+ssh://[e-mail]/bugzilla/trunk/
modified sanitycheck.cgi
modified Bugzilla/Bug.pm
modified Bugzilla/Search.pm
modified Bugzilla/DB/Schema.pm
modified Bugzilla/Install/DB.pm
modified template/en/default/admin/sanitycheck/messages.html.tmpl
modified template/en/default/global/messages.html.tmpl
Committed revision 7121.

Comment 37

[Tiago Mello [:timello]]

I'm getting the follow error after submitting the a new bug:

Invalid Parameter

keywords is not a valid parameter for the Bugzilla::Bug::create function.

Comment 38

[Max Kanat-Alexander]

Added to the release notes in bug 604256.