Last Comment Bug 42946 - Bugs filed at same time causing one to be duplicated.
: Bugs filed at same time causing one to be duplicated.
Status: RESOLVED FIXED
: selenium
Product: Bugzilla
Classification: Server Software
Component: Creating/Changing Bugs (show other bugs)
: 2.10
: All All
: P2 normal with 3 votes (vote)
: Bugzilla 3.0
Assigned To: Olav Vitters
: default-qa
:
Mentors:
: 46843 99812 120724 121848 148223 158549 171205 286647 292084 297468 340837 340838 388228 532865 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2000-06-17 21:28 PDT by John Dobbins
Modified: 2011-02-05 12:48 PST (History)
21 users (show)
justdave: approval+
LpSolit: testcase+
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
first draft to show the idea (7.51 KB, patch)
2004-01-30 11:16 PST, Stephen Reindl
no flags Details | Diff | Splinter Review
Missing files to complete the patch above (10.00 KB, application/x-tar)
2004-01-30 11:17 PST, Stephen Reindl
no flags Details
Second try (7.86 KB, patch)
2004-02-17 09:25 PST, Stephen Reindl
no flags Details | Diff | Splinter Review
Patch v1: Bugzilla HEAD 2006-05-20 (6.75 KB, patch)
2006-05-20 14:56 PDT, Olav Vitters
LpSolit: review+
Details | Diff | Splinter Review
Patch v1.3: Fix nits, terms.bug (7.46 KB, patch)
2006-06-04 16:16 PDT, Olav Vitters
bugzilla-mozilla: review+
Details | Diff | Splinter Review
Patch v1.4: Really fix nits (7.50 KB, patch)
2006-06-04 16:35 PDT, Olav Vitters
bugzilla-mozilla: review+
Details | Diff | Splinter Review
selenium test, v1 (1.02 KB, patch)
2011-02-05 12:47 PST, Frédéric Buclin
LpSolit: review+
Details | Diff | Splinter Review

Description John Dobbins 2000-06-17 21:28:34 PDT
Lately I've noticed a lot of bugs being duplicated. There will be 2 (or more) 
bug reports that are exact duplicate execpt for the assigned number. I've looked 
at several and noticed that the lowest numbered bug of the pair has the same 
filing time as diferent bug filed by another reporter.
Comment 1 Dave Miller [:justdave] (justdave@bugzilla.org) 2000-06-17 23:04:29 PDT
Sounds to me like the table was locked while the update action from the first 

person was being performed, and the person submitting the second bug got 

impatient and hit the refresh button to repost it.  Except that it had already 

been posted, it was just waiting for a database lock.  So now it gets posted 

twice.

Comment 2 Jesse Ruderman 2000-10-05 18:12:44 PDT
There is some discussion of this problem in bug 22353.
Comment 3 Doron Rosenberg (IBM) 2000-11-18 10:53:22 PST
I might have a patch against this. It checks the last bug submitted by the user
and compares the "subjects".  This however adds an additional mysql call per bug
submission. I'll attach the patch once I test the sql.
Comment 4 Henrik Gemal 2000-11-19 03:48:16 PST
*** Bug 46843 has been marked as a duplicate of this bug. ***
Comment 5 timeless 2000-11-21 19:13:43 PST
Please also compare the Description.  I know that subjects should be globaly 
unique --blah blah blah--, but perhaps someone is writing 3 different cases 
that deserve the same subject. [Possibly give a warning, hold the data and 
allow submission.]
Comment 6 Matthew Tuck [:CodeMachine] 2001-03-23 04:49:21 PST
I think this could happen if you hit the Submit button twice in succession
accidentally.  I've done that sort of thing before on show_bug.cgi and midaired
myself.
Comment 7 Zach Lipton [:zach] 2001-08-25 17:57:13 PDT
moving
Comment 8 Dave Miller [:justdave] (justdave@bugzilla.org) 2001-09-15 15:26:13 PDT
*** Bug 99812 has been marked as a duplicate of this bug. ***
Comment 9 Gervase Markham [:gerv] 2001-10-14 20:40:02 PDT
doron: do you have the patch for this? If so, attach it and put the milestone
back :-)

Gerv
Comment 10 Doron Rosenberg (IBM) 2001-10-15 06:31:55 PDT
This got lost in the great harddisk crash of march 2001...
Comment 11 Jerome Lacoste 2001-11-22 07:24:41 PST
I think this feature is nice. I see almost once a day one of those 'double
entry' duplicates. It would be really nice to detect them. 
If this feature can be (re-)implemented in a relatively short time, it would
save a lot of time to the whole community.
It would even allow to diminish the bug rate that is so much taken into account
by the in-famous MozillaQuest site ;)
Comment 12 Dave Miller [:justdave] (justdave@bugzilla.org) 2001-12-09 20:29:53 PST
Someone elsewhere talking about something completely different mentioned
something that gave me a good idea for this...

Embed a unique identifier of some sort in the enter_bug.cgi html as a form
element...  probably the current timestamp would be sufficient as it's very
unlikely that the same user would be able to load two different forms in
different windows within a second of each other.

stash the IDs in a separate table (perhaps the tokens table?) and expire them
after 24 hours.  When the bug is posted, delete the token.  If the token doesn't
exist, bounce the submission as a duplicate.

On the other hand, that may cause problems with automated bug submissions... it
might work better to stash the submitted tokens for a short period of time, and
bounce the submissions as a duplicate if the token already exists in the table
(thus has already been used recently).
Comment 13 timeless 2001-12-11 02:22:37 PST
just to reiterate points from Comment 5:
msie makes it very easy to open the same exact page in multiple windows.
and I use session history (in mozilla2-5).

If i submit 5 bugs based on a single template which i'm only using once, then 
i'll quickly run into your thing, and i'll complain.

I don't want to make a template, i just want to use fields that i've already 
filled out.


Comment 14 Dave Miller [:justdave] (justdave@bugzilla.org) 2001-12-11 08:35:48 PST
OK, so the obvious thing to do is to store the ticket number that was created
with the token when we file it as having been used, and compare the ticket with
the new one if the token gets reused, and only flag an error if they actually
match...
Comment 15 xyzzy 2002-01-17 23:56:51 PST
*** Bug 120724 has been marked as a duplicate of this bug. ***
Comment 16 Matthew Tuck [:CodeMachine] 2002-05-30 18:56:48 PDT
*** Bug 148223 has been marked as a duplicate of this bug. ***
Comment 17 Bradley Baetz (:bbaetz) 2002-09-28 21:40:49 PDT
*** Bug 171205 has been marked as a duplicate of this bug. ***
Comment 18 Dave Miller [:justdave] (justdave@bugzilla.org) 2002-11-24 13:27:55 PST
*** Bug 121848 has been marked as a duplicate of this bug. ***
Comment 19 timeless 2002-11-24 13:45:03 PST
ok, on irc i basically suggested comment 12 + a variant which would satisfy
comment 13: instead of simply bouncing bug submissions when we think it's an
errant accidental filing, we should provide a link to the successful report and
provide the filer with the filled bug submission form and a new submission id.
if the user wants to file the new bug, the user simply clicks commit again.

the other thing we could do is provide an editable field which would factor into
the token. that way filers who behave as described in comment 13 could simply
change their portion of the token to avoid the extra roundtrip. I don't think
it'd be very easy to explain such a field to the end user, so it's just an idea...

I wonder what we should do if one user sends a bug submission with a token owned
by another user.  normally i'd say this is an invasion of privacy, but if
someone actually posted a bug submission link complete with a token then perhaps
it would be best to complain...  I think what we should do is say that the bug
report used a token assigned to another user and simply reproduce the bug entry
form with a new token without providing a link to the filed bug. this should
solve both the privacy factor and the filing duplicate factor.
Comment 20 Dave Miller [:justdave] (justdave@bugzilla.org) 2003-01-31 08:58:05 PST
*** Bug 158549 has been marked as a duplicate of this bug. ***
Comment 21 Stephen Reindl 2003-11-17 09:56:15 PST
Coming back to comment #12: instead of only applying the timestamp, reduce the
risk by adding a random number (or the time fraction), anyhow a small risk
remains ...

On comment #19: I did a hack for a customer some time ago implementing somwthing
like this for a order tracking system. Maybe I can help introducing this here
also...
Comment 22 Stephen Reindl 2004-01-30 11:16:11 PST
Created attachment 140247 [details] [diff] [review]
first draft to show the idea

Hi all,

the applied patch plus the other two files in the next attachment will give an
overview my proposal for a solution. Please review and give comments. 

Be aware that this solution is using the UUID library. In case you don't like
this we can change this to rand() (I was unsure if this is really rand()).
Comment 23 Stephen Reindl 2004-01-30 11:17:44 PST
Created attachment 140248 [details]
Missing files to complete the patch above
Comment 24 Myk Melez [:myk] [@mykmelez] 2004-01-30 12:57:18 PST
>Embed a unique identifier of some sort in the enter_bug.cgi html as a form
>element...  probably the current timestamp would be sufficient as it's very
>unlikely that the same user would be able to load two different forms in
>different windows within a second of each other.

Note that the advent of bookmark tab groups makes this more likely (although
it's still unlikely to happen), as it's now possible to hit one bookmark and
simultaneously open several tabs, each of which could be pointed at the
enter_bug.cgi.  So it's worth adding a random number.

Nonetheless, wouldn't comparing the description of the most recently posted
couple of bugs be a much easier solution technically and handle 99% of the cases?
Comment 25 Stephen Reindl 2004-01-31 00:21:25 PST
I basically agree. But there're some more points here: 
 
1. In case two users are generating an issue with the same summary, the summary 
check would fail or we have to include the creator in the query (fine for me), 
because this is handled by standard duplicate check functionality. 
 
2. The short_desc field is handled as a fulltext search field (at least in mysql). In 
case of installations like bugzilla.mozilla.org or other installations the lookup for 
recent summaries will take more time. 
 
3. The unique/radom id prevents to check for same issue entries only to one sender. 
In case of anonymous bug entries this would prevent topic 1. 
Comment 26 Stephen Reindl 2004-02-17 09:25:34 PST
Created attachment 141627 [details] [diff] [review]
Second try

Attached there's another patch that is handling this issue. I used this in an
installation for some weeks (with a lot on IE users) and the number of
duplicated bugs due to pressing the refresh button or other issues has been
reduced 'dramatically' (about 2-3 bugs per day before -> now ...zarro...)
Comment 27 Dave Miller [:justdave] (justdave@bugzilla.org) 2004-03-19 21:53:43 PST
Pushing out bugs that aren't blockers.  If someone's working on one of these, we
can move it back.
Comment 28 Rob Siklos [:robzilla] 2004-06-14 10:41:14 PDT
here's a solution which may be better:

Instead of messing around with tokens (creating the situation in comment 13), 
why not just store a checksum of all the form data posted to post_bug.cgi in 
the db?   Then, if someone posts again, just check for any recent entry in the 
db with the same checksum.  

This solution has the advantages that you don't need to worry about token 
uniqueness or randomness, or which user is posting; and you don't have to store 
arbitrary data (like short_desc) in db tables
Comment 29 Stephen Reindl 2004-06-14 11:11:26 PDT
(In reply to comment #28)
> here's a solution which may be better:
> 
> Instead of messing around with tokens (creating the situation in comment 13), 
> why not just store a checksum of all the form data posted to post_bug.cgi in 
> the db?   Then, if someone posts again, just check for any recent entry in the 
> db with the same checksum.  
> 
... but not any date values. In addition, what means recent? An option is to
provide a new field (e.g. creation_hash) that contains a hash of most of the
fields and is unique. 

In case we find a hit:

- there might be the same issue twice
- another issue with the same hash (very unlikely, but ... ): should be checked
by comparing descriptions
Comment 30 Jeff Lawson 2004-06-14 11:22:09 PDT
That would not solve the most common duplication problem that I see:  The person
submits the bug.  She realizes that some details or left out or not enough
people were on the CC list or something trivial, so browser back button is used
to go and edit the original report, and hit submit again.  Slightly different
bug, but the same thing.

She should have just submitted changes to the bug that was just opened, but
teaching users is difficult.  I've seen this behavior in multiple people (note
if people do the "back button" mistake on process_bug.cgi, they'll get a midair
collision against themself so this protection already exists for open bugs).  It
would be nice to be able to protect the database from stupid users and force
them to visit the new bug instead.
Comment 31 Max Kanat-Alexander 2005-03-05 04:53:10 PST
This bug has not been touched by its owner in over six months, even though it is
targeted to 2.20, for which the freeze is 10 days away. Unsetting the target
milestone, on the assumption that nobody is actually working on it or has any
plans to soon.

If you are the owner, and you plan to work on the bug, please give it a real
target milestone. If you are the owner, and you do *not* plan to work on it,
please reassign it to nobody@bugzilla.org or a .bugs component owner. If you are
*anybody*, and you get this comment, and *you* plan to work on the bug, please
reassign it to yourself if you have the ability.
Comment 32 Dave Miller [:justdave] (justdave@bugzilla.org) 2005-04-27 07:41:08 PDT
*** Bug 292084 has been marked as a duplicate of this bug. ***
Comment 33 Frédéric Buclin 2005-06-05 16:15:40 PDT
(In reply to comment #12)
> stash the IDs in a separate table (perhaps the tokens table?) and expire them
> after 24 hours.  When the bug is posted, delete the token.  If the token doesn't
> exist, bounce the submission as a duplicate.
> 
> On the other hand, that may cause problems with automated bug submissions... it
> might work better to stash the submitted tokens for a short period of time, and
> bounce the submissions as a duplicate if the token already exists in the table
> (thus has already been used recently).

What I had in mind was something like:
- enter_bug.cgi creates an entry in the 'tokens' table;
- post_bug.cgi removes the token when the new bug has been created;
- if the user reloads the page, post_bug.cgi detects that the token is missing
and refuses to create a new bug.

Does this work?
Comment 34 Frédéric Buclin 2005-06-05 16:16:49 PDT
*** Bug 286647 has been marked as a duplicate of this bug. ***
Comment 35 Stephen Lee 2005-06-07 09:02:34 PDT
But what about the case where, rather than explicitly creating a bookmarkable
template for a series of similar bugs, you just press the back button adjust the
relevant details and post again?

Could something fancy be done client-side (e.g. javascript) so that on pressing
the "Submit" button you get something like:
 "You have already submitted this bug form. Do you really want to create an
additional bug report?"
Comment 36 Frédéric Buclin 2005-06-12 02:09:55 PDT
*** Bug 297468 has been marked as a duplicate of this bug. ***
Comment 37 Olav Vitters 2006-05-20 09:02:31 PDT
I want to implement what LpSolit said in comment #33. With the addition that if the cgi->params doesn't have the token param, the token checking is not done (do allow automated scripts to work.. even if they might have a webservices interface in future)

Still thinking about instead of removing the token, storing the new bug id there. Then we can make the message like: "Visit the bug". Perhaps also another button to force submission of the bug? (another param that updates the original token). This would work when the user really did submit a totally new bug and is far easier than figuring out how to detect such cases.
Comment 38 Olav Vitters 2006-05-20 14:56:37 PDT
Created attachment 222746 [details] [diff] [review]
Patch v1: Bugzilla HEAD 2006-05-20

This adds a hidden token to create.html.tmpl and create-guided.html.tmpl. If the token field is NOT present in the form submission, post_bug.cgi will always allow the bug to be created. This means that custom create templates will need to be changed (relnote). Allowing the token field not to be there is a feature to ensure any existing bug submission scripts are not broken.

When the bug has been created, the token data is updated to include the bugnr that was just created.

When post_bug.cgi detects that the token has been used before, it retrieves the old bugnr and passes that on to an 'are-you-really-sure?' template. This template encourages the user to visit the existing bug.

The template also allows the user to file the bug anyway. For this purpose it includes another hidden field called 'ignore_token' which has the older bugnr.

The ignore_token is essentially another token, because otherwise the following situation could be possible:
1. User posts a bug
2. User refreshes page to see latest comments
3. User sees 'are-you-sure-page'.
4. User doesn't read the text and clicks 'File bug again'
5. User refreshes bug again

In step #5, the ignore_token is already set. Without that being in some way unique, Bugzilla will in step #5 silently create new bugs without warning the user  (because of the ignore_token).

I could also have created a new token in the 'are-you-sure' page instead of adding the ignore_token parameter. The ignore_token param however allows me to check if the user is constantly refreshing the bug and clicking 'File bug again'...
Because ignore_token is only added by the 'are-you-sure-template', the ignore_token verification can only fail if the user is refreshing the page / submitting the same bug[1]. If the user had used the back button and the existing enter_bug.cgi form, there wouldn't be an ignore_token field which could fail (a new ignore_token would be generated). Meaning: if ignore_token fails, the user had no way of changing any of the fields, so I can be certain 'File bug anyway' should NOT be allowed. To enforce that, the 'are-you-sure?' template will NOT allow the user to submit the bug again when ignore_token fails to verify.

[1] Well.. except in one theoretical case that will not happen in practice.
Comment 39 Frédéric Buclin 2006-06-04 15:50:42 PDT
Comment on attachment 222746 [details] [diff] [review]
Patch v1: Bugzilla HEAD 2006-05-20

>Index: post_bug.cgi

>+    trick_taint($token);

Nit: you don't use $token in a SQL query here. So you should detaint it later.


>+    my $prev_bugnr = $event;

Nit: if you used $prev_bugpr directly when calling GetTokenData(), you would save this line.
Nit: I highly prefer $old_bug_id than $prev_bugnr. Please fix that before committing your patch.


>+if ($token) {
>+    $dbh->do('UPDATE tokens SET eventdata = ? WHERE token = ?', undef, 
>+             ("createbug:$id", $token));
>+}

Nit: You should detaint $token here as you now use it.



>Index: template/en/default/bug/create/confirm-create-dupe.html.tmpl

>+[% PROCESS global/variables.none.tmpl %]
>+[% PROCESS "global/field-descs.none.tmpl" %]

field-descs.none.tmpl already processes variables.none.tmpl, so calling variables.none.tmpl here is useless.


>+  <form name="Create" id="Create" method="post" action="post_bug.cgi">

Nit: 'create' instead of 'Create'.


>+  <input type="hidden" name="ignore_token" value="[% bugid FILTER html %]">
>+  <input type="submit" value="File bug again">
>+  </form>

Nit: the indentation is incorrect. Write:
<form>
  <input>
  <input>
</form>


Please fix these nits in a new patch and carry my r+ forward. r=LpSolit
Comment 40 Olav Vitters 2006-06-04 16:16:00 PDT
Created attachment 224383 [details] [diff] [review]
Patch v1.3: Fix nits, terms.bug

Fixes nits. Carrying forward r+ as per comment 39.

Also add forgotten terms.bug:
<LpSolit> not ok 163 - template/en/default/bug/create/confirm-create-dupe.html.tmpl contains invalid bare words (e.g. 'bug') --WARNING
<LpSolit>   <input type="submit" value="File bug again">
<LpSolit> bkor: File $terms.bug again
<LpSolit> bkor: there are several places in this template where you use bug alone
<LpSolit>   <p>If you are sure you used the same form to submit a new bug, click 'File bug
<LpSolit>   again'.<p>
<LpSolit> bkor: should be value="File [% terms.bug %] again" btw, not what I wrote above
<bkor> LpSolit: already changed that, running tests again
<bkor> ok, passes all tests
<LpSolit> bkor: is the UI still correct? ;)
<LpSolit> no missing whitespaces? ;)
<bkor> yeah, unexpected missing whitespace is annoying, but looks fine
Comment 41 Olav Vitters 2006-06-04 16:35:14 PDT
Created attachment 224385 [details] [diff] [review]
Patch v1.4: Really fix nits

Seems I made the patch before I saved all the files, because the terms.bug template changes where missing. :-(

Another fixed nit:
<LpSolit> bkor: another nit I missed in my previous review: $cgi->param('ignore_token') ne $old_bug_id
<LpSolit> bkor: both are integers, so use !=
Comment 42 Frédéric Buclin 2006-06-05 03:59:15 PDT
Checking in enter_bug.cgi;
/cvsroot/mozilla/webtools/bugzilla/enter_bug.cgi,v  <--  enter_bug.cgi
new revision: 1.134; previous revision: 1.133
done
Checking in post_bug.cgi;
/cvsroot/mozilla/webtools/bugzilla/post_bug.cgi,v  <--  post_bug.cgi
new revision: 1.146; previous revision: 1.145
done
RCS file: /cvsroot/mozilla/webtools/bugzilla/template/en/default/bug/create/confirm-create-dupe.html.tmpl,v
done
Checking in template/en/default/bug/create/confirm-create-dupe.html.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/bug/create/confirm-create-dupe.html.tmpl,v  <--  confirm-create-dupe.html.tmpl
initial revision: 1.1
done
Checking in template/en/default/bug/create/create-guided.html.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/bug/create/create-guided.html.tmpl,v  <--  create-guided.html.tmpl
new revision: 1.30; previous revision: 1.29
done
Checking in template/en/default/bug/create/create.html.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/bug/create/create.html.tmpl,v  <--  create.html.tmpl
new revision: 1.58; previous revision: 1.57
done
Comment 43 Vlad Dascalu 2006-06-08 09:39:24 PDT
*** Bug 340837 has been marked as a duplicate of this bug. ***
Comment 44 Vlad Dascalu 2006-06-08 09:46:17 PDT
*** Bug 340838 has been marked as a duplicate of this bug. ***
Comment 45 Frédéric Buclin 2006-06-22 13:16:34 PDT
*** Bug 46843 has been marked as a duplicate of this bug. ***
Comment 46 Max Kanat-Alexander 2006-09-07 17:54:04 PDT
Added to relnotes in bug 349423. Please let me know if I missed any critical information about this bug in the release notes.
Comment 47 Frédéric Buclin 2007-07-15 09:10:13 PDT
*** Bug 388228 has been marked as a duplicate of this bug. ***
Comment 48 Vitaly Fedrushkov 2009-12-04 01:55:51 PST
*** Bug 532865 has been marked as a duplicate of this bug. ***
Comment 49 Frédéric Buclin 2011-02-05 12:47:51 PST
Created attachment 510043 [details] [diff] [review]
selenium test, v1

Committing to: bzr+ssh://lpsolit%40gmail.com@bzr.mozilla.org/bugzilla/qa/4.0/
modified t/test_bug_edit.t
Committed revision 173.

Committing to: bzr+ssh://lpsolit%40gmail.com@bzr.mozilla.org/bugzilla/qa/3.6/
modified t/test_bug_edit.t
Committed revision 144.

Note You need to log in before you can comment on or make changes to this bug.