701585 - Mysteriously wrong build dates for some products

Status: RESOLVED FIXED

(Socorro :: Webapp, task)

Description

[[:jberkus] Josh Berkus]

Some products are showing up with wrong build dates, often several weeks off.  Examining the code, it is not clear how these could have gotten into the system:

* the cronjob does not appear capable of creating a wrong build date
* and there seems to be no reason why a user would edit the build dates to be wrong
* we tested the admin interface for a bug which would cause wrong build dates to be accidentally inserted, and did not reproduce any such issue

This is a serious problem because it can cause products to be excluded from reports and crash stats.  Currently there are 9 such products, of which 5 were set to the same wrong build date: 2011-09-23

Comment 1

[Robert Kaiser]

Counld you tell us which ones are actually affect, just out of interest?

Comment 2

[[:jberkus] Josh Berkus]

product_name | version_string | build_date 
--------------+----------------+------------
 Fennec       | 5.0b5          | 2011-06-09
 Firefox      | 6.0.1          | 2011-09-23
 Firefox      | 7.0            | 2011-09-23
 Fennec       | 7.0            | 2011-09-23
 Firefox      | 7.0(beta)      | 2011-08-17
 Firefox      | 6.0.2          | 2011-09-23
 Fennec       | 6.0.2          | 2011-09-23
 Firefox      | 8.0            | 2011-11-07
 Thunderbird  | 7.0.1          | 2011-11-07

Note that the last has already been corrected in prod.

Comment 3

[[:jberkus] Josh Berkus]

rhelmer and I discussed this, and it turns out the list of users who have access to the admin interface is quite large.  So user error now seems like the most likely cause.

Accordingly, in 2.3.4, we'll be adding logging to edit-product-info so that we can see if the cause *is* user error.

Comment 4

[[:jberkus] Josh Berkus]

Opening blocking bug for the locking

Comment 5

[[:jberkus] Josh Berkus]

Moving to 2.5.  We'll need several weeks of log data to verify if this is user error.

Comment 6

[[:jberkus] Josh Berkus]

Will look at this after ESR.

Comment 7

[[:jberkus] Josh Berkus]

OK, I've checked the data from this.

(1) We have not had an occurance of wrong build dates in the database since 12/28.

(2) Nobody has modified the dates using the admin UI since 12/28.

(3) Unknown people did modify the build dates using the admin UI during the period 12/20 to 12/28, and these modified records correspond to the only wrong build date in the database since 12/10.

THERFORE: wrong build dates are occurring as a consequence of erroneous user input.  Because all such input happened before we tracked user names, there is no way to find the user at fault.

This bug will be solved by implementation of bug 721135.

Comment 8

[[:jberkus] Josh Berkus]

This change isn't version-specific, so removing version target.

Comment 9

[[:jberkus] Josh Berkus]

This problem has not recurred since we changed the admin interface.  As such, closing as resolved.