668425 - statusdb not getting results/endtimes sometimes

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Chris AtLee [:catlee]]

e.g.

mysql> select * from builds where id=5307062;
+---------+-------------+------------+----------+-----------+---------------------+---------------------+--------+-----------+-----------+------+
| id      | buildnumber | builder_id | slave_id | master_id | starttime           | endtime             | result | reason    | source_id | lost |
+---------+-------------+------------+----------+-----------+---------------------+---------------------+--------+-----------+-----------+------+
| 5307062 |        3434 |       3298 |      645 |         8 | 2011-06-28 18:03:44 | 1970-01-01 00:00:01 |   NULL | scheduler |   2102540 |    0 | 
+---------+-------------+------------+----------+-----------+---------------------+---------------------+--------+-----------+-----------+------+
1 row in set (0.00 sec)

mysql> select * from masters where id=8;
+----+---------------------------------------------------+----------------------+
| id | url                                               | name                 |
+----+---------------------------------------------------+----------------------+
|  8 | http://production-master03.build.mozilla.org:8010 | Production Master 03 | 
+----+---------------------------------------------------+----------------------+
1 row in set (0.00 sec)

mysql> select * from builders where id=3298;
+------+--------------------------+-----------+-----------------+
| id   | name                     | master_id | category        |
+------+--------------------------+-----------+-----------------+
| 3298 | mozilla-central-macosx64 |         8 | mozilla-central | 
+------+--------------------------+-----------+-----------------+

On the master this build is finished.

The pickle file claims that it finished:
>>> b.getTimes()
(1309284224.192858, 1309296638.0811191)

Comment 1

[Chris AtLee [:catlee]]

A reconfig happened at 2011-06-28 12:39:59-0700 pacific...so this build spanned the reconfig.

Comment 2

[Chris AtLee [:catlee]]

What's happening is that update_from_files.py is failing to find the old build and so we end up with duplicate rows:

mysql> select * from builds where buildnumber=3434 and builder_id=3298 and slave_id=645;
+---------+-------------+------------+----------+-----------+---------------------+---------------------+--------+-----------+-----------+------+
| id      | buildnumber | builder_id | slave_id | master_id | starttime           | endtime             | result | reason    | source_id | lost |
+---------+-------------+------------+----------+-----------+---------------------+---------------------+--------+-----------+-----------+------+
| 5307062 |        3434 |       3298 |      645 |         8 | 2011-06-28 18:03:44 | 1970-01-01 00:00:01 |   NULL | scheduler |   2102540 |    0 | 
| 5308571 |        3434 |       3298 |      645 |         8 | 2011-06-28 18:03:44 | 2011-06-28 21:30:38 |      0 | scheduler |   2102540 |    0 | 
+---------+-------------+------------+----------+-----------+---------------------+---------------------+--------+-----------+-----------+------+

Comment 3

[Chris AtLee [:catlee]]

Attachment: don't use end time to match existing builds

What is happening is the in-progress build is being written to disk (with no results or finish time) during the reconfig.

When the build finishes, it gets written again, so it does get seen from update_from_files.py. However the query to find previous instances of the build fails because the query includes the build's finish time, which the db doesn't have.

I believe matching against start time, build number and builder on the same master is sufficient for uniqueness.