758174 - All Fedora 12 mozilla-inbound pgo mochitest/reftest jobs stuck pending for last 21 hours

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Ed Morley [:emorley]]

I'm not sure how much of this has to do with bug 757911, but the last Fedora 12 mozilla-inbound pgo mochitest/reftest runs that show as completed on TBPL are:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=Fedora%2012%20mozilla-inbound%20pgo&rev=7ffffcb45b94

Everything since is stuck as pending:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=Fedora%2012%20mozilla-inbound%20pgo

All other platforms/runs seem ok, it's just pgo mochitest/reftest and just Fedora 32.

As well as not having coverage for 21+ hours, this is also blocking the next merge from inbound to mozilla-central.

Comment 1

[Chris AtLee [:catlee]]

yes, definitely caused by bug 757911.

the problem was that after replication failed a bunch of data was deleted. several of the pending mozilla-inbound buildrequests were pointing to data that had been deleted, which was causing exceptions on the buildbot masters.

SQL:
select * from buildsets left outer join sourcestamps on sourcestampid = sourcestamps.id where sourcestamps.id is null and complete = 0;
+---------+-------------------+-----------+---------------+--------------+----------+-------------+---------+----
--+--------+----------+---------+------------+---------+
| id      | external_idstring | reason    | sourcestampid | submitted_at | complete | complete_at | results | id   | branch | revision | patchid | repository | project |
+---------+-------------------+-----------+---------------+--------------+----------+-------------+---------+------+--------+----------+---------+------------+---------+
| 4029643 | NULL              | scheduler |       3244070 |   1337802993 |        0 |        NULL |    NULL | NUL
L | NULL   | NULL     |    NULL | NULL       | NULL    | 
+---------+-------------------+-----------+---------------+--------------+----------+-------------+---------+------+--------+----------+---------+------------+---------+

select * from buildrequests where buildsetid = 4029643;

+----------+------------+----------------------------------------------------------+----------+------------+------------------------------------------------------------------------------+-------------------------+----------+---------+--------------+-------------+
| id       | buildsetid | buildername                                              | priority | claimed_at | clai
med_by_name                                                              | claimed_by_incarnation  | complete | r
esults | submitted_at | complete_at |
+----------+------------+----------------------------------------------------------+----------+------------+-----
-------------------------------------------------------------------------+-------------------------+----------+--
-------+--------------+-------------+
| 11697922 |    4029643 | Firefox mozilla-central macosx64 l10n nightly            |        0 | 1337805207 | buil
dbot-master32.srv.releng.scl3.mozilla.com:/builds/buildbot/build1/master | pid21520-boot1336271274 |        1 |  
     0 |   1337802991 |  1337805593 | 
| 11697927 |    4029643 | Rev3 Fedora 12 mozilla-inbound pgo test mochitests-1/5   |        0 |          0 | NULL
                                                                         | NULL                    |        0 |  
  NULL |   1337802993 |        NULL | 
| 11697928 |    4029643 | Rev3 Fedora 12 mozilla-inbound pgo test mochitests-2/5   |        0 |          0 | NULL
                                                                         | NULL                    |        0 |  
  NULL |   1337802993 |        NULL | 
| 11697929 |    4029643 | Rev3 Fedora 12 mozilla-inbound pgo test mochitests-3/5   |        0 |          0 | NULL
                                                                         | NULL                    |        0 |  
  NULL |   1337802993 |        NULL | 
| 11697930 |    4029643 | Rev3 Fedora 12 mozilla-inbound pgo test mochitests-4/5   |        0 |          0 | NULL
                                                                         | NULL                    |        0 |  
  NULL |   1337802993 |        NULL | 
| 11697931 |    4029643 | Rev3 Fedora 12 mozilla-inbound pgo test mochitests-5/5   |        0 |          0 | NULL
                                                                         | NULL                    |        0 |  
  NULL |   1337802993 |        NULL | 
| 11697932 |    4029643 | Rev3 Fedora 12 mozilla-inbound pgo test mochitest-other  |        0 |          0 | NULL
                                                                         | NULL                    |        0 |  
  NULL |   1337802993 |        NULL | 
| 11697933 |    4029643 | Rev3 Fedora 12 mozilla-inbound pgo test reftest          |        0 |          0 | NULL
                                                                         | NULL                    |        0 |  
  NULL |   1337802993 |        NULL | 
| 11697934 |    4029643 | Rev3 Fedora 12 mozilla-inbound pgo test crashtest        |        0 |          0 | NULL
                                                                         | NULL                    |        0 |  
  NULL |   1337802993 |        NULL | 
| 11697935 |    4029643 | Rev3 Fedora 12 mozilla-inbound pgo test xpcshell         |        0 |          0 | NULL
                                                                         | NULL                    |        0 |  
  NULL |   1337802993 |        NULL | 
| 11697936 |    4029643 | Rev3 Fedora 12 mozilla-inbound pgo test jsreftest        |        0 |          0 | NULL
                                                                         | NULL                    |        0 |  
  NULL |   1337802993 |        NULL | 
| 11697937 |    4029643 | Rev3 Fedora 12 mozilla-inbound pgo test reftest-ipc      |        0 |          0 | NULL
                                                                         | NULL                    |        0 |  
  NULL |   1337802993 |        NULL | 
| 11697938 |    4029643 | Rev3 Fedora 12 mozilla-inbound pgo test reftest-no-accel |        0 |          0 | NULL
                                                                         | NULL                    |        0 |  
  NULL |   1337802993 |        NULL | 
| 11697939 |    4029643 | Rev3 Fedora 12 mozilla-inbound pgo test crashtest-ipc    |        0 |          0 | NULL
                                                                         | NULL                    |        0 |  
  NULL |   1337802993 |        NULL | 
| 11697940 |    4029643 | Rev3 Fedora 12 mozilla-inbound pgo test jetpack          |        0 |          0 | NULL
                                                                         | NULL                    |        0 |  
  NULL |   1337802993 |        NULL | 
| 11697941 |    4029643 | Rev3 Fedora 12 mozilla-inbound pgo test peptest          |        0 |          0 | NULL
                                                                         | NULL                    |        0 |  
  NULL |   1337802993 |        NULL | 
+----------+------------+----------------------------------------------------------+----------+------------+-----

mysql> delete from buildrequests where buildsetid = 4029643;                                                     
Query OK, 16 rows affected (0.01 sec)

mysql> delete from buildsets where id = 4029643;
Query OK, 1 row affected (0.00 sec)