Closed Bug 758174 Opened 13 years ago Closed 13 years ago

All Fedora 12 mozilla-inbound pgo mochitest/reftest jobs stuck pending for last 21 hours

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
x86
Linux
Type:
defect
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: emorley, Assigned: catlee)

Details

I'm not sure how much of this has to do with bug 757911, but the last Fedora 12 mozilla-inbound pgo mochitest/reftest runs that show as completed on TBPL are: https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=Fedora%2012%20mozilla-inbound%20pgo&rev=7ffffcb45b94 Everything since is stuck as pending: https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=Fedora%2012%20mozilla-inbound%20pgo All other platforms/runs seem ok, it's just pgo mochitest/reftest and just Fedora 32. As well as not having coverage for 21+ hours, this is also blocking the next merge from inbound to mozilla-central.
yes, definitely caused by bug 757911. the problem was that after replication failed a bunch of data was deleted. several of the pending mozilla-inbound buildrequests were pointing to data that had been deleted, which was causing exceptions on the buildbot masters. SQL: select * from buildsets left outer join sourcestamps on sourcestampid = sourcestamps.id where sourcestamps.id is null and complete = 0; +---------+-------------------+-----------+---------------+--------------+----------+-------------+---------+---- --+--------+----------+---------+------------+---------+ | id | external_idstring | reason | sourcestampid | submitted_at | complete | complete_at | results | id | branch | revision | patchid | repository | project | +---------+-------------------+-----------+---------------+--------------+----------+-------------+---------+------+--------+----------+---------+------------+---------+ | 4029643 | NULL | scheduler | 3244070 | 1337802993 | 0 | NULL | NULL | NUL L | NULL | NULL | NULL | NULL | NULL | +---------+-------------------+-----------+---------------+--------------+----------+-------------+---------+------+--------+----------+---------+------------+---------+ select * from buildrequests where buildsetid = 4029643; +----------+------------+----------------------------------------------------------+----------+------------+------------------------------------------------------------------------------+-------------------------+----------+---------+--------------+-------------+ | id | buildsetid | buildername | priority | claimed_at | clai med_by_name | claimed_by_incarnation | complete | r esults | submitted_at | complete_at | +----------+------------+----------------------------------------------------------+----------+------------+----- -------------------------------------------------------------------------+-------------------------+----------+-- -------+--------------+-------------+ | 11697922 | 4029643 | Firefox mozilla-central macosx64 l10n nightly | 0 | 1337805207 | buil dbot-master32.srv.releng.scl3.mozilla.com:/builds/buildbot/build1/master | pid21520-boot1336271274 | 1 | 0 | 1337802991 | 1337805593 | | 11697927 | 4029643 | Rev3 Fedora 12 mozilla-inbound pgo test mochitests-1/5 | 0 | 0 | NULL | NULL | 0 | NULL | 1337802993 | NULL | | 11697928 | 4029643 | Rev3 Fedora 12 mozilla-inbound pgo test mochitests-2/5 | 0 | 0 | NULL | NULL | 0 | NULL | 1337802993 | NULL | | 11697929 | 4029643 | Rev3 Fedora 12 mozilla-inbound pgo test mochitests-3/5 | 0 | 0 | NULL | NULL | 0 | NULL | 1337802993 | NULL | | 11697930 | 4029643 | Rev3 Fedora 12 mozilla-inbound pgo test mochitests-4/5 | 0 | 0 | NULL | NULL | 0 | NULL | 1337802993 | NULL | | 11697931 | 4029643 | Rev3 Fedora 12 mozilla-inbound pgo test mochitests-5/5 | 0 | 0 | NULL | NULL | 0 | NULL | 1337802993 | NULL | | 11697932 | 4029643 | Rev3 Fedora 12 mozilla-inbound pgo test mochitest-other | 0 | 0 | NULL | NULL | 0 | NULL | 1337802993 | NULL | | 11697933 | 4029643 | Rev3 Fedora 12 mozilla-inbound pgo test reftest | 0 | 0 | NULL | NULL | 0 | NULL | 1337802993 | NULL | | 11697934 | 4029643 | Rev3 Fedora 12 mozilla-inbound pgo test crashtest | 0 | 0 | NULL | NULL | 0 | NULL | 1337802993 | NULL | | 11697935 | 4029643 | Rev3 Fedora 12 mozilla-inbound pgo test xpcshell | 0 | 0 | NULL | NULL | 0 | NULL | 1337802993 | NULL | | 11697936 | 4029643 | Rev3 Fedora 12 mozilla-inbound pgo test jsreftest | 0 | 0 | NULL | NULL | 0 | NULL | 1337802993 | NULL | | 11697937 | 4029643 | Rev3 Fedora 12 mozilla-inbound pgo test reftest-ipc | 0 | 0 | NULL | NULL | 0 | NULL | 1337802993 | NULL | | 11697938 | 4029643 | Rev3 Fedora 12 mozilla-inbound pgo test reftest-no-accel | 0 | 0 | NULL | NULL | 0 | NULL | 1337802993 | NULL | | 11697939 | 4029643 | Rev3 Fedora 12 mozilla-inbound pgo test crashtest-ipc | 0 | 0 | NULL | NULL | 0 | NULL | 1337802993 | NULL | | 11697940 | 4029643 | Rev3 Fedora 12 mozilla-inbound pgo test jetpack | 0 | 0 | NULL | NULL | 0 | NULL | 1337802993 | NULL | | 11697941 | 4029643 | Rev3 Fedora 12 mozilla-inbound pgo test peptest | 0 | 0 | NULL | NULL | 0 | NULL | 1337802993 | NULL | +----------+------------+----------------------------------------------------------+----------+------------+----- mysql> delete from buildrequests where buildsetid = 4029643; Query OK, 16 rows affected (0.01 sec) mysql> delete from buildsets where id = 4029643; Query OK, 1 row affected (0.00 sec)
Assignee: nobody → catlee
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.