895943 - duplicate jobs triggered in 23.0b6 b7 and b8

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

Details incoming.

Comment 1

[[e-mail] (:bhearsum)]

Examples from each release:
23.0b6:
- 6 linux partner repacks:
-- http://buildbot-master58:8001/builders/release-mozilla-beta-linux_partner_repack/builds/3 & 4 & 5
--- all started at Mon Jul 15 19:14:50 2013
-- http://buildbot-master64:8001/builders/release-mozilla-beta-linux_partner_repack/builds/0 & 1 & 2
--- all started at Mon Jul 15 19:15:12 2013

- 4 linux64 partner repacks:
-- http://buildbot-master58:8001/builders/release-mozilla-beta-linux64_partner_repack/builds/1
--- started at Mon Jul 15 18:47:56 2013
-- http://buildbot-master62:8001/builders/release-mozilla-beta-linux64_partner_repack/builds/2
--- started at Mon Jul 15 18:47:32 2013
-- http://buildbot-master64:8001/builders/release-mozilla-beta-linux64_partner_repack/builds/0
--- started at Mon Jul 15 18:47:42 2013
-- http://buildbot-master66.srv.releng.usw2.mozilla.com:8001/builders/release-mozilla-beta-linux64_partner_repack/builds/3
--- started at Mon Jul 15 18:47:20 2013

- 3 macosx64 partner repacks
-- http://buildbot-master58:8001/builders/release-mozilla-beta-macosx64_partner_repack/builds/1
--- started at Mon Jul 15 18:48:31 2013
-- http://buildbot-master66.srv.releng.usw2.mozilla.com:8001/builders/release-mozilla-beta-macosx64_partner_repack/builds/1 & 2
--- started at Mon Jul 15 18:48:11 2013 & Mon Jul 15 18:48:12 2013

23.0b7:
- 2 linux64 partner repacks:
-- http://buildbot-master64:8001/builders/release-mozilla-beta-linux64_partner_repack/builds/1
--- started at Thu Jul 18 21:44:12 2013
-- http://buildbot-master66.srv.releng.usw2.mozilla.com:8001/builders/release-mozilla-beta-linux64_partner_repack/builds/4
--- started at Thu Jul 18 21:44:20 2013

- 2 win32 update verifies:
-- http://buildbot-master62:8001/builders/release-mozilla-beta-win32_update_verify_1%2F4/builds/3 and 4
--- both started at Fri Jul 19 01:56:37 2013

There were lots of other duplicates too, looking at https://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/23.0b6-candidates/build1/logs/ and https://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/23.0b7-candidates/build1/logs/ is an easy way to find them.

I tried to find additional details of these in the buildbot db, but unfortunately my sql-fu wasn't good enough. Going to dump some twistd.log info for the 23.0b7 ones in the next comment, though.

Comment 2

[[e-mail] (:bhearsum)]

From bm62:
twistd.log.6:2013-07-19 01:56:36-0700 [-] prioritizeBuilders: 0.27s prioritized 12 builder(s): [((0, 0L, 100, 1374224177L), 'release-mozilla-beta-win32_update_verify_1/4'), ((0, 0L, 100, 1374224177L), 'release-mozilla-beta-win32_update_verify_3/4'), ((0, 0L, 100, 1374224183L), 'release-mozilla-beta-linux_update_verify_2/4'), ((0, 0L, 100, 1374224183L), 'release-mozilla-beta-linux64_update_verify_4/4'), ((0, 0L, 100, 1374224183L), 'release-mozilla-beta-linux_update_verify_1/4'), ((0, 0L, 100, 1374224183L), 'release-mozilla-beta-linux64_update_verify_3/4'), ((0, 0L, 100, 1374224183L), 'release-mozilla-beta-linux64_update_verify_1/4'), ((0, 0L, 100, 1374224183L), 'release-mozilla-beta-linux64_update_verify_2/4'), ((0, 0L, 100, 1374224183L), 'release-mozilla-beta-win32_update_verify_2/4'), ((0, 0L, 100, 1374224183L), 'release-mozilla-beta-win32_update_verify_4/4'), ((0, 0L, 100, 1374224183L), 'release-mozilla-beta-linux_update_verify_3/4'), ((0, 0L, 100, 1374224183L), 'release-mozilla-beta-linux_update_verify_4/4')]
twistd.log.6:2013-07-19 01:56:36-0700 [-] prioritizeBuilders: 0.28s important builder release-mozilla-beta-win32_update_verify_1/4 (p == (0, 0L, 100, 1374224177L))
twistd.log.6:2013-07-19 01:56:36-0700 [-] prioritizeBuilders: 0.28s found 10 important builder(s): ['release-mozilla-beta-linux_update_verify_3/4', 'release-mozilla-beta-linux64_update_verify_4/4', 'release-mozilla-beta-linux64_update_verify_3/4', 'release-mozilla-beta-linux_update_verify_1/4', 'release-mozilla-beta-linux64_update_verify_1/4', 'release-mozilla-beta-win32_update_verify_3/4', 'release-mozilla-beta-win32_update_verify_1/4', 'release-mozilla-beta-linux64_update_verify_2/4', 'release-mozilla-beta-linux_update_verify_4/4', 'release-mozilla-beta-linux_update_verify_2/4']
twistd.log.6:2013-07-19 01:56:37-0700 [-] starting build <Build release-mozilla-beta-win32_update_verify_1/4> using slave <SlaveBuilder builder='release-mozilla-beta-win32_update_verify_1/4' slave='w64-ix-slave06'>
twistd.log.6:2013-07-19 01:56:37-0700 [-] starting build <Build release-mozilla-beta-win32_update_verify_1/4>.. pinging the slave <SlaveBuilder builder='release-mozilla-beta-win32_update_verify_1/4' slave='w64-ix-slave06'>
twistd.log.6:2013-07-19 01:56:37-0700 [-] starting build <Build release-mozilla-beta-win32_update_verify_1/4> using slave <SlaveBuilder builder='release-mozilla-beta-win32_update_verify_1/4' slave='w64-ix-slave119'>
twistd.log.6:2013-07-19 01:56:37-0700 [-] starting build <Build release-mozilla-beta-win32_update_verify_1/4>.. pinging the slave <SlaveBuilder builder='release-mozilla-beta-win32_update_verify_1/4' slave='w64-ix-slave119'>
twistd.log.6:2013-07-19 01:56:37-0700 [Broker,6779,10.12.40.151] <Build release-mozilla-beta-win32_update_verify_1/4>.startBuild
twistd.log.6:2013-07-19 01:56:37-0700 [Broker,6897,10.12.40.26] <Build release-mozilla-beta-win32_update_verify_1/4>.startBuild


From bm64:
2013-07-18 21:44:11-0700 [-] prioritizeBuilders: 0.10s prioritized 2 builder(s): [((0, 0L, 100, 1374209048L), 'release-mozilla-beta-linux64_partner_repack'), ((6, 0L, 100, 1373746552L), 'fuzzer-linux64')]
2013-07-18 21:44:11-0700 [-] prioritizeBuilders: 0.10s important builder release-mozilla-beta-linux64_partner_repack (p == (0, 0L, 100, 1374209048L))
2013-07-18 21:44:11-0700 [-] prioritizeBuilders: 0.10s found 2 important builder(s): ['fuzzer-linux64', 'release-mozilla-beta-linux64_partner_repack']
2013-07-18 21:44:12-0700 [-] starting build <Build release-mozilla-beta-linux64_partner_repack> using slave <SlaveBuilder builder='release-mozilla-beta-linux64_partner_repack' slave='bld-lion-r5-074'>
2013-07-18 21:44:12-0700 [-] starting build <Build release-mozilla-beta-linux64_partner_repack>.. pinging the slave <SlaveBuilder builder='release-mozilla-beta-linux64_partner_repack' slave='bld-lion-r5-074'>
2013-07-18 21:44:12-0700 [Broker,41718,10.26.52.94] <Build release-mozilla-beta-linux64_partner_repack>.startBuild
2013-07-18 21:44:41-0700 [-]  <Build release-mozilla-beta-linux64_partner_repack>: build finished

From bm66:
twistd.log.4:2013-07-18 21:44:19-0700 [-] prioritizeBuilders: 0.11s prioritized 1 builder(s): [((0, 0L, 100, 1374209054L), 'release-mozilla-beta-linux64_partner_repack')]
twistd.log.4:2013-07-18 21:44:19-0700 [-] prioritizeBuilders: 0.11s important builder release-mozilla-beta-linux64_partner_repack (p == (0, 0L, 100, 1374209054L))
twistd.log.4:2013-07-18 21:44:19-0700 [-] prioritizeBuilders: 0.11s found 1 important builder(s): ['release-mozilla-beta-linux64_partner_repack']
twistd.log.4:2013-07-18 21:44:20-0700 [-] starting build <Build release-mozilla-beta-linux64_partner_repack> using slave <SlaveBuilder builder='release-mozilla-beta-linux64_partner_repack' slave='bld-lion-r5-080'>
twistd.log.4:2013-07-18 21:44:20-0700 [-] starting build <Build release-mozilla-beta-linux64_partner_repack>.. pinging the slave <SlaveBuilder builder='release-mozilla-beta-linux64_partner_repack' slave='bld-lion-r5-080'>
twistd.log.4:2013-07-18 21:44:20-0700 [Broker,31897,10.26.52.100] <Build release-mozilla-beta-linux64_partner_repack>.startBuild
twistd.log.4:2013-07-18 21:44:47-0700 [-]  <Build release-mozilla-beta-linux64_partner_repack>: build finished

From bm81 (the scheduler):
master/twistd.log.4:2013-07-18 21:44:08-0700 [-] AggregatingScheduler(release-mozilla-beta-linux64_firefox_l10n_done) <id=50825480>: new buildset: branch=releases/mozilla-beta, ssid=5574135, builders: release-mozilla-beta-linux64_partner_repack
master/twistd.log.4:2013-07-18 21:44:14-0700 [-] AggregatingScheduler(release-mozilla-beta-linux64_firefox_l10n_done) <id=50825480>: new buildset: branch=releases/mozilla-beta, ssid=5574136, builders: release-mozilla-beta-linux64_partner_repack
master/twistd.log.3:2013-07-19 01:56:17-0700 [-] AggregatingScheduler(release-mozilla-beta-updates_done) <id=50775824>: new buildset: branch=releases/mozilla-beta, ssid=5575297, builders: release-mozilla-beta-linux_update_verify_1/4, release-mozilla-beta-linux_update_verify_2/4, release-mozilla-beta-linux_update_verify_3/4, release-mozilla-beta-linux_update_verify_4/4, release-mozilla-beta-linux64_update_verify_1/4, release-mozilla-beta-linux64_update_verify_2/4, release-mozilla-beta-linux64_update_verify_3/4, release-mozilla-beta-linux64_update_verify_4/4, release-mozilla-beta-macosx64_update_verify_1/4, release-mozilla-beta-macosx64_update_verify_2/4, release-mozilla-beta-macosx64_update_verify_3/4, release-mozilla-beta-macosx64_update_verify_4/4, release-mozilla-beta-win32_update_verify_1/4, release-mozilla-beta-win32_update_verify_2/4, release-mozilla-beta-win32_update_verify_3/4, release-mozilla-beta-win32_update_verify_4/4
master/twistd.log.3:2013-07-19 01:56:22-0700 [-] AggregatingScheduler(release-mozilla-beta-updates_done) <id=50775824>: new buildset: branch=releases/mozilla-beta, ssid=5575298, builders: release-mozilla-beta-linux_update_verify_1/4, release-mozilla-beta-linux_update_verify_2/4, release-mozilla-beta-linux_update_verify_3/4, release-mozilla-beta-linux_update_verify_4/4, release-mozilla-beta-linux64_update_verify_1/4, release-mozilla-beta-linux64_update_verify_2/4, release-mozilla-beta-linux64_update_verify_3/4, release-mozilla-beta-linux64_update_verify_4/4, release-mozilla-beta-macosx64_update_verify_1/4, release-mozilla-beta-macosx64_update_verify_2/4, release-mozilla-beta-macosx64_update_verify_3/4, release-mozilla-beta-macosx64_update_verify_4/4, release-mozilla-beta-win32_update_verify_1/4, release-mozilla-beta-win32_update_verify_2/4, release-mozilla-beta-win32_update_verify_3/4, release-mozilla-beta-win32_update_verify_4/4


The double "new buildset" entries are pretty indicative of a problem. Makes me suspect either time problems on the scheduler master or dataloss in the db. I'm not smart enough to back either of those up though.

Comment 3

[[e-mail] (:bhearsum)]

This is almost certainly because of bug 894201.

Comment 4

[[e-mail] (:bhearsum)]

http://buildbot-master66:8001/builders/release-mozilla-beta-push_to_mirrors/builds/2 and http://buildbot-master65.srv.releng.usw2.mozilla.com:8001/builders/release-mozilla-beta-push_to_mirrors/builds/5 hit this overnight.

Comment 5

[[e-mail] (:bhearsum)]

Didn't get any duplicate jobs in b9. I'm pretty sure that bug 894201 fixed this.