1462429 - Thunderbird 52.8.0 build1 - no notification of steps/progress after manual rerun of repack 4, which finished two hours ago

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Wayne Mery (:wsmwk)]

"completed repack_4/10 on win32" two hours ago, 12:10pm EST. (I had manually rerun the repack after it failed with an exception the previous night.)

No release automation mail since then.

Comment 1

[Jordan Lund (:jlund)]

jlund looks at tb
15:08:34 
<•nthomas> win32 repack 6/10 seems to be awol on the mailing list, neither complete nor failed
15:08:49 
<wsmwk> i could point out that we don't normally get exception status.  so perhaps that is related
15:08:58 nthomas: hmm. i missed that
15:09:18 
<•jlund> Jordan Lund yeah, can't see it in recent
15:10:02 
<•nthomas> zombie pending/running ?
15:10:05 
<•jlund> Jordan Lund [release] Thunderbird 52.8.0 build1: completed repack_6/10 on win32
15:10:11 found it in email
15:10:29 
<•nthomas> oh, much earlier. that'll teach me to trust my eyeballs
15:11:18 no '[release] Thunderbird 52.8.0 build1: All win32 builds now available' though
15:11:37 
<•jlund> Jordan Lund no win64 either
15:12:04 
<•nthomas> don't think that's enabled for TB
15:12:05 
<•jlund> Jordan Lund no win64 at all
15:12:07 ah
15:12:19 
<•nthomas> confirmed - https://archive.mozilla.org/pub/thunderbird/releases/52.7.0/
15:12:54 
<wsmwk> we dont have win64
15:13:13 
<•jlund> Jordan Lund so the fact that 4 failed originally, it seems related
15:13:23 wonder if rerun was triggered correctly
15:13:31 
<•nthomas> I vaguely recall we have an AggregatingScheduler that sends the all now available message, which runs on the scheduler masters
15:13:43 was wondering that too
15:13:52 sometimes a reconfig can confuse things too
15:14:04 not that we do those often now
15:14:39 
<•jlund> Jordan Lund we killed a bunch of comm-beta and comm-central builders since last 52.0 tb release
15:14:58 but can't see anything wrong with those commits or how they would affect 52*
15:16:31 http://buildbot-master73.bb.releng.usw2.mozilla.com:8001/builders/release-comm-esr52-win32_repack_4%2F10/builds/6/steps/run_script/logs/stdio
15:16:46 hm, rerun was green but end of log suggests it shouldn't have exited 0
15:17:07 
<•nthomas> ah
15:17:26 
<•jlund> Jordan Lund still, not sure how that would block the graph
15:18:06 anyway, there is something new about 'gl' locale no?
15:18:12 `Couldn't find locale identified by: Thunderbird-52.8.0-build1, WINNT_x86-msvc, gl`
15:20:12 
<•nthomas> looks like that occurs for other locales in the same log too
15:20:51 
<•jlund> Jordan Lund I'm going to rerun while I look
15:20:59 
<•nthomas> sounds good
15:21:04 there's WINNT data in https://aus4-admin.mozilla.org/releases#Thunderbird-52.8.0
15:21:29 even for locales like fr and fy-NL in that rerun job
15:21:59 
<•jlund> Jordan Lund http://buildbot-master73.bb.releng.usw2.mozilla.com:8001/builders/release-comm-esr52-win32_repack_4%2F10/builds/7

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

That rerun was green too, but still no notification of 'all win32 builds available' (or similar), which points the finger at the buildbot schedulers. The build scheduler was paying attention until:

2018-05-17 06:40:19-0700 [-] AggregatingScheduler(release-comm-esr52-thunderbird_deliverables_ready) <id=139686839497216>: new builds: ((u'release-comm-esr52-linux64_repack_complete', 155868479L, 1526529210L),) since 1526529150 (lastCheck: 1526529210, lastReset: 1526506538.17)

then no more matching lines. It was having trouble digesting a merge into jamun with many changesets at the same time. Restarting buildbot unwedged it and we got the email about all win32 builds, and downstreams started.

The test scheduler wasn't very happy, throwing many exceptions trying to read a scheduler state from the db. Also corrected by restarting buildbot.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Forgot to mention I did verify the win32 gl files were on archive.m.o, with sensible timestamps (via bucketlister). And that Balrog had the right hashes for those, so the log lines we reacted to earlier were red herrings.