1635321 - 77.0b1 build2 partial generation error for mac tr, in green job

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

ie https://firefoxci.taskcluster-artifacts.net/aChpVxEcQze8Y8dKMeh8yw/0/public/logs/live_backing.log

2020-05-04 20:42:43,004 - WARNING - target-76.0b6.partial.mar: /tmp/tmp8exnbf5c/common.sh: line 215: temp-dirlist: No such file or directory
2020-05-04 20:42:43,004 - WARNING - target-76.0b6.partial.mar: rm: cannot remove 'temp-dirlist': No such file or directory
2020-05-04 20:42:43,007 - WARNING - target-76.0b7.partial.mar: /tmp/tmp8exnbf5c/common.sh: line 215: temp-dirlist: No such file or directory
2020-05-04 20:42:43,007 - WARNING - target-76.0b7.partial.mar: rm: cannot remove 'temp-dirlist': No such file or directory
2020-05-04 20:42:43,011 - WARNING - target-76.0b8.partial.mar: rm: cannot remove 'temp-filelist': No such file or directory

This had exit status 0 so the graph continued.

Later there were update verify issues (https://firefoxci.taskcluster-artifacts.net/Al0Xz4KKSg2h8PKJiS0Emg/0/public/logs/live_backing.log)

[task 2020-05-04T21:24:02.866Z] FINISH PATCH Contents/MacOS/crashreporter.app/Contents/Info.plist
[task 2020-05-04T21:24:02.866Z] FINISH PATCH Contents/MacOS/XUL
[task 2020-05-04T21:24:02.866Z] FINISH PATCH Contents/Library/LaunchServices/org.mozilla.updater
[task 2020-05-04T21:24:02.866Z] FINISH PATCH Contents/Info.plist
[task 2020-05-04T21:24:02.866Z] FINISH ADD Contents/CodeResources
[task 2020-05-04T21:24:02.866Z] succeeded
[task 2020-05-04T21:24:02.867Z] calling QuitProgressUI
[task 2020-05-04T21:24:03.887Z] Comparing source/Firefox.app with target/Firefox.app...
[task 2020-05-04T21:24:03.887Z] Files only in target/Firefox.app:
[task 2020-05-04T21:24:03.887Z]   Contents/MacOS/liboggwasm.dylib
[task 2020-05-04T21:24:03.887Z] TEST-UNEXPECTED-FAIL: differences found after update
[task 2020-05-04T21:24:03.888Z] TEST-UNEXPECTED-FAIL: [76.0 tr partial] check_updates returned failure for Darwin_x86_64-gcc3-u-i386-x86_64 downloads/Firefox 76.0b8.dmg vs. downloads/Firefox 77.0b1.dmg: 1

and similar for 76.0b6 and b7 partials (log, log).

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

We opted to do a build3 instead of rerunning ~10 tasks and cache busting the files in the candidates directory.

I can't tell from the log why only Contents/MacOS/liboggwasm.dylib was left out - that's a new file in the 77 cycle - but that's the only difference in updatev3.manifest between the build2 and build3 partials.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Simon, do you have any ideas on what happened here, and how we can avoid it in future ?

Comment 3

[Simon Fraser [:sfraser] ⌚️GMT]

This looks like make_incremental_update.sh and common.sh not being completely safe to run more than once with the same target directory - it's hardcoding some temporary file names. I'll swap to mktemp and submit a patch.

Comment 4

[Simon Fraser [:sfraser] ⌚️GMT]

Attachment: Bug 1635321 update-packaging race condition fixes

Since we now only download the 'to' mar once, temp-filelist and temp-dirlist were created, filled and then removed in the same directory. This is a safer way.

Comment 5

[Simon Fraser [:sfraser] ⌚️GMT]

Attachment: Bug 1635321 update-packaging race condition fixes

conflicts with D75072, we should choose which we prefer based on bash version availability

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d711609b40a5
update-packaging race condition fixes r=nthomas

Comment 7

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/d711609b40a5

Comment 8

[Julien Cristau [:jcristau] (back August 18)]

https://hg.mozilla.org/releases/mozilla-release/rev/74c70d8f058a