812947 - thunderbird mac da 15.0.1 -> 17.0 partial broken

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, defect)

Description

[Kim Moir [:kmoir] ET]

END 20:15:09 (12 seconds elapsed): /pub/mozilla.org/thunderbird/nightly/17.0-candidates/build2/win32/zh-TW/Thunderbird Setup 17.0.exe

Command failed for the following files:
  /pub/mozilla.org/thunderbird/nightly/17.0-candidates/build2/update/mac/da/thunderbird-15.0.1-17.0.partial.mar
command: ERROR
Traceback (most recent call last):
  File "/builds/slave/tb-rel-c-rel-av/scripts/scripts/release/../../lib/python/util/commands.py", line 40, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python-2.5.1/lib/python2.5/subprocess.py", line 461, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['ssh', '-l', 'tbirdbld', '-i', '/home/cltbld/.ssh/tbirdbld_dsa', 'stage.mozilla.org', 'nice', 'ionice', '-c2', '-n7', 'extract_and_run_command.py', '-j2', 'clamdscan', '-m', '--no-summary', '--', '/pub/mozilla.org/thunderbird/nightly/17.0-candidates/build2/']' returned non-zero exit status 1
command: END (4762.32s elapsed)

Traceback (most recent call last):
  File "/builds/slave/tb-rel-c-rel-av/scripts/scripts/release/stage-tasks.py", line 232, in <module>
    buildNumber=buildNumber)
  File "/builds/slave/tb-rel-c-rel-av/scripts/scripts/release/stage-tasks.py", line 98, in runAntivirusCheck
    sshKey=stageSshKey)
  File "/builds/slave/tb-rel-c-rel-av/scripts/scripts/release/../../lib/python/util/commands.py", line 59, in run_remote_cmd
    return run_cmd(cmd_prefix + cmd, **kwargs)
  File "/builds/slave/tb-rel-c-rel-av/scripts/scripts/release/../../lib/python/util/commands.py", line 40, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python-2.5.1/lib/python2.5/subprocess.py", line 461, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ssh', '-l', 'tbirdbld', '-i', '/home/cltbld/.ssh/tbirdbld_dsa', 'stage.mozilla.org', 'nice', 'ionice', '-c2', '-n7', 'extract_and_run_command.py', '-j2', 'clamdscan', '-m', '--no-summary', '--', '/pub/mozilla.org/thunderbird/nightly/17.0-candidates/build2/']' returned non-zero exit status 1
program finished with exit code 1
elapsedTime=4775.115991

Comment 1

[[e-mail] (:bhearsum)]

Strangely, I don't see the script actually trying to do anything with that file. These are the only references to it:
START 19:52:58: /pub/mozilla.org/thunderbird/nightly/17.0-candidates/build2/update/mac/da/thunderbird-15.0.1-17.0.partial.mar.asc
/pub/mozilla.org/thunderbird/nightly/17.0-candidates/build2/update/mac/da/thunderbird-15.0.1-17.0.partial.mar.asc: OK
END 19:52:58 (0 seconds elapsed): /pub/mozilla.org/thunderbird/nightly/17.0-candidates/build2/update/mac/da/thunderbird-15.0.1-17.0.partial.mar.asc


However, update verify for that MAR failed with:
SOURCE DIRECTORY ../../update
DESTINATION DIRECTORY .
UPDATE TYPE partial
PREPARE ADD precomplete
PREPARE PATCH Contents/_CodeSignature/CodeResources
PREPARE PATCH Contents/MacOS/updater.ini
PREPARE PATCH Contents/MacOS/updater.app/Contents/MacOS/updater
PREPARE PATCH Contents/MacOS/update-settings.ini
PREPARE PATCH Contents/MacOS/thunderbird-bin
PREPARE PATCH Contents/MacOS/thunderbird
PREPARE PATCH Contents/MacOS/searchplugins/wikipedia-da.xml
PREPARE PATCH Contents/MacOS/searchplugins/amazon-co-uk.xml
PREPARE PATCH Contents/MacOS/plugin-container.app/Contents/MacOS/plugin-container
PREPARE PATCH Contents/MacOS/platform.ini
PREPARE PATCH Contents/MacOS/omni.ja
failed: 8
calling QuitProgressUI
FAIL: update status was not succeeded: failed: 8

Which confirms we have a broken MAR for whatever reason. At this point I'm going to suggest that we simply remove the partial snippet + MAR rather than try to fix this, as it's not worth our time for a single broken from 15.0.1. Kim, can you relay this to the tb-drivers list to make sure this is an OK course of action before proceeding?

Comment 2

[[e-mail] (:bhearsum)]

Attachment: logs from repack

I don't see anything out of the ordinary in these logs.

Comment 3

[[e-mail] (:bhearsum)]

I tried to find an unsigned version of this file to see if it was broken out of box, or only after it was signed. It had already been deleted from the signing server.

cc'ing nick, because he often has insight into issues like this.

Comment 4

[Kim Moir [:kmoir] ET]

Standard8 

Since this issue only relates to one update from 15.0.1 to 17.0, we're thinking it would make sense just to delete the files related to this MAR, not rebuild for this issue.

This is just a corrupt MAR file, not related to a real virus scan issue. 

Thoughts?

Comment 5

[Mark Banner (:standard8)]

Yes, lets drop that file, we don't have many users on 15.0.1 now anyway and I wasn't really expecting partials from 15.0.1 either.

Comment 6

[Kim Moir [:kmoir] ET]

The files have been deleted.  I'll start the push to mirrors for Thunderbird 17.0 once the one for FF has completed.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

I did some checking around but didn't get to the bottom of it. Update verify checks the mar file matches the hash and size in the update.xml, and it fails later than that so we know the mar was bad somewhere on the slave. After it's signed everything is handled consistently so probably before that. The mar utility doesn't have any trouble unpacking the file, so this smells like some problem with the omni.ja patch itself. From a quick read of the updater source I can't see how we'd get a return value of 8 in an update verify log, because that's now deprecated in favor of more finegrained errors (in toolkit/mozapps/update/ see: common/errors.h, updater/updater.cpp, updater/archivereader.cpp).

bld-lion-r5-053 doesn't have a particularly orange or red build history, and the slave tracking is fairly clean - just a reimage early November.

So lets spin the BOFH excuse wheel of fortune and come up with .... static from plastic slide rulers.