794144 - antivirus scan broken on dev-stage01

Status: RESOLVED WONTFIX

(Release Engineering :: Release Automation, defect, P3)

Description

[[e-mail] (:bhearsum)]

command: START
command: ssh -l ffxbld -i /Users/cltbld/.ssh/ffxbld_dsa dev-stage01.srv.releng.scl3.mozilla.com nice ionice -c2 -n7 extract_and_run_command.py -j2 clamdscan -m --no-summary -- /pub/mozilla.org/firefox/nightly/15.0.1-candidates/build1/
command: cwd: /builds/slave/rel-m-rel-av
command: output:
Traceback (most recent call last):
  File "/usr/local/bin/extract_and_run_command.py", line 35, in ?
    from mar import BZ2MarFile
  File "/home/ffxbld/checkouts/tools.clean/stage/../buildfarm/utils/mar.py", line 13, in ?
    import struct, os, bz2, hashlib, tempfile
ImportError: No module named hashlib
command: ERROR
Traceback (most recent call last):
  File "/builds/slave/rel-m-rel-av/scripts/scripts/release/../../lib/python/util/commands.py", line 42, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python-2.7.2/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['ssh', '-l', 'ffxbld', '-i', '/Users/cltbld/.ssh/ffxbld_dsa', 'dev-stage01.srv.releng.scl3.mozilla.com', 'nice', 'ionice', '-c2', '-n7', 'extract_and_run_command.py', '-j2', 'clamdscan', '-m', '--no-summary', '--', '/pub/mozilla.org/firefox/nightly/15.0.1-candidates/build1/']' returned non-zero exit status 1
command: END (0.29s elapsed)

Traceback (most recent call last):
  File "/builds/slave/rel-m-rel-av/scripts/scripts/release/stage-tasks.py", line 226, in <module>
    buildNumber=buildNumber)
  File "/builds/slave/rel-m-rel-av/scripts/scripts/release/stage-tasks.py", line 96, in runAntivirusCheck
    sshKey=stageSshKey)
  File "/builds/slave/rel-m-rel-av/scripts/scripts/release/../../lib/python/util/commands.py", line 60, in run_remote_cmd
    run_cmd(cmd_prefix + cmd, **kwargs)
  File "/builds/slave/rel-m-rel-av/scripts/scripts/release/../../lib/python/util/commands.py", line 42, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python-2.7.2/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ssh', '-l', 'ffxbld', '-i', '/Users/cltbld/.ssh/ffxbld_dsa', 'dev-stage01.srv.releng.scl3.mozilla.com', 'nice', 'ionice', '-c2', '-n7', 'extract_and_run_command.py', '-j2', 'clamdscan', '-m', '--no-summary', '--', '/pub/mozilla.org/firefox/nightly/15.0.1-candidates/build1/']' returned non-zero exit status 1


I worked around this temporarily with this diff in the tools checkout:
[root@dev-stage01 tools]# /tools/python/bin/hg diff
diff -r 755e58ebc9d4 stage/extract_and_run_command.py
--- a/stage/extract_and_run_command.py	Mon Sep 24 16:39:52 2012 -0700
+++ b/stage/extract_and_run_command.py	Tue Sep 25 10:12:27 2012 -0700
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/tools/python/bin/python
 """\
 Usage: extract_and_run_command.py [-j N] [command to run] -- [files and/or directories]
     -j is the number of workers to start, defaulting to 1.

Comment 1

[[e-mail] (:bhearsum)]

Extremely temporary workaround for this is to locally modify /usr/local/bin/clamdscan to point at /tools/python/bin/python. Puppet overrides it every 30min though.

Comment 2

[[e-mail] (:bhearsum)]

Rail and I talked about this and couldn't figure out an easy fix. A couple of options:
- Upgrade the system to CentOS 6 (or replace it with a newly built VM)
- Change the default python to 2.5 or higher.

Comment 3

[[e-mail] (:bhearsum)]

Moving to S3 is going to make this not matter.