751455 - Adjust virus scanning for new stage servers

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Both post_upload.py and extract_and_run_command.py (from tools/stage) are living in sysadmins repo on the new upload servers (at opsec's request). The extract script needs to import mar.py, which it does by looking at the right relative-path in a tools checkout:

sys.path.append(path.join(path.dirname(path.realpath(__file__)), "../buildfarm/utils"))
...
from mar import BZ2MarFile

Since we no longer have a tools checkout I propose we get puppet to manage mar.py too, putting it in /usr/lib64/python2.6/site-packages (on sys.path), and then drop the sys.path modification.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

The files I mentioned live at: 
http://hg.mozilla.org/build/tools/file/default/stage/extract_and_run_command.py
http://hg.mozilla.org/build/tools/file/default/buildfarm/utils/mar.py

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

I committed rev 36509 to add mar.py to /usr/lib64/python2.6/site-packages/, but it hasn't appeared on upload1 in the 50 minutes since. If puppet is erroring then this is a good thing to check on.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

A virus scan works with the change in comment #2, plus installing p7zip (to provide 7za, also in puppet). 1h 18 minutes elapsed on upload1, compared to 3hrs 14 to do the virus scan + dummy rsync for 13.0b2 (ie more work, but not many minutes worth).

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] Add header block to help people with deployment

The idea with these comment blocks is to help us remember that builds/tools is the canonical location for these scripts, but that they're managed by puppet on stage.m.o. I put this after the docstrings because that seemed vaguely better, but means it doesn't always go right at the very top of the file. Looks like you can do the other way around if that's seems better to you.

Comment 5

[[e-mail] (:bhearsum)]

Comment on attachment 623624 [details] [diff] [review]
[tools] Add header block to help people with deployment

Review of attachment 623624 [details] [diff] [review]:
-----------------------------------------------------------------

This seems good enough to me.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 623624 [details] [diff] [review]
[tools] Add header block to help people with deployment

http://hg.mozilla.org/build/tools/rev/2fd823c26d35

Sending        bin/extract_and_run_command.py
Sending        bin/mar.py
Sending        bin/post_upload.py
Transmitting file data ...
Committed revision 37822.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

All done here.