1313958 - Provide clamav >= v0.98 in our yum repos

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Bug 1313954 boiled down to using an older version of clamav, which is no longer supported and can't load the current virus definitions. This breaks antivirus checks for Thunderbird releases (no longer used for Firefox).

Please provide a yum repo/update our repos with >= v0.98 so that we can generate valid golden AMI av checks. Three packages are needed - clamav, clamav-db, and clamd - and I found v0.99.2 from EPEL6 worked fine.

Comment 1

[Amy Rich [:arr] [:arich]]

We don't have cycles to work on this, but buildduty has been working on package updates and might.

Comment 2

[Chris Cooper [:coop] (he/him)]

We can add it to their queue.

Amy: are there docs for buildduty on updating our yum setup?

Comment 3

[Amy Rich [:arr] [:arich]]

https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Packages

Comment 4

[Alin Selagea [:aselagea]]

It is true that we updated packages placed on the custom repos (which are not mirrored from anywhere). But it seems that the 3 clam* packages mentioned by Nick can be found on EPEL repo:
    - "clam*-0.97.3-3.el6.x86_64.rpm" => repos/yum/mirrors/epel/6/2012-03-07/*
    - "clam*-0.98.1-1.el6.x86_64.rpm" => repos/yum/mirrors/epel/6/2014-05-06/*

One note here would be that the "latest" symlink points to the older folder (2012-03-07/). Grabing an AWS loaner revealed that we indeed have the "0.97" version of those packages installed.
 
https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Packages provides instructions on how to sync the mirrors and my understanding is that it would result in another folder (like 2016-11-15) containing all the latest packages. Given the note above, I would tend to believe that upgrading all the packages is not a good thing since it would cause package version conflicts.

So I'm wondering how to to the upgrade - two solutions came to my mind (may be far from the real solution though):
    - either copy 2012-03-07/ into a new folder and only replace the intended packages with the upgraded version, then switch the symlink to point to that folder
    - create a custom repo and then upgrade puppet manifests 

ni-ing Callek here for some suggestions.

Comment 5

[Amy Rich [:arr] [:arich]]

I suggest you follow the example of things like openssl and create a custom repo.

Comment 6

[Justin Wood (:Callek)]

(In reply to Amy Rich [:arr] [:arich] from comment #5)
> I suggest you follow the example of things like openssl and create a custom
> repo.

++

Comment 7

[Alin Selagea [:aselagea]]

At this point, I created a custom repo at /repo/yum/custom/clamav and included the following rpms:
    - clamav-db-0.99.2-1.el6
    - clamd-0.99.2-1.el6
    - clamav-0.99.2-1.el6
    - clamav-devel-0.99.2-1.el6

However, we don't seem to install them by default on our linux builders. 
@Nick: do you know how to test if this really works so that we don't encounter the same issues as in bug 1313954?

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

I'd suggest spinning up an instance, and pointing it a puppet environment with [1] modified to include your new repo. Then run puppet to make sure it pulls in the updated clamav. You could then reproduce the steps in the most recent log [2] to verify the whole process.

It's a little tricky to spin up an instance, since we haven't implemented loaners for av-linux64. IIRC I made a dev-av-linux64 by copying from dev-linux64, and merging difference between bld-linux64 and av-linux64.

[1] https://hg.mozilla.org/build/puppet/file/default/modules/packages/manifests/setup.pp
[2] https://archive.mozilla.org/pub/thunderbird/candidates/50.0b3-candidates/build1/logs/release-comm-beta-thunderbird_antivirus-bm77-build1-build4.txt.gz

Comment 9

[Alin Selagea [:aselagea]]

I used a custom config file that points to a recent spot-av-linux64 ami and managed to spin up an instance (av-linux64-ec2-aselagea). Going further, I applied the puppet changes to my environment and ran puppet on the loaner => it looks fine, the intended packages are indeed installed.

[[e-mail] ~]# yum list installed | grep clam
clamav.x86_64                      0.99.2-1.el6              @clamav
clamav-db.x86_64                   0.99.2-1.el6              @clamav
clamd.x86_64                       0.99.2-1.el6              @clamav

I created a personal build master on dev-master2, added the entries for it in slavealloc and started it. Also added the loaner to slavealloc and locked it to my master. For some reason, it won't connect to it though.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

I think that's looking pretty great already. You could avoid the "fun" of setting up a master by connecting to the instance and running: 

mkdir -p /builds/slave/tb-rel-c-beta-av-0000000000000/
cd /builds/slave/tb-rel-c-beta-av-0000000000000/
bash -c 'wget -Oarchiver_client.py --no-check-certificate --tries=10 --waitretry=3 https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/utils/archiver_client.py'
bash -c 'python archiver_client.py mozharness --repo releases/mozilla-beta --tag THUNDERBIRD_50_0b3_RELEASE --destination scripts --debug'
python2.7 scripts/scripts/release/antivirus.py --product thunderbird --version 50.0b3 --build-number 1 --bucket-name net-mozaws-prod-delivery-archive --tools-revision THUNDERBIRD_50_0b3_RELEASE --tools-repo https://hg.mozilla.org/build/tools

Please comment here where you got the rpms from. Was it an EPEL6 repo ?

Comment 11

[Alin Selagea [:aselagea]]

(In reply to Nick Thomas [:nthomas] from comment #10)

> Please comment here where you got the rpms from. Was it an EPEL6 repo ?

Yup, I got the rpms from an EPEL6 repo - it's actually the same location you mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=1313954#c3

Comment 12

[Alin Selagea [:aselagea]]

Attachment: log_raw.zip

Used my loaner and ran the commands mentioned by Nick above. Taking a look at the logs revealed that the clam-related errors were no longer present. However, the build ended with an error as there were 3 files for which the command failed:

03:59:15     INFO -  END 03:59:15 (5 seconds elapsed): /builds/slave/tb-rel-c-beta-av-0000000000000/cache/win32/gd/Thunderbird Setup 50.0b3.exe
03:59:15     INFO -  Command failed for the following files:
03:59:15     INFO -    /builds/slave/tb-rel-c-beta-av-0000000000000/cache/update/linux-x86_64/fr/thunderbird-50.0b1-50.0b3.partial.mar
03:59:15     INFO -    /builds/slave/tb-rel-c-beta-av-0000000000000/cache/update/linux-x86_64/fr/thunderbird-50.0b3.complete.mar
03:59:15     INFO -    /builds/slave/tb-rel-c-beta-av-0000000000000/cache/update/linux-x86_64/fi/thunderbird-50.0b3.complete.mar
03:59:15    ERROR - Return code: 1

That doesn't seem related to the new clam* rpms though.

Comment 13

[Alin Selagea [:aselagea]]

Attachment: Bug 1313958 - Provide clamav >= v0.98 in our yum repos;

Review commit: https://reviewboard.mozilla.org/r/97190/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/97190/

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8816439 [details]
Bug 1313958 - Provide clamav >= v0.98 in our yum repos;

https://reviewboard.mozilla.org/r/97190/#review97762

Looks good to me.