1259423 - freshclam fails to update the DB

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Rail Aliiev [:rail]]

See https://tools.taskcluster.net/task-inspector/#cifkVsTBSbGDL9TvVxgj2g/0

Comment 1

[Kim Moir [:kmoir] ET]

I pulled down the image locally and was able to run the command to update the db.  Perhaps a intermittent network issue? Or we can't connect to this db address from the container when it runs in the build.

Comment 2

[Rail Aliiev [:rail]]

Yeah, it's intermittent. Probably related to the load on the clamav mirrors.

Some ideas:

1) we can regenerate the images regularly
 * pros: no need to run freshclam
 * cons: need to use ":latest" in the tasks or use some service to tell the last good sha256; need to schedule this

2) don't fail if freshclam fails. Probably the easiest way to go. Failures are not frequent, so it won't probably hurt.

Comment 3

[Rail Aliiev [:rail]]

We can probably refresh the images as a separate task and publish the images as artifacts, but I'm not sure how to use them

Comment 4

[Kim Moir [:kmoir] ET]

Yeah, I looked at how to run a commmand in docker (freshclam) and not fail starting the image if it failed but didn't find a way to do that.

Comment 5

[Rail Aliiev [:rail]]

another idea:

1) create a task to generate an image and use dockerSave feature (see http://docs.taskcluster.net/workers/docker-worker/)

2) reference the image as in https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/tests/eslint-gecko.yml#11-14. Not sure if the dependencies are tracked automatically

Comment 6

[Rail Aliiev [:rail]]

https://gist.github.com/rail/fae3cf4b1eccee3d8ecd may be something that would address the "don't fail if freshclam fails". I haven't tested it though.

Comment 7

[Kim Moir [:kmoir] ET]

Attachment: bug1259423.patch

I tested it and it works

Comment 8

[Rail Aliiev [:rail]]

Comment on attachment 8735490 [details] [diff] [review]
bug1259423.patch

I think we also need to update all beetmover calls in releasetasks and add "--refresh-antivirus". Would be great to have add a test to catch beetmover tasks without "--refresh-antivirus" passed

Comment 9

[Kim Moir [:kmoir] ET]

We don't need to update the beetmover calls in releasetasks.  When I tested it, I just used an existing taskid and the same call to mh (with a bogus bucket/credentials).  Beetmover is just called like this via releasetasks:

python scripts/release/beet_mover.py --template configs/beetmover/source.yml.tmpl --platform linux64 --version {{ version }} --locale en-US --taskid {{ stableSlugId(buildername) }} --build-num build{{ buildNumber }} --bucket {{ beetmover_candidates_bucket }}

The specific mh actions aren't referenced in the call.

Comment 10

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/2f84b90500fe

Comment 11

[Kim Moir [:kmoir] ET]

We need a new image for rail/funsize-update-generator too but perhaps this should be part of bug 1259425

Comment 12

[Rail Aliiev [:rail]]

(In reply to Pulsebot from comment #10)
> https://hg.mozilla.org/integration/mozilla-inbound/rev/2f84b90500fe

Can you uplift this patch also to aurora, beta, and esr45?

(In reply to Kim Moir [:kmoir] from comment #11)
> We need a new image for rail/funsize-update-generator too but perhaps this
> should be part of bug 1259425

We can tackle that separately. Updated the image:

Digest: sha256:d1c88af7f13d124e08a37d10203e9e60655a231301123e5f2127fab5468ef5dd

Comment 13

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/2f84b90500fe

Comment 14

[Rail Aliiev [:rail]]

keeping open to address the remaining issues.

Comment 15

[Kim Moir [:kmoir] ET]

Attachment: bug1259423beta_and_esr45.patch

other changes were needed to be added for beta and esr45 since funsize files were missing there. 

I landed the relevant changes on m-a and m-b but m-esr45 is closed now so will try again later

Comment 16

[Kim Moir [:kmoir] ET]

merged this pull request and landed changes on mozilla-esr45 too now that it is open