1499077 - reduce downloads from tooltool on occ instances

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: OpenCloudConfig, task)

Description

[Rob Thijssen [:grenade (EET/UTC+0300)]]

in occ commit https://github.com/mozilla-releng/OpenCloudConfig/commit/3876c69,
download source of procmon and procexp was moved from github to tooltool (by adding the sha512 hash), however because an occ checksum download component is used instead of a regular download component, the file is downloaded from tooltool on every instance reboot, causing uneccesary load on the tooltool servers.

change the component to a regular download, to prevent such frequent downloads.

Comment 1

[Rob Thijssen [:grenade (EET/UTC+0300)]]

having looked at the code, i believe i was mistaken about the cause of the problem.

the component (a ZipInstall) is actually checking for pre-existence of the file being downloaded and skipping the download if it's already on the filesystem.

the excessive downloads are probably caused by the change to the sha512 hash in occ manifests without an ami redeploy. each newly created ec2 instance (for worker types that have not had an ami deploy since the change) may be downloading the artifact on first boot instead of finding the artifact already downloaded inside its base ami.

if this is the case, it should be fixed (downloads reduced to just those that occur on ami creation instances) by generating new amis for any worker types that haven't been redeployed in the last week.

Comment 2

[Rob Thijssen [:grenade (EET/UTC+0300)]]

ami redeploy in progress.

- https://github.com/mozilla-releng/OpenCloudConfig/commit/c7fee73
- https://tools.taskcluster.net/task-group-inspector/#/FI-AtiWYRuG_gO8cm-VZ3w

no change to occ or worker manifests, just redeploying (by pushing with commit syntax) so that base amis in each region will already contain the procmon and procexp artifacts on the base ami filesystem so that individual instances can skip downloading of these artifacts.

Comment 3

[Rob Thijssen [:grenade (EET/UTC+0300)]]

catlee: are we seeing any higher than expected load on tooltool buckets? i believe the issue we discussed last month (https://mozilla.logbot.info/ci/20181015#c15466612) should be resolved. please let me know if that's not the case.

Comment 4

[Chris AtLee [:catlee]]

the tooltool hit rate has returned to normal levels, so I think we're good here.