Closed Bug 1303153 Opened 9 years ago Closed 9 years ago

Analyze usage of S3 storage for artifacts

Categories

(Taskcluster :: General, defect)

Product:
Taskcluster
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dustin, Assigned: dustin)

References

Details

We probably have some try artifact kicking around from back in the day when we were keeping them for >1 year. We should torch those.
Some preliminary analysis by branch: 18.29% done category seen projected gaia-try 0.00B 0.00B try-taskcluster 0.00B 0.00B bugzilla/bugzilla-5_0 312.17KB 1.66MB bugzilla-4_2 680.91KB 3.63MB community-builds 1.87MB 10.26MB buildbot-bridge 7.37MB 40.32MB mozilla-bteam/bmo-upstream-merge 13.66MB 74.72MB mozilla-bteam/bmo-development 13.67MB 74.76MB mozilla-b2g/gaia-master 21.11MB 115.46MB bugzilla-4_4 27.38MB 149.77MB bmo-development 33.80MB 184.88MB bugzilla/bugzilla-master 57.85MB 316.38MB bugzilla-5_0 75.98MB 415.51MB mozilla-bteam/bmo-master 101.35MB 554.26MB mozilla-b2g/gaia 115.87MB 633.67MB bmo-upstream-merge 184.32MB 1007.97MB bmo-master 231.14MB 1.23GB bugzilla-master 295.66MB 1.57GB alder 447.85MB 2.39GB cratertest 884.59MB 4.72GB gecko-bb 1013.68MB 5.41GB alder-bb 1.91GB 10.45GB nss-try 2.79GB 15.27GB pine-bb 2.83GB 15.50GB buildbot-bb 3.07GB 16.81GB nss 4.86GB 26.60GB stylo-try 19.41GB 106.19GB mshal-testing 26.50GB 144.94GB mozilla-esr45 28.78GB 157.41GB elm-bb 35.04GB 191.66GB cedar 46.82GB 256.08GB stylo 54.51GB 298.14GB gaia-master 75.08GB 410.59GB b2g-ota 78.90GB 431.50GB elm 85.37GB 466.90GB cedar-bb 96.14GB 525.77GB jamun 114.64GB 626.94GB larch-bb 114.76GB 627.62GB mshal-testing-bb 116.37GB 636.39GB larch 129.22GB 706.68GB ash 134.03GB 732.98GB oak 144.69GB 791.27GB mozilla-esr45-bb 149.43GB 817.23GB date 161.53GB 883.37GB mozilla-release 225.56GB 1.20TB mozilla-b2g44_v2_5 231.37GB 1.23TB gaia 328.07GB 1.75TB oak-bb 329.11GB 1.75TB pine 461.09GB 2.46TB mozilla-release-bb 493.05GB 2.63TB ash-bb 606.78GB 3.24TB jamun-bb 650.44GB 3.47TB date-bb 1.09TB 5.97TB mozilla-beta 1.23TB 6.72TB b2g-inbound-bb 1.99TB 10.89TB mozilla-aurora 2.54TB 13.93TB mozilla-beta-bb 2.58TB 14.13TB autoland-bb 3.98TB 21.81TB unknown 4.12TB 22.57TB mozilla-central 6.07TB 33.20TB b2g-inbound 7.35TB 40.21TB mozilla-central-bb 8.73TB 47.78TB fx-team-bb 13.18TB 72.11TB mozilla-aurora-bb 13.94TB 76.24TB autoland 14.11TB 77.20TB fx-team 19.82TB 108.42TB s3-only 24.61TB 134.59TB try 34.57TB 189.05TB mozilla-inbound-bb 44.18TB 241.60TB try-bb 56.86TB 310.98TB mozilla-inbound 62.52TB 341.91TB total projected size: 1.75PB The projection is based on the reasonable assumption that the remaining 82% of the data gathered will be similar to this 18%. The "*-bb" branches are uploads via mozharness (c.f. bug 1303214). "s3-only" is probably inaccurate right now, so please ignore. All of this data is for tasks over 14 days old. I imagine we can shrink try by 100% and try-bb by 90%. If we turned down retention on autoland and inbound to 14 days as well, we could shrink both of those by 90%. From that, the savings would be about 1PB, well over $25k/mo. That's all very back-of-the-envelope, but I'm quite confident we can cut our S3 storage cost in half without much difficulty.
Script I'm using to gather task categorizations, and S3 sizes (this is stuck inside queue's src/main.js): let scan = async ({cfg, Artifact, Task}) => { let route_pattern = /^(index\.garbage\.staging|index\.buildbot\.branches|index\.gecko\.v[12]|tc-treeherder(-stage)?(\.v2)?)\.([^.]*)\..*/; let tcbranch = (task) => { let branch; task.routes.forEach((r) => { let res = route_pattern.exec(r); if (res) { branch = res[4]; } }); return branch; }; // more than 14 days old let before = new Date(new Date().getTime() - 15 * 24 * 3600 * 1000); await Task.scan({ created: base.Entity.op.greaterThan(before), }, { limit: 100, handler: async (task) => { let branch; if (task.workerType == 'cratertest') { branch = 'cratertest'; } else if (task.workerType == 'buildbot-bridge') { branch = 'buildbot-bridge'; } else { branch = tcbranch(task); if (branch && task.provisionerId === 'null-provisioner') { branch = branch + '-bb'; } } if (!branch) { branch = 'unknown'; } console.log(task.taskId, branch, task.created.toJSON()); }, }); }; let scans3 = async ({cfg, publicArtifactBucket}) => { let s3 = publicArtifactBucket.s3; let params = {}; let task = {}; while (1) { let res = await s3.listObjects(params).promise(); let lastKey; res.data.Contents.forEach((obj) => { let taskId = obj.Key.slice(0, 22); if (taskId != task.taskId) { if (task.taskId) { console.log(task.taskId, task.bytes); } task.taskId = taskId; task.bytes = 0; } task.bytes += obj.Size; lastKey = obj.Key; }); if (!res.data.IsTruncated) { return; } if (res.data.NextMarker) { params.Marker = res.data.NextMarker; } else { params.Marker = lastKey; } }; console.log(task.taskId, task.bytes); };
Summary: manually delete old try job artifacts → Analyze usage of S3 storage for artifacts
I think that the projections were off by a constant factor, although i haven't figured out what. I now have data for all tasks in the last year, and the new per-branch analysis is branch seen gaia-try 596.00B bugzilla/bugzilla-4_4 21.77KB bugzilla-4_2 2.29MB bugzilla/bugzilla-5_0 4.45MB community-builds 6.09MB buildbot-bridge 23.74MB mozilla-bteam/bmo-upstream-merge 26.32MB bugzilla-4_4 75.15MB mozilla-b2g/gaia-master 92.59MB bmo-development 182.14MB mozilla-bteam/bmo-development 185.09MB mozilla-b2g/gaia 270.86MB bugzilla-5_0 295.34MB mozilla-bteam/bmo-master 422.51MB bugzilla/bugzilla-master 516.41MB bmo-upstream-merge 544.29MB bmo-master 578.38MB bugzilla-master 808.15MB alder 1.41GB gecko-bb 2.31GB cratertest 2.54GB alder-bb 4.21GB buildbot-bb 6.74GB pine-bb 7.17GB nss 39.99GB nss-try 42.74GB stylo-try 65.05GB mshal-testing 81.38GB mozilla-esr45 105.01GB elm-bb 113.10GB cedar 120.03GB gaia-master 223.43GB b2g-ota 224.33GB cedar-bb 242.79GB stylo 270.15GB mshal-testing-bb 282.31GB elm 288.90GB larch-bb 350.50GB jamun 428.39GB date 478.56GB mozilla-esr45-bb 514.95GB larch 515.43GB ash 524.58GB 571.28GB oak 572.59GB mozilla-b2g44_v2_5 704.64GB gaia 960.48GB oak-bb 1.04TB mozilla-release 1.04TB pine 1.47TB mozilla-release-bb 1.68TB ash-bb 1.81TB jamun-bb 2.50TB date-bb 3.12TB mozilla-beta 3.93TB b2g-inbound-bb 5.77TB mozilla-beta-bb 7.71TB mozilla-aurora 8.76TB autoland-bb 13.80TB mozilla-central 19.75TB b2g-inbound 21.65TB unknown 23.37TB mozilla-central-bb 26.79TB fx-team-bb 39.52TB mozilla-aurora-bb 42.51TB autoland 52.50TB fx-team 61.00TB try 117.82TB mozilla-inbound-bb 131.41TB try-bb 172.14TB mozilla-inbound 195.40TB
So, those numbers add up to 964TB, no the 1.59PB that CloudHealth counts. I've verified that the sum of all of the object sizes as recorded by scans3, above, is really 964TB. AWS Cloudwatch says 1,804,605,437,027,661 bytes, though, which is 1.60PB, matching CloudHealth. Where are those extra 600TB?? I've started a second run of the listObjects iteration (code is above) to see if it produces substantially different results from above.
I just re-ran this, just to see if something funny happened with the listObjects run. (v0.12.9) dustin@lamport ~/p/taskcluster-queue [master*] $ NODE_ENV=production node lib/main.js scans3 > task-sizes2.txt (v0.12.9) dustin@lamport ~/p/taskcluster-queue [master*] $ ls -al task-sizes* -rw-rw-r-- 1 dustin dustin 561469540 Sep 19 07:00 task-sizes2.txt -rw-rw-r-- 1 dustin dustin 565532329 Sep 17 11:45 task-sizes.txt (v0.12.9) dustin@lamport ~/p/taskcluster-queue [master*] $ python Python 2.7.12 (default, Jul 1 2016, 15:12:24) [GCC 5.4.0 20160609] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> total = 0 >>> for l in open("task-sizes2.txt"): ... size = int(l.strip().split(' ')[1]) ... total += size ... >>> total 1056927727495201 >>> total / (1024 ** 4) 961 So 961TB, again 600TB short of what AWS says we have.
After talking with Richard, it was suggested that perhaps the discrepancy could come from versioning being enabled on the bucket. I have confirmed that it is indeed enabled. I'm not sure why we have versioning enabled, but have pinged others to find out. If we determine that this size difference is indeed caused by versioning and that we do not need it, we could come up with a lifecycle plan or a script to remove older versions.
Yep, that's the ticket. We have never, ever, deleted a byte of artifacts. I've restarted the script to get S3 object sizes, adjusted to use listObjectVersions and to count delete markers against the storge total too (the key length). That will probably take another two days to run. That said if we assume that we can turn off versioning and delete all of the already-deleted data, then we save 600TB. That's 37% of the storage used for this bucket, for which we are paying about $45k/mo, so savings would amount to about $17k/mo. Based on the numbers in comment 3, we can add to that savings $3k for try, $4k for try-bb, and $10k for the integration branches (a conservative 90% savings by deleting everything between 1 year and 14 days old). That's about $34k/mo.
We should also think about using lifecycle to ensure deletions, and possibly moving objects to IA after some time period.
Let's add a lifecycle policy to permanently delete-markers/previous-versions after 1 day. The queue won't serve artifacts after it has deleted them. So IMO disabling versioning would be fine, having things around 1 day after they are deleted might be smart as we can restore them in case of something really bad happening.
those followup actions got their own bugs, chained off the parent of this one
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.