Closed Bug 1530728 Opened 5 years ago Closed 5 years ago

audit all types of releases associated artifacts

Categories

(Release Engineering :: General, enhancement, P1)

Product:
Release Engineering
Component:
General
Type:
enhancement

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mtabara, Assigned: mtabara)

References

(Depends on 1 open bug)

Details

Attachments

(2 files)

List of artifacts that we publish alongside a normal release
4.69 KB, patch
Details | Diff | Splinter Review
Bug 1530728 - remove txt files as part of the release artifacts cleanup. r=catlee
47 bytes, text/x-phabricator-request
Details | Review

TC migration project completed last year and in the meantime we had several attempts to glance at the artifacts we're actually transferring for various types of releases in the gecko family. Given some of the boilerplate code that is shared in parts of the beetmover's transforms (tool that taslk to TC and S3 in order to ensure the pretty naming + transferring of the files) we suspect that there might have been accidental unneeded artifacts across different types of releases (e.g. nightly artifacts that have been naturally ported to beta/release without a real need of that beforehand, no longer used artifacts (e.g. awsy files?), etc).

As a technical debt or simply to better understand our downstream usage, we're aiming to perform an artifacts audit.

Done right, we should be able to identify the list of artifacts needed for a certain type of release. Cleaning up and minimizing those lists of artifacts has some beneficial side-effects that include but are not limited to:

  1. reducing end-to-end overall times of a release time by avoiding to download of unnecessary artifacts in CoT and beetmover upload

  2. S3 storage space $$$

  3. simplifying declarative artifacts manifests

  4. sanitization/cleanup as technical debt

Marking this as sensitive bug until further notice.

CloudOps confirmed today that there are S3 logs turned on for our prod deliver bucket and that they can provide us access to those buckets to access programatically.

We're aiming to look at this in Q2 once declarative artifacts work is fully in-tree.

Depends on: 1520244

I like this. We should prioritize this in Q2 and bubble it up as a potential OKR

Priority: -- → P1

making public for now. can make private if needed later.

Group: mozilla-employee-confidential, releng-security

At first glance, after some investigation around, there's two workaround here:

  1. we can get the information via querying via aws cli which takes a lot of time and network bandwidth as logs are per day.

  2. we can get this information easier via Athena (which looks at cloudfront logs) but there’s cost associated to it; also potential management decision with respect to who is to run those commands. CloudOps can give RelEng access or RelEng to provide a comprehensive list of artifacts for which we want data.

If we were to go for latter, we could filter out the artifacts so that we filter out artifacts that we know for sure are used by automation or are by downstream consumers.

Simon did some great investigation work a few weeks back and mentioned how slow log inspection can be. The data is not greatly organized and definitely not easiest to play with. Today I repeated the procedure to better understand if we can refine this further but my early conclusion is that this won't scale very well.

Instead, plan going forward is this:

  1. draft out a full dump of all the artifacts that a release produces
  2. filter out all the files that we know for sure are consumed in some way by RelEng automation or downstream users
  3. out of the remaining files, get Athena logs for their usage/count, for at least 5 (or more?) last releases. (that means we should run the athena logs for at least our last five major releases (including a dot release), last five beta releases and last five nightlies

Depending on the data, we may be able to trim some of the artifacts of the release process to save $$$ and end-to-end time.

See Also: → 1552485
Depends on: 1552485
See Also: 1552485 →
Assignee: nobody → mtabara
Status: NEW → ASSIGNED

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #7)

I used the aws cli tool to list all the artifacts present in our most recent release, 66.0.5 and collapsed the artifacts under {platform}/{locale} to be able to look at them from perspective. They lie under https://bugzilla.mozilla.org/attachment.cgi?id=9065738&action=diff . The total storage for all of them is about 15 GB.

Out of them, the following subset of artifacts are either required as part of the release or needed by various automation pieces.

pub/firefox/candidates/66.0.5-candidates/build1/KEY

pub/firefox/candidates/66.0.5-candidates/build1/SHA*SUMMARY
pub/firefox/candidates/66.0.5-candidates/build1/SHA*SUMS
pub/firefox/candidates/66.0.5-candidates/build1/SHA*SUMS.asc

pub/firefox/candidates/66.0.5-candidates/build1/beetmover-checksums/{platform}/{locale}/firefox-66.0.5.checksums.beet
pub/firefox/candidates/66.0.5-candidates/build1/beetmover-checksums/{platform}/xpi/{locale}.checksums.beet
pub/firefox/candidates/66.0.5-candidates/build1/beetmover-checksums/{platform}-EME-free/{locale}/firefox-66.0.5.checksums.beet
pub/firefox/candidates/66.0.5-candidates/build1/beetmover-checksums/source/firefox-66.0.5.checksums.asc
pub/firefox/candidates/66.0.5-candidates/build1/beetmover-checksums/source/firefox-66.0.5.checksums.beet

pub/firefox/candidates/66.0.5-candidates/build1/{linux-i686,linux-x86_64}/{locale}/firefox-66.0.5.tar.bz2
pub/firefox/candidates/66.0.5-candidates/build1/{linux-i686,linux-x86_64}/{locale}/firefox-66.0.5.tar.bz2.asc
pub/firefox/candidates/66.0.5-candidates/build1/mac/{locale}/Firefox 66.0.5.dmg
pub/firefox/candidates/66.0.5-candidates/build1/win32/{locale}/Firefox Installer.exe
pub/firefox/candidates/66.0.5-candidates/build1/{win32,win64}/{locale}/Firefox Setup 66.0.5.exe
pub/firefox/candidates/66.0.5-candidates/build1/{win32,win64}/{locale}/Firefox Setup 66.0.5.msi
pub/firefox/candidates/66.0.5-candidates/build1/{win32,win64}/{locale}/firefox-66.0.5.zip

pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/buildhub.json

pub/firefox/candidates/66.0.5-candidates/build1/{platform}xpi/{locale}.xpi

pub/firefox/candidates/66.0.5-candidates/build1/{platform}_info.txt
pub/firefox/candidates/66.0.5-candidates/build1/jsshell/jsshell-{platform}.zip

pub/firefox/candidates/66.0.5-candidates/build1/mac-EME-free/ach/Firefox 66.0.5.dmg
pub/firefox/candidates/66.0.5-candidates/build1/mac-EME-free/ach/Firefox 66.0.5.dmg.asc
pub/firefox/candidates/66.0.5-candidates/build1/{win32,win64}-EME-free/ach/Firefox Setup 66.0.5.exe
pub/firefox/candidates/66.0.5-candidates/build1/{win32,win64}-EME-free/ach/Firefox Setup 66.0.5.exe.asc

pub/firefox/candidates/66.0.5-candidates/build1/mar-tools/{platform}/{mar,mar.exe}
pub/firefox/candidates/66.0.5-candidates/build1/mar-tools/{platform}/{mbsdiff,mbsdiff.exe}

pub/firefox/candidates/66.0.5-candidates/build1/partner-repacks/*

pub/firefox/candidates/66.0.5-candidates/build1/source/firefox-66.0.5.source.tar.xz
pub/firefox/candidates/66.0.5-candidates/build1/source/firefox-66.0.5.source.tar.xz.asc

pub/firefox/candidates/66.0.5-candidates/build1/update/{platform}/{locale}/*.partial.mar
pub/firefox/candidates/66.0.5-candidates/build1/update/{platform}/{locale}/*.complete.mar

On the other hand, dealing with the following is tracked under bug 1552485.

pub/firefox/candidates/66.0.5-candidates/build1/beetmover-checksums/{platform}/{locale}/firefox-66.0.5.checksums.asc

Which leaves the following artifacts to be investigated. Each set takes ~ 0.5 Gb for each platform, so a total of 2.5 Gb out of the ~15 Gb we have in total for 66.0.5.

pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.awsy.tests.tar.gz
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.common.tests.tar.gz
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.cppunittest.tests.tar.gz
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.crashreporter-symbols.zip
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.json
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.mochitest.tests.tar.gz
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.mozinfo.json
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.reftest.tests.tar.gz
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.talos.tests.tar.gz
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.test_packages.json
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.txt
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.web-platform.tests.tar.gz
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.xpcshell.tests.tar.gz
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/mozharness.zip

Data team will help us identify the url count on these artifacts to better understand their value and their potential removal.

I wonder if we can stop publishing the mar/mbsdiff binaries, and use them like taskcluster toolchain references.

(In reply to Chris AtLee [:catlee] from comment #11)

I wonder if we can stop publishing the mar/mbsdiff binaries, and use them like taskcluster toolchain references.

That's a good point. I filed bug 1552646 to track this.
I'm also wondering if there's value in publishing the jssshell as well:

pub/firefox/candidates/66.0.5-candidates/build1/jsshell/jsshell-{platform}.zip

I might file another bug to track that. Or see what Athena logs look like, maybe we can get rid of that as well.

Depends on: 1552646

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #12)

(In reply to Chris AtLee [:catlee] from comment #11)

I wonder if we can stop publishing the mar/mbsdiff binaries, and use them like taskcluster toolchain references.

That's a good point. I filed bug 1552646 to track this.
I'm also wondering if there's value in publishing the jssshell as well:

pub/firefox/candidates/66.0.5-candidates/build1/jsshell/jsshell-{platform}.zip

FWIW, we publish the jsshell files on purpose: https://bugzilla.mozilla.org/show_bug.cgi?id=1336514

Jason granted me access in bug 1552542 to play with Athena \o/

At first glance observations:

  • buildhub.json is heavily used - expected given Buildhub
  • *_info.txt are used as well - expected given UV
  • all the files that I assumed are unused in comment 1 are actually quiet queried for major releases but (almost) untouched for betas
  • I wonder whether I should change the query for the ~releases instead of ~candidates
  • Tom mentioned he seems to have recalled seeing some of them get used for testing things like extensions

To be continued.

Last time I touched based on this, out of all files, some were either mandatory for downstream consumers or needed by automation. The leftovers in doubt were:

for platform in (linux-i686, linux-x86_64, mac, win32, win64):

pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.awsy.tests.tar.gz
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.common.tests.tar.gz
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.cppunittest.tests.tar.gz
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.crashreporter-symbols.zip
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.json
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.mochitest.tests.tar.gz
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.mozinfo.json
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.reftest.tests.tar.gz
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.talos.tests.tar.gz
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.test_packages.json
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.txt
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.web-platform.tests.tar.gz
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/firefox-66.0.5.xpcshell.tests.tar.gz
pub/firefox/candidates/66.0.5-candidates/build1/{platform}/en-US/mozharness.zip

I've been doing more digging in the logs for 66*, 67* and 68* releases and based on the results, it seems that, the files that are used the least are:

  1. awsy.tests.tar.gz - 1-2 or not at all for 66* and 67*, mostly in major releases, seldom in beta. 68* counterpart though was queried more than 10 times for some platforms. The 68.0es counterpart too so there might be some use for this still.

  2. test_packages.json - 1-3 calls per major release. seldom used in betas.

  3. firefox-$VERSION.txt - used in all major releases and betas, but only once (rarely twice). There could be some crawlers out there querying for this data (contains buildid + revision)?

  4. mozinfo - 1-3 calls per major release, seldom used in betas.

The rest of the files, such as the *tests* suites are used in major releases and sometimes on beta. mozharness and firefox-$VERSION.json are also used across releases.

Given the results above, it seems like the assumptions I have made here were not valid. We make good use of the artifacts, either in automation or by downstream consumers.

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #0)>

  1. reducing end-to-end overall times of a release time by avoiding to download of unnecessary artifacts in CoT and beetmover upload

  2. S3 storage space $$$

  3. simplifying declarative artifacts manifests

  4. sanitization/cleanup as technical debt

Out of the list above, 1) and 2) are no longer in question. In the interest of 3) and 4), we could still remove test_packages.json, firefox-$VERSION.txt and mozinfo files. This could slightly reduce the payloads in beetmover jobs but are tiny files so the storage space is inexistent.

Pushed by mtabara@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/fe9a9de06733
remove txt files as part of the release artifacts cleanup. r=catlee

(In reply to Pulsebot from comment #18)

Pushed by mtabara@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/fe9a9de06733
remove txt files as part of the release artifacts cleanup. r=catlee

Grafted to beta, riding trains to release:
https://hg.mozilla.org/releases/mozilla-beta/rev/1e0df0529dec

To conclude, most of the assumptions we've had about artifacts proved to be wrong. They are indeed used either by automation or downstream consumers. The sole file we could remove was the txt file. That information within it resides in other files so we trimmed it from automation to make the payloads slightly smaller.

With this, we can call the auditing done.

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED

Note to self: when I did this work, I assumed everything was declarative-artifacts, but I was wrong. At that time, Devedition was not, I was supposed to remove it from https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/taskgraph/transforms/beetmover_repackage.py#l55 as well.

Am taking this change along in bug 1537713.

See Also: → 1537713
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: