Closed Bug 1432591 Opened 7 years ago Closed 7 years ago

source tarball isn't provided anymore

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(firefox61 fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox61 --- fixed

People

(Reporter: gaston, Assigned: mozilla)

References

Details

Attachments

(6 files)

beetmoverscript patch to add source.tar.xz{,.asc} to the manifests
58 bytes, text/x-github-pull-request
mtabara
: review+
Details | Review
bug 1432591 - use a simple source package name.
59 bytes, text/x-review-board-request
froydnj
: review+
Details
bug 1432591 - mozharness - back out source-related changes from 605111b6f51f.
59 bytes, text/x-review-board-request
tomprince
: review+
Details
bug 1432591 - mozharness - back out source readme changes from 35d738241395.
59 bytes, text/x-review-board-request
Callek
: review+
Details
bug 1432591 - taskgraph changes for source tarball.
59 bytes, text/x-review-board-request
bhearsum
: review+
Details
beetmoverscript - pretty naming
58 bytes, text/x-github-pull-request
mtabara
: review+
Details | Review
In bug 1422203/bug 749312 it was decided to stop shipping a source tarball for betas and replace it by https://releases.mozilla.org/pub/firefox/releases/59.0b3/SOURCE, which ended up being in production more or less for 59.0b3. This is very sad, since downstream packagers (we also package betas for QA) might use the previously-provided tarball, and now we only get to hammer hg.m.o which isnt made for this, and is very slow. But if you like us hammering hg that generates tarball on the fly, then i guess we'll get to live with it.
Blocks: 1422203, 749312
Summary: source tarball isn't provided anymore → source tarball isn't provided anymore for betas
I'm not sure it was meant this way, but it seems there's zero caching of the tarball on hg or somewhere else in the holy cloud. Fetching a release tarball is fine: >> Fetch https://releases.mozilla.org/pub/mozilla.org/firefox/releases/57.0.4/source/firefox-57.0.4.source.tar.xz firefox-57.0.4.source.... 100% |***********************************************************************| 238 MB 00:17 Fetching a beta is.. s. l.. o... w..: [20:41] c64:~/mystuff/www/mozilla-firefox/ $make makesum ===> Checking files for firefox-59.0beta3 >> Fetch https://hg.mozilla.org/releases/mozilla-beta/archive/c94e1fd9fe7a47c51c421db2a7052292ac411e3a.tar.bz2 [20:56] c64:~/ $du -sh /usr/ports/distfiles/mozilla/firefox-59.0b3.tar.bz2.part 75.2M /usr/ports/distfiles/mozilla/firefox-59.0b3.tar.bz2.part In 15mn, i got 75Mb.
So in the end it took a full hour to fetch a whooping 300Mb, i suppose the size increase is due to tests additions ? Either way, that's sad. A real killer. In the past 5 or 6 years i got used to package/test all betas in the cycle, now maybe i'll do only one every two or three..
Also, building a random mercurial commit isnt the same thing for some details, packaging-wise. - you need to go through autohell to generate configure from configure.in (if you're using the old-style configure and not mach) - if you install, you install in $PREFIX/lib/firefox and not $PREFIX/lib/firefox-${VERSION} like it used to be the case for beta tarballs, which were actual 'release tarballs'. So, lots of subtle differencies (there might be more ,that's just the two i stumbled upon right now) but it's diverting from "building something which is close to what's going to be released for real" which was the case for 'real' betas tarballs.
(In reply to Landry Breuil (:gaston) from comment #3) > Also, building a random mercurial commit isnt the same thing for some > details, packaging-wise. > > - you need to go through autohell to generate configure from configure.in > (if you're using the old-style configure and not mach) configure is a plain copy of configure.in, you don't need to go through autohell. > - if you install, you install in $PREFIX/lib/firefox and not > $PREFIX/lib/firefox-${VERSION} like it used to be the case for beta > tarballs, which were actual 'release tarballs'. That's a deliberate and unrelated change: bug 445128
- hg.mozilla.org archives don't support tar.xz - Slow download speed is probably bug 1045866 mitigation A workaround is to use GitHub snapshots and translate hg to git commit via git-cinnabar.
Slow download speed is probably trying to get a bz2 archive. Try gz.
(In reply to Mike Hommey [:glandium] from comment #4) > (In reply to Landry Breuil (:gaston) from comment #3) > > Also, building a random mercurial commit isnt the same thing for some > > details, packaging-wise. > > > > - you need to go through autohell to generate configure from configure.in > > (if you're using the old-style configure and not mach) > > configure is a plain copy of configure.in, you don't need to go through > autohell. Then why are there comments about autoconf in it ? :) ok, fine for that part.. > > - if you install, you install in $PREFIX/lib/firefox and not > > $PREFIX/lib/firefox-${VERSION} like it used to be the case for beta > > tarballs, which were actual 'release tarballs'. > > That's a deliberate and unrelated change: bug 445128 okay, good thing. I don't know if something has been done on the backend/hg side or if i've just been lucky because someone else fetched it before me and it got cached, but fetching 59.0b4/783507b1b80d927d5ef408c6ff87f8c52f2c8af9.tar.bz2 has been very fast.
Just a followup, as i doubt anything will happen to change this sorry state of things, fetching b7 took ~6mn right now, from https://hg.mozilla.org/releases/mozilla-beta/archive/1a24837f9ed232d8d2dc4535d11ee53c9847b109.tar.bz2
Greg, this is something you should probably be aware of if you aren't already, since it might be impacting the mercurial web heads.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → WORKSFORME
reopening, it seems tarballs served by hg are not deterministic. Oh, and chris, when you close a bug, can you at least explain why ? We're trying to ship your software to endusers too, so we're on the same boat..
Flags: needinfo?(catlee)
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
(In reply to Mike Hommey [:glandium] from comment #9) > Greg, this is something you should probably be aware of if you aren't > already, since it might be impacting the mercurial web heads. Yes, yes it is. I've actually been wanting to disable support for obtaining most source archives from hg.mozilla.org because it is a DoS vector. And, this bug delivers insight as to why we've been seeing more and more requests for archives on hg.mo lately. We can't have automated mechanisms making repeated queries to obtain dynamically-generated source archives because it is too much load on the hg.mo servers. catlee: we need to come up with a solution that doesn't rely on people hitting /archive/ URLs on hg.mozilla.org. I'm not sure what the reasons were to stop producing dedicated source archives. But my opinion is we should start producing them again. If you want this to live as part of the build system and the in-tree TC CI, we can support that on the build side of the world.
(In reply to Gregory Szorc [:gps] from comment #11) > I'm not sure what the reasons were to stop producing dedicated source archives. "Rationale" was from bug #749312 and implem in bug #1422203 . Us packagers will be happy to use whatever stable url is given us for beta source tarballs. As i said in my first comment, i dont like unnecessarly hammering hg.
(In reply to Landry Breuil (:gaston) from comment #10) > it seems tarballs served by hg are not deterministic. Do you use gz? If so it's probably due to timestamps changing once cached copy is pruned after a while. bz2/zip should be unaffected. $ echo foo | gzip >a.gz $ echo foo | gzip >b.gz $ md5 ?.gz MD5 (a.gz) = 86f84543062eb1cdd56e84de670c6267 MD5 (b.gz) = e9d1b245d3c2733f0ef8f4d6f48858db
hg.mozilla.org is using Mercurial's built-in support for generating archives. That uses Python's stdlib packages. It shouldn't be much work to fix Mercurial to produce deterministic archives. But that may be out of scope for this bug. We do have code in mozilla-central for producing deterministic tarballs. See https://hg.mozilla.org/mozilla-central/file/861067332bac/python/mozbuild/mozpack/archive.py. It would be trivial to hook that code up to a TC task that takes the output of `hg files` to produce a tarball.
(In reply to Jan Beich from comment #13) > (In reply to Landry Breuil (:gaston) from comment #10) > > it seems tarballs served by hg are not deterministic. > > Do you use gz? If so it's probably due to timestamps changing once cached > copy is pruned after a while. bz2/zip should be unaffected. > > $ echo foo | gzip >a.gz > $ echo foo | gzip >b.gz > $ md5 ?.gz > MD5 (a.gz) = 86f84543062eb1cdd56e84de670c6267 > MD5 (b.gz) = e9d1b245d3c2733f0ef8f4d6f48858db i fetch .tar.bz2 archives from hg, dunno for gentoo (from whom the non-deterministic archives issue came first on irc)
(In reply to Landry Breuil (:gaston) from comment #15) > (In reply to Jan Beich from comment #13) > > (In reply to Landry Breuil (:gaston) from comment #10) > > > it seems tarballs served by hg are not deterministic. > > > > Do you use gz? If so it's probably due to timestamps changing once cached > > copy is pruned after a while. bz2/zip should be unaffected. > > > > $ echo foo | gzip >a.gz > > $ echo foo | gzip >b.gz > > $ md5 ?.gz > > MD5 (a.gz) = 86f84543062eb1cdd56e84de670c6267 > > MD5 (b.gz) = e9d1b245d3c2733f0ef8f4d6f48858db > > i fetch .tar.bz2 archives from hg, dunno for gentoo (from whom the > non-deterministic archives issue came first on irc) We are using the tar.bz2 as well. I created the digest for gentoo this morning around 8 CST and by 10CST the digest was no longer correct. !!! Got: 305353563 !!! Expected: 305357047
I just poked through Mercurial's source code and it looks deterministic to me. Using the changeset's time for the mtime of entries. Files in sorted order. The one thing it isn't normalizing is the user/group and uid/gid values. Those could vary depending on the server. But they shouldn't. I downloaded .tar.gz and .zip archives for mozilla-beta and they were identical. So I'm not sure where the non-determinism is coming from. As long as the URL has the full SHA-1 hash (and isn't a symbolic symbol), you should get the same data...
$ LC_ALL=en_US.UTF8 ls -l ff* 305531522 Feb 23 09:04 ff59b12-201802230852-411KBs.tar.bz2 354857397 Feb 23 09:22 ff59b12-201802230852-495KBs.tar.gz 305528067 Feb 24 12:56 ff59b12-201802241244-423KBs.tar.bz2 354857487 Feb 24 13:20 ff59b12-201802241244-488KBs.tar.gz $ LC_ALL=en_US.UTF8 sha1sum ff* 8f150f68d395141028529bf2f79b06d9e3869059 ff59b12-201802230852-411KBs.tar.bz2 bdb3d1fa03ab50d7f4b838a8380f3ad309b985d6 ff59b12-201802230852-495KBs.tar.gz 7db90725425058859f56994592937ef3b13124f4 ff59b12-201802241244-423KBs.tar.bz2 ba28b2fa7a40b63c79efe36e5bf1783d13a6db71 ff59b12-201802241244-488KBs.tar.gz 1. Really slow 2. Different $ diff -Naur mozilla-beta-abc5f0d35082241616975246fd770c77359a0d5e-1 mozilla-beta-abc5f0d35082241616975246fd770c77359a0d5e diff -Naur mozilla-beta-abc5f0d35082241616975246fd770c77359a0d5e-1/.hg_archival.txt mozilla-beta-abc5f0d35082241616975246fd770c77359a0d5e/.hg_archival.txt --- mozilla-beta-abc5f0d35082241616975246fd770c77359a0d5e-1/.hg_archival.txt 2018-02-20 22:24:47.000000000 +0300 +++ mozilla-beta-abc5f0d35082241616975246fd770c77359a0d5e/.hg_archival.txt 2018-02-20 22:24:47.000000000 +0300 @@ -1,11 +1,7 @@ repo: 8ba995b74e18334ab3707f27e9eb8f4e37ba3d29 node: abc5f0d35082241616975246fd770c77359a0d5e branch: default -latesttag: DEVEDITION_59_0b11_BUILD1 -latesttag: DEVEDITION_59_0b11_RELEASE -latesttag: FENNEC_59_0b11_BUILD1 -latesttag: FENNEC_59_0b11_RELEASE -latesttag: FIREFOX_59_0b11_BUILD1 -latesttag: FIREFOX_59_0b11_RELEASE -latesttagdistance: 70 -changessincelatesttag: 70 +tag: DEVEDITION_59_0b12_BUILD1 +tag: DEVEDITION_59_0b12_RELEASE +tag: FIREFOX_59_0b12_BUILD1 +tag: FIREFOX_59_0b12_RELEASE
Fwiw, i've just realized source tarballs werent generated for candidates either, so now to test the 59.0RCs i'll just hammer mozilla-release repo. Yay.
Hmmm. So there's no source tarball for releases either ? REALLY ? Without stable tarballs with consistent checksums downstreams cant package anything. I hope this is a misunderstanding and there will be a real tarball when 59 if officially RTM. https://releases.mozilla.org/pub/firefox/releases/59.0/SOURCE
Another argument for properly generated tarballs like before: size. Now we're > 300Mb with hg archives, while we were around 210Mb for proper source tarballs.
Summary: source tarball isn't provided anymore for betas → source tarball isn't provided anymore
(In reply to Maxim Britov from comment #18) > $ diff -Naur mozilla-beta-abc5f0d35082241616975246fd770c77359a0d5e-1 > mozilla-beta-abc5f0d35082241616975246fd770c77359a0d5e > diff -Naur > mozilla-beta-abc5f0d35082241616975246fd770c77359a0d5e-1/.hg_archival.txt > mozilla-beta-abc5f0d35082241616975246fd770c77359a0d5e/.hg_archival.txt > --- mozilla-beta-abc5f0d35082241616975246fd770c77359a0d5e-1/.hg_archival.txt > 2018-02-20 22:24:47.000000000 +0300 > +++ mozilla-beta-abc5f0d35082241616975246fd770c77359a0d5e/.hg_archival.txt > 2018-02-20 22:24:47.000000000 +0300 > @@ -1,11 +1,7 @@ > repo: 8ba995b74e18334ab3707f27e9eb8f4e37ba3d29 > node: abc5f0d35082241616975246fd770c77359a0d5e > branch: default > -latesttag: DEVEDITION_59_0b11_BUILD1 > -latesttag: DEVEDITION_59_0b11_RELEASE > -latesttag: FENNEC_59_0b11_BUILD1 > -latesttag: FENNEC_59_0b11_RELEASE > -latesttag: FIREFOX_59_0b11_BUILD1 > -latesttag: FIREFOX_59_0b11_RELEASE > -latesttagdistance: 70 > -changessincelatesttag: 70 > +tag: DEVEDITION_59_0b12_BUILD1 > +tag: DEVEDITION_59_0b12_RELEASE > +tag: FIREFOX_59_0b12_BUILD1 > +tag: FIREFOX_59_0b12_RELEASE This is very useful! So everything appears to be deterministic except for .hg_archival.txt. That file is generated from a Mercurial template: repo: {root} node: {ifcontains(rev, revset("wdir()"), "{p1node}{dirty}", "{node}")} branch: {branch|utf8} {ifeq(latesttagdistance, 0, join(latesttag % "tag: {tag}", "\n"), separate("\n", join(latesttag % "latesttag: {tag}", "\n"), "latesttagdistance: {latesttagdistance}", "changessincelatesttag: {changessincelatesttag}"))} The output of that template *should* be deterministic as long as the repository doesn't change. But since it is printing info related to the "latest tag," if a new tag appears on the repo, the contents of this file will change. We can swap in a different template that doesn't refer to "latest tag" and make archive generation deterministic. We can also disable generation of the .hg_archival.txt file completely. Although I'm not sure what implications this may have. I'm sure someone, somewhere is depending on it. But I'd rather we restore proper source tarball distributions because I don't want these expensive requests hitting hg.mozilla.org. catlee: I think this would be in your court to enable source archives again
An additional difference between source tarballs and mercurial archives is that the former contained the `configure` script while the latter doesn't.
Are mercurial clones an acceptable solution for downstream packagers? That would give you the same revision. We can edit the SOURCE file template [1] to add cloning instructions if that's a workable solution. https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/configs/builds/releng_sub_linux_configs/64_source.py#l4
I would prefer a simple tarball. I am always uncomfortable in packaging having to checkout sources to build a packages (so many more things can happen than a simple curl/wget).
The Mercurial bundles that we generate and host at https://hg.cdn.mozilla.net/ could also be used. However, we generate those daily and auto expire the old ones after a few days. Plus they are quite large - much larger than a source distribution. Hosting Mercurial bundles for long amounts of time on hg.cdn.mozilla.net is certainly doable. However, I'd prefer to limit the retention period and limit which bundles are retained for that long. I would also not want a manual process to be required to generate these "special" bundles. i.e. if we wanted to create special bundles for just releases that were retained much longer - possibly even indefinitely - that process should be fully automated. But at the point you implement that, you might as well go back to using source tarballs because I think they are a better solution to the problem at hand - which is that downstream packagers need a reliable and deterministic-over-time mechanism to obtain Firefox's source code. We /could/ tell them "use version control." I'm not sure if that is viable though. Plus standalone artifacts have better security properties: you can record their size, hash, etc, and sign that. If you are actively using VCS for content distribution, you rely on the VCS to implement content integrity. Being a significant contributor to VCS tools, I think it is better to put your security trust in cryptographically proven hashing algorithms, PGP, etc rather than VCS. What I'm trying to say is that I think the preference for downstream consumers is (from highest to lowest): 1) Hosted static source archives 2) Hosted static Mercurial bundles 3) Hosted dynamic source archives (assuming they are deterministic) 4) Hosted dynamic Mercurial bundles (assuming they are deterministic) 5) VCS `clone` You can record lengths and hashes of 1-4 to prove content integrity. #5 can theoretically get you content integrity. But you have to trust VCS. I don't trust VCS. Not even GPG signed tags (because they can be subverted).
An acceptable solution is to provide a source tarball as it was before bug 1422203 landed - i understand it might be a burden for releng to rollback/rethink such changes, but in the opensource world, when you want to package something, it's expected to have a stable *source tarball* - as it has been the case for 20+ years maybe ? The slides from https://fosdem.org/2018/schedule/event/how_to_make_package_managers_cry/ are a good reminder of the do's and dont's... Granted, as said in bug 749312 the MPL says that 'providing means to download sources' is enough, but in the real world a source tarball is a common practice.. and we don't need the extra size/complexity provided by all the hg overhead, if i want the history i know how to get a bundle, unbundle it and update it, but this is not needed to provide os-specific packages/binaries to users on platforms not supported by mozilla - or linux distros which prefer to package firefox from source instead of advertising mozilla binaries. Sadly, nowadays ppl just dump their crap on github, tag stuff, and expect downstreams to figure out things (same thing, github doesnt provide stable tarballs from git tags, only "releases"/assets posted explicitely on the github releases page) I'm not saying we should have a source tarball for *every* build produced, but at least releases and betas... and in the era of CDN's, distribution of files shouldnt be an issue.
Assignee: nobody → aki
Status: REOPENED → ASSIGNED
Flags: needinfo?(catlee)
Attachment #8959026 - Flags: review?(mtabara)
Attachment #8959037 - Flags: review?(core-build-config-reviews)
Comment on attachment 8959040 [details] bug 1432591 - taskgraph changes for source tarball. https://reviewboard.mozilla.org/r/227920/#review233864 ::: taskcluster/ci/release-source/kind.yml:1 (Diff revision 1) > # This Source Code Form is subject to the terms of the Mozilla Public What I see here seems fine, but I'm a bit confused as to how this gets the source builder running again - I don't see any changes to target tasks, shipping-*, or other things that influence task selection. Also, do we need any changes to beetmover-cdns to make sure the tarballs sync to /releases/?
Comment on attachment 8959038 [details] bug 1432591 - mozharness - back out source-related changes from 605111b6f51f. https://reviewboard.mozilla.org/r/227916/#review233894 ::: testing/mozharness/mozharness/mozilla/building/buildbase.py:1277 (Diff revision 1) > else: > self.warning("mozbuild_path could not be determined. skipping " > "creating it.") > > + def checkout_sources(self): > + self._checkout_source() I don't think this (or the above code it is called) is needed on taskcluster, since the checkout is already handled by taskcluster/`run-task`.
Attachment #8959038 - Flags: review?(mozilla) → review-
Comment on attachment 8959039 [details] bug 1432591 - mozharness - back out source readme changes from 35d738241395. https://reviewboard.mozilla.org/r/227918/#review233896 ::: testing/mozharness/configs/builds/releng_sub_linux_configs/64_source.py:10 (Diff revision 1) > - > -The source code used to build these binaries is specified by the following unique URL: > -{repo}/rev/{revision} > - > - zip: {repo}/archive/{revision}.zip > - tar.gz: {repo}/archive/{revision}.tar.gz > + 'package-source', > + ], > + 'objdir': 'obj-firefox', > + 'stage_platform': 'source', # Not used, but required by the script > + 'buildbot_json_path': 'buildprops.json', > + 'vcs_share_base': '/builds/hg-shared', This appears to be [redundant](https://searchfox.org/mozilla-central/rev/8976abf9cab8eb4661665cc86bd355cd08238011/testing/mozharness/mozharness/base/vcs/mercurial.py#356) with `HG_SHARE_BASE_DIR`. ::: testing/mozharness/scripts/fx_desktop_build.py:38 (Diff revision 1) > buildscript_kwargs = { > 'config_options': BUILD_BASE_CONFIG_OPTIONS + copy.deepcopy(try_config_options), > 'all_actions': [ > 'get-secrets', > 'clobber', > + 'checkout-sources', As mentioned in my other review, this isn't needed.
Attachment #8959039 - Flags: review+
Comment on attachment 8959040 [details] bug 1432591 - taskgraph changes for source tarball. https://reviewboard.mozilla.org/r/227920/#review233864 > What I see here seems fine, but I'm a bit confused as to how this gets the source builder running again - I don't see any changes to target tasks, shipping-*, or other things that influence task selection. > > Also, do we need any changes to beetmover-cdns to make sure the tarballs sync to /releases/? > I'm a bit confused as to how this gets the source builder running again It is already running, but it was just generating files like [this](http://ftp.mozilla.org/pub/firefox/releases/59.0/SOURCE).
Comment on attachment 8959040 [details] bug 1432591 - taskgraph changes for source tarball. https://reviewboard.mozilla.org/r/227920/#review233902 ::: taskcluster/ci/release-source/kind.yml:34 (Diff revision 1) > + env: > + TOOLTOOL_MANIFEST: "browser/config/tooltool-manifests/linux64/releng.manifest" > + PKG_SRCPACK_BASENAME: source > + run: > + using: mozharness > + actions: [clobber checkout-sources package-source] `checkout-sources` is not required.
Comment on attachment 8959040 [details] bug 1432591 - taskgraph changes for source tarball. https://reviewboard.mozilla.org/r/227920/#review233920
Attachment #8959040 - Flags: review?(bhearsum) → review+
Attachment #8959026 - Flags: review?(mtabara) → review+
Comment on attachment 8959039 [details] bug 1432591 - mozharness - back out source readme changes from 35d738241395. https://reviewboard.mozilla.org/r/227918/#review233966 ::: testing/mozharness/configs/builds/releng_sub_linux_configs/64_source.py:4 (Diff revision 1) > -import os > - > config = { > - "tmpl": """The binaries in this directory are made available to you under the Mozilla Public License v.2 > -(MPL 2): > -http://www.mozilla.org/MPL/2.0/ > + 'default_actions': [ > + 'clobber', > + 'checkout-sources', echo-ing tomprince, checkout-sources is unneeded
Attachment #8959039 - Flags: review?(bugspam.Callek) → review+
In staging, we now have a source/firefox-59.0b17.source.tar.xz{,.asc}, after addressing the above review comments and creating another beetmover patch. Anyone miss the .bundle file terribly, or is this sufficient?
Flags: needinfo?(landry)
Flags: needinfo?(gps)
> Anyone miss the .bundle file terribly, or is this sufficient? It looks like it generates but doesn't upload the bundle file. I don't know if anybody cares about it (but the discussion here suggests probably not). If that is the case, we should rip out the code that causes the bundle to be built (not least, because it appears to take at least 15m of the 24m the tasks takes). (I'm specifically referring to the mozharness code that calls hg-bundle).
Attachment #8959377 - Flags: review?(mtabara)
Attachment #8959377 - Flags: review?(mtabara) → review+
(In reply to Tom Prince [:tomprince] from comment #41) > > Anyone miss the .bundle file terribly, or is this sufficient? > > It looks like it generates but doesn't upload the bundle file. I don't know > if anybody cares about it (but the discussion here suggests probably not). > If that is the case, we should rip out the code that causes the bundle to be > built (not least, because it appears to take at least 15m of the 24m the > tasks takes). > > (I'm specifically referring to the mozharness code that calls hg-bundle). I agree. Let's ditch the bundle.
Ditched the bundle and checkout-sources.
Comment on attachment 8959038 [details] bug 1432591 - mozharness - back out source-related changes from 605111b6f51f. https://reviewboard.mozilla.org/r/227916/#review234076
Attachment #8959038 - Flags: review?(mozilla) → review+
Comment on attachment 8959039 [details] bug 1432591 - mozharness - back out source readme changes from 35d738241395. https://reviewboard.mozilla.org/r/227918/#review234078 ::: testing/mozharness/configs/builds/releng_sub_linux_configs/64_source.py:3 (Diff revision 2) > -import os > - > config = { > - "tmpl": """The binaries in this directory are made available to you under the Mozilla Public License v.2 > -(MPL 2): > + 'default_actions': [ > + 'clobber', `clobber` is unneed on taskcluster
Attachment #8959039 - Flags: review+
Comment on attachment 8959040 [details] bug 1432591 - taskgraph changes for source tarball. https://reviewboard.mozilla.org/r/227920/#review234080 ::: taskcluster/ci/release-source/kind.yml:34 (Diff revision 2) > + env: > + TOOLTOOL_MANIFEST: "browser/config/tooltool-manifests/linux64/releng.manifest" > + PKG_SRCPACK_BASENAME: source > + run: > + using: mozharness > + actions: [clobber package-source] `clobber` isn't needed on taskcluster
That looks good to me naming-wise from a distributor pov - what's the layout of files inside the tarball ? firefox-${VERSION}/ toplevel dir, and all the needed files (including the generated configure) as before ? The bundle isnt needed. Where is the staging public url so i can test a resulting archive ? Thanks aki for working on this!
Flags: needinfo?(landry)
https://queue.taskcluster.net/v1/task/IGQNnVbMSmu1dJLyiRhLDQ/runs/0/artifacts/public/build/source.tar.xz is an example, straight from the source builder task. It gets renamed to firefox-59.0b17.source.tar.xz in the beetmover task.
Thanks, the content of the archive looks good to me, there's the configure script, everything is in a firefox-59.0 subdir, the size went down a bit (270mb..) and it fetches fast (43s).
Excellent. Once https://bugzilla.mozilla.org/attachment.cgi?id=8959037 gets build peer review, we should be able to land.
Attachment #8959037 - Flags: review?(core-build-config-reviews) → review?(nfroyd)
Comment on attachment 8959037 [details] bug 1432591 - use a simple source package name. https://reviewboard.mozilla.org/r/227914/#review234818 This is not exactly my area, but the discussion makes it sound like we *do* get a versioned tarball out at the end of the release process, so if this makes that process easier, then let's do that.
Attachment #8959037 - Flags: review?(nfroyd) → review+
Pushed by asasaki@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/a5f31b2455db use a simple source package name. r=froydnj https://hg.mozilla.org/integration/autoland/rev/c949c9e84c03 mozharness - back out source-related changes from 605111b6f51f. r=tomprince https://hg.mozilla.org/integration/autoland/rev/5a220ead873c mozharness - back out source readme changes from 35d738241395. r=Callek,tomprince https://hg.mozilla.org/integration/autoland/rev/f0732d969844 taskgraph changes for source tarball. r=bhearsum
Thanks! Pushed out beetmoverscript 5.0.1; triggered autoland. We're going to have to uplift these patches if we want source tarballs on beta/release before Fx 61.
It looks like 60.0b5 build 1 is off a previous revision, so we'll still get SOURCE files.
Let's ditch the bundle and all the code associated with it. FWIW, the bulk of the CPU time generating the bundle is bzip2. Switch to zstd (`hg bundle -t zstd-v2`) and it will be much, much faster. We stopped generating bzip2 bundles for hg.mozilla.org because they were larger and a *lot* slower than zstd on both ends.
Flags: needinfo?(gps)
https://hg.mozilla.org/mozilla-central/rev/a5f31b2455db https://hg.mozilla.org/mozilla-central/rev/f0732d969844
Status: ASSIGNED → RESOLVED
Closed: 7 years ago → 7 years ago
status-firefox61: --- → fixed
Resolution: --- → FIXED
60 Beta format is slightly different from 58 Beta. Extracted directory lacks bN suffix e.g., - firefox-58.0b6.tar.xz extracts firefox-58.0b6 directory - firefox-60.0b6.tar.xz extracts firefox-60.0 directory
(In reply to Jan Beich from comment #67) > 60 Beta format is slightly different from 58 Beta. Extracted directory lacks > bN suffix e.g., > > - firefox-58.0b6.tar.xz extracts firefox-58.0b6 directory > - firefox-60.0b6.tar.xz extracts firefox-60.0 directory Right. That i think i can live with/adapt. I'm just glad to have source tarballs back :)
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: