Migrate `MozillaBuild` source hosting to `GitHub`
Categories
(Firefox Build System :: MozillaBuild, task, P3)
Tracking
(Not tracked)
People
(Reporter: ahochheiden, Unassigned)
Details
It currently lives at https://hg.mozilla.org/mozilla-build and requires hg to make changes to. Now that we've migrated to git for mozilla-central, we should make MozillaBuild follow suit. I don't think this is urgent, and will probably be done sometime between the releases of MozillaBuild 4.2 and MozillaBuild 4.3.
| Comment hidden (obsolete) |
After consideration we shouldn't use LFS, mostly because it isn't built into Git so would require extra attention across our tooling.
Instead I propose:
- starting on Git with a fresh repository
- stop storing archives and binaries in repo, instead upload these files to archive.mozilla.org
- update the build script to pull down files from archive.mozilla.org
Alex - any thoughts?
|
Reporter | |
Comment 3•8 days ago
|
||
(In reply to :glob ✱ from comment #2)
- stop storing archives and binaries in repo, instead upload these files to archive.mozilla.org
- update the build script to pull down files from archive.mozilla.org
That seems fine. I was considering something similar already to get around bug 1965558. I was thinking of storing them at https://ftp.mozilla.org/pub/mozilla/libraries/win32/src/ (or a slightly different path to keep the msys/non-msys dependencies separate). Whoever is doing the MozillaBuild release will need access to that anyway, and that would keep the process similar.
We can keep the files locally at .mozbuild/<tbd> and only download if they're not there, so the testing part isn't slowed down by big downloads every time.
- starting on Git with a fresh repository
The repo history isn't that important, but I was hoping to preserve it using git-cinnabar. Though, I have done little to no investigating into how feasible/time consuming that would be to do. If it's a significant task, I'm fine with losing it and starting fresh.
(In reply to Alex Hochheiden [:ahochheiden] from comment #3)
The repo history isn't that important, but I was hoping to preserve it using
git-cinnabar. Though, I have done little to no investigating into how feasible/time consuming that would be to do. If it's a significant task, I'm fine with losing it and starting fresh.
It's mostly to avoid having to deal with those large binaries in the repo at all.
Of course, the hg repo's history will still be accessible.
|
|
Reporter | |
Comment 5•8 days ago
|
||
(In reply to :glob ✱ from comment #4)
(In reply to Alex Hochheiden [:ahochheiden] from comment #3)
The repo history isn't that important, but I was hoping to preserve it using
git-cinnabar. Though, I have done little to no investigating into how feasible/time consuming that would be to do. If it's a significant task, I'm fine with losing it and starting fresh.It's mostly to avoid having to deal with those large binaries in the repo at all.
That makes sense. I was only thinking about the current state of the repo, but you're right, if we keep the history we'd keep the historical binaries. Which is currently 1.7gb for me on disk, and includes many variants of emacs and python distributions going back to version 2.5.
We could create some weird amalgamation where we only keep the commits without the binaries, but that might not be worth the effort. Starting fresh is probably the right call.
|
|
Reporter | |
Comment 6•7 days ago
|
||
Glandium says he will do the conversion that will keep the history, but drop all the archives/binaries.
|
||
Comment 7•7 days ago
|
||
With the following commands:
git clone hg::https://hg.mozilla.org/mozilla-build
git cinnabar clear
git filter-repo --force --invert-paths --path-glob "*.nsi" --path-glob "*.exe" --path-glob "*.7z" --path-glob "*.tar.lzma" --path-glob "*.tar.bz2" --path-glob "*.tar.gz" --path-glob "*.tar.xz" --path-glob "*.tgz" --path-glob "*.msi" --path-glob "*.zip" --path-glob "*.egg"
git gc --aggressive
I get a 684K repo.
(git filter-repo from https://github.com/newren/git-filter-repo)
|
|
Reporter | |
Comment 8•7 days ago
|
||
So I reworked D247519 to get around bug 1965558 and it's essentially the first draft of what's proposed here. Download all the dependencies instead of committing them to the repo. Glandium is keen on using fetch-content (what we use in CI to fetch dependencies, verifies sha256/file size, etc) which is essentially just a more robust implementation of what I have.
Which got me thinking, what's stopping us from just making this all ./mach command in the firefox repo? If we follow this through to the end, that's essentially all it is. A single command that makes an installer. It's not really necessary for a lot of people to have easy access to doing so via a ./mach command, but I could make that argument for a lot of commands.
I wouldn't need to duplicate that fetch-content script. It'd be trivial to use .mozbuild to store the downloaded dependencies (otherwise we'd be guessing where it was, or duplicating a bunch of code again to know for sure).
We'd probably lose the history of packageit.py (unless there's some way to graft it onto whatever mach_commands.py file we end up using?). I posed the short version of this thought on Slack to Glandium, and decided to post the long version here before he replied.
Description
•