922241 - windows builds should only produce tarballs instead of both zip and exes

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect, P3)

Description

[[e-mail] (:bhearsum)]

Our zip builds are not officially supported. We don't ship them as part of betas or releases. Today we discovered that there's even some differences between the that cause certain updates to apply differently (bug 922227). There's probably other differences we haven't thought of, too.

We should stop building and shipping these because:
1) They take up machine time unnecessarily
2) We don't require them to build l10n or MARs anymore
3) It's confusing to offer the zip and the installer

Some people have mentioned that it's convenient to have them because they're a little easier to work with for testing, but our installers are very simple to extract by hand these days;
1) 7z x firefox-installer.exe
2) you have a runnable firefox in the "core" directory

In my opinion, the benefits outweigh the negatives.

Comment 1

[Ed Morley [:emorley]]

Things like mozregression will need updating to support the installers; have notified a few people :-)

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

You don't necessarily need 7z to extract the installer. You can also use the silent installer at least for the full build (not stub installer) like ´firefox-setup.exe /S /D=path´ or use mozinstall (pip install mozinstall).

Comment 3

[Axel Hecht [:Pike]]

Are there corporate setups that would prevent the installer to run, but let the zip work? I don't know jack about both Windows and corporate setups, but I recall something like that being listed as use-case for the zips.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

AFAIK we only offer zip files for nightly builds but not releases, beta, nor candidate builds.

Comment 5

[[e-mail] (:bhearsum)]

Sounds like we need a post to the newsgroups before we can do this - there's probably other use cases we're missing here. I'll try to send that out in the next couple of weeks.

Comment 6

[Ed Morley [:emorley]]

Earlier today I emailed the internal auto-tools@ alias and also posted to mozilla.tools, though guess we may need further outreach at some point :-)

https://groups.google.com/d/msg/mozilla.tools/1EUBGcPW4Rw/4J9XJJtPkxsJ

Comment 7

[Andrew Halberstadt [:ahal]]

We'll need to update mozharness here at least: http://mxr.mozilla.org/build/source/mozharness/scripts/desktop_unittest.py#208

Comment 8

[(not currently active) Ted Mielczarek]

In general I believe all of our automated testing relies on downloading the zip file on Windows and not the installer, so we'd need to sort that all out.

On a more personal note, I've always used the zip files because they're less of a hassle than the installer. I could get over myself though if there are compelling reasons to drop them.

Comment 9

[Andrew Halberstadt [:ahal]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #8)
> In general I believe all of our automated testing relies on downloading the
> zip file on Windows and not the installer, so we'd need to sort that all out.
> 
> On a more personal note, I've always used the zip files because they're less
> of a hassle than the installer. I could get over myself though if there are
> compelling reasons to drop them.

Mozharness should handle installing from a .zip vs .exe automatically (via mozinstall), it gets the url from the buildprops.json, so I'm not sure what changes if any would need to be done there.

Comment 10

[(no longer active)]

(In reply to Ed Morley [:edmorley UTC+1] from comment #6)
> Earlier today I emailed the internal auto-tools@ alias and also posted to
> mozilla.tools, though guess we may need further outreach at some point :-)
> 
> https://groups.google.com/d/msg/mozilla.tools/1EUBGcPW4Rw/4J9XJJtPkxsJ

I don't think that newsgroup is the right forum, this will affect more than the target audience of that newsgroup.

I object to us dropping zip packages, since not everybody's system has 7z and not everybody knows that's what they should do to extract the contents of the package (I didn't know until a few minutes ago.)

Comment 11

[[e-mail] (:bhearsum)]

(In reply to :Ehsan Akhgari (needinfo? me!) from comment #10)
> (In reply to Ed Morley [:edmorley UTC+1] from comment #6)
> > Earlier today I emailed the internal auto-tools@ alias and also posted to
> > mozilla.tools, though guess we may need further outreach at some point :-)
> > 
> > https://groups.google.com/d/msg/mozilla.tools/1EUBGcPW4Rw/4J9XJJtPkxsJ
> 
> I don't think that newsgroup is the right forum, this will affect more than
> the target audience of that newsgroup.
> 
> I object to us dropping zip packages, since not everybody's system has 7z
> and not everybody knows that's what they should do to extract the contents
> of the package (I didn't know until a few minutes ago.)

Where would you suggest we have the discussion?

Comment 12

[(no longer active)]

(In reply to comment #11)
> (In reply to :Ehsan Akhgari (needinfo? me!) from comment #10)
> > (In reply to Ed Morley [:edmorley UTC+1] from comment #6)
> > > Earlier today I emailed the internal auto-tools@ alias and also posted to
> > > mozilla.tools, though guess we may need further outreach at some point :-)
> > > 
> > > https://groups.google.com/d/msg/mozilla.tools/1EUBGcPW4Rw/4J9XJJtPkxsJ
> > 
> > I don't think that newsgroup is the right forum, this will affect more than
> > the target audience of that newsgroup.
> > 
> > I object to us dropping zip packages, since not everybody's system has 7z
> > and not everybody knows that's what they should do to extract the contents
> > of the package (I didn't know until a few minutes ago.)
> 
> Where would you suggest we have the discussion?

At the very least, dev.platform, and maybe firefox-dev as well.

Comment 13

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I've frequently used the zip files for debugging Windows crashes on Linux when all we had to debug was information in crash-stats.  All the tools needed to do this come with typical Linux distributions (objdump on Linux handles Windows binaries), so I could correlate disassembly to a crash report for a Windows build based on downloading the zip file.  If there are extra tools that need to be downloaded (rather than using the distro's package installer), that makes debugging Windows crashes a good bit harder for me.

Comment 14

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(Is the 7z command provided by the Ubuntu package p7zip-full the one you refer to in comment 0?)

Comment 15

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Oh, then again, I have a command alias:

alias unmar-full-update='MAR=~/builds/mozilla-central/obj/firefox-debugopt/dist/host/bin/mar ~/builds/mozilla-central/mozilla/tools/update-packaging/unwrap_full_update.sh'

which suggests that when I ran into this problem for releases, I unpackaged the full update to get what I needed.

I'm not crazy about having so many steps for developers to figure this stuff out; it's extra knowledge about how to get around in our ecosystem that needs to be cargo-culted around.  But I suppose this one change doesn't make things that much worse for the crash-debugging case.

Comment 16

[Gregory Szorc [:gps]]

And that's why we have mach commands, Python packages like mozdownload and mozregression, etc. Somebody writes it once, posts it on PyPI or in the tree, and the technically hard part of the problem is solved.

If the tools suck, update the tools. Bad or non-existent tooling for developers shouldn't be a blocker for this not happening. Delaying it, perhaps.

Comment 17

[Chris AtLee [:catlee]]

Kim, is this something the build team could look into?

Comment 18

[(not currently active) Ted Mielczarek]

This seems at odds with the proposal in bug 1494788. It seems fine to stop *shipping* Windows zip files, and that's something that RelEng could do all on your own--just make beetmover stop putting the Windows zips in places where people download them. If the build tasks still generate them as artifacts that's just an internal implementation detail.

More germane to the points in bug 1494788, maybe we should simply make the build system produce a tarball on all platforms, and then post-signing repackaging tasks could be responsible for producing the final user-visible packages like installers/DMGs. (Which they would continue to do with the assistance of the build system.)

Comment 19

[Tom Prince [:tomprince]]

> maybe we should simply make the build system produce a tarball on all platforms, and then post-signing repackaging tasks could be responsible for producing the final user-visible packages like installers/DMGs. (Which they would continue to do with the assistance of the build system.)

This definitely seems like a sensible approach. Particularly since we currently only consume the zip on windows, and use the installer from a latter task anyway.

Comment 20

[Kim Moir [:kmoir] ET]

catlee: Is this the approach you'd like your team to take? If so do you have any suggestions of timeframes for this work so we could coordinate the work between teams? This will help me decide who I'll ask to work on it.  We discussed this bug in our team meeting today and Greg also suggested using zstandard tarballs for better compression.

Comment 21

[Chris AtLee [:catlee]]

(In reply to Ted Mielczarek [:ted] [:ted.mielczarek] from comment #18)
> This seems at odds with the proposal in bug 1494788. It seems fine to stop
> *shipping* Windows zip files, and that's something that RelEng could do all
> on your own--just make beetmover stop putting the Windows zips in places
> where people download them. If the build tasks still generate them as
> artifacts that's just an internal implementation detail.

I think the main point of this bug is that there's no reason why builds on Windows should always produce *two* archives containing the Firefox binaries. One is sufficient. If we want to standardize on tarballs for automation, that also makes sense.

> More germane to the points in bug 1494788, maybe we should simply make the
> build system produce a tarball on all platforms, and then post-signing
> repackaging tasks could be responsible for producing the final user-visible
> packages like installers/DMGs. (Which they would continue to do with the
> assistance of the build system.)

+1

(In reply to Kim Moir [:kmoir] ET from comment #20)
> catlee: Is this the approach you'd like your team to take? If so do you have
> any suggestions of timeframes for this work so we could coordinate the work
> between teams? This will help me decide who I'll ask to work on it.  We
> discussed this bug in our team meeting today and Greg also suggested using
> zstandard tarballs for better compression.

Yeah, sounds like the right direction. We have no particular time frame in mind...it's just a slight inefficiency in the build/release process that it would be nice to fix up someday.

zstd tarballs are an interesting idea. Do tools like mozregression support unpacking zstd archives as well?

Comment 22

[Chris AtLee [:catlee]]

Re-naming the bug based on the latest discussion here.

Comment 23

[Kim Moir [:kmoir] ET]

ni mshal and chmanchester so can provide any feedback on this approach

Comment 24

[Michael Shal [:mshal]]

It looks like mozregression uses mozinstall to unpack things, and they would both need to be updated to support zstd and/or a new extension if we need to be able to pull those packages from the build task. However since it sounds like the proposal is to just use tarballs internally, we could have the build system always generate a zstd tarball and then have repackage convert to the user-facing file format at the end. I think we'd need to then sort out how the repackage task works for non-signed builds, or if we just need to update mozregression and artifact builds to be able to use tarballs as well.

Comment 25

[Chris Manchester (limited bugmail, email directly)]

If we're vigilante about updating clients (and providing a simple way to unpack and run a build with widely available tools) I don't see any problem with this approach.

Comment 26

[Aki Sasaki (not active)]

Aiui, we still produce zips in CI, but we no longer ship them.

In nine years there hasn't been enough momentum to fully resolve this bug. I'm happy to mentor anyone who wants to drive this to 100% done: changing the format we create; changing any signing logic; any beetmover logic; any test download/extract logic, etc., etc. But in the absence of someone to own this, let's -> RESO INCOMPLETE