stop building/providing zip files for windows

NEW
Unassigned

Status

P3
normal
5 years ago
a month ago

People

(Reporter: bhearsum, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Our zip builds are not officially supported. We don't ship them as part of betas or releases. Today we discovered that there's even some differences between the that cause certain updates to apply differently (bug 922227). There's probably other differences we haven't thought of, too.

We should stop building and shipping these because:
1) They take up machine time unnecessarily
2) We don't require them to build l10n or MARs anymore
3) It's confusing to offer the zip and the installer

Some people have mentioned that it's convenient to have them because they're a little easier to work with for testing, but our installers are very simple to extract by hand these days;
1) 7z x firefox-installer.exe
2) you have a runnable firefox in the "core" directory

In my opinion, the benefits outweigh the negatives.

Comment 1

5 years ago
Things like mozregression will need updating to support the installers; have notified a few people :-)
You don't necessarily need 7z to extract the installer. You can also use the silent installer at least for the full build (not stub installer) like ´firefox-setup.exe /S /D=path´ or use mozinstall (pip install mozinstall).
OS: Linux → Windows 7
Hardware: x86_64 → All

Comment 3

5 years ago
Are there corporate setups that would prevent the installer to run, but let the zip work? I don't know jack about both Windows and corporate setups, but I recall something like that being listed as use-case for the zips.
AFAIK we only offer zip files for nightly builds but not releases, beta, nor candidate builds.
Sounds like we need a post to the newsgroups before we can do this - there's probably other use cases we're missing here. I'll try to send that out in the next couple of weeks.

Comment 6

5 years ago
Earlier today I emailed the internal auto-tools@ alias and also posted to mozilla.tools, though guess we may need further outreach at some point :-)

https://groups.google.com/d/msg/mozilla.tools/1EUBGcPW4Rw/4J9XJJtPkxsJ
In general I believe all of our automated testing relies on downloading the zip file on Windows and not the installer, so we'd need to sort that all out.

On a more personal note, I've always used the zip files because they're less of a hassle than the installer. I could get over myself though if there are compelling reasons to drop them.
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #8)
> In general I believe all of our automated testing relies on downloading the
> zip file on Windows and not the installer, so we'd need to sort that all out.
> 
> On a more personal note, I've always used the zip files because they're less
> of a hassle than the installer. I could get over myself though if there are
> compelling reasons to drop them.

Mozharness should handle installing from a .zip vs .exe automatically (via mozinstall), it gets the url from the buildprops.json, so I'm not sure what changes if any would need to be done there.

Comment 10

5 years ago
(In reply to Ed Morley [:edmorley UTC+1] from comment #6)
> Earlier today I emailed the internal auto-tools@ alias and also posted to
> mozilla.tools, though guess we may need further outreach at some point :-)
> 
> https://groups.google.com/d/msg/mozilla.tools/1EUBGcPW4Rw/4J9XJJtPkxsJ

I don't think that newsgroup is the right forum, this will affect more than the target audience of that newsgroup.

I object to us dropping zip packages, since not everybody's system has 7z and not everybody knows that's what they should do to extract the contents of the package (I didn't know until a few minutes ago.)
(In reply to :Ehsan Akhgari (needinfo? me!) from comment #10)
> (In reply to Ed Morley [:edmorley UTC+1] from comment #6)
> > Earlier today I emailed the internal auto-tools@ alias and also posted to
> > mozilla.tools, though guess we may need further outreach at some point :-)
> > 
> > https://groups.google.com/d/msg/mozilla.tools/1EUBGcPW4Rw/4J9XJJtPkxsJ
> 
> I don't think that newsgroup is the right forum, this will affect more than
> the target audience of that newsgroup.
> 
> I object to us dropping zip packages, since not everybody's system has 7z
> and not everybody knows that's what they should do to extract the contents
> of the package (I didn't know until a few minutes ago.)

Where would you suggest we have the discussion?
Flags: needinfo?(ehsan)

Comment 12

5 years ago
(In reply to comment #11)
> (In reply to :Ehsan Akhgari (needinfo? me!) from comment #10)
> > (In reply to Ed Morley [:edmorley UTC+1] from comment #6)
> > > Earlier today I emailed the internal auto-tools@ alias and also posted to
> > > mozilla.tools, though guess we may need further outreach at some point :-)
> > > 
> > > https://groups.google.com/d/msg/mozilla.tools/1EUBGcPW4Rw/4J9XJJtPkxsJ
> > 
> > I don't think that newsgroup is the right forum, this will affect more than
> > the target audience of that newsgroup.
> > 
> > I object to us dropping zip packages, since not everybody's system has 7z
> > and not everybody knows that's what they should do to extract the contents
> > of the package (I didn't know until a few minutes ago.)
> 
> Where would you suggest we have the discussion?

At the very least, dev.platform, and maybe firefox-dev as well.

Updated

5 years ago
Flags: needinfo?(ehsan)
I've frequently used the zip files for debugging Windows crashes on Linux when all we had to debug was information in crash-stats.  All the tools needed to do this come with typical Linux distributions (objdump on Linux handles Windows binaries), so I could correlate disassembly to a crash report for a Windows build based on downloading the zip file.  If there are extra tools that need to be downloaded (rather than using the distro's package installer), that makes debugging Windows crashes a good bit harder for me.
(Is the 7z command provided by the Ubuntu package p7zip-full the one you refer to in comment 0?)
Oh, then again, I have a command alias:

alias unmar-full-update='MAR=~/builds/mozilla-central/obj/firefox-debugopt/dist/host/bin/mar ~/builds/mozilla-central/mozilla/tools/update-packaging/unwrap_full_update.sh'

which suggests that when I ran into this problem for releases, I unpackaged the full update to get what I needed.

I'm not crazy about having so many steps for developers to figure this stuff out; it's extra knowledge about how to get around in our ecosystem that needs to be cargo-culted around.  But I suppose this one change doesn't make things that much worse for the crash-debugging case.
And that's why we have mach commands, Python packages like mozdownload and mozregression, etc. Somebody writes it once, posts it on PyPI or in the tree, and the technically hard part of the problem is solved.

If the tools suck, update the tools. Bad or non-existent tooling for developers shouldn't be a blocker for this not happening. Delaying it, perhaps.

Updated

2 years ago
Priority: -- → P3
(Assignee)

Updated

6 months ago
Component: General Automation → General
Product: Release Engineering → Release Engineering
Kim, is this something the build team could look into?
Flags: needinfo?(kmoir)
This seems at odds with the proposal in bug 1494788. It seems fine to stop *shipping* Windows zip files, and that's something that RelEng could do all on your own--just make beetmover stop putting the Windows zips in places where people download them. If the build tasks still generate them as artifacts that's just an internal implementation detail.

More germane to the points in bug 1494788, maybe we should simply make the build system produce a tarball on all platforms, and then post-signing repackaging tasks could be responsible for producing the final user-visible packages like installers/DMGs. (Which they would continue to do with the assistance of the build system.)
> maybe we should simply make the build system produce a tarball on all platforms, and then post-signing repackaging tasks could be responsible for producing the final user-visible packages like installers/DMGs. (Which they would continue to do with the assistance of the build system.)

This definitely seems like a sensible approach. Particularly since we currently only consume the zip on windows, and use the installer from a latter task anyway.

Updated

a month ago
See Also: → bug 1494788
catlee: Is this the approach you'd like your team to take? If so do you have any suggestions of timeframes for this work so we could coordinate the work between teams? This will help me decide who I'll ask to work on it.  We discussed this bug in our team meeting today and Greg also suggested using zstandard tarballs for better compression.
Flags: needinfo?(kmoir) → needinfo?(catlee)
(In reply to Ted Mielczarek [:ted] [:ted.mielczarek] from comment #18)
> This seems at odds with the proposal in bug 1494788. It seems fine to stop
> *shipping* Windows zip files, and that's something that RelEng could do all
> on your own--just make beetmover stop putting the Windows zips in places
> where people download them. If the build tasks still generate them as
> artifacts that's just an internal implementation detail.

I think the main point of this bug is that there's no reason why builds on Windows should always produce *two* archives containing the Firefox binaries. One is sufficient. If we want to standardize on tarballs for automation, that also makes sense.

> More germane to the points in bug 1494788, maybe we should simply make the
> build system produce a tarball on all platforms, and then post-signing
> repackaging tasks could be responsible for producing the final user-visible
> packages like installers/DMGs. (Which they would continue to do with the
> assistance of the build system.)

+1

(In reply to Kim Moir [:kmoir] ET from comment #20)
> catlee: Is this the approach you'd like your team to take? If so do you have
> any suggestions of timeframes for this work so we could coordinate the work
> between teams? This will help me decide who I'll ask to work on it.  We
> discussed this bug in our team meeting today and Greg also suggested using
> zstandard tarballs for better compression.

Yeah, sounds like the right direction. We have no particular time frame in mind...it's just a slight inefficiency in the build/release process that it would be nice to fix up someday.

zstd tarballs are an interesting idea. Do tools like mozregression support unpacking zstd archives as well?
Flags: needinfo?(catlee)
You need to log in before you can comment on or make changes to this bug.