Closed Bug 814009 Opened 12 years ago Closed 11 years ago

Only generate windows 64 builds where supported

Categories

(Release Engineering :: General, defect, P2)

x86_64
Windows 7
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: benjamin, Assigned: armenzg)

References

Details

(Whiteboard: Current plan in https://groups.google.com/forum/#!msg/mozilla.dev.apps.firefox/DOihL2429NM/ZxXhgVt7Dm0J)

Attachments

(5 files)

Per newsgroup discussion. Please stop building windows 64 builds and tests. This includes the following subtasks, which I'm not filing specific bugs on but you may want to break these out:

* stop building win64 nightlies
* repatriate existing win64 nightly users onto win32 builds using a custom update
* stop doing win64 "hourly" builds on mozilla-central and other branches
* disable the win64 option in try/trychooser

This bug is not the place to argue about this decision, which has already been made. If there is critical data which you think should be heard about this decision, please post it to mozilla.dev.apps.firefox.
Sounds good. I had started this same thread last week to reduce load and increase capacity.

The only part I don't know how to do is the migration of users. I will ask around and see who can pick it up.
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #0)
> * repatriate existing win64 nightly users onto win32 builds using a custom
> update

Do we know that there's a worthwhile number of users to bother with this? If so, I suggest we file a separate bug for this, and possibly do it after we turn off builds.
Last I heard, it's 50% of our nightly testers.
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #0)
> Per newsgroup discussion. Please stop building windows 64 builds and tests.

> * repatriate existing win64 nightly users onto win32 builds using a custom
> update

Best way is to remove the update button, adding small info that Mozilla no longer support Win64 Nightly builds and add a link to http://nightly.mozilla.org. 
And not by force them to use win32 Nightly, i'm pretty sure, most of the Win64 will no longer use firefox Nightly.
Just to include for my own future reference (probably stated elsewhere as well)

<Callek|buildduty> bsmedberg: as a curiosity to help inform my mind here, will we still want w64 for w8-based builds+tests?
<bsmedberg> no
<Callek|buildduty> bsmedberg: so even for w8 tests w32 is all we need?
<bsmedberg> yes
Depends on: 814116
Depends on: 814180
Depends on: 812966
(In reply to Phil Ringnalda (:philor) from comment #3)
> Last I heard, it's 50% of our nightly testers.

Yes, of course.

There was no 64bit release.

(In reply to SpeciesX from comment #4)
> (In reply to Benjamin Smedberg  [:bsmedberg] from comment #0)
> > Per newsgroup discussion. Please stop building windows 64 builds and tests.
> 
> > * repatriate existing win64 nightly users onto win32 builds using a custom
> > update
> 
> Best way is to remove the update button, adding small info that Mozilla no
> longer support Win64 Nightly builds and add a link to
> http://nightly.mozilla.org. 
> And not by force them to use win32 Nightly, i'm pretty sure, most of the
> Win64 will no longer use firefox Nightly.

Sure. I for one won't be running Firefox on Windows at all. I upgraded to a 64bit nightly after I hit the 32bit address space limit with Mozilla's resource usage.

Time to look for a new browser for Windows I guess. Now since most nifty Firefox extensions either don't work even in Firefox or are ported to other browsers as well this should not be much of a problem.
(In reply to Phil Ringnalda (:philor) from comment #3)
> Last I heard, it's 50% of our nightly testers.
So, how many of them Mozilla ready to loose after this change? )
(In reply to Michal 'hramrach' Suchanek from comment #6)

(In reply to Phoenix from comment #7)

Bugzilla is not the venue to discuss this, the decision has been made. If you contest with the decision please see the thread where it *was* discussed which is in the public at https://groups.google.com/d/topic/mozilla.dev.planning/EmXonELGjVM/discussion

Commenting here with "me too"'s or "why's" or "please don't" type comments is disruptive to those of us who actually work on the problem, and may result in your account getting banned.

For more information regarding the ettiquette of Bugzilla please see: https://bugzilla.mozilla.org/page.cgi?id=etiquette.html
Whiteboard: [see comment#8 before commenting]
Will an update from Win64 to Win32 be offered to users as standard update or it will be too much burden and we need to update it manually by reinstalling Win32 build ?
(In reply to Virtual_ManPL [:Virtual] from comment #9)
> Will an update from Win64 to Win32 be offered to users as standard update or
> it will be too much burden and we need to update it manually by reinstalling
> Win32 build ?

That is indeed planned, and is what Ben had meant with "* repatriate existing win64 nightly users onto win32 builds using a custom update"

We're still figuring out timeframe/whats needed to do that though. But we intend to do it at about the same time as when we actually turn off w64 nightlies.
I have a question. Will Firefox ever have any 64-bit in future? As in, is there any chance that, as time changes, we will start seeing 64-bit builds again? Or is this a forever nail to the coffin?

Anyway, worth noting is, IE 64-bit never seemed Microsoft's priority (IE9 64-bit for example, lacking feature(s) that enhance the performance), but, Opera for example seems to be doing well with it's newly introduced, stable builds.

And yes, half of nightly users might stop using nightly builds if 64-bit builds are removed.

So yea, in the time when the browsers are moving so fast, will we see any re-consideration in the unseen future.
Summary: Disable windows 64 builds → Disable windows 64 builds for now
Why? This has to be one of the lamest things to do. Opera has even come out with a 64 Bit Version.
As a reminder folks, this is a work item bug for the Release Engineering team. We are NOT the ones that made this decision, and discussing the issue here is not the right venue. IRC and newsgroups should be your venues for that.
(In reply to Ben Hearsum [:bhearsum] from comment #13)
>We are NOT the ones that made this decision, and discussing the issue
> here is not the right venue. IRC and newsgroups should be your venues for
> that.

Only one person has decided to kill Win64 build...
The Products team made the recommendation back in March https://groups.google.com/d/msg/mozilla.dev.planning/EmXonELGjVM/fRynmbLObUoJ

I reply to the thread hoping we do something slightly different.
:(
then a LOT of work needs to be done informing folks out there that Firefox 32bit will run fine on a 64bit OS like Win7. 

NO where have I seen that info in any of the mostly negative press.
It's f...gly amazing. You guys manage to waste 100M$/year on software development and then... fail to create 64-bit build? In all these years?! Holy ****, maintainers of Linux distro I'm also using were able to create working 64-bit builds for at least 8 years at much smaller budget. Why the heck some third-party guys can do your job better than you? ****, what you're doing when wasting 100M$? Buying some expensive yachts to top managers or something like that? You've also ditched Thunderbird. But where to you waste all these 100M bucks??? And where is any effect of this waste? I think you should really fire some of top managers, etc. Else you will become part of history.
Can the platform be changed from Windows 7 to Windows in general or include Windows 8?
How does are 64-bit releases like Waterfox and Palemoon affected by killing the 64-bit build?
What is the future of SpiderMonkey ? I am using recent version of this JavaScript engine and I depend on x64 bit builds for Windows.
All - I understand many of you are upset this decision, and have questions. Please take them to dev.apps.firefox (https://groups.google.com/forum/?fromgroups#!forum/mozilla.dev.apps.firefox) or another newsgroup. Most of the folks who can answer them are not subscribed to this bug.
In a german article on heise.de, it says that you stop providing win64 builds partly because plugins are not available in 64bit versions and thus the user experience would be bad.
Now, that you chicken out of providing a win64 build, we really have a chicken-egg problem going on. Now, companies like Adobe can further delay any effort in making a 64Bit Flash plug-in or a 64Bit adobe reader plugin. Isn't that a pretty bad strategy?

Also, what happened to the plugin-container concept? Shouldn't you be able to run 32Bit plugin in an external process? I believe opera does it like that.
The people that can address your concerns are not on this page.

Please please please... move the discussions to https://groups.google.com/forum/?fromgroups#!forum/mozilla.dev.apps.firefox

We can easily lock this bug from further commenting but we don't want to do that.
We can hear you on the forums and it is much easier to discuss this with the people that can make changes to the proposal.
Whiteboard: [see comment#8 before commenting] → Please move the discussions to https://groups.google.com/forum/?fromgroups#!forum/mozilla.dev.apps.firefox
>  Please move the discussions to https://groups.google.com/forum/
And why I'm supposed to have google account to be able to comment?
(In reply to js8080 from comment #25)
> >  Please move the discussions to https://groups.google.com/forum/
> And why I'm supposed to have google account to be able to comment?

You don't, it's just the most convenient interface. You may also post through the "mozilla.dev.apps.firefox" NNTP group. Or you can subscribe via the mailing list here: https://lists.mozilla.org/listinfo/dev-apps-firefox
bhearsum, it's not THAT simple, at least in some countries. These said NNTP groups are accessible from some providers, yes; but to post, you will need a paid account. Many ISPs consider Usenet simply 'clinically dead' and will no longer support it in a read-write fashion unless you agree on extra option which costs you extra money, too.

The new Google Groups interface is just horrible, and there is no longer a way to opt out - so for me, I will not use these to post for sure.

... and to comment 24:
"Please please please... move the discussions to https://groups.google.com/forum/?fromgroups#!forum/mozilla.dev.apps.firefox"

Just read what bsmedberg wrote on Nov 22 (read: YESTERDAY)
https://groups.google.com/forum/#!msg/mozilla.dev.planning/EmXonELGjVM/rchkN2iMdL8J

"Please let us consider this discussion closed unless there is critical new information which needs to be presented."

Ahah. So just shortly after it has been started, he wants to consider the discussion closed again. And now you come along suggesting that we should continue the discussion on m.d.a.f. anyway.
There's something here that doesn't make sense logically, IMHO.

Plus, just my 2 cents, this ridiculous number of 27 posts can't really be called a discussion. To me it rather sounds like "there was way too many negative comments than I expected, so let's play the stick-in-the-mud in order not to let user-side bashing no longer come near you." Way to go, I must say.
some major pubic relation blunders going on here ... 

after seeing in the news that Mozilla is dropping Thunderbird and 64 bit Firefox .. a LOT of folks wondering what is going on. 

matters not if it's true or false, it's the perception that something major is going wrong with Mozilla. 

perhaps the folks that made this decision should be joining in here for some frank feedback.
Disgusting
****************************************************************************************
Go to forums or mailing lists or newsgroups (all mentioned above already) or your blog or a random internet page to post your useless statements that don't add ANY value to this report.
I'm interested in following the process of this bug report, not comments addressed to Mozilla management that will likely NOT read this technical report anyway.
Please understand what bugtrackers are for, and even more what they ARE NOT for. Thanks.
****************************************************************************************
How long does it take you to dump the x64 branch? Why is this "bug" here in the first place? Complain about this bug being here, not us commenting.
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #0)
> * disable the win64 option in try/trychooser

Is this necessary? Are we removing win64 builders entirely? Or is this just an artifact of trychooser's inability to have default-off options (bug 691177)? If the latter, this would provide me (even better: someone else!) incentive to implement that.
(In reply to Steve Fink [:sfink] from comment #32)
> (In reply to Benjamin Smedberg  [:bsmedberg] from comment #0)
> > * disable the win64 option in try/trychooser
> 
> Is this necessary? Are we removing win64 builders entirely? Or is this just
> an artifact of trychooser's inability to have default-off options (bug
> 691177)? If the latter, this would provide me (even better: someone else!)
> incentive to implement that.

We're removing them entirely.
Assignee: nobody → coop
Status: NEW → ASSIGNED
Priority: -- → P2
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #0) 
> * repatriate existing win64 nightly users onto win32 builds using a custom
> update

I'm going to break this one out into a separate bug. It seems that most 64-bit users who have commented thus far would *not* like to end up on 32-bit by default, so this will require more work in the custom update.

> * stop building win64 nightlies
> * stop doing win64 "hourly" builds on mozilla-central and other branches
> * disable the win64 option in try/trychooser

bsmedberg: is it OK if I leave win64 as a non-default option in trychooser, i.e. you would need to explicitly select the option to get a win64 build? Anyone who wants to continue to work on win64 builds in the interim would still be able to.
Would that mean that the default trychooser syntax would be "try: -b do -p linux,linux64,macosx64,win32,android,android-armv6,android-noion,ics_armv7a_gecko,panda,otoro,unagi -u all -t none" and that people who either type -p all or get it from a trychooser hg extension would be always getting win64 (even after we inevitably break it), or did you mean that you're going to fix bug 691177?
(In reply to Phil Ringnalda (:philor) from comment #35)
> or did you mean that you're going to fix bug 691177?

Perhaps not me personally, but yes, I think fixing that is important. Making it easier to not build things by default attacks the load issue where it starts.
Depends on: 737661
Blocks: 784681
tweaking summary, per bsmedberg, we'll still generate 64bit windows builds as follows:

1) generate 64bit builds on mozilla-central, but not on mozilla-inbound, m-a/m-b/m-r or any project branches
** 64bit builds would be nightly builds only, not per checkin builds

2) 64bit builds on try, but not-by-default

3) 64bit builds only, there will not be any tests run on these 64bit builds
Summary: Disable windows 64 builds for now → Only generate windows 64 builds where supported
Attachment #698072 - Flags: review?(nthomas)
Attachment #698073 - Flags: review?(nthomas)
Attachment #698074 - Flags: review?(nthomas)
Depends on: 826010
Hey Armen, can you also kill the Win64 opt build on elm when you get a chance?
Bug 691177, which flopped on landing once but I believe has functional patches now, would allow turning off the try tests by default too. I don't know if that capability was factored into point #3 of comment 37 above.
Attachment #698072 - Flags: review?(nthomas) → review+
Attachment #698073 - Flags: review?(nthomas) → review+
Attachment #698074 - Flags: review?(nthomas) → review+
Attachment #698072 - Flags: checked-in+
Attachment #698073 - Flags: checked-in+
Attachment #698074 - Flags: checked-in+
None of these code changes where to disable builds but to remove the code that the 5 windows 7 64-bit machines were running (They were running just in case testing on that OS became tier1 again)

http://hg.mozilla.org/build/tools/rev/abf7cfe92ecf
http://hg.mozilla.org/build/buildbot-configs/rev/b93785e0b0da
http://hg.mozilla.org/build/puppet-manifests/diff/2a7328c269ea/modules/buildmaster/templates/BuildSlaves-tests.py.erb
in production
Is this why I'm stuck at Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:21.0) Gecko/20130109 Firefox/21.0 ID:20130109030942 ?
(In reply to alex_mayorga from comment #46)
> Is this why I'm stuck at Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:21.0)
> Gecko/20130109 Firefox/21.0 ID:20130109030942 ?

No, nightlies are still being generated.

The problem in your case is that the last few Nightlies have been broken:
https://tbpl.mozilla.org/php/getParsedLog.php?id=18669831&tree=Firefox#error3
Mr Morley,

And that's bug 812218 apparently, right?
Should I file a separate bug for the lack of updates since Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:21.0) Gecko/20130109 Firefox/21.0 ID:20130109030942 ?
(In reply to alex_mayorga from comment #49)
> Should I file a separate bug for the lack of updates since Mozilla/5.0
> (Windows NT 6.1; Win64; x64; rv:21.0) Gecko/20130109 Firefox/21.0
> ID:20130109030942 ?

There's no updates because the builds are broken. Looks like that's https://bugzilla.mozilla.org/show_bug.cgi?id=829738 or https://bugzilla.mozilla.org/show_bug.cgi?id=745093.
bug 830676 causes many websites broken on win64
(In reply to Yuan Pengfei from comment #51)
> bug 830676 causes many websites broken on win64

Hi Yuan, please follow up on that bug and not on this one. Remember that win64 is not tier1 for Mozilla and it will not get someone to fix it unless someone volunteers for it. Convincing Mozilla that win64 should be tier1 will require conversation on the mailing lists and not on this bug.

joduinn gives a summary of the current plan for this bug in comment 37:
https://bugzilla.mozilla.org/show_bug.cgi?id=814009#c37

The extended explanation by bsmedberg is in here:
https://groups.google.com/forum/#!msg/mozilla.dev.apps.firefox/DOihL2429NM/ZxXhgVt7Dm0J
Whiteboard: Please move the discussions to https://groups.google.com/forum/?fromgroups#!forum/mozilla.dev.apps.firefox → Current plan in https://groups.google.com/forum/#!msg/mozilla.dev.apps.firefox/DOihL2429NM/ZxXhgVt7Dm0J
(In reply to Armen Zambrano G. [:armenzg] from comment #52)
> (In reply to Yuan Pengfei from comment #51)
> > bug 830676 causes many websites broken on win64
> 
> Hi Yuan, please follow up on that bug and not on this one. Remember that
> win64 is not tier1 for Mozilla and it will not get someone to fix it unless
> someone volunteers for it. Convincing Mozilla that win64 should be tier1
> will require conversation on the mailing lists and not on this bug.
> 

I am trying to express that even if win64 builds are generated, they will not be usable.
Depends on: 834891
Status: ASSIGNED → NEW
Priority: P2 → P3
We're currently not being able to keep up with our current development load and it's about time to deal with this part of the plan.

On another note, would it make sense to book one of our branches only for Win64?
They could get nightly updates and they could backout any merges from mozilla-central that would cause a regression.


(In reply to John O'Duinn [:joduinn] from comment #37)
> tweaking summary, per bsmedberg, we'll still generate 64bit windows builds
> as follows:
> 
> 1) generate 64bit builds on mozilla-central, but not on mozilla-inbound,
> m-a/m-b/m-r or any project branches
> ** 64bit builds would be nightly builds only, not per checkin builds
> 
> 2) 64bit builds on try, but not-by-default
>
Filed as bug 855056.

> 
> 3) 64bit builds only, there will not be any tests run on these 64bit builds
This was completed on comment 44.
Assignee: coop → armenzg
Depends on: 855056
Priority: P3 → P2
DISCLAIMER: This patch does not touch the win64 nightly builds.

This leaves the win64 dep builds running:
https://tbpl.mozilla.org/?jobname=WINNT 6.1 x86-64&showall=1

This would allow for win64 to be triggered on try with this syntax:
"try: -b o -p win64 -u none -t none" (tested it on my local machine)

This tackles points 1 and 2 of the current plan:
> 1) generate 64bit builds on mozilla-central, but not on mozilla-inbound,
> m-a/m-b/m-r or any project branches
> ** 64bit builds would be nightly builds only, not per checkin builds
> 2) 64bit builds on try, but not-by-default
Attachment #730681 - Flags: review?
Attachment #730681 - Flags: review? → review?(aki)
Attachment #730681 - Flags: review?(aki) → review+
Comment on attachment 730681 [details] [diff] [review]
leave win64 dep builds running on mozilla-central and try (when specified explicitely)

http://hg.mozilla.org/build/buildbot-configs/rev/b2d6983bf944

This should go live in the next reconfiguration of our masters.
Attachment #730681 - Flags: checked-in+
Zombie pending builds on mozilla-inbound have been removed from the scheduler DB.
1) We still generate win64 nightly updates
2) We have addressed all points on comment 37
3) Updating nightly win64 users to win32 is to happen on bug 834891
4) The trychooser website has been updated to not show win64 as a default platform 

Added this info to the mailing list:
https://groups.google.com/d/msg/mozilla.dev.apps.firefox/DOihL2429NM/ehfqgsoqqREJ

[1] http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/

(In reply to John O'Duinn [:joduinn] from comment #37)
> tweaking summary, per bsmedberg, we'll still generate 64bit windows builds
> as follows:
> 
> 1) generate 64bit builds on mozilla-central, but not on mozilla-inbound,
> m-a/m-b/m-r or any project branches
> ** 64bit builds would be nightly builds only, not per checkin builds
> 
> 2) 64bit builds on try, but not-by-default
> 
> 3) 64bit builds only, there will not be any tests run on these 64bit builds

[4] http://trychooser.pub.build.mozilla.org
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
We need win64 Thunderbird nightly builds.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Attachment #732476 - Flags: review?(aki) → review+
Comment on attachment 732476 [details] [diff] [review]
add comm-central on-change and nightly builds

This will be live in our next reconfiguration of our masters.
Attachment #732476 - Flags: checked-in+
Depends on: 857486
latest patch is in production
Status: REOPENED → RESOLVED
Closed: 11 years ago11 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.