Configure single buildslave in production to generate nightly Windows 64-bit builds

RESOLVED FIXED

Status

Release Engineering
General
P2
normal
RESOLVED FIXED
8 years ago
4 years ago

People

(Reporter: coop, Assigned: armenzg)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [win64][automation][q1goal])

Attachments

(2 attachments, 7 obsolete attachments)

(Reporter)

Description

8 years ago
We've run into a bit of a chicken-and-egg problem with getting the Windows 64-bit builds into a decent state. Without builds and logs (and eventually symbols) for developers to look at, we haven't been able to get much help from developers in debugging issues and getting builds to a nightly state. 

Armen has been able to generate a 64bit Windows build, so he has a toolchain that works. The build crashes right away though. We don't necessarily want to deploy the current toolchain across a collection of buildslaves until we're sure generated builds are usable.

I'm going to suggest we try something new/different.

After talking with joduinn this morning, we'd like to hook up Armen's single Win64 builder to production-master and have it build nightly builds on trunk. By doing this, we can iterate on the toolchain as required, trigger builds as required, and generate builds and logs for developers to use as we go.

Once we have working builds from an agreed-upon toolchain, we can scale this up.

We can put the Win64 nightly builds into the experimental/ dir until they (at least) start up. We should disable updates, testing, etc in the mozconfig to start.
(Assignee)

Updated

8 years ago
Blocks: 558448
Priority: P4 → P2
(Assignee)

Comment 1

8 years ago
I have hit bug 567937 (bustage on x64 build due to jsnativestack.cpp) as I was running builds this morning.

I should have builds being posted on ftp and being reported to tinderbox at the EOD.
Status: NEW → ASSIGNED
(Assignee)

Comment 2

8 years ago
Created attachment 447351 [details] [diff] [review]
[WIP] enable Win64 bit builds on staging

This patch enables Win64 nightly, dependent and debug builds for win 64 on staging.

The difference from win32 is that I have unit tests disabled, I have added "WINNT 6.1 x86-64" to the basename and "WINNT_x86_64-msvc" as the update channel.

I will ask for review once we start having green builds (it currently does not build because of bug 567937).

I am going to be using the production keys and pushing to ftp. Once I am satisfied I will be reporting to the Firefox's tinderbox page and ask for reviews then.
Attachment #447351 - Flags: feedback?(ccooper)
(Assignee)

Comment 3

8 years ago
Created attachment 447352 [details] [diff] [review]
[WIP] buildbotcustom patch

I still haven't deal with the debug builds but I will get to it.
(Reporter)

Comment 4

8 years ago
Comment on attachment 447351 [details] [diff] [review]
[WIP] enable Win64 bit builds on staging

As we discussed on our call, I just want to make sure that we only have nightlies enabled for opt/debug/xulrunner, and no dep builds (for now). We (buildduty) can force build the nightly builder by hand if we need interim builds.
Attachment #447351 - Flags: feedback?(ccooper) → feedback+
Did you mean just opt and xulrunner ? We're not doing debug nightlies for any other platforms right now.
(Reporter)

Comment 6

8 years ago
(In reply to comment #5)
> Did you mean just opt and xulrunner ? We're not doing debug nightlies for any
> other platforms right now.

We're constrained by only having a single builder. If we can get debug builds on strictly a nightly basis, I think that would be worthwhile, but we certainly should be building debug dep builds. Note that a debug "nightly" wouldn't need to be a true nightly (with branding, updates, etc.), I just want it to build once a day. 

In order of preference, I want:

* nightly opt
* nightly XR (a subset of the code, might be easier to debug early issues)
* nightly debug
Ah, I see what you mean.
(Assignee)

Comment 8

8 years ago
Created attachment 449096 [details] [diff] [review]
[WIP] add Win64 to configs (plus local modifications)
Attachment #447351 - Attachment is obsolete: true
(Assignee)

Comment 9

8 years ago
Created attachment 449097 [details] [diff] [review]
[WIP] buildbotcustom changes for Win64 (plus local changes)
Attachment #447352 - Attachment is obsolete: true
(Assignee)

Comment 10

8 years ago
Created attachment 449099 [details] [diff] [review]
[WIP] mozconfigs for Win64
(Assignee)

Comment 11

8 years ago
I just wanted to refresh the patches and keep track of everything that staging-master04 is running.

I will keep on working on bug 565402 so we can start cloning machines.

The machine is running once a day and posting everything as if it was a production machine.
I will keep the bug open but reduce the priority.
Priority: P2 → P4
(Assignee)

Comment 12

7 years ago
Created attachment 503969 [details] [diff] [review]
mozconfig files for win64

I need to get the mozconfig files into buildbot-configs so I don't blow away my user repo again and make the win64 nightly builds burn.
Attachment #449099 - Attachment is obsolete: true
Attachment #503969 - Flags: review?(bhearsum)
Comment on attachment 503969 [details] [diff] [review]
mozconfig files for win64

Do we really need to specify target/host? Shouldn't those be autodetected? I wouldn't mind getting Ted's feedback on this, too.
Attachment #503969 - Flags: review?(ted.mielczarek)
Attachment #503969 - Flags: review?(bhearsum)
Attachment #503969 - Flags: review+
Comment on attachment 503969 [details] [diff] [review]
mozconfig files for win64

This looks mostly fine, I just have a few comments. Some of these are things that I know you probably copy/pasted from the existing Windows mozconfigs, but I think we should clean some things up instead of just cargo-culting forever.

>diff --git a/mozilla2/win64/birch/debug/mozconfig b/mozilla2/win64/birch/debug/mozconfig
>new file mode 100644
>--- /dev/null
>+++ b/mozilla2/win64/birch/debug/mozconfig
>@@ -0,0 +1,14 @@
>+ac_add_options --target=x86_64-pc-mingw32
>+ac_add_options --host=x86_64-pc-mingw32
>+
>+ac_add_options --enable-application=browser
>+#ac_add_options --enable-jemalloc

Just leave this out instead of commenting it out.

>+ac_add_options --disable-optimize
>+ac_add_options --enable-debug
>+ac_add_options --enable-trace-malloc
>+ac_add_options --enable-debug-symbols

enable-debug-sybmols is redundant with enable-debug.

>diff --git a/mozilla2/win64/birch/nightly/mozconfig b/mozilla2/win64/birch/nightly/mozconfig
>new file mode 100644
>--- /dev/null
>+++ b/mozilla2/win64/birch/nightly/mozconfig
>@@ -0,0 +1,14 @@
>+ac_add_options --target=x86_64-pc-mingw32
>+ac_add_options --host=x86_64-pc-mingw32
>+
>+# It was not liked by Win64 
>+#mk_add_options PROFILE_GEN_SCRIPT='$(PYTHON) $(MOZ_OBJDIR)/_profile/pgo/profileserver.py'

Even if PGO builds don't work with Win64, this option shouldn't harm anything. (If you don't call "make -f client.mk profiledbuild", it's never used.)

>diff --git a/mozilla2/win64/birch/xulrunner/mozconfig b/mozilla2/win64/birch/xulrunner/mozconfig
>new file mode 100644
>--- /dev/null
>+++ b/mozilla2/win64/birch/xulrunner/mozconfig
>@@ -0,0 +1,16 @@
>+ac_add_options --target=x86_64-pc-mingw32
>+ac_add_options --host=x86_64-pc-mingw32
>+
>+export MOZILLA_OFFICIAL=1
>+export JAVA_HOME=/d/jdk1.6.0_14
>+
>+# For NSS symbols
>+export MOZ_DEBUG_SYMBOLS=1

Use --enable-debug-symbols instead.
Attachment #503969 - Flags: review?(ted.mielczarek) → review+
(In reply to comment #13)
> Comment on attachment 503969 [details] [diff] [review]
> mozconfig files for win64
> 
> Do we really need to specify target/host? Shouldn't those be autodetected? I
> wouldn't mind getting Ted's feedback on this, too.

I think we do since we're using the same MozillaBuild either way, so uname gives you the same result.
(Assignee)

Updated

7 years ago
Priority: P4 → P5
(Assignee)

Updated

7 years ago
Priority: P5 → P3
Whiteboard: [win64][automation] → [win64][automation][q1goal]
(Assignee)

Updated

7 years ago
Priority: P3 → P2
(Assignee)

Comment 16

7 years ago
Created attachment 513522 [details] [diff] [review]
[mozconfig] win64 mozconfigs v1

I have reduced it to just mozilla-central to make it cleaner.
Attachment #503969 - Attachment is obsolete: true
Attachment #513522 - Flags: review?(ted.mielczarek)
(Assignee)

Comment 17

7 years ago
Created attachment 513526 [details] [diff] [review]
[buildbot-configs] add win64 configuration parameters
Attachment #449096 - Attachment is obsolete: true
(Assignee)

Comment 18

7 years ago
Created attachment 513527 [details] [diff] [review]
[buildbotcustom] add win64 to our custom libraries
Attachment #449097 - Attachment is obsolete: true
Attachment #513527 - Flags: review?(bhearsum)
Comment on attachment 513527 [details] [diff] [review]
[buildbotcustom] add win64 to our custom libraries

I really don't like that the build system uses "win64-x86_64", but it's not worth changing at this point. I found some more things that need updating, though:
http://mxr.mozilla.org/build-central/source/buildbotcustom/status/generators.py#64
http://mxr.mozilla.org/build-central/source/buildbotcustom/steps/test.py#210
http://mxr.mozilla.org/build-central/source/buildbotcustom/steps/test.py#232
http://mxr.mozilla.org/build-central/source/buildbotcustom/steps/unittest.py#262
http://mxr.mozilla.org/build-central/source/buildbotcustom/process/factory.py#445
http://mxr.mozilla.org/build-central/source/buildbotcustom/l10n.py#96


We'll also have to update the partner-repack code at some point in the future, but that doesn't need to be addressed here.
Attachment #513527 - Flags: review?(bhearsum) → review-
Comment on attachment 513522 [details] [diff] [review]
[mozconfig] win64 mozconfigs v1

>diff --git a/mozilla2/win64/mozilla-central/debug/mozconfig b/mozilla2/win64/mozilla-central/debug/mozconfig
>new file mode 100644
>--- /dev/null
>+++ b/mozilla2/win64/mozilla-central/debug/mozconfig
>+ac_add_options --enable-debug-symbols
>+ac_add_options --enable-tests

Get rid of --enable-debug-symbols (it's implied with --enable-debug), and get rid of --enable-tests (it's the default).

>+
>+# For NSS symbols
>+export MOZ_DEBUG_SYMBOLS=1

This is no longer necessary.  (--enable-debug and --enable-debug-symbols set this properly for NSS on trunk)

>diff --git a/mozilla2/win64/mozilla-central/nightly/mozconfig b/mozilla2/win64/mozilla-central/nightly/mozconfig
>new file mode 100644
>--- /dev/null
>+++ b/mozilla2/win64/mozilla-central/nightly/mozconfig
>+ac_add_options --enable-tests

--enable-tests is the default. Let's not specify default options here, and leave them to configure.

>+# For NSS symbols
>+export MOZ_DEBUG_SYMBOLS=1

Use --enable-debug-symbols instead.

>diff --git a/mozilla2/win64/mozilla-central/xulrunner/mozconfig b/mozilla2/win64/mozilla-central/xulrunner/mozconfig
>new file mode 100644
>--- /dev/null
>+++ b/mozilla2/win64/mozilla-central/xulrunner/mozconfig
>+# For NSS symbols
>+ac_add_options --enable-debug-symbols

I don't think the comment is really necessary here, it's fairly obvious what this option does.

r=me with those few things fixed.
Attachment #513522 - Flags: review?(ted.mielczarek) → review+
(Assignee)

Comment 21

7 years ago
Comment on attachment 513522 [details] [diff] [review]
[mozconfig] win64 mozconfigs v1

Thanks ted! Taken your comments and committed to "default" with:
http://hg.mozilla.org/build/buildbot-configs/rev/61f8b2f31012
Attachment #513522 - Flags: checked-in+
(Assignee)

Comment 22

7 years ago
Comment on attachment 513527 [details] [diff] [review]
[buildbotcustom] add win64 to our custom libraries

I will be moving all the work of this bug into bug 565402 to consolidate all the work in the same place.

Attachment 520752 [details] [diff] obsoletes this one.
Review has been requested there.
Attachment #513527 - Attachment is obsolete: true
(Assignee)

Comment 23

7 years ago
We can produce nightly builds.
Any remaining work will continue in bug 565402 since I have moved any patches used in here over there.
Status: ASSIGNED → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.