1243448 - beta release source builder failing to run configure

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Jordan Lund (:jlund)]

because SingleSourceFactory doesn't use tooltool.

e.g. failure http://buildbot-master74.bb.releng.usw2.mozilla.com:8001/builders/release-mozilla-beta-firefox_source/builds/7

proposed solutions:

1) use tooltool in singlesourcefactory
2) force it to use a custom mozconfig
3) replace 'make source-package' with a custom script so we don't need configure at all

full irc (#releng):
07:49:00 <Callek> mshal: any insight? https://ftp.mozilla.org/pub/firefox/candidates/45.0b1-candidates/build1/logs/release-mozilla-beta-firefox_source-bm74-build1-build7.txt.gz
07:52:29 <mshal> Callek: did tooltool run? I see it get copied into the mock environment, but not actually run & download packages
07:52:35 <mshal> which is why it can't find gtk3's pkg-config
07:53:41 <Callek> mshal: bah! no..... we hadn't run tooltool in the source builder before...
07:53:48 <mshal> ahh
07:53:50 <Callek> mshal: this is also partly because we 'package' the source
07:54:13 <mshal> well I think this is the primary error: /builds/slave/rel-m-beta-fx_source-000000000/mozilla-beta/configure: line 18050: /builds/slave/rel-m-beta-fx_source-000000000/mozilla-beta/./gtk3/usr/local/bin/pkg-config: No such file or directory
07:54:21 <mshal> which happens because you don't have gtk3 downloaded from tooltool
07:54:53 <Callek> rail: bhearsum: any memory of why we run configure for the source builder?
07:55:19 <•bhearsum> because we use "make source-package" IIRC?  
08:00:14 <Callek> rail: mshal: sooo, to fix source builders we need to both run tooltool as part of the source builder now, and adjust excludes at https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/upload-files.mk#798
08:00:29 <Callek> rail: mshal: unless we can get away with --disable-compile-environment for source to avoid pkg-config?
08:01:45 <mshal> does it need to have gtk3 enabled in configure?
08:02:26 <Callek> mshal: it probably doesn't need/want gtk3 enabled
08:02:26 <rail> I guess we can use different .mozconfig for sources
08:02:49 <Callek> either way though, if we *require* gtk3 in configure we'll barf without gtk3
08:03:04 <mshal> is it hard to make it use a different mozconfig?
08:05:37 <rail> mshal: not at all: https://dxr.mozilla.org/build-central/source/buildbotcustom/process/release.py#458-460
08:06:59 <mshal> ahh, well unless there's some reason source builds need gtk3 enabled in configure, I think a custom mozconfig is probably the easiest
08:07:23 <mshal> (I don't know too much about what source builders actually do, though)
08:07:55 <rail> iirc just ./configure && make source-package hg-bundle 
08:10:30 <mshal> the configure is just needed to convert the top-level Makefile.in and such so the Makefile actually exists?
08:10:54 <mshal> or do we also ship the configuration with the source package?
08:11:49 <mshal> hmm, looks like .mozconfig* is excluded
08:17:08 <mshal> maybe we should just replace 'make source-package' with a small shell or python script :)
08:18:15 <rail-brb> yeah, that would be great

Comment 1

[Jordan Lund (:jlund)]

Attachment: 160127_source_builder_gtk_mozconfig-gecko.patch

courtesy of callek. I am proposing to land this as is now for two reasons:

1) it's somewhat tested
2) it fixes beta1

long term though, nick raised the point of releng naturally 'owning' this mozconfig and thus also having this bite us down the line.

callek: is there a way we can edit this mozconfig so essentially we do something like:

cat $source_mozconfig

# take all of the original
. $topsrcdir/browser/config/mozconfigs/linux64/release
# add/remove the custom bits we need
ac_add_options --disable-compile-environment

I guess it doesn't really matter if we bloat this mozconfig with things we don't need but what may restrict us is including things we can't have..

if r+, I'll land on m-b directly and re-tag

Comment 2

[Jordan Lund (:jlund)]

Attachment: 160127_source_builder_gtk_mozconfig-bbotcustom.patch

Comment 3

[Jordan Lund (:jlund)]

Attachment: 160127_source_builder_gtk_mozconfig-gecko-mozharness.patch

this is for promotion to pick up the fix.

interestingly, it uses 'nightly' mozconfig right now..

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8712895 [details] [diff] [review]
160127_source_builder_gtk_mozconfig-gecko.patch

r+ to get us back on track, but I'd like to avoid this divergence in the longterm (ie mozharness source script runs tooltool to provide gtk3 (or whatever), and we go back to the linux64 opt mozconfig).

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8712896 [details] [diff] [review]
160127_source_builder_gtk_mozconfig-bbotcustom.patch

This is a global change which I think is going to break release & esr38 for Firefox. Thunderbird is OK because it defines source_mozconfig already, but those two Firefox don't. 

There's two options I can see
* land this, then fix m-r and m-esr38 by setting source_mozconfig in their configs back to ../release; remove source_config for m-r when we merge in a few weeks
* not land this, and set source_mozconfig for beta instead; then set it for release when we merge in a few weeks

It's not clear which of those works better for esr45, we're going to have to pay attention either way, and anyway this is going to be superceded by release promotion. So lets take option 2, since it's marginally less work to land.

Comment 6

[Jordan Lund (:jlund)]

Attachment: 160127_source_builder_gtk_mozconfig-bbotcustom.patch

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8712972 [details] [diff] [review]
160127_source_builder_gtk_mozconfig-bbotcustom.patch

Looks like attachment 8712896 [details] [diff] [review] again.

Comment 8

[Jordan Lund (:jlund)]

Attachment: 160127_source_builder_gtk_mozconfig-bbotcfg.patch

lol oops

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 8713033 [details] [diff] [review]
160127_source_builder_gtk_mozconfig-bbotcfg.patch

hijacking!

Comment 10

[Jordan Lund (:jlund)]

Comment on attachment 8712895 [details] [diff] [review]
160127_source_builder_gtk_mozconfig-gecko.patch

landed directly on beta for now. going to backport to inbound and uplift to m-a if this works

Comment 11

[Jordan Lund (:jlund)]

Comment on attachment 8713033 [details] [diff] [review]
160127_source_builder_gtk_mozconfig-bbotcfg.patch

thanks! https://hg.mozilla.org/build/buildbot-configs/rev/8b8b715b6951

Comment 12

[Jordan Lund (:jlund)]

this worked! need to land on other repos tomorrow and follow up with different mozconfig

Comment 13

[Justin Wood (:Callek)]

(In reply to Jordan Lund (:jlund) from comment #12)
> this worked! need to land on other repos tomorrow and follow up with
> different mozconfig

I can confirm; if the *only* thing in the mozconfig is:

ac_add_options --disable-compile-environment

We actually are just fine. I did get a slightly different tar.xz sha compared to the bigger version here, though I can't seem to explain that difference considering after unpacking the .tar.xz's into two different directories `diff -r --brief <dir1> <dir2>` yielded *zero* differences.

As did a listing of md5sums compared from both archives extracted files: `find firefox-44.0b4 -type f -print0 | xargs -I{} -0 md5sum {}`

Comment 14

[Wayne Mery (:wsmwk)]

callek indicates we need this for Thunderbird 45 beta, which we'd like to do very soon this week.

Comment 15

[Jordan Lund (:jlund)]

Attachment: 160201_bug_1243448_source_builder_mozconfig-gecko.patch

to double check, as per our discussion over irc, you are suggesting that a mozconfig with only this will work?

Comment 16

[Justin Wood (:Callek)]

Comment on attachment 8714413 [details] [diff] [review]
160201_bug_1243448_source_builder_mozconfig-gecko.patch

Review of attachment 8714413 [details] [diff] [review]:
-----------------------------------------------------------------

Yes I could confirm that it worked!

Comment 17

[Justin Wood (:Callek)]

Comment on attachment 8714413 [details] [diff] [review]
160201_bug_1243448_source_builder_mozconfig-gecko.patch

Review of attachment 8714413 [details] [diff] [review]:
-----------------------------------------------------------------

::: browser/config/mozconfigs/linux64/source
@@ +1,1 @@
> +ac_add_options --disable-compile-environment

Though admittedly I would mark a comment (something like):

# The source "build" only needs a mozconfig because we use the build system as our script for generating it. This allows us to run configure without any extra dependencies on specific toolchains, e.g. gtk3.

Comment 18

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/fb8eef56e9de

Comment 19

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/fb8eef56e9de