1350011 - comm-central Linux builds fail with: make[5]: *** [force-cargo-library-build] Error 101

Status: RESOLVED FIXED

(Thunderbird :: Build Config, defect)

Description

[Jorg K (CEST = GMT+2)]

https://treeherder.mozilla.org/#/jobs?repo=comm-central&revision=ea538d555890a7fa33b97de780bb5d09b230f428&selectedJob=85973554

From the log:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error(PkgConfig(Failure { command: "\"pkg-config\" \"--libs\" \"--cflags\" \"x11 >= 1.4.99.1\"", output: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "Package x11 was not found in the pkg-config search path.\nPerhaps you should add the directory containing `x11.pc\'\nto the PKG_CONFIG_PATH environment variable\nNo package \'x11\' found\n" } }), State { next_error: None })', /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libcore/result.rs:837
stack backtrace:

Comment 1

[Jorg K (CEST = GMT+2)]

Any hints what we need to configure/port? Aryx said something to me on IRC (which I didn't understand), but I didn't log the channel and lost it.

Comment 2

[Rail Aliiev [:rail]]

Sounds like missing x11 headers/libraries. I'd expect "configure" to fail, not cargo.

Comment 3

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Sorry, am no help here.

So this is from https://dxr.mozilla.org/mozilla-central/rev/e03e0c60462c775c7558a1dc9d5cf2076c3cd1f9/third_party/rust/x11/Cargo.toml#39

error: failed to run custom build command for `x11 v2.12.1`
process didn't exit successfully: `/builds/slave/tb-c-cen-lx-000000000000000000/build/objdir-tb/toolkit/library/release/build/x11-9858dee221fb8695/build-script-build` (exit code: 101)

2.12.0 is 3 months old: https://github.com/rust-lang/crates.io-index

Comment 4

[Jorg K (CEST = GMT+2)]

Anyone has any idea what's going on here? Rail moved this to Thunderbird::Build Config, and then said:
"Sounds like missing x11 headers/libraries." So where/why/when are they or did they go missing and how do we restore them? After all, this runs on the treeherder servers, or does it also fail locally now?

Comment 5

[:aceman]

I don't get this failure locally (on linux). Has it been fixed in m-c since? I have rustc 0.16, not 0.15.1 as on the buid servers.

Comment 6

[Jorg K (CEST = GMT+2)]

Aceman and I have been looking at some logs:

TB Tinderbox:
https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-central-linux/1490274624/comm-central-linux-bm72-build1-build1.txt.gz
libX11                  i686   1.3-2.el6           centos6               585 k
libX11                  x86_64 1.3-2.el6           centos6               582 k

Required is > 1.4.99 to fit with Rust.

On the FF build machines which use Taskcluster we see:
https://public-artifacts.taskcluster.net/TlUuz9jsS6ywcFr9g4FUYw/0/public/logs/live_backing.log
[task 2017-03-23T21:57:47.645888Z] 21:57:47     INFO - 0xf14a8000 - 0xf15e1fff  libX11.so.6.3.0
which according to the internet equates to a package version of 1.6.3.

Sorry, Rail, is this really a Thunderbird::Build config issue?

It looks to me that Rust got upgraded and all those Tinderbox Linux machines are now incompatible since they still have X11 1.3.

Can they be upgraded? Or is this the end of Thunderbird on Linux unless it moves to Taskcluster?

Comment 7

[Edmund Wong (:ewong)]

(In reply to Jorg K (GMT+1) from comment #4)
> Anyone has any idea what's going on here? Rail moved this to
> Thunderbird::Build Config, and then said:
> "Sounds like missing x11 headers/libraries." So where/why/when are they or
> did they go missing and how do we restore them? After all, this runs on the
> treeherder servers, or does it also fail locally now?

I'll take a gander..  not entirely sure what's going on.

Comment 8

[Edmund Wong (:ewong)]

(In reply to Jorg K (GMT+1) from comment #6)
> Aceman and I have been looking at some logs:
> 
> TB Tinderbox:
> https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-central-
> linux/1490274624/comm-central-linux-bm72-build1-build1.txt.gz
> libX11                  i686   1.3-2.el6           centos6               585
> k
> libX11                  x86_64 1.3-2.el6           centos6               582
> k
> 
> Required is > 1.4.99 to fit with Rust.
> 
> On the FF build machines which use Taskcluster we see:
> https://public-artifacts.taskcluster.net/TlUuz9jsS6ywcFr9g4FUYw/0/public/
> logs/live_backing.log
> [task 2017-03-23T21:57:47.645888Z] 21:57:47     INFO - 0xf14a8000 -
> 0xf15e1fff  libX11.so.6.3.0
> which according to the internet equates to a package version of 1.6.3.
> 
> Sorry, Rail, is this really a Thunderbird::Build config issue?
> 
> It looks to me that Rust got upgraded and all those Tinderbox Linux machines
> are now incompatible since they still have X11 1.3.
> 
> Can they be upgraded? Or is this the end of Thunderbird on Linux unless it
> moves to Taskcluster?

From what I've discerned from both logs:

TB uses mock.
Firefox doesn't and ergo uses the x11 libs from the system itself.

Now if someone in Releng can confirm(or deny), TB and Firefox uses AWS 
instances and possibly similar AMIs and ergo similiar operating systems
(if not the same).

That said, I'm wondering if disabling Mock on Linux32 might work? 

i.e. 
https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/thunderbird_config.py#l140
            'use_mock': True,

change that to 'use_mock': False

(Again, just a left-field suggestion since I know absolutely nothing
of the stuff that goes in AMIs).

Comment 9

[Edmund Wong (:ewong)]

fwiw..  SeaMonkey's also hitting this, but unlike TB, our infra isn't that
up to date so I doubt we'll be able to depend on the system's X11 libs.

Comment 10

[Rail Aliiev [:rail]]

(In reply to Jorg K (GMT+1) from comment #6)
 > Sorry, Rail, is this really a Thunderbird::Build config issue?

Now it is clear to say that is a build environment issue, not build config.


> It looks to me that Rust got upgraded and all those Tinderbox Linux machines
> are now incompatible since they still have X11 1.3.
> 
> Can they be upgraded? Or is this the end of Thunderbird on Linux unless it
> moves to Taskcluster?

Upgrading this would be not trivial. Mock based builds use Centos 6.2, which is very ancient. Probably there is a way to use Centos 6.8 in mock (similar to https://dxr.mozilla.org/build-central/source/puppet/modules/runner/files/mockbuild-config-templates/mozilla-centos6-x86_64.cfg), but it's completely untested.


(In reply to Edmund Wong (:ewong) from comment #8)
> TB uses mock.
> Firefox doesn't and ergo uses the x11 libs from the system itself.
> 
> Now if someone in Releng can confirm(or deny), TB and Firefox uses AWS 
> instances and possibly similar AMIs and ergo similiar operating systems
> (if not the same).

Unfortunately, the Tc-based build environment is a bit different from what we use for buildbot. TC uses Centos6 latest (6.8 atm) https://dxr.mozilla.org/mozilla-central/source/taskcluster/docker/centos6-build/Dockerfile, buildbot uses Centos 6.2 (https://s3-external-1.amazonaws.com/mozilla-releng-mock-centos-us-east-1/mirrors/centos/6/latest/centos-version.txt)

> 
> That said, I'm wondering if disabling Mock on Linux32 might work? 
> 
> i.e. 
> https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/
> thunderbird_config.py#l140
>             'use_mock': True,
> 
> change that to 'use_mock': False

I don't think this is going to work. With mock disabled, buildbot will use the existing host environment and won't be able to install any packages (it's not running as root).

Sounds like we finally hit the wall :/ Fixing the existing build environment is not trivial, neither porting to TC...

Comment 11

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Would it be possible to use tooltool or similar to include a version of rustc that works, maybe statically compiled to not depend on the system libX11?

Comment 12

[Jorg K (CEST = GMT+2)]

Attachment: 1350011-linux-x11.patch

This appears to work (so far, usually stopped after 14 minutes):
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=27f10ccf5e60c61065210b5778e008caec2152c0
Idea of Frank-Rainer Grahl.

Comment 13

[Jorg K (CEST = GMT+2)]

Assigned this accidentally.

Comment 14

[Jorg K (CEST = GMT+2)]

Landed temporary fix:
https://hg.mozilla.org/comm-central/rev/725206ff24cc9e4933df23f1db7ce465114fa22d

Comment 15

[Jorg K (CEST = GMT+2)]

Attachment: 1350011-rename.patch

OK, let's see whether renaming the file will avoid the "build file copies are not in sync" problem.

Comment 16

[Jorg K (CEST = GMT+2)]

https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=fc10376c2c40420abf5b9dcde7f477bc7296f48e

Comment 17

[Richard Marti (:Paenglab)]

I have my doubt as I think whole directories are compared and /build/unix/mozconfig.* are also compared which shows more files not in sync.

It seems the files in im/config/mozconfigs/linux mail/config/mozconfigs/linux and suite/config/mozconfigs/linux and the linux64 ones aren't compared. What about trying to move the config option to this files?

Comment 18

[Richard Marti (:Paenglab)]

Try with my proposal: https://hg.mozilla.org/try-comm-central/rev/2d022f5f1b7c14f2f0930c13af46acc8a136f14e

Comment 19

[Jorg K (CEST = GMT+2)]

Comment on attachment 8851035 [details] [diff] [review]
1350011-rename.patch

(In reply to Richard Marti (:Paenglab) from comment #17)
> How about trying to move the config option to these files?
Sure, since my attempt was a complete disaster :-(

Comment 20

[Richard Marti (:Paenglab)]

Attachment: moveConfig.patch

Try is green.

Comment 21

[Jorg K (CEST = GMT+2)]

(In reply to Richard Marti (:Paenglab) from comment #18)
> Try with my proposal:
Green as expected. How many files do you want to change? debug, nightly, l10n, perhaps release?, also IB and SM, 32 and 64 bit. 24 files. We can do that. We did already to disable Rust temporarily:
https://hg.mozilla.org/comm-central/rev/615bdf95952f15f6fa7020ba8b48e9863f5eabe5

Comment 22

[Jorg K (CEST = GMT+2)]

Comment on attachment 8851067 [details] [diff] [review]
moveConfig.patch

I think you need many more files, see previous comment. Also, let me back out my changeset. I proved the point, but we got lots of oranges as expected.

Comment 23

[Richard Marti (:Paenglab)]

Attachment: moveConfig.patch

Added the l10n files. Are the other files used in c-c or only for release etc.?

Comment 24

[Jorg K (CEST = GMT+2)]

Backout of first temporary solution:
https://hg.mozilla.org/comm-central/rev/01a0dc6906d9ac6f435dd9f089577b559285219e

Comment 25

[Jorg K (CEST = GMT+2)]

Comment on attachment 8851068 [details] [diff] [review]
moveConfig.patch

Review of attachment 8851068 [details] [diff] [review]:
-----------------------------------------------------------------

As I said, debug for for debug builds. I'm not sure about release for the time being. And please note my backout.

Comment 26

[Richard Marti (:Paenglab)]

Attachment: Bug1350011.patch

This should now address all needed files.

Comment 27

[Jorg K (CEST = GMT+2)]

Comment on attachment 8851073 [details] [diff] [review]
Bug1350011.patch

Nice. What are we going to do with the "release" files: 1 in im/, 2 in mail/ and 4 in suite/.
I believe they are needed for beta and above. Do we care right now?

Anyway, let's land this after the next M-C merge.

Comment 28

[Jorg K (CEST = GMT+2)]

Comment on attachment 8851073 [details] [diff] [review]
Bug1350011.patch

Frank-Rainer, you can r+ this for SM. Do you want the release files treated, too?

Comment 29

[Frank-Rainer Grahl (:frg)]

Comment on attachment 8851073 [details] [diff] [review]
Bug1350011.patch

Well we mailed last week about webrender and I prefer to see cursors in edit fields which I lost after enabling it locally so its best that it stays out anyway for now until more mature.

Comment 30

[Jorg K (CEST = GMT+2)]

https://hg.mozilla.org/comm-central/rev/6b1f2f9821292d9a46b718d959e177120af0199f

Let's close this for now.