Closed Bug 1350011 Opened 3 years ago Closed 3 years ago

comm-central Linux builds fail with: make[5]: *** [force-cargo-library-build] Error 101

Categories

(Thunderbird :: Build Config, defect)

All
Linux
defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED
Thunderbird 55.0

People

(Reporter: jorgk-bmo, Assigned: Paenglab)

Details

(Whiteboard: [Thunderbird-testfailure: B Linux all])

Attachments

(2 files, 3 obsolete files)

https://treeherder.mozilla.org/#/jobs?repo=comm-central&revision=ea538d555890a7fa33b97de780bb5d09b230f428&selectedJob=85973554

From the log:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error(PkgConfig(Failure { command: "\"pkg-config\" \"--libs\" \"--cflags\" \"x11 >= 1.4.99.1\"", output: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "Package x11 was not found in the pkg-config search path.\nPerhaps you should add the directory containing `x11.pc\'\nto the PKG_CONFIG_PATH environment variable\nNo package \'x11\' found\n" } }), State { next_error: None })', /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libcore/result.rs:837
stack backtrace:
OS: Unspecified → Linux
Hardware: Unspecified → All
Component: General Automation → Build Config
Product: Release Engineering → Thunderbird
QA Contact: catlee
Any hints what we need to configure/port? Aryx said something to me on IRC (which I didn't understand), but I didn't log the channel and lost it.
Flags: needinfo?(rail)
Flags: needinfo?(aryx.bugmail)
Sounds like missing x11 headers/libraries. I'd expect "configure" to fail, not cargo.
Flags: needinfo?(rail)
Sorry, am no help here.

So this is from https://dxr.mozilla.org/mozilla-central/rev/e03e0c60462c775c7558a1dc9d5cf2076c3cd1f9/third_party/rust/x11/Cargo.toml#39

error: failed to run custom build command for `x11 v2.12.1`
process didn't exit successfully: `/builds/slave/tb-c-cen-lx-000000000000000000/build/objdir-tb/toolkit/library/release/build/x11-9858dee221fb8695/build-script-build` (exit code: 101)

2.12.0 is 3 months old: https://github.com/rust-lang/crates.io-index
Flags: needinfo?(aryx.bugmail)
Anyone has any idea what's going on here? Rail moved this to Thunderbird::Build Config, and then said:
"Sounds like missing x11 headers/libraries." So where/why/when are they or did they go missing and how do we restore them? After all, this runs on the treeherder servers, or does it also fail locally now?
Flags: needinfo?(philipp)
Flags: needinfo?(ewong)
Flags: needinfo?(acelists)
I don't get this failure locally (on linux). Has it been fixed in m-c since? I have rustc 0.16, not 0.15.1 as on the buid servers.
Flags: needinfo?(acelists)
Aceman and I have been looking at some logs:

TB Tinderbox:
https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-central-linux/1490274624/comm-central-linux-bm72-build1-build1.txt.gz
libX11                  i686   1.3-2.el6           centos6               585 k
libX11                  x86_64 1.3-2.el6           centos6               582 k

Required is > 1.4.99 to fit with Rust.

On the FF build machines which use Taskcluster we see:
https://public-artifacts.taskcluster.net/TlUuz9jsS6ywcFr9g4FUYw/0/public/logs/live_backing.log
[task 2017-03-23T21:57:47.645888Z] 21:57:47     INFO - 0xf14a8000 - 0xf15e1fff  libX11.so.6.3.0
which according to the internet equates to a package version of 1.6.3.

Sorry, Rail, is this really a Thunderbird::Build config issue?

It looks to me that Rust got upgraded and all those Tinderbox Linux machines are now incompatible since they still have X11 1.3.

Can they be upgraded? Or is this the end of Thunderbird on Linux unless it moves to Taskcluster?
Flags: needinfo?(rail)
(In reply to Jorg K (GMT+1) from comment #4)
> Anyone has any idea what's going on here? Rail moved this to
> Thunderbird::Build Config, and then said:
> "Sounds like missing x11 headers/libraries." So where/why/when are they or
> did they go missing and how do we restore them? After all, this runs on the
> treeherder servers, or does it also fail locally now?

I'll take a gander..  not entirely sure what's going on.
Flags: needinfo?(ewong)
(In reply to Jorg K (GMT+1) from comment #6)
> Aceman and I have been looking at some logs:
> 
> TB Tinderbox:
> https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-central-
> linux/1490274624/comm-central-linux-bm72-build1-build1.txt.gz
> libX11                  i686   1.3-2.el6           centos6               585
> k
> libX11                  x86_64 1.3-2.el6           centos6               582
> k
> 
> Required is > 1.4.99 to fit with Rust.
> 
> On the FF build machines which use Taskcluster we see:
> https://public-artifacts.taskcluster.net/TlUuz9jsS6ywcFr9g4FUYw/0/public/
> logs/live_backing.log
> [task 2017-03-23T21:57:47.645888Z] 21:57:47     INFO - 0xf14a8000 -
> 0xf15e1fff  libX11.so.6.3.0
> which according to the internet equates to a package version of 1.6.3.
> 
> Sorry, Rail, is this really a Thunderbird::Build config issue?
> 
> It looks to me that Rust got upgraded and all those Tinderbox Linux machines
> are now incompatible since they still have X11 1.3.
> 
> Can they be upgraded? Or is this the end of Thunderbird on Linux unless it
> moves to Taskcluster?

From what I've discerned from both logs:

TB uses mock.
Firefox doesn't and ergo uses the x11 libs from the system itself.

Now if someone in Releng can confirm(or deny), TB and Firefox uses AWS 
instances and possibly similar AMIs and ergo similiar operating systems
(if not the same).

That said, I'm wondering if disabling Mock on Linux32 might work? 

i.e. 
https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/thunderbird_config.py#l140
            'use_mock': True,

change that to 'use_mock': False

(Again, just a left-field suggestion since I know absolutely nothing
of the stuff that goes in AMIs).
fwiw..  SeaMonkey's also hitting this, but unlike TB, our infra isn't that
up to date so I doubt we'll be able to depend on the system's X11 libs.
(In reply to Jorg K (GMT+1) from comment #6)
 > Sorry, Rail, is this really a Thunderbird::Build config issue?

Now it is clear to say that is a build environment issue, not build config.


> It looks to me that Rust got upgraded and all those Tinderbox Linux machines
> are now incompatible since they still have X11 1.3.
> 
> Can they be upgraded? Or is this the end of Thunderbird on Linux unless it
> moves to Taskcluster?

Upgrading this would be not trivial. Mock based builds use Centos 6.2, which is very ancient. Probably there is a way to use Centos 6.8 in mock (similar to https://dxr.mozilla.org/build-central/source/puppet/modules/runner/files/mockbuild-config-templates/mozilla-centos6-x86_64.cfg), but it's completely untested.


(In reply to Edmund Wong (:ewong) from comment #8)
> TB uses mock.
> Firefox doesn't and ergo uses the x11 libs from the system itself.
> 
> Now if someone in Releng can confirm(or deny), TB and Firefox uses AWS 
> instances and possibly similar AMIs and ergo similiar operating systems
> (if not the same).

Unfortunately, the Tc-based build environment is a bit different from what we use for buildbot. TC uses Centos6 latest (6.8 atm) https://dxr.mozilla.org/mozilla-central/source/taskcluster/docker/centos6-build/Dockerfile, buildbot uses Centos 6.2 (https://s3-external-1.amazonaws.com/mozilla-releng-mock-centos-us-east-1/mirrors/centos/6/latest/centos-version.txt)

> 
> That said, I'm wondering if disabling Mock on Linux32 might work? 
> 
> i.e. 
> https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/
> thunderbird_config.py#l140
>             'use_mock': True,
> 
> change that to 'use_mock': False

I don't think this is going to work. With mock disabled, buildbot will use the existing host environment and won't be able to install any packages (it's not running as root).

Sounds like we finally hit the wall :/ Fixing the existing build environment is not trivial, neither porting to TC...
Flags: needinfo?(rail)
Would it be possible to use tooltool or similar to include a version of rustc that works, maybe statically compiled to not depend on the system libX11?
Flags: needinfo?(philipp) → needinfo?(rail)
Attached patch 1350011-linux-x11.patch β€” β€” Splinter Review
This appears to work (so far, usually stopped after 14 minutes):
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=27f10ccf5e60c61065210b5778e008caec2152c0
Idea of Frank-Rainer Grahl.
Assignee: nobody → jorgk
Assigned this accidentally.
Assignee: jorgk → nobody
Attached patch 1350011-rename.patch (obsolete) β€” β€” Splinter Review
OK, let's see whether renaming the file will avoid the "build file copies are not in sync" problem.
I have my doubt as I think whole directories are compared and /build/unix/mozconfig.* are also compared which shows more files not in sync.

It seems the files in im/config/mozconfigs/linux mail/config/mozconfigs/linux and suite/config/mozconfigs/linux and the linux64 ones aren't compared. What about trying to move the config option to this files?
Comment on attachment 8851035 [details] [diff] [review]
1350011-rename.patch

(In reply to Richard Marti (:Paenglab) from comment #17)
> How about trying to move the config option to these files?
Sure, since my attempt was a complete disaster :-(
Attachment #8851035 - Attachment is obsolete: true
Attached patch moveConfig.patch (obsolete) β€” β€” Splinter Review
Try is green.
Attachment #8851067 - Flags: review?(jorgk)
(In reply to Richard Marti (:Paenglab) from comment #18)
> Try with my proposal:
Green as expected. How many files do you want to change? debug, nightly, l10n, perhaps release?, also IB and SM, 32 and 64 bit. 24 files. We can do that. We did already to disable Rust temporarily:
https://hg.mozilla.org/comm-central/rev/615bdf95952f15f6fa7020ba8b48e9863f5eabe5
Comment on attachment 8851067 [details] [diff] [review]
moveConfig.patch

I think you need many more files, see previous comment. Also, let me back out my changeset. I proved the point, but we got lots of oranges as expected.
Attached patch moveConfig.patch (obsolete) β€” β€” Splinter Review
Added the l10n files. Are the other files used in c-c or only for release etc.?
Attachment #8851067 - Attachment is obsolete: true
Attachment #8851067 - Flags: review?(jorgk)
Attachment #8851068 - Flags: review?(jorgk)
Comment on attachment 8851068 [details] [diff] [review]
moveConfig.patch

Review of attachment 8851068 [details] [diff] [review]:
-----------------------------------------------------------------

As I said, debug for for debug builds. I'm not sure about release for the time being. And please note my backout.
Attached patch Bug1350011.patch β€” β€” Splinter Review
This should now address all needed files.
Assignee: nobody → richard.marti
Attachment #8851068 - Attachment is obsolete: true
Attachment #8851068 - Flags: review?(jorgk)
Attachment #8851073 - Flags: review?(jorgk)
Comment on attachment 8851073 [details] [diff] [review]
Bug1350011.patch

Nice. What are we going to do with the "release" files: 1 in im/, 2 in mail/ and 4 in suite/.
I believe they are needed for beta and above. Do we care right now?

Anyway, let's land this after the next M-C merge.
Attachment #8851073 - Flags: review?(jorgk) → review+
Comment on attachment 8851073 [details] [diff] [review]
Bug1350011.patch

Frank-Rainer, you can r+ this for SM. Do you want the release files treated, too?
Attachment #8851073 - Flags: review?(frgrahl)
Comment on attachment 8851073 [details] [diff] [review]
Bug1350011.patch

Well we mailed last week about webrender and I prefer to see cursors in edit fields which I lost after enabling it locally so its best that it stays out anyway for now until more mature.
Attachment #8851073 - Flags: review?(frgrahl) → review+
https://hg.mozilla.org/comm-central/rev/6b1f2f9821292d9a46b718d959e177120af0199f

Let's close this for now.
Status: NEW → RESOLVED
Closed: 3 years ago
Flags: needinfo?(rail)
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 55.0
You need to log in before you can comment on or make changes to this bug.