Closed Bug 1627412 Opened 4 months ago Closed 4 months ago

Thunderbird 68.6.1 toolchain build failures

Categories

(Thunderbird :: Build Config, defect, P1)

defect

Tracking

(thunderbird_esr68 fixed)

RESOLVED FIXED
Thunderbird 68.0
Tracking Status
thunderbird_esr68 --- fixed

People

(Reporter: rjl, Assigned: rjl)

References

Details

Attachments

(1 file)

sccache will not build: Task ID Xyb5gJRZR66_8P2Zcq20DA

The short version is it's trying to download an old version of openssl and the URL has changed.

Also clang does not build. It looks like the svn checkout command is having issues, but doesn't give a clear error.

Task ID TAh7I4a8QHWZCTrmYv9CAg

For sccache, it's probably easiest to fix the URL in the script. That way nothing else changes.
For clang, I suspect the svn connection is being blocked as I can run the failing command locally. Switching to a fetch task would be ideal, but it looks like a lot of changes.

This is blocking builds on comm-esr68, and has potential to affect Firefox builds on m-esr68 if there's a need to rebuild these tools for whatever reason.

dmajor: is this something you can advise on?

Flags: needinfo?(dmajor)

I retriggered just in case svn had a bad day yesterday but it's still failing. I can run the checkout command fine locally. Maybe try removing -q to see if it prints anything more verbose: https://dxr.mozilla.org/mozilla-esr68/source/build/build-clang/build-clang.py#200

Flags: needinfo?(dmajor)

The error from svn co:
svn: OPTIONS of 'https://llvm.org/svn/llvm-project/llvm/tags/RELEASE_800/final': SSL handshake failed: SSL alert received: Handshake failed (https://llvm.org)\n"

The Subversion client in the toolchain-build image on esr68 is linked to an old version of GNUTLS that does not support the GCM ciphers that llvm.org requires to connect.
There is a newer version of Subversion that is linked to openssl in the backports repository. I was able to install it on a local copy of the toolchain-build image and the svn co command worked.

These changes were made to address problems when rebuilding toolchains for
the Thunderbird 68.6.1 release.
This newer Subversion client supports the ciphers required to connect to
llvm.org when building Clang.
The OpenSSL download URL changed for the version used to build Sccache.

Try build looks promising:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=d6a15abff4391824d60e6316f1c0af6f26cb1252

I know this won't be able to get into m-esr68 default without
going through proper procedures. I'd like to get it on 
THUNDERBIRD_68_VERBRANCH on m-esr68 as we're unable build at
the moment.
Attachment #9138305 - Flags: review?(dmajor)
Attachment #9138305 - Flags: review?(dmajor) → review?(mh+mozilla)
Status: NEW → ASSIGNED
Status: ASSIGNED → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 68.0
Attachment #9138305 - Flags: review?(mh+mozilla)

I'm going to work on a proper uplift fix just in case.

The suggestions from glandium are:
mozilla-central@df808b9785c66bee376a610d0bd45d3d0ed716f0 / Bug 1570541 - Use git fetch tasks for clang. r=froydnj
mozilla-central@4e2bac5f0f3b509a3cb4644ec6e01c37ae11f52e / Bug 1617143 - Do not build openssl ourselves for sccache. r=froydnj

See Also: → 1628031

Looking back at this, I don't know why you needed the sccache change, because bug 1449629 already landed a workaround for the fact that the url changed (it's actually redirecting), and running the sccache task on the esr68 tree without your change currently works.

As for the second half, I pointed to the wrong bug, what's needed is actually bug 1566336, which I'll request an uplift for.

(In reply to Mike Hommey [:glandium] from comment #7)

Looking back at this, I don't know why you needed the sccache change, because bug 1449629 already landed a workaround for the fact that the url changed (it's actually redirecting), and running the sccache task on the esr68 tree without your change currently works.

Interesting... the output isn't clear on what happened:

[task 2020-04-03T21:53:13.634Z] + OPENSSL_TARBALL=openssl-1.1.0g.tar.gz
[task 2020-04-03T21:53:13.634Z] + curl -L -O https://www.openssl.org/source/openssl-1.1.0g.tar.gz
[task 2020-04-03T21:53:13.795Z]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[task 2020-04-03T21:53:13.795Z]                                  Dload  Upload   Total   Spent    Left  Speed
[task 2020-04-03T21:53:13.795Z] 
[task 2020-04-03T21:53:14.152Z]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[task 2020-04-03T21:53:14.152Z] 100  4182  100  4182    0     0   8128      0 --:--:-- --:--:-- --:--:-- 11747
[task 2020-04-03T21:53:14.152Z] + cat
[task 2020-04-03T21:53:14.153Z] + cat openssl-1.1.0g.tar.gz.sha256sum
[task 2020-04-03T21:53:14.153Z] de4d501267da39310905cb6dc8c6121f7a2cad45a7707f76df828fe1b85073af  openssl-1.1.0g.tar.gz
[task 2020-04-03T21:53:14.154Z] + sha256sum -c openssl-1.1.0g.tar.gz.sha256sum
[task 2020-04-03T21:53:14.154Z] openssl-1.1.0g.tar.gz: FAILED
[task 2020-04-03T21:53:14.154Z] sha256sum: WARNING: 1 computed checksum did NOT match

What stands out is that "4182" value for bytes received, which is clearly not correct.

Sounds like a fluke, like a server disconnection or something. I'd expect a retrigger to work.

You need to log in before you can comment on or make changes to this bug.