761421 - Upgrade clang to 158160

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

r157958 fixed PR12979 which was casing problems with our CheckedInt.h.

I will upload the patches in a sec.

Comment 1

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: Update clang to 157958 on os x

Comment 2

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: upgrade clang on linux to 157958

The new patch is because the version of glibc on centos 5 does not define O_CLOEXEC.

Comment 3

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Testing on 32 bit linux found llvm.org/pr13048. I have fixed an restarted the tests.

Comment 4

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: upgrade clang on linux to 158158

Comment 5

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: Update clang to 158158 on os x

Comment 6

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

32 bit linux is OK now, so I have checked in the patches:

https://hg.mozilla.org/build/rpm-sources/rev/d9b3401ac0ba
https://hg.mozilla.org/build/braindump/rev/662fbcab17f8

will attach the puppet ones in a sec.

Comment 7

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: puppet change 1: install the new packages

Comment 8

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: support for building with "old" xcode

This is not needed to build with xcode 4.3, but is with xcode 4.0.

If I remember correctly, we have xcode 4.1 in the bots. I don't know if it needs this patch or not. Would you mind testing and r+ or r- the patch depending on the result?

Thanks

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 631522 [details] [diff] [review]
support for building with "old" xcode

Yeah, this needed even on the lion builders...

Comment 10

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Comment on attachment 631522 [details] [diff] [review]
support for building with "old" xcode

https://hg.mozilla.org/build/braindump/rev/049fb30ef0b3

Comment 11

[Rail Aliiev [:rail]]

I'm still waiting for some outstanding release related jobs. I hope to start deploying later today or tomorrow.

Comment 12

[Rail Aliiev [:rail]]

Comment on attachment 631371 [details] [diff] [review]
puppet change 1: install the new packages

Sorry about the delay, we had a couple of chemspill releases last week.

I'll land the patch and deploy the packages today.

BTW, would you consider to use tooltool (https://github.com/jhford/tooltool) to deploy clang packages next time? This will require building relocatable tarballs with clang (not rpms and dmgs), in a similar way how we build gonk gor b2g builds.

Comment 13

[Rail Aliiev [:rail]]

Comment on attachment 631371 [details] [diff] [review]
puppet change 1: install the new packages

http://hg.mozilla.org/build/puppet-manifests/rev/587ce9e7e906

Comment 14

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

> BTW, would you consider to use tooltool (https://github.com/jhford/tooltool)
> to deploy clang packages next time? This will require building relocatable
> tarballs with clang (not rpms and dmgs), in a similar way how we build gonk
> gor b2g builds.

That lets us use the same format for OS X and linux? That is awesome! I will try it next time, yes!

Comment 15

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: puppet change 2: remove the old packages

This should only be checked in after a try run confirms the new packages are OK.

Comment 16

[Rail Aliiev [:rail]]

(In reply to Rafael Ávila de Espíndola (:espindola) from comment #14)
> That lets us use the same format for OS X and linux? That is awesome! I will
> try it next time, yes!

yes, you can use multiple versions and use tooltool manifests in the source tree to point to the particular version. Let's use it next time!

Comment 17

[Rail Aliiev [:rail]]

Comment on attachment 634152 [details] [diff] [review]
puppet change 2: remove the old packages

lgtm, let me know when you want me to deploy this.

Comment 18

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

A try push is at

https://tbpl.mozilla.org/?tree=Try&rev=c2228487f25a

It looks like there is a problem in mochitest-4. It is not clear if it is a regression in clang or if new firefox code found the problem. I did a push with the old version in

https://tbpl.mozilla.org/?tree=Try&rev=ca9e8fc2105f

to find out.

Comment 19

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

The failure on try in caused by llvm's r158126, which was fixed by r158160, so we just missed it :-(

Rail, what do you think we should do? If the old packages were still in place I would say just close this bug as "wontfix" and I would open a new one with that bug (and another one Ehsan found) fixed.

Since the OS X packages have been replaced, I can see these options:

* Rollback the OS X packages.
* Live with this bug for now. Might cause some confusion, but this is still not in production, so we can probably live with it.
* Rebuild the packages at r158160. Somewhat annoying since we know we will do another upgrade and we want to change the package format.

Comment 20

[Rail Aliiev [:rail]]

What do you think about this:

1) rail backs out http://hg.mozilla.org/build/puppet-manifests/rev/587ce9e7e906
2) espindola bui1ds 58160 for linux, linux64 and mac do we can use it by tooltool. We'll need to package the files into a tarball and unpack it by setup.sh (so it will be accessible from /builds/slave/xxxx/clang(-version?).

Try is aware of tooltool manifests (there are empty ones in browser/config/tooltool-manifests), but we'll need to ask IT to put the package and setup.sh to http://runtime-binaries.pvt.build.mozilla.org/tooltool (someday we'll have a webapp for this! :) ).

What do you think?

Comment 21

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

OK, so you suggestion is to upgrade to 158160 using tooltool. I can do that. Is there documentation somewhere on what exactly the tarball has to look like?

Comment 22

[Rail Aliiev [:rail]]

There is a Wiki page here: https://wiki.mozilla.org/ReleaseEngineering/Applications/Tooltool

The main idea is having a simple tool which fetches files and runs a command to setup the environment properly.

From my understanding you'll need to build a relocatable package and have a setup.sh script to unpack it and probably setup symlinks (ln -s clang-xxx clang), then update your manifests and mozconfigs. You may want to take a look at the b2g build logs to see how they deploy gonk toolchain.

Comment 23

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: clang build script

The build is still going, but we should be able to do the code review in parallel.

This script is a reduced version of the generic toolchain building one. In is *not* reproducible, but the spec file was not.

This version is linux (centos 5) only. I will update it to support os X as soon as the linux build finishes.

Comment 24

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: Small fixes to the build script

* The old tar in centos has no --mtime
* Install in "clang", not "inst"
* We are only doing a two stage build.

With these changes, it is possible to build a clang.tar.bz2 in centos 5.

Comment 25

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/fd73f9eb62b7

Comment 26

[Rail Aliiev [:rail]]

Attachment: rm 158158

Not tested yet.

Comment 27

[Rail Aliiev [:rail]]

FTR.

pkgdmg provider is a little bit dumb, and doesn't cleanup in /var/db after upgrading a package. It was thinking that r155417 is installed because there is /var/db/.puppet_pkgdmg_installed_clang-3.0-r155417.moz0.dmg left after r155417 -> r158158 upgrade.


I removed /var/db/.puppet_pkgdmg_installed_clang-3.0-r155417.moz0.dmg on bld-lion-r5-0{01..40} (try lion build slaves). They should pickup r155417 today.

Comment 28

[Rail Aliiev [:rail]]

Comment on attachment 635293 [details] [diff] [review]
rm 158158

Worked fine in staging.

Comment 29

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: Add support to OS X in the build script

necessary changes:

* Different set of patches
* Different host compiler and options
* The tar in OS X is in /usr/bin and doesn't support --owner

Comment 30

[Rail Aliiev [:rail]]

Comment on attachment 635293 [details] [diff] [review]
rm 158158

http://hg.mozilla.org/build/puppet-manifests/rev/40f20eb317ca

deployed

Comment 31

[Rail Aliiev [:rail]]

Comment on attachment 635313 [details] [diff] [review]
Add support to OS X in the build script

lgtm

Comment 32

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: add a setup.sh script for use with tooltool

Comment 33

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Bug 766708 is still ongoing, so I uploaded the clang packages to http://people.mozilla.org/~respindola. Rail, can you deploy them to relengweb1.dmz.scl3.mozilla.com or do you have to build them again?

Comment 34

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/6f63ad04afb1

Comment 35

[Rail Aliiev [:rail]]

FTR,

$ wget https://raw.github.com/jhford/tooltool/master/tooltool.py
$ wget https://hg.mozilla.org/integration/mozilla-inbound/raw-file/84dffec9742c/build/unix/build-clang/setup.sh
$ wget http://people.mozilla.org/~respindola/clang-linux.tar.bz2
$ wget http://people.mozilla.org/~respindola/clang-osx.tar.bz2

$ python2.6 tooltool.py -m linux.manifest add clang-linux.tar.bz2 setup.sh
$ python2.6 tooltool.py -m osx.manifest add clang-osx.tar.bz2 setup.sh
$ sed -i 's/clang-.*.tar.bz2/clang.tar.bz2/g' *.manifest

$ cat linux.manifest 
[
{
"size": 47, 
"digest": "2005a41fe97a5e00997063705f39d42b6a43b1cf7ba306cbc7b1513de34cdcd050fc6326efa2107f19ba0cc67914745dbf13154fa748010a93cf072481ef4aaa", 
"algorithm": "sha512", 
"filename": "setup.sh"
}, 
{
"size": 71534797, 
"digest": "66bd11bea6e1f07090e9e03c833e107088097605611fd455e80b280ce2b71ca71ff9841a66614f62da162469b222b5eefd5535373b199c60fd485959889b5dcb", 
"algorithm": "sha512", 
"filename": "clang.tar.bz2"
}
]

$ cat osx.manifest 
[
{
"size": 47, 
"digest": "2005a41fe97a5e00997063705f39d42b6a43b1cf7ba306cbc7b1513de34cdcd050fc6326efa2107f19ba0cc67914745dbf13154fa748010a93cf072481ef4aaa", 
"algorithm": "sha512", 
"filename": "setup.sh"
}, 
{
"size": 63015959, 
"digest": "e0ac132a77b052d6494d4bf02f17aba475138ffcff07a5f92f17f45b1d5f27b7b0cea36c29473965271e60910af82ffd4989df94c10d24794d1bf8362bcb785b", 
"algorithm": "sha512", 
"filename": "clang.tar.bz2"
}
]

$ sudo cp -vi setup.sh /var/www/html/runtime-binaries/tooltool/sha512/2005a41fe97a5e00997063705f39d42b6a43b1cf7ba306cbc7b1513de34cdcd050fc6326efa2107f19ba0cc67914745dbf13154fa748010a93cf072481ef4aaa
`setup.sh' -> `/var/www/html/runtime-binaries/tooltool/sha512/2005a41fe97a5e00997063705f39d42b6a43b1cf7ba306cbc7b1513de34cdcd050fc6326efa2107f19ba0cc67914745dbf13154fa748010a93cf072481ef4aaa'

$ sudo cp -vi clang-linux.tar.bz2 /var/www/html/runtime-binaries/tooltool/sha512/66bd11bea6e1f07090e9e03c833e107088097605611fd455e80b280ce2b71ca71ff9841a66614f62da162469b222b5eefd5535373b199c60fd485959889b5dcb
`clang-linux.tar.bz2' -> `/var/www/html/runtime-binaries/tooltool/sha512/66bd11bea6e1f07090e9e03c833e107088097605611fd455e80b280ce2b71ca71ff9841a66614f62da162469b222b5eefd5535373b199c60fd485959889b5dcb'

$ sudo cp -vi clang-osx.tar.bz2 /var/www/html/runtime-binaries/tooltool/sha512/e0ac132a77b052d6494d4bf02f17aba475138ffcff07a5f92f17f45b1d5f27b7b0cea36c29473965271e60910af82ffd4989df94c10d24794d1bf8362bcb785b
`clang-osx.tar.bz2' -> `/var/www/html/runtime-binaries/tooltool/sha512/e0ac132a77b052d6494d4bf02f17aba475138ffcff07a5f92f17f45b1d5f27b7b0cea36c29473965271e60910af82ffd4989df94c10d24794d1bf8362bcb785b'

Comment 36

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

I forgot to build a 32 bit linux package. I will fix that in a sec, but I think we are also missing a chmod:


retry: Calling <function run_with_timeout at 0x2aaaaed12ed8> with args: (['/tools/tooltool.py', '--url', 'http://runtime-binaries.pvt.build.mozilla.org/tooltool', '--overwrite', '-m', 'browser/config/tooltool-manifests/linux64/releng.manifest', 'fetch'], 1260, None, None, False, True), kwargs: {}, attempt #1
Executing: ['/tools/tooltool.py', '--url', 'http://runtime-binaries.pvt.build.mozilla.org/tooltool', '--overwrite', '-m', 'browser/config/tooltool-manifests/linux64/releng.manifest', 'fetch']
ERROR - failed to fetch 'setup.sh': HTTP Error 403: Forbidden
ERROR - failed to fetch 'clang.tar.bz2': HTTP Error 403: Forbidden
ERROR - The following files failed: 'setup.sh', clang.tar.bz2'

Comment 37

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

The linux32 package is at
http://people.mozilla.org/~respindola/clang-linux32.tar.bz2

Comment 38

[Rail Aliiev [:rail]]

$ wget http://people.mozilla.org/~respindola/clang-linux32.tar.bz2
$ python2.6 tooltool.py -m linux32.manifest add clang-linux32.tar.bz2 setup.sh
$ sed -i 's/clang-.*.tar.bz2/clang.tar.bz2/g' linux32.manifest 
$ cat linux32.manifest 
[
{
"size": 47, 
"digest": "2005a41fe97a5e00997063705f39d42b6a43b1cf7ba306cbc7b1513de34cdcd050fc6326efa2107f19ba0cc67914745dbf13154fa748010a93cf072481ef4aaa", 
"algorithm": "sha512", 
"filename": "setup.sh"
}, 
{
"size": 72350187, 
"digest": "7d2fbe08aca3ae740e33b8aee872705a3b5229681dd0617ceffd6619fba75cb3cb7e1c3a071218f7cfd464003e5cd773cd8e67d16f78df9c50218fb6671580c6", 
"algorithm": "sha512", 
"filename": "clang.tar.bz2"
}
]

$ sudo cp -vi clang-linux32.tar.bz2 /var/www/html/runtime-binaries/tooltool/sha512/7d2fbe08aca3ae740e33b8aee872705a3b5229681dd0617ceffd6619fba75cb3cb7e1c3a071218f7cfd464003e5cd773cd8e67d16f78df9c50218fb6671580c6
`clang-linux32.tar.bz2' -> `/var/www/html/runtime-binaries/tooltool/sha512/7d2fbe08aca3ae740e33b8aee872705a3b5229681dd0617ceffd6619fba75cb3cb7e1c3a071218f7cfd464003e5cd773cd8e67d16f78df9c50218fb6671580c6'

$ sudo chmod 644 /var/www/html/runtime-binaries/tooltool/sha512/*

Comment 39

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/c42521d54b79
https://hg.mozilla.org/mozilla-central/rev/84dffec9742c

Comment 40

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

A new try push with all tests and talos is at

https://tbpl.mozilla.org/?tree=Try&rev=fca5d53846e3

Comment 41

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

OK, looks like it works :-)

Should we remove the old clang packages now or should we fix 767508 first?

Comment 42

[Rail Aliiev [:rail]]

(In reply to Rafael Ávila de Espíndola (:espindola) from comment #41)
> OK, looks like it works :-)

\o/
 
> Should we remove the old clang packages now or should we fix 767508 first?

Yeah, let's remove it. I hope bug 767508 will be very simple to solve with $topsrcdir.

Comment 43

[Rail Aliiev [:rail]]

Attachment: remove clang packages

TODO:
1) test in staging
2) deploy when Rafael is OK with package removal

Comment 44

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Comment on attachment 635868 [details] [diff] [review]
remove clang packages

OK. Just wait for me to send an email on how to use the new packages.

Comment 45

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: add clang manifests

Add the current clang manifests to the tree. This makes it easy to build a patch that uses them on try, just copy them over the releng.manifest.

Comment 46

[Rail Aliiev [:rail]]

Comment on attachment 635868 [details] [diff] [review]
remove clang packages

http://hg.mozilla.org/build/puppet-manifests/rev/fd06119e5fa6

Comment 47

[Rail Aliiev [:rail]]

Feel free to close the bug.

Comment 48

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Fixed :-)

Comment 49

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/23818f9b65a2