Closed Bug 761421 Opened 9 years ago Closed 9 years ago

Upgrade clang to 158160

Categories

(Release Engineering :: General, defect)

x86
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: espindola, Assigned: espindola)

References

Details

Attachments

(5 files, 9 obsolete files)

3.35 KB, patch
espindola
: review+
Details | Diff | Splinter Review
5.97 KB, patch
rail
: review+
Details | Diff | Splinter Review
448 bytes, patch
rail
: review+
Details | Diff | Splinter Review
4.85 KB, patch
espindola
: review+
Details | Diff | Splinter Review
2.32 KB, patch
rail
: review+
Details | Diff | Splinter Review
r157958 fixed PR12979 which was casing problems with our CheckedInt.h.

I will upload the patches in a sec.
Attached patch Update clang to 157958 on os x (obsolete) — Splinter Review
Assignee: nobody → respindola
Status: NEW → ASSIGNED
Attachment #630033 - Flags: review?(rail)
Attached patch upgrade clang on linux to 157958 (obsolete) — Splinter Review
The new patch is because the version of glibc on centos 5 does not define O_CLOEXEC.
Attachment #630295 - Flags: review?(rail)
Attachment #630033 - Flags: review?(rail) → review+
Attachment #630295 - Flags: review?(rail) → review+
Summary: Upgrade clang to 157958 → Upgrade clang to 158158
Testing on 32 bit linux found llvm.org/pr13048. I have fixed an restarted the tests.
Attached patch upgrade clang on linux to 158158 (obsolete) — Splinter Review
Attachment #630295 - Attachment is obsolete: true
Attachment #631067 - Flags: review?(rail)
Attachment #631067 - Flags: review?(rail) → review+
Attached patch Update clang to 158158 on os x (obsolete) — Splinter Review
Attachment #630033 - Attachment is obsolete: true
Attachment #631068 - Flags: review?(rail)
Attachment #631068 - Flags: review?(rail) → review+
32 bit linux is OK now, so I have checked in the patches:

https://hg.mozilla.org/build/rpm-sources/rev/d9b3401ac0ba
https://hg.mozilla.org/build/braindump/rev/662fbcab17f8

will attach the puppet ones in a sec.
This is not needed to build with xcode 4.3, but is with xcode 4.0.

If I remember correctly, we have xcode 4.1 in the bots. I don't know if it needs this patch or not. Would you mind testing and r+ or r- the patch depending on the result?

Thanks
Attachment #631522 - Flags: review?(rail)
Comment on attachment 631522 [details] [diff] [review]
support for building with "old" xcode

Yeah, this needed even on the lion builders...
Attachment #631522 - Flags: review?(rail) → review+
I'm still waiting for some outstanding release related jobs. I hope to start deploying later today or tomorrow.
Comment on attachment 631371 [details] [diff] [review]
puppet change 1: install the new packages

Sorry about the delay, we had a couple of chemspill releases last week.

I'll land the patch and deploy the packages today.

BTW, would you consider to use tooltool (https://github.com/jhford/tooltool) to deploy clang packages next time? This will require building relocatable tarballs with clang (not rpms and dmgs), in a similar way how we build gonk gor b2g builds.
Attachment #631371 - Flags: review?(rail) → review+
> BTW, would you consider to use tooltool (https://github.com/jhford/tooltool)
> to deploy clang packages next time? This will require building relocatable
> tarballs with clang (not rpms and dmgs), in a similar way how we build gonk
> gor b2g builds.

That lets us use the same format for OS X and linux? That is awesome! I will try it next time, yes!
This should only be checked in after a try run confirms the new packages are OK.
Attachment #634152 - Flags: review?(rail)
(In reply to Rafael Ávila de Espíndola (:espindola) from comment #14)
> That lets us use the same format for OS X and linux? That is awesome! I will
> try it next time, yes!

yes, you can use multiple versions and use tooltool manifests in the source tree to point to the particular version. Let's use it next time!
Comment on attachment 634152 [details] [diff] [review]
puppet change 2: remove the old packages

lgtm, let me know when you want me to deploy this.
Attachment #634152 - Flags: review?(rail) → review+
A try push is at

https://tbpl.mozilla.org/?tree=Try&rev=c2228487f25a

It looks like there is a problem in mochitest-4. It is not clear if it is a regression in clang or if new firefox code found the problem. I did a push with the old version in

https://tbpl.mozilla.org/?tree=Try&rev=ca9e8fc2105f

to find out.
The failure on try in caused by llvm's r158126, which was fixed by r158160, so we just missed it :-(

Rail, what do you think we should do? If the old packages were still in place I would say just close this bug as "wontfix" and I would open a new one with that bug (and another one Ehsan found) fixed.

Since the OS X packages have been replaced, I can see these options:

* Rollback the OS X packages.
* Live with this bug for now. Might cause some confusion, but this is still not in production, so we can probably live with it.
* Rebuild the packages at r158160. Somewhat annoying since we know we will do another upgrade and we want to change the package format.
What do you think about this:

1) rail backs out http://hg.mozilla.org/build/puppet-manifests/rev/587ce9e7e906
2) espindola bui1ds 58160 for linux, linux64 and mac do we can use it by tooltool. We'll need to package the files into a tarball and unpack it by setup.sh (so it will be accessible from /builds/slave/xxxx/clang(-version?).

Try is aware of tooltool manifests (there are empty ones in browser/config/tooltool-manifests), but we'll need to ask IT to put the package and setup.sh to http://runtime-binaries.pvt.build.mozilla.org/tooltool (someday we'll have a webapp for this! :) ).

What do you think?
OK, so you suggestion is to upgrade to 158160 using tooltool. I can do that. Is there documentation somewhere on what exactly the tarball has to look like?
There is a Wiki page here: https://wiki.mozilla.org/ReleaseEngineering/Applications/Tooltool

The main idea is having a simple tool which fetches files and runs a command to setup the environment properly.

From my understanding you'll need to build a relocatable package and have a setup.sh script to unpack it and probably setup symlinks (ln -s clang-xxx clang), then update your manifests and mozconfigs. You may want to take a look at the b2g build logs to see how they deploy gonk toolchain.
Summary: Upgrade clang to 158158 → Upgrade clang to 158160
Attached patch clang build script (obsolete) — Splinter Review
The build is still going, but we should be able to do the code review in parallel.

This script is a reduced version of the generic toolchain building one. In is *not* reproducible, but the spec file was not.

This version is linux (centos 5) only. I will update it to support os X as soon as the linux build finishes.
Attachment #634911 - Flags: review?(rail)
Attachment #634911 - Flags: review?(rail) → review+
Whiteboard: [leave open]
Attached patch Small fixes to the build script (obsolete) — Splinter Review
* The old tar in centos has no --mtime
* Install in "clang", not "inst"
* We are only doing a two stage build.

With these changes, it is possible to build a clang.tar.bz2 in centos 5.
Attachment #634911 - Attachment is obsolete: true
Attachment #635004 - Flags: review?(rail)
Attachment #635004 - Flags: review?(rail) → review+
Attached patch rm 158158Splinter Review
Not tested yet.
FTR.

pkgdmg provider is a little bit dumb, and doesn't cleanup in /var/db after upgrading a package. It was thinking that r155417 is installed because there is /var/db/.puppet_pkgdmg_installed_clang-3.0-r155417.moz0.dmg left after r155417 -> r158158 upgrade.


I removed /var/db/.puppet_pkgdmg_installed_clang-3.0-r155417.moz0.dmg on bld-lion-r5-0{01..40} (try lion build slaves). They should pickup r155417 today.
Comment on attachment 635293 [details] [diff] [review]
rm 158158

Worked fine in staging.
Attachment #635293 - Flags: review?(respindola)
Attachment #635293 - Flags: review?(respindola) → review+
necessary changes:

* Different set of patches
* Different host compiler and options
* The tar in OS X is in /usr/bin and doesn't support --owner
Attachment #635313 - Flags: review?(rail)
Comment on attachment 635313 [details] [diff] [review]
Add support to OS X in the build script

lgtm
Attachment #635313 - Flags: review?(rail) → review+
Attachment #635390 - Flags: review?(rail) → review+
Bug 766708 is still ongoing, so I uploaded the clang packages to http://people.mozilla.org/~respindola. Rail, can you deploy them to relengweb1.dmz.scl3.mozilla.com or do you have to build them again?
FTR,

$ wget https://raw.github.com/jhford/tooltool/master/tooltool.py
$ wget https://hg.mozilla.org/integration/mozilla-inbound/raw-file/84dffec9742c/build/unix/build-clang/setup.sh
$ wget http://people.mozilla.org/~respindola/clang-linux.tar.bz2
$ wget http://people.mozilla.org/~respindola/clang-osx.tar.bz2

$ python2.6 tooltool.py -m linux.manifest add clang-linux.tar.bz2 setup.sh
$ python2.6 tooltool.py -m osx.manifest add clang-osx.tar.bz2 setup.sh
$ sed -i 's/clang-.*.tar.bz2/clang.tar.bz2/g' *.manifest

$ cat linux.manifest 
[
{
"size": 47, 
"digest": "2005a41fe97a5e00997063705f39d42b6a43b1cf7ba306cbc7b1513de34cdcd050fc6326efa2107f19ba0cc67914745dbf13154fa748010a93cf072481ef4aaa", 
"algorithm": "sha512", 
"filename": "setup.sh"
}, 
{
"size": 71534797, 
"digest": "66bd11bea6e1f07090e9e03c833e107088097605611fd455e80b280ce2b71ca71ff9841a66614f62da162469b222b5eefd5535373b199c60fd485959889b5dcb", 
"algorithm": "sha512", 
"filename": "clang.tar.bz2"
}
]

$ cat osx.manifest 
[
{
"size": 47, 
"digest": "2005a41fe97a5e00997063705f39d42b6a43b1cf7ba306cbc7b1513de34cdcd050fc6326efa2107f19ba0cc67914745dbf13154fa748010a93cf072481ef4aaa", 
"algorithm": "sha512", 
"filename": "setup.sh"
}, 
{
"size": 63015959, 
"digest": "e0ac132a77b052d6494d4bf02f17aba475138ffcff07a5f92f17f45b1d5f27b7b0cea36c29473965271e60910af82ffd4989df94c10d24794d1bf8362bcb785b", 
"algorithm": "sha512", 
"filename": "clang.tar.bz2"
}
]

$ sudo cp -vi setup.sh /var/www/html/runtime-binaries/tooltool/sha512/2005a41fe97a5e00997063705f39d42b6a43b1cf7ba306cbc7b1513de34cdcd050fc6326efa2107f19ba0cc67914745dbf13154fa748010a93cf072481ef4aaa
`setup.sh' -> `/var/www/html/runtime-binaries/tooltool/sha512/2005a41fe97a5e00997063705f39d42b6a43b1cf7ba306cbc7b1513de34cdcd050fc6326efa2107f19ba0cc67914745dbf13154fa748010a93cf072481ef4aaa'

$ sudo cp -vi clang-linux.tar.bz2 /var/www/html/runtime-binaries/tooltool/sha512/66bd11bea6e1f07090e9e03c833e107088097605611fd455e80b280ce2b71ca71ff9841a66614f62da162469b222b5eefd5535373b199c60fd485959889b5dcb
`clang-linux.tar.bz2' -> `/var/www/html/runtime-binaries/tooltool/sha512/66bd11bea6e1f07090e9e03c833e107088097605611fd455e80b280ce2b71ca71ff9841a66614f62da162469b222b5eefd5535373b199c60fd485959889b5dcb'

$ sudo cp -vi clang-osx.tar.bz2 /var/www/html/runtime-binaries/tooltool/sha512/e0ac132a77b052d6494d4bf02f17aba475138ffcff07a5f92f17f45b1d5f27b7b0cea36c29473965271e60910af82ffd4989df94c10d24794d1bf8362bcb785b
`clang-osx.tar.bz2' -> `/var/www/html/runtime-binaries/tooltool/sha512/e0ac132a77b052d6494d4bf02f17aba475138ffcff07a5f92f17f45b1d5f27b7b0cea36c29473965271e60910af82ffd4989df94c10d24794d1bf8362bcb785b'
I forgot to build a 32 bit linux package. I will fix that in a sec, but I think we are also missing a chmod:


retry: Calling <function run_with_timeout at 0x2aaaaed12ed8> with args: (['/tools/tooltool.py', '--url', 'http://runtime-binaries.pvt.build.mozilla.org/tooltool', '--overwrite', '-m', 'browser/config/tooltool-manifests/linux64/releng.manifest', 'fetch'], 1260, None, None, False, True), kwargs: {}, attempt #1
Executing: ['/tools/tooltool.py', '--url', 'http://runtime-binaries.pvt.build.mozilla.org/tooltool', '--overwrite', '-m', 'browser/config/tooltool-manifests/linux64/releng.manifest', 'fetch']
ERROR - failed to fetch 'setup.sh': HTTP Error 403: Forbidden
ERROR - failed to fetch 'clang.tar.bz2': HTTP Error 403: Forbidden
ERROR - The following files failed: 'setup.sh', clang.tar.bz2'
$ wget http://people.mozilla.org/~respindola/clang-linux32.tar.bz2
$ python2.6 tooltool.py -m linux32.manifest add clang-linux32.tar.bz2 setup.sh
$ sed -i 's/clang-.*.tar.bz2/clang.tar.bz2/g' linux32.manifest 
$ cat linux32.manifest 
[
{
"size": 47, 
"digest": "2005a41fe97a5e00997063705f39d42b6a43b1cf7ba306cbc7b1513de34cdcd050fc6326efa2107f19ba0cc67914745dbf13154fa748010a93cf072481ef4aaa", 
"algorithm": "sha512", 
"filename": "setup.sh"
}, 
{
"size": 72350187, 
"digest": "7d2fbe08aca3ae740e33b8aee872705a3b5229681dd0617ceffd6619fba75cb3cb7e1c3a071218f7cfd464003e5cd773cd8e67d16f78df9c50218fb6671580c6", 
"algorithm": "sha512", 
"filename": "clang.tar.bz2"
}
]

$ sudo cp -vi clang-linux32.tar.bz2 /var/www/html/runtime-binaries/tooltool/sha512/7d2fbe08aca3ae740e33b8aee872705a3b5229681dd0617ceffd6619fba75cb3cb7e1c3a071218f7cfd464003e5cd773cd8e67d16f78df9c50218fb6671580c6
`clang-linux32.tar.bz2' -> `/var/www/html/runtime-binaries/tooltool/sha512/7d2fbe08aca3ae740e33b8aee872705a3b5229681dd0617ceffd6619fba75cb3cb7e1c3a071218f7cfd464003e5cd773cd8e67d16f78df9c50218fb6671580c6'

$ sudo chmod 644 /var/www/html/runtime-binaries/tooltool/sha512/*
OK, looks like it works :-)

Should we remove the old clang packages now or should we fix 767508 first?
(In reply to Rafael Ávila de Espíndola (:espindola) from comment #41)
> OK, looks like it works :-)

\o/
 
> Should we remove the old clang packages now or should we fix 767508 first?

Yeah, let's remove it. I hope bug 767508 will be very simple to solve with $topsrcdir.
TODO:
1) test in staging
2) deploy when Rafael is OK with package removal
Attachment #635868 - Flags: review?(respindola)
Comment on attachment 635868 [details] [diff] [review]
remove clang packages

OK. Just wait for me to send an email on how to use the new packages.
Attachment #635868 - Flags: review?(respindola) → review+
Add the current clang manifests to the tree. This makes it easy to build a patch that uses them on try, just copy them over the releng.manifest.
Attachment #635970 - Flags: review?(rail)
Attachment #635970 - Flags: review?(rail) → review+
Feel free to close the bug.
Fixed :-)
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Whiteboard: [leave open]
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.