Status

Release Engineering
General
RESOLVED FIXED
6 years ago
4 years ago

People

(Reporter: espindola, Assigned: espindola)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(5 attachments, 9 obsolete attachments)

3.35 KB, patch
espindola
: review+
Details | Diff | Splinter Review
5.97 KB, patch
rail
: review+
Details | Diff | Splinter Review
448 bytes, patch
rail
: review+
Details | Diff | Splinter Review
4.85 KB, patch
espindola
: review+
Details | Diff | Splinter Review
2.32 KB, patch
rail
: review+
Details | Diff | Splinter Review
r157958 fixed PR12979 which was casing problems with our CheckedInt.h.

I will upload the patches in a sec.
Created attachment 630033 [details] [diff] [review]
Update clang to 157958 on os x
Assignee: nobody → respindola
Status: NEW → ASSIGNED
Attachment #630033 - Flags: review?(rail)
Created attachment 630295 [details] [diff] [review]
upgrade clang on linux to 157958

The new patch is because the version of glibc on centos 5 does not define O_CLOEXEC.
Attachment #630295 - Flags: review?(rail)
Attachment #630033 - Flags: review?(rail) → review+
Attachment #630295 - Flags: review?(rail) → review+
Summary: Upgrade clang to 157958 → Upgrade clang to 158158
Testing on 32 bit linux found llvm.org/pr13048. I have fixed an restarted the tests.
Created attachment 631067 [details] [diff] [review]
upgrade clang on linux to 158158
Attachment #630295 - Attachment is obsolete: true
Attachment #631067 - Flags: review?(rail)
Attachment #631067 - Flags: review?(rail) → review+
Created attachment 631068 [details] [diff] [review]
Update clang to 158158 on os x
Attachment #630033 - Attachment is obsolete: true
Attachment #631068 - Flags: review?(rail)
Attachment #631068 - Flags: review?(rail) → review+
32 bit linux is OK now, so I have checked in the patches:

https://hg.mozilla.org/build/rpm-sources/rev/d9b3401ac0ba
https://hg.mozilla.org/build/braindump/rev/662fbcab17f8

will attach the puppet ones in a sec.
Created attachment 631371 [details] [diff] [review]
puppet change 1: install the new packages
Attachment #631371 - Flags: review?(rail)
Created attachment 631522 [details] [diff] [review]
support for building with "old" xcode

This is not needed to build with xcode 4.3, but is with xcode 4.0.

If I remember correctly, we have xcode 4.1 in the bots. I don't know if it needs this patch or not. Would you mind testing and r+ or r- the patch depending on the result?

Thanks
Attachment #631522 - Flags: review?(rail)
Comment on attachment 631522 [details] [diff] [review]
support for building with "old" xcode

Yeah, this needed even on the lion builders...
Attachment #631522 - Flags: review?(rail) → review+
Comment on attachment 631522 [details] [diff] [review]
support for building with "old" xcode

https://hg.mozilla.org/build/braindump/rev/049fb30ef0b3
Attachment #631522 - Flags: checked-in+
I'm still waiting for some outstanding release related jobs. I hope to start deploying later today or tomorrow.
Comment on attachment 631371 [details] [diff] [review]
puppet change 1: install the new packages

Sorry about the delay, we had a couple of chemspill releases last week.

I'll land the patch and deploy the packages today.

BTW, would you consider to use tooltool (https://github.com/jhford/tooltool) to deploy clang packages next time? This will require building relocatable tarballs with clang (not rpms and dmgs), in a similar way how we build gonk gor b2g builds.
Attachment #631371 - Flags: review?(rail) → review+
Comment on attachment 631371 [details] [diff] [review]
puppet change 1: install the new packages

http://hg.mozilla.org/build/puppet-manifests/rev/587ce9e7e906
Attachment #631371 - Flags: checked-in+
> BTW, would you consider to use tooltool (https://github.com/jhford/tooltool)
> to deploy clang packages next time? This will require building relocatable
> tarballs with clang (not rpms and dmgs), in a similar way how we build gonk
> gor b2g builds.

That lets us use the same format for OS X and linux? That is awesome! I will try it next time, yes!
Created attachment 634152 [details] [diff] [review]
puppet change 2: remove the old packages

This should only be checked in after a try run confirms the new packages are OK.
Attachment #634152 - Flags: review?(rail)
(In reply to Rafael Ávila de Espíndola (:espindola) from comment #14)
> That lets us use the same format for OS X and linux? That is awesome! I will
> try it next time, yes!

yes, you can use multiple versions and use tooltool manifests in the source tree to point to the particular version. Let's use it next time!
Comment on attachment 634152 [details] [diff] [review]
puppet change 2: remove the old packages

lgtm, let me know when you want me to deploy this.
Attachment #634152 - Flags: review?(rail) → review+
A try push is at

https://tbpl.mozilla.org/?tree=Try&rev=c2228487f25a

It looks like there is a problem in mochitest-4. It is not clear if it is a regression in clang or if new firefox code found the problem. I did a push with the old version in

https://tbpl.mozilla.org/?tree=Try&rev=ca9e8fc2105f

to find out.
The failure on try in caused by llvm's r158126, which was fixed by r158160, so we just missed it :-(

Rail, what do you think we should do? If the old packages were still in place I would say just close this bug as "wontfix" and I would open a new one with that bug (and another one Ehsan found) fixed.

Since the OS X packages have been replaced, I can see these options:

* Rollback the OS X packages.
* Live with this bug for now. Might cause some confusion, but this is still not in production, so we can probably live with it.
* Rebuild the packages at r158160. Somewhat annoying since we know we will do another upgrade and we want to change the package format.
What do you think about this:

1) rail backs out http://hg.mozilla.org/build/puppet-manifests/rev/587ce9e7e906
2) espindola bui1ds 58160 for linux, linux64 and mac do we can use it by tooltool. We'll need to package the files into a tarball and unpack it by setup.sh (so it will be accessible from /builds/slave/xxxx/clang(-version?).

Try is aware of tooltool manifests (there are empty ones in browser/config/tooltool-manifests), but we'll need to ask IT to put the package and setup.sh to http://runtime-binaries.pvt.build.mozilla.org/tooltool (someday we'll have a webapp for this! :) ).

What do you think?
OK, so you suggestion is to upgrade to 158160 using tooltool. I can do that. Is there documentation somewhere on what exactly the tarball has to look like?
There is a Wiki page here: https://wiki.mozilla.org/ReleaseEngineering/Applications/Tooltool

The main idea is having a simple tool which fetches files and runs a command to setup the environment properly.

From my understanding you'll need to build a relocatable package and have a setup.sh script to unpack it and probably setup symlinks (ln -s clang-xxx clang), then update your manifests and mozconfigs. You may want to take a look at the b2g build logs to see how they deploy gonk toolchain.
Summary: Upgrade clang to 158158 → Upgrade clang to 158160
Attachment #631067 - Attachment is obsolete: true
Attachment #631068 - Attachment is obsolete: true
Attachment #631371 - Attachment is obsolete: true
Attachment #631522 - Attachment is obsolete: true
Attachment #634152 - Attachment is obsolete: true
Created attachment 634911 [details] [diff] [review]
clang build script

The build is still going, but we should be able to do the code review in parallel.

This script is a reduced version of the generic toolchain building one. In is *not* reproducible, but the spec file was not.

This version is linux (centos 5) only. I will update it to support os X as soon as the linux build finishes.
Attachment #634911 - Flags: review?(rail)
Attachment #634911 - Flags: review?(rail) → review+
Whiteboard: [leave open]
Created attachment 635004 [details] [diff] [review]
Small fixes to the build script

* The old tar in centos has no --mtime
* Install in "clang", not "inst"
* We are only doing a two stage build.

With these changes, it is possible to build a clang.tar.bz2 in centos 5.
Attachment #634911 - Attachment is obsolete: true
Attachment #635004 - Flags: review?(rail)
Attachment #635004 - Flags: review?(rail) → review+
https://hg.mozilla.org/mozilla-central/rev/fd73f9eb62b7
Created attachment 635293 [details] [diff] [review]
rm 158158

Not tested yet.
FTR.

pkgdmg provider is a little bit dumb, and doesn't cleanup in /var/db after upgrading a package. It was thinking that r155417 is installed because there is /var/db/.puppet_pkgdmg_installed_clang-3.0-r155417.moz0.dmg left after r155417 -> r158158 upgrade.


I removed /var/db/.puppet_pkgdmg_installed_clang-3.0-r155417.moz0.dmg on bld-lion-r5-0{01..40} (try lion build slaves). They should pickup r155417 today.
Comment on attachment 635293 [details] [diff] [review]
rm 158158

Worked fine in staging.
Attachment #635293 - Flags: review?(respindola)
Attachment #635293 - Flags: review?(respindola) → review+
Created attachment 635313 [details] [diff] [review]
Add support to OS X in the build script

necessary changes:

* Different set of patches
* Different host compiler and options
* The tar in OS X is in /usr/bin and doesn't support --owner
Attachment #635313 - Flags: review?(rail)
Attachment #635004 - Attachment is obsolete: true
Comment on attachment 635293 [details] [diff] [review]
rm 158158

http://hg.mozilla.org/build/puppet-manifests/rev/40f20eb317ca

deployed
Attachment #635293 - Flags: checked-in+
Comment on attachment 635313 [details] [diff] [review]
Add support to OS X in the build script

lgtm
Attachment #635313 - Flags: review?(rail) → review+
Created attachment 635390 [details] [diff] [review]
add a setup.sh script for use with tooltool
Attachment #635390 - Flags: review?(rail)
Attachment #635390 - Flags: review?(rail) → review+
Bug 766708 is still ongoing, so I uploaded the clang packages to http://people.mozilla.org/~respindola. Rail, can you deploy them to relengweb1.dmz.scl3.mozilla.com or do you have to build them again?
https://hg.mozilla.org/mozilla-central/rev/6f63ad04afb1
FTR,

$ wget https://raw.github.com/jhford/tooltool/master/tooltool.py
$ wget https://hg.mozilla.org/integration/mozilla-inbound/raw-file/84dffec9742c/build/unix/build-clang/setup.sh
$ wget http://people.mozilla.org/~respindola/clang-linux.tar.bz2
$ wget http://people.mozilla.org/~respindola/clang-osx.tar.bz2

$ python2.6 tooltool.py -m linux.manifest add clang-linux.tar.bz2 setup.sh
$ python2.6 tooltool.py -m osx.manifest add clang-osx.tar.bz2 setup.sh
$ sed -i 's/clang-.*.tar.bz2/clang.tar.bz2/g' *.manifest

$ cat linux.manifest 
[
{
"size": 47, 
"digest": "2005a41fe97a5e00997063705f39d42b6a43b1cf7ba306cbc7b1513de34cdcd050fc6326efa2107f19ba0cc67914745dbf13154fa748010a93cf072481ef4aaa", 
"algorithm": "sha512", 
"filename": "setup.sh"
}, 
{
"size": 71534797, 
"digest": "66bd11bea6e1f07090e9e03c833e107088097605611fd455e80b280ce2b71ca71ff9841a66614f62da162469b222b5eefd5535373b199c60fd485959889b5dcb", 
"algorithm": "sha512", 
"filename": "clang.tar.bz2"
}
]

$ cat osx.manifest 
[
{
"size": 47, 
"digest": "2005a41fe97a5e00997063705f39d42b6a43b1cf7ba306cbc7b1513de34cdcd050fc6326efa2107f19ba0cc67914745dbf13154fa748010a93cf072481ef4aaa", 
"algorithm": "sha512", 
"filename": "setup.sh"
}, 
{
"size": 63015959, 
"digest": "e0ac132a77b052d6494d4bf02f17aba475138ffcff07a5f92f17f45b1d5f27b7b0cea36c29473965271e60910af82ffd4989df94c10d24794d1bf8362bcb785b", 
"algorithm": "sha512", 
"filename": "clang.tar.bz2"
}
]

$ sudo cp -vi setup.sh /var/www/html/runtime-binaries/tooltool/sha512/2005a41fe97a5e00997063705f39d42b6a43b1cf7ba306cbc7b1513de34cdcd050fc6326efa2107f19ba0cc67914745dbf13154fa748010a93cf072481ef4aaa
`setup.sh' -> `/var/www/html/runtime-binaries/tooltool/sha512/2005a41fe97a5e00997063705f39d42b6a43b1cf7ba306cbc7b1513de34cdcd050fc6326efa2107f19ba0cc67914745dbf13154fa748010a93cf072481ef4aaa'

$ sudo cp -vi clang-linux.tar.bz2 /var/www/html/runtime-binaries/tooltool/sha512/66bd11bea6e1f07090e9e03c833e107088097605611fd455e80b280ce2b71ca71ff9841a66614f62da162469b222b5eefd5535373b199c60fd485959889b5dcb
`clang-linux.tar.bz2' -> `/var/www/html/runtime-binaries/tooltool/sha512/66bd11bea6e1f07090e9e03c833e107088097605611fd455e80b280ce2b71ca71ff9841a66614f62da162469b222b5eefd5535373b199c60fd485959889b5dcb'

$ sudo cp -vi clang-osx.tar.bz2 /var/www/html/runtime-binaries/tooltool/sha512/e0ac132a77b052d6494d4bf02f17aba475138ffcff07a5f92f17f45b1d5f27b7b0cea36c29473965271e60910af82ffd4989df94c10d24794d1bf8362bcb785b
`clang-osx.tar.bz2' -> `/var/www/html/runtime-binaries/tooltool/sha512/e0ac132a77b052d6494d4bf02f17aba475138ffcff07a5f92f17f45b1d5f27b7b0cea36c29473965271e60910af82ffd4989df94c10d24794d1bf8362bcb785b'
I forgot to build a 32 bit linux package. I will fix that in a sec, but I think we are also missing a chmod:


retry: Calling <function run_with_timeout at 0x2aaaaed12ed8> with args: (['/tools/tooltool.py', '--url', 'http://runtime-binaries.pvt.build.mozilla.org/tooltool', '--overwrite', '-m', 'browser/config/tooltool-manifests/linux64/releng.manifest', 'fetch'], 1260, None, None, False, True), kwargs: {}, attempt #1
Executing: ['/tools/tooltool.py', '--url', 'http://runtime-binaries.pvt.build.mozilla.org/tooltool', '--overwrite', '-m', 'browser/config/tooltool-manifests/linux64/releng.manifest', 'fetch']
ERROR - failed to fetch 'setup.sh': HTTP Error 403: Forbidden
ERROR - failed to fetch 'clang.tar.bz2': HTTP Error 403: Forbidden
ERROR - The following files failed: 'setup.sh', clang.tar.bz2'
The linux32 package is at
http://people.mozilla.org/~respindola/clang-linux32.tar.bz2
$ wget http://people.mozilla.org/~respindola/clang-linux32.tar.bz2
$ python2.6 tooltool.py -m linux32.manifest add clang-linux32.tar.bz2 setup.sh
$ sed -i 's/clang-.*.tar.bz2/clang.tar.bz2/g' linux32.manifest 
$ cat linux32.manifest 
[
{
"size": 47, 
"digest": "2005a41fe97a5e00997063705f39d42b6a43b1cf7ba306cbc7b1513de34cdcd050fc6326efa2107f19ba0cc67914745dbf13154fa748010a93cf072481ef4aaa", 
"algorithm": "sha512", 
"filename": "setup.sh"
}, 
{
"size": 72350187, 
"digest": "7d2fbe08aca3ae740e33b8aee872705a3b5229681dd0617ceffd6619fba75cb3cb7e1c3a071218f7cfd464003e5cd773cd8e67d16f78df9c50218fb6671580c6", 
"algorithm": "sha512", 
"filename": "clang.tar.bz2"
}
]

$ sudo cp -vi clang-linux32.tar.bz2 /var/www/html/runtime-binaries/tooltool/sha512/7d2fbe08aca3ae740e33b8aee872705a3b5229681dd0617ceffd6619fba75cb3cb7e1c3a071218f7cfd464003e5cd773cd8e67d16f78df9c50218fb6671580c6
`clang-linux32.tar.bz2' -> `/var/www/html/runtime-binaries/tooltool/sha512/7d2fbe08aca3ae740e33b8aee872705a3b5229681dd0617ceffd6619fba75cb3cb7e1c3a071218f7cfd464003e5cd773cd8e67d16f78df9c50218fb6671580c6'

$ sudo chmod 644 /var/www/html/runtime-binaries/tooltool/sha512/*
https://hg.mozilla.org/mozilla-central/rev/c42521d54b79
https://hg.mozilla.org/mozilla-central/rev/84dffec9742c
A new try push with all tests and talos is at

https://tbpl.mozilla.org/?tree=Try&rev=fca5d53846e3
OK, looks like it works :-)

Should we remove the old clang packages now or should we fix 767508 first?
(In reply to Rafael Ávila de Espíndola (:espindola) from comment #41)
> OK, looks like it works :-)

\o/
 
> Should we remove the old clang packages now or should we fix 767508 first?

Yeah, let's remove it. I hope bug 767508 will be very simple to solve with $topsrcdir.
Created attachment 635868 [details] [diff] [review]
remove clang packages

TODO:
1) test in staging
2) deploy when Rafael is OK with package removal
Attachment #635868 - Flags: review?(respindola)
Comment on attachment 635868 [details] [diff] [review]
remove clang packages

OK. Just wait for me to send an email on how to use the new packages.
Attachment #635868 - Flags: review?(respindola) → review+
Created attachment 635970 [details] [diff] [review]
add clang manifests

Add the current clang manifests to the tree. This makes it easy to build a patch that uses them on try, just copy them over the releng.manifest.
Attachment #635970 - Flags: review?(rail)
Attachment #635970 - Flags: review?(rail) → review+
Comment on attachment 635868 [details] [diff] [review]
remove clang packages

http://hg.mozilla.org/build/puppet-manifests/rev/fd06119e5fa6
Attachment #635868 - Flags: checked-in+
Feel free to close the bug.
Fixed :-)
Status: ASSIGNED → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
https://hg.mozilla.org/mozilla-central/rev/23818f9b65a2
Blocks: 768405

Updated

5 years ago
Whiteboard: [leave open]
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.