1042132 - Allow clang-cl to be used on try

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[(no longer active)]

I'm hoping that we can use tooltool to download clang-cl binaries so that we can start doing clang-cl builds on try.

What I need to do is to add a package to tooltool which can be unzipped into a known location so that we can use the path to the compiler in our mozconfigs on try.

John, do you know how I can achieve this?

Comment 1

[John Ford [:jhford] CET/CEST Berlin Time]

Unless you have access to the host that does the downloads, I think the process is that you post the package somewhere that Releng can download it and ask someone on Releng to upload it to tooltool's server.


Documentation on how the uploading works:

https://wiki.mozilla.org/ReleaseEngineering/Applications/Tooltool#How_to_upload_to_tooltool

Example bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1025731#c5

Comment 2

[(no longer active)]

Thanks a lot, John.  I'll try to do that tomorrow.

Another question, since clang-cl is a young project and we are fixing bugs in the compiler that we have caught during this effort, we may need to do periodic updates to the compiler.  How convenient is it to upload new tooltool packages?  Is it reasonable to assume that we can update the package a few times a week or so?

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

I can't really speak to whether multiple times a week is an option.  If the tooltool manifest is stored in buildbot-configs, it'll be really painful to update.  If it's stored in the gecko tree, the process will be to get the sha512 checksum of the uploaded file and update the manifest in a gecko push when you're feeling ready.

Comment 4

[(no longer active)]

Sounds great!  I'll figure out a way to store it in the Gecko tree to make things easier.

Comment 5

[(not currently active) Ted Mielczarek]

Existing clang tooltool manifests are in-tree, FYI:
http://mxr.mozilla.org/mozilla-central/source/browser/config/tooltool-manifests/linux32/clang.manifest

Comment 6

[(no longer active)]

(In reply to comment #5)
> Existing clang tooltool manifests are in-tree, FYI:
> http://mxr.mozilla.org/mozilla-central/source/browser/config/tooltool-manifests/linux32/clang.manifest

Yeah, thanks, I chatted about this with catlee and rail offline, and they told me pretty much everything there is to know aout this process.  :-)  I forgot to update the bug...

Comment 7

[(no longer active)]

OK, so I uploaded an initial version of clang-cl here: <http://people.mozilla.org/~eakhgari/LLVM-3.6.0svn-r217871-win32.zip>

Can someone please upload that to tooltool?  Thanks!

Comment 8

[(no longer active)]

ping?  If we can move forward with this, please let me know and I will provide a new build.

Comment 9

[Chris AtLee [:catlee]]

sorry, lost track of this.

this is now up on tooltool. You can use this in your tooltool manifestes:
{
"size": 187464180,
"digest": "d84497b53cc30a36129cf97eaf4556dc33c66b79a5e78fefa5b6b5dc6b5db2ac7b53d1881736e9e5ed9fbaaea2d58e468be2653c126fb4cccc125fc2e94b694b",
"algorithm": "sha512",
"filename": "LLVM-3.6.0svn-r217871-win32.zip"
}

Comment 10

[Chris AtLee [:catlee]]

Incidentally, it's possible to upload to tooltool yourself!
https://wiki.mozilla.org/ReleaseEngineering/Applications/Tooltool#How_to_upload_to_tooltool

Comment 11

[(no longer active)]

{
"size": 157145505,
"digest": "c79b420b034493bac467f918d2a3c34dd28614319a18a9176a9718f906010e5ee2c856104dbca70e6e9c628383840a8b40637bf0c43879f8b640be546327b154",
"algorithm": "sha512",
"filename": "LLVM-3.6.0svn-r219740-win32.zip"
}

Comment 12

[(no longer active)]

This setup.sh knows how to extract the LLVM package:

{
"size": 117,
"digest": "dd08739366c3a779f1fa4ff41b63c0c68a2df256cc5c87f3ade46f35891768291c51bc55e98df994d40e1f2b5e5d002cf68623e97b73ac5185769d4d4864352e",
"algorithm": "sha512",
"filename": "setup.sh"
}

Comment 13

[(no longer active)]

clang-cl with assertions turned on, same revision as in comment 11:

{
"size": 188983493,
"digest": "b36855156ab370fe9d49e3e95fa14a9a56a64a2229b79835d7d1e30f7c06611fac3e9eb2c7297088747eed1d7bfb45aae031cdc8ca062d2d53dd66454b767a90",
"algorithm": "sha512",
"filename": "LLVM-3.6.0svn-r219740-win32.zip"
}

Comment 14

[Rail Aliiev [:rail]]

Ehsan, is there anything needed to be done here?

Comment 15

[(no longer active)]

(In reply to Rail Aliiev [:rail] from comment #14)
> Ehsan, is there anything needed to be done here?

My last attempt here lead into sadness because I hit the buildbot maximum log limit.  AFAIK that is not possible to override, right?

Comment 16

[Rail Aliiev [:rail]]

Yeah, IIRC, it won't be that easy to make it happen. Maybe this is a good opportunity to implement this job in Taskcluster. Spidermoney is almost done in bug 1164656.

Comment 17

[(no longer active)]

Maybe.  Does TaskCluster have full working support for Windows?

Comment 18

[Nathan Froyd [:froydnj]]

(In reply to [Vacation: Nov 3-19] from comment #15)
> (In reply to Rail Aliiev [:rail] from comment #14)
> > Ehsan, is there anything needed to be done here?
> 
> My last attempt here lead into sadness because I hit the buildbot maximum
> log limit.  AFAIK that is not possible to override, right?

Did you hit this due to warning spam?  I've got some patches that make the warning spam much more reasonable.

Also, ni? to Rail for comment 17.

Comment 19

[Rail Aliiev [:rail]]

(In reply to [Vacation: Nov 3-19] from comment #17)
> Maybe.  Does TaskCluster have full working support for Windows?

Not yet. The current plan is to add windows support in Q2 2016.

Comment 20

[(no longer active)]

Attachment: Part 1: Port build-clang.py to Windows

This is useful for deploying clang-cl to the infrastructure.

Comment 21

[(no longer active)]

Attachment: Part 2: Add tooltool manifests for clang-cl

Comment 22

[(no longer active)]

Attachment: Part 3: Add a MOZCONFIG helper script that lets you pick clang-cl

Comment 23

[(no longer active)]

I have been successfully running clang-cl on try with these patches.

Comment 24

[Rail Aliiev [:rail]]

Comment on attachment 8716786 [details] [diff] [review]
Part 1: Port build-clang.py to Windows

Review of attachment 8716786 [details] [diff] [review]:
-----------------------------------------------------------------

wow, windows!!! \o/

Comment 25

[Rail Aliiev [:rail]]

Comment on attachment 8716787 [details] [diff] [review]
Part 2: Add tooltool manifests for clang-cl

Review of attachment 8716787 [details] [diff] [review]:
-----------------------------------------------------------------

stamp!

Comment 26

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/13d0b431096a
https://hg.mozilla.org/integration/mozilla-inbound/rev/a21e103469af
https://hg.mozilla.org/integration/mozilla-inbound/rev/18c149301a73

Comment 27

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/13d0b431096a
https://hg.mozilla.org/mozilla-central/rev/a21e103469af
https://hg.mozilla.org/mozilla-central/rev/18c149301a73