Last Comment Bug 768879 - Have a world readable tooltool repository.
: Have a world readable tooltool repository.
Status: RESOLVED FIXED
[tooltool]
:
Product: Release Engineering
Classification: Other
Component: Tools (show other bugs)
: other
: x86_64 Linux
: -- normal (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
: Hal Wine [:hwine] (use NI)
Mentors:
Depends on:
Blocks: fx-reproducible-build 768123 774109 775539 776261 882712
  Show dependency treegraph
 
Reported: 2012-06-27 07:10 PDT by Rafael Ávila de Espíndola (:espindola) (not reading bugmail)
Modified: 2013-10-23 02:30 PDT (History)
13 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
public httpd conf file (891 bytes, patch)
2012-08-25 17:09 PDT, Justin Wood (:Callek) (Away until Aug 29)
dustin: review-
Details | Diff | Splinter Review

Description Rafael Ávila de Espíndola (:espindola) (not reading bugmail) 2012-06-27 07:10:57 PDT
Currently the tooltool repository used in the builds is not readable from the outside world.

Some of the tools are open source (clang for example), and we should make then globally visible. This would allow others to reproduce results they get on try. Currently whoever builds the packages has to remember to permanently keep an otherwise redundant copy on people.mozilla.org.
Comment 1 Dustin J. Mitchell [:dustin] 2012-07-09 13:07:28 PDT
One option here may be to have *multiple* tooltool repositories that the tool checks in sequence.  Then we can use
  http://tooltool.pvt.build.mozilla.org
  http://tooltool.pub.build.mozilla.org
with the option of later spreading the private repositories out over multiple hosts for resiliency.
Comment 2 Chris AtLee [:catlee] 2012-07-10 10:35:59 PDT
Yeah, that's exactly where we're headed towards since try will need its own tooltool repo as well. This will require minor changes to the tooltool client.
Comment 3 Dustin J. Mitchell [:dustin] 2012-08-07 09:16:24 PDT
To extend that a little, I'll set up the following tooltool roots on the new and old web clusters:

  http://tooltool.pvt.build.mozilla.org/build - existing content
  http://tooltool.pub.build.mozilla.org/try - empty

with the option to add more directories easily.
Comment 4 Dustin J. Mitchell [:dustin] 2012-08-07 11:03:02 PDT
Running on the old and new clusters, out of new directories on the mount.

Both of these are reading from /mnt/netapp/relengweb/tooltool, rather than the old runtime-binaries directory.  I've set up a host-local rsync cronjob (every 5 minutes) from the old to the new on relengweb1, but once we can take the old hostname down, I'll need to remove that rsync.

For the moment, the upload process for respindola remains the same, and it can do so even after config changes to use the new vhosts.
Comment 5 Rafael Ávila de Espíndola (:espindola) (not reading bugmail) 2012-08-07 11:22:22 PDT
(In reply to Dustin J. Mitchell [:dustin] from comment #3)
> To extend that a little, I'll set up the following tooltool roots on the new
> and old web clusters:
> 
>   http://tooltool.pvt.build.mozilla.org/build - existing content
>   http://tooltool.pub.build.mozilla.org/try - empty
> 
> with the option to add more directories easily.

Why is one directory named try and the other one build? Should I open a bug about moving the open source packages to the public one (the clang package at least I know can/should be moved)?
Comment 6 Dustin J. Mitchell [:dustin] 2012-08-07 11:30:13 PDT
I took a guess that the try repo will be public.  It's easy to change that.  Don't make any moves yet, since the script is still not looking at *either* of these URLs.  In other words, nothing for you to worry about here, yet.
Comment 7 Justin Wood (:Callek) (Away until Aug 29) 2012-08-25 17:09:09 PDT
Created attachment 655377 [details] [diff] [review]
public httpd conf file

(In reply to Dustin J. Mitchell [:dustin] from comment #6)
> I took a guess that the try repo will be public.  It's easy to change that. 
> Don't make any moves yet, since the script is still not looking at *either*
> of these URLs.  In other words, nothing for you to worry about here, yet.

I just updated http://tooltool.pub.build.mozilla.org/ to have a build/ directory like the pvt and threw in it the setup.sh script and the 9 current (m-c and m-a) clang files. The use-case for this manual change is SeaMonkey right now. So only things that I *know* can be public will be put there, it does have the benefit that others than SeaMonkey can use these files, but I do not want to resolve this bug or advertise it too much until 'we' come up with a better system to do this.

If I had more time atm I could/would drive a proper resolution here, instead, but I don't just yet.

The attached is the changes to teh conf file I think are necessary to handle this (my initial changes were overwrote by puppet, and I don't seem to have access to the svn repo section where this is defined)
Comment 8 Dustin J. Mitchell [:dustin] 2012-08-26 09:55:37 PDT
Comment on attachment 655377 [details] [diff] [review]
public httpd conf file

We definitely don't want ExecCGI turned on here.  I assume the Directory is present mainly to enable Indexes, but the internal repo doesn't have that, and I don't think we want people trolling the repo for binaries, so I'd prefer to leave it off.  So, just adding the alias is probably the best bet.  Can we name it "temp-sm-stuff" instead, so it doesn't become de-facto production?

I'll land the Alias addition now as you've given it, but let me know if we can rename and I will (and rename the corresponding dir, too).
Comment 9 Dustin J. Mitchell [:dustin] 2012-08-26 09:56:29 PDT
Oh, and until we turn up the new cluster (bug 774354), we have to make these changes in two places, so if you do get svn access, be careful.  (The other place is the webapp module, which covers *all* webapps at Mozilla).
Comment 10 Justin Wood (:Callek) (Away until Aug 29) 2012-08-26 12:21:32 PDT
(In reply to Dustin J. Mitchell [:dustin] from comment #8)
> Comment on attachment 655377 [details] [diff] [review]
> public httpd conf file
> 
> We definitely don't want ExecCGI turned on here.  I assume the Directory is
> present mainly to enable Indexes,

Correct, iirc in my skim of the code, before it tries to download the sha512 sum-file, it makes sure it is THERE, which is why it needs the index.

> but the internal repo doesn't have that,

It does, (on relengweb1) and is exactly where I copied it from.

> and I don't think we want people trolling the repo for binaries, so I'd
> prefer to leave it off.  So, just adding the alias is probably the best bet.

I don't think we need to worry too much about that, at least right now.

> Can we name it "temp-sm-stuff" instead, so it doesn't become de-facto
> production?

I'm not strongly opposed, but I like the current naming I did anyway as both for-seamonkey and a future "de-facto production". So I'd say about 65% in favor of what i did, though "do as you like, just let me know what you do".

> I'll land the Alias addition now as you've given it, but let me know if we
> can rename and I will (and rename the corresponding dir, too).

I want to figure out the index thing though as well, fwiw.
Comment 11 Dustin J. Mitchell [:dustin] 2012-08-27 05:05:15 PDT
I don't see any fetching of the index in
 https://github.com/jhford/releng-rpms/blob/master/tooltool.py
It verifies whether the file is on disk locally - maybe that's what you're seeing?

That said, I'm not horribly opposed to the indexes, so I'll turn them on.

Given that you've described /builds/ as "de-facto production", I'm now 100% opposed to it, so I'll change that to /temp-sm-stuff/.
Comment 12 Dustin J. Mitchell [:dustin] 2013-06-13 08:49:02 PDT
So, we have a world-readable repo now.  Bug 882712 will point production tooltool users to it.  So I think this is done.
Comment 13 :Ehsan Akhgari 2013-06-13 09:51:33 PDT
(In reply to comment #12)
> So, we have a world-readable repo now.  Bug 882712 will point production
> tooltool users to it.  So I think this is done.

Where is it?
Comment 14 Dustin J. Mitchell [:dustin] 2013-06-13 10:05:54 PDT
http://tooltool.pub.build.mozilla.org/try (comment 5)

It's still empty - that part's above my pay grade :)
Comment 15 :Ehsan Akhgari 2013-06-13 11:02:22 PDT
(In reply to comment #14)
> http://tooltool.pub.build.mozilla.org/try (comment 5)
> 
> It's still empty - that part's above my pay grade :)

Is there a bug that tracks making the "content' of that repo public?  A public empty repo isn't exactly useful...
Comment 16 Dustin J. Mitchell [:dustin] 2013-06-13 12:26:58 PDT
I don't know - check with simone?
Comment 17 :Ehsan Akhgari 2013-06-13 13:02:02 PDT
Simone, can you please see comment 15?  Thanks!
Comment 18 Simone Bruno [:simone] 2013-10-23 02:30:50 PDT
Hi all,

There has been some work done on tooltool in the past weeks, now all the relevant code changes have been implemented and will be deployed soon.

The status of the tooltool activities is summed up in Bug 920485.

I will soon add to that bug a couple of links to relevant documentation which will explain, in particular, how these changes will make possible to manage several tooltool repositories.

Hope this helps!

Note You need to log in before you can comment on or make changes to this bug.