Note: There are a few cases of duplicates in user autocompletion which are being worked on.

Have a world readable tooltool repository.

RESOLVED FIXED

Status

Release Engineering
General
RESOLVED FIXED
5 years ago
2 months ago

People

(Reporter: espindola, Unassigned)

Tracking

(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [tooltool])

Attachments

(1 attachment)

Currently the tooltool repository used in the builds is not readable from the outside world.

Some of the tools are open source (clang for example), and we should make then globally visible. This would allow others to reproduce results they get on try. Currently whoever builds the packages has to remember to permanently keep an otherwise redundant copy on people.mozilla.org.
One option here may be to have *multiple* tooltool repositories that the tool checks in sequence.  Then we can use
  http://tooltool.pvt.build.mozilla.org
  http://tooltool.pub.build.mozilla.org
with the option of later spreading the private repositories out over multiple hosts for resiliency.
Yeah, that's exactly where we're headed towards since try will need its own tooltool repo as well. This will require minor changes to the tooltool client.

Updated

5 years ago
Blocks: 774109

Updated

5 years ago
Blocks: 776261

Updated

5 years ago
Component: Release Engineering → Release Engineering: Developer Tools
QA Contact: lsblakk
Whiteboard: [tooltool]

Updated

5 years ago
Blocks: 775539
QA Contact: lsblakk → hwine
To extend that a little, I'll set up the following tooltool roots on the new and old web clusters:

  http://tooltool.pvt.build.mozilla.org/build - existing content
  http://tooltool.pub.build.mozilla.org/try - empty

with the option to add more directories easily.
Running on the old and new clusters, out of new directories on the mount.

Both of these are reading from /mnt/netapp/relengweb/tooltool, rather than the old runtime-binaries directory.  I've set up a host-local rsync cronjob (every 5 minutes) from the old to the new on relengweb1, but once we can take the old hostname down, I'll need to remove that rsync.

For the moment, the upload process for respindola remains the same, and it can do so even after config changes to use the new vhosts.
(In reply to Dustin J. Mitchell [:dustin] from comment #3)
> To extend that a little, I'll set up the following tooltool roots on the new
> and old web clusters:
> 
>   http://tooltool.pvt.build.mozilla.org/build - existing content
>   http://tooltool.pub.build.mozilla.org/try - empty
> 
> with the option to add more directories easily.

Why is one directory named try and the other one build? Should I open a bug about moving the open source packages to the public one (the clang package at least I know can/should be moved)?
I took a guess that the try repo will be public.  It's easy to change that.  Don't make any moves yet, since the script is still not looking at *either* of these URLs.  In other words, nothing for you to worry about here, yet.
Blocks: 768123
Created attachment 655377 [details] [diff] [review]
public httpd conf file

(In reply to Dustin J. Mitchell [:dustin] from comment #6)
> I took a guess that the try repo will be public.  It's easy to change that. 
> Don't make any moves yet, since the script is still not looking at *either*
> of these URLs.  In other words, nothing for you to worry about here, yet.

I just updated http://tooltool.pub.build.mozilla.org/ to have a build/ directory like the pvt and threw in it the setup.sh script and the 9 current (m-c and m-a) clang files. The use-case for this manual change is SeaMonkey right now. So only things that I *know* can be public will be put there, it does have the benefit that others than SeaMonkey can use these files, but I do not want to resolve this bug or advertise it too much until 'we' come up with a better system to do this.

If I had more time atm I could/would drive a proper resolution here, instead, but I don't just yet.

The attached is the changes to teh conf file I think are necessary to handle this (my initial changes were overwrote by puppet, and I don't seem to have access to the svn repo section where this is defined)
Attachment #655377 - Flags: review?(dustin)
Comment on attachment 655377 [details] [diff] [review]
public httpd conf file

We definitely don't want ExecCGI turned on here.  I assume the Directory is present mainly to enable Indexes, but the internal repo doesn't have that, and I don't think we want people trolling the repo for binaries, so I'd prefer to leave it off.  So, just adding the alias is probably the best bet.  Can we name it "temp-sm-stuff" instead, so it doesn't become de-facto production?

I'll land the Alias addition now as you've given it, but let me know if we can rename and I will (and rename the corresponding dir, too).
Attachment #655377 - Flags: review?(dustin) → review-
Oh, and until we turn up the new cluster (bug 774354), we have to make these changes in two places, so if you do get svn access, be careful.  (The other place is the webapp module, which covers *all* webapps at Mozilla).
(In reply to Dustin J. Mitchell [:dustin] from comment #8)
> Comment on attachment 655377 [details] [diff] [review]
> public httpd conf file
> 
> We definitely don't want ExecCGI turned on here.  I assume the Directory is
> present mainly to enable Indexes,

Correct, iirc in my skim of the code, before it tries to download the sha512 sum-file, it makes sure it is THERE, which is why it needs the index.

> but the internal repo doesn't have that,

It does, (on relengweb1) and is exactly where I copied it from.

> and I don't think we want people trolling the repo for binaries, so I'd
> prefer to leave it off.  So, just adding the alias is probably the best bet.

I don't think we need to worry too much about that, at least right now.

> Can we name it "temp-sm-stuff" instead, so it doesn't become de-facto
> production?

I'm not strongly opposed, but I like the current naming I did anyway as both for-seamonkey and a future "de-facto production". So I'd say about 65% in favor of what i did, though "do as you like, just let me know what you do".

> I'll land the Alias addition now as you've given it, but let me know if we
> can rename and I will (and rename the corresponding dir, too).

I want to figure out the index thing though as well, fwiw.
I don't see any fetching of the index in
 https://github.com/jhford/releng-rpms/blob/master/tooltool.py
It verifies whether the file is on disk locally - maybe that's what you're seeing?

That said, I'm not horribly opposed to the indexes, so I'll turn them on.

Given that you've described /builds/ as "de-facto production", I'm now 100% opposed to it, so I'll change that to /temp-sm-stuff/.
So, we have a world-readable repo now.  Bug 882712 will point production tooltool users to it.  So I think this is done.
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Blocks: 882712
(In reply to comment #12)
> So, we have a world-readable repo now.  Bug 882712 will point production
> tooltool users to it.  So I think this is done.

Where is it?
http://tooltool.pub.build.mozilla.org/try (comment 5)

It's still empty - that part's above my pay grade :)
(In reply to comment #14)
> http://tooltool.pub.build.mozilla.org/try (comment 5)
> 
> It's still empty - that part's above my pay grade :)

Is there a bug that tracks making the "content' of that repo public?  A public empty repo isn't exactly useful...
I don't know - check with simone?
Simone, can you please see comment 15?  Thanks!
Flags: needinfo?(sbruno)

Updated

4 years ago
Blocks: 885777
(Assignee)

Updated

4 years ago
Product: mozilla.org → Release Engineering
Hi all,

There has been some work done on tooltool in the past weeks, now all the relevant code changes have been implemented and will be deployed soon.

The status of the tooltool activities is summed up in Bug 920485.

I will soon add to that bug a couple of links to relevant documentation which will explain, in particular, how these changes will make possible to manage several tooltool repositories.

Hope this helps!
Flags: needinfo?(sbruno)
(Assignee)

Updated

2 months ago
Component: Tools → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.