Closed Bug 738040 Opened 12 years ago Closed 12 years ago

need to add mozilla tools to PATH with puppetagain

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

x86
macOS
task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jhford, Assigned: dustin)

References

Details

Attachments

(3 files)

We have tools that use an alternate --prefix value.  Lets include the ones we want, in the order we want in the system PATH value.
changeset:   135:f7c3d7cb6ca8
tag:         tip
user:        John Ford <jhford@mozilla.com>
date:        Wed Mar 21 15:01:12 2012 -0700
summary:     bug 738040 - add mozilla tools to system-wide PATH
Assignee: nobody → jhford
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Was this for user convenience, or do builds depend on these values?
I'm moving this to users::global, and documenting it.
Assignee: jhford → dustin
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
changeset:   138:49205f58184b
user:        Dustin J. Mitchell <dustin@mozilla.com>
date:        Fri Mar 23 21:57:25 2012 -0700
summary:     Bug 738040: configure system-global $PATH in users::global

I'm assuming that this is for user convenience.  If it *is* for builds, let me know and I'll find a way to set those variables in the buildbot startup, instead.
This was not for user convenience.  It is required for the builds to work with the software installed in the special prefixes without changing the build system all over the place.  Please revert 49205f58184b to add the PATH additions back
Look at the patch - it didn't change behavior.

However, given what you've said I think that the PATH entries should be added specifically for Buildbot at startup, not globally.
This is *appending* to the path, which means that system installs will override the releng-specific packages:

[root@relabs07 ~]# which hg
/tools/python27-mercurial/bin/hg
[root@relabs07 ~]# which python
/usr/bin/python
[root@relabs07 ~]# which python2.6
/usr/bin/python2.6
[root@relabs07 ~]# which python2.7
/tools/python27/bin/python2.7
[root@relabs07 ~]# 

but if something pulls in the system's hg, it will prefer that.

Contrary to what I said in comment 6, I think there's value in having the PATH the same for cltbld's interactive shell as for buildbot.  I *don't* like the idea of having this environment available for root, if possible.

Also, in bug 769780, kmoir's patch adds /usr/local/bin/hg as a symlink with puppet, which shouldn't be necessary if this PATH addition was working correctly (and maybe it wasn't necessary?)

My proposal is this:
 - set PATH=/usr/local/bin:$PATH in cltbld's shell config (note it's prepended)
 - adjust custom RPMs and DMGs to add symlinks in /usr/local/bin/

Any reason not to do so?
Status: REOPENED → ASSIGNED
Things seem positive in IRC.

Further question: a bunch of packages::mozilla::* aren't used right now.  Should I upgrade them to install /usr/local/bin links, delete the manifests, or just leave them as they are?
I'd leave them present and upgrade to do the /usr/local/bin links for now, we can have an |# XXX Unused| comment if you want, and be sure to remove the comment when/if we start using them.

I think leaving them present will help us short-term, in the potential that we WILL use them, rather than need to reinvent our wheel when we do start needing to use it.
I'm worried about adding new versions of unused packages, without much testing.  I think I'll take a middle road, and add a comment to the manifests for each unused package:
  # NOTE: this package does not install links in /usr/local/bin; see bug 738040

For the remainder, the upgrade is pretty straightforward - in %install:

# add /usr/local/bin links
mkdir -p $RPM_BUILD_ROOT/usr/local/bin
(
    cd $RPM_BUILD_ROOT/%{_prefix}/bin/
    for f in *; do
        ln -s %{_prefix}/bin/$f $RPM_BUILD_ROOT/usr/local/bin
    done
)

and in %files:

/usr/local/bin
Attached patch bug738040.patchSplinter Review
Attachment #640655 - Flags: review?(kmoir)
Comment on attachment 640655 [details] [diff] [review]
bug738040.patch

Looks good.
Attachment #640655 - Flags: review?(kmoir) → review+
I accidentally pushed the patch before the packages, which will leave hosts with PATH=/usr/local/bin, but without the files there.  I've temporarily reverted the patch, and will re-commit it when the packages are sync'd (so, in 36m or so).
Actually, I need to wait until yum repository caches have been invalidated, which is 6h.  So I'll re-land this tomorrow.
Sooo.. in my own work, I got hit by the unconditional ln's here for python. Linux foopy installs py26 and py27 (moz version) into /tools (which is done that way to avoid the name conflicts)

But we end up trying to link 2to3 and other binaries twice, (for py27 and py26 versions) and getting puppetizing faults due to that. Spoke with dustin on IRC and he's fixing the RPMs now to only link py26 and py27 binaries directly, and ignore the 2to3/etc stuff in /bin.

He'll then update the repo(s) on our puppetmaster, and I'll unblock myself with a manual yum clean.

[root@linux-foopy-test ~]# puppet agent --test --environment=jwood
info: Caching catalog for linux-foopy-test.build.mtv1.mozilla.com
info: Applying configuration version '420394838d95'
notice: /Stage[main]/Packages::Mozilla::Python27/Package[mozilla-python27]/ensure: ensure changed '2
.7.2-3.el6' to '0:2.7.2-4.el6'
notice: /Stage[main]/Packages::Mozilla::Py27_mercurial/Package[mozilla-python27-mercurial]/ensure: e
nsure changed '2.1.1-3.el6' to '0:2.1.1-4.el6'
err: /Stage[main]/Packages::Mozilla::Python26/Package[mozilla-python26]/ensure: change from 2.6.7-3.
el6 to 0:2.6.7-4.el6 failed: Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y install mozil
la-python26' returned 1:

Transaction Check Error:
  file /usr/local/bin/2to3 from install of mozilla-python26-2.6.7-4.el6.x86_64 conflicts with file f
rom package mozilla-python27-2.7.2-4.el6.x86_64
  file /usr/local/bin/idle from install of mozilla-python26-2.6.7-4.el6.x86_64 conflicts with file f
rom package mozilla-python27-2.7.2-4.el6.x86_64
  file /usr/local/bin/pydoc from install of mozilla-python26-2.6.7-4.el6.x86_64 conflicts with file
from package mozilla-python27-2.7.2-4.el6.x86_64
  file /usr/local/bin/smtpd.py from install of mozilla-python26-2.6.7-4.el6.x86_64 conflicts with fi
le from package mozilla-python27-2.7.2-4.el6.x86_64

Error Summary
-------------

 at /etc/puppet/environments/jwood/env/modules/packages/manifests/mozilla/python26.pp:7
Attached patch bug738040.patchSplinter Review
only add python2.x links in /usr/local/bin; r=callek

idle, 2to3, pydoc, etc. aren't necessary, and conflict between
mozilla-python26 and mozilla-python27 anyway.

These packages are built and being deployed now.  I'm still waiting until tomorrow to land the manifest changes, but I'll land this with them.
Attachment #640832 - Flags: review?(bugspam.Callek)
(In reply to Dustin J. Mitchell [:dustin] from comment #16)
> Created attachment 640832 [details] [diff] [review]
> bug738040.patch
> 
> only add python2.x links in /usr/local/bin; r=callek
> 
> idle, 2to3, pydoc, etc. aren't necessary, and conflict between
> mozilla-python26 and mozilla-python27 anyway.
> 
> These packages are built and being deployed now.  I'm still waiting until
> tomorrow to land the manifest changes, but I'll land this with them.

success.

[root@linux-foopy-test ~]# puppet agent --test --environment=jwood
info: Caching catalog for linux-foopy-test.build.mtv1.mozilla.com
info: Applying configuration version '420394838d95'
notice: /Stage[main]/Packages::Mozilla::Python27/Package[mozilla-python27]/ensure: created
notice: /Stage[main]/Packages::Mozilla::Py27_mercurial/Package[mozilla-python27-mercurial]/ensure: c
reated
notice: /Stage[main]/Packages::Mozilla::Python26/Package[mozilla-python26]/ensure: created
notice: Finished catalog run in 42.90 seconds
Attachment #640832 - Flags: review?(bugspam.Callek) → review+
I spot-checked a few HP's, and they have the updated RPMs installed.

Both patches are (re-)landed, along with bug 772599, so even if the updated RPMs are *not* installed, Puppet will install them on the next run.
Status: ASSIGNED → RESOLVED
Closed: 12 years ago12 years ago
Resolution: --- → FIXED
I thin it's time to remove the temporary /etc/profile/releng-path.sh
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Attached patch bug738040Splinter Review
Attachment #649268 - Flags: review?(bugspam.Callek)
Attachment #649268 - Flags: review?(bugspam.Callek) → review+
Status: REOPENED → RESOLVED
Closed: 12 years ago12 years ago
Resolution: --- → FIXED
Attachment #649268 - Flags: checked-in+
Product: mozilla.org → Release Engineering
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: