Add more logic in handling the version of Mozmill in our environment

RESOLVED INCOMPLETE

Status

RESOLVED INCOMPLETE
7 years ago
5 years ago

People

(Reporter: whimboo, Assigned: whimboo)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [mozmill-env])

Attachments

(1 attachment)

(Assignee)

Description

7 years ago
Over the weekend I have made some more thoughts about how we could better handle the version of Mozmill in our shipped environments. Right now we include a fixed version. It's safe but would require us to continuously have to update the environments across platforms whenever the version of Mozmill changes. One idea I came up with is described below, and I would kinda like to get feedback from you.

* We do not ship Mozmill and its dependencies with the environment. That means the environment only gets an update when base components have to be replaced.

* Given that we will have a bash shell across all platforms we can add an initialization routine to the .profile file. It would check dependent on a simple text file on mozqa.com, if the currently supported version of Mozmill is installed. If not Mozmill has to be installed/upgraded/downgraded. After each call to pip an additional script will be executed which updates the paths of all .py scripts to be relocatable. 

* There will be an ENV variable which will allow users to override the supported version of Mozmill. It will allow us to let users test beta versions of upcoming Mozmill versions.

Benefits
* Lesser maintenance work on any of our machines.
* Integrated logic to have an always relocatable environment
* No need to require an user to download a new versions of the environment when Mozmill gets upgraded/downgraded
* Users can install additional tools from pypi or any other side (easy_install and pip are not accessible right now)
(Assignee)

Updated

7 years ago
Blocks: 623204
(Assignee)

Comment 1

7 years ago
Created attachment 573886 [details] [diff] [review]
Allow to install/upgrade/uninstall modules v1

I wanted to upload that already yesterday for feedback. With this change we add a pip wrapper that always forces the install/upgrade of modules into the environmental location. From the CLI side the user can still use "pip install whatever".

The PoC has been implemented in Python so it can be re-used across platforms with a bit of modification. What do you think? I would appreciate feedback before I will continue later next week for Linux and Windows.
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Attachment #573886 - Flags: feedback?(gmealer)
(Assignee)

Updated

7 years ago
Attachment #573886 - Flags: feedback?(dburns)
Comment on attachment 573886 [details] [diff] [review]
Allow to install/upgrade/uninstall modules v1

So here's the thing, I don't know whether to + or -.

Making the upgrade as easy as possible makes sense. 

However, I don't think you want to do it automatically at login (via either build.sh or .profile, which applies to login shells) or via cron, because you don't know whether a test is running at that time. 

I think the right way to go is either:

A) Update manually, as we do now. It's not -that- much maintenance, we just need to have a "mozmill changed, now what" checklist. It could be driven with Puppet after bringing a cluster down for maintenance to cut interaction.

-or-

B) Make this process part of the environment setup that's deployed at test runtime and then removed again, and isolated only to that test run. IOW, don't install into any site libraries or central virtualenvs, only in one-off site libraries.

That's the only way I know to have this happen without affecting existing runs.

I'm unsure whether you need this wrapper if you're not doing it automatically. The code's fine, and if that were the only question I'd f+.
(Assignee)

Comment 3

7 years ago
Geo, please keep in mind that the current patch does not involve any strategy in updating the environment at login. It's really only about being able to install/upgrade/remove Python packages inside the environment.

Further, which I think you forgot about, the environment will not only be used by ourselves for any automation purpose, but also by Mozmill Crowd, or any tester who wants to have an isolated environment.

Any discussion on how to upgrade the environment for our own needs I would kindly like to punt to end of next or overnext week. At this time please only give feedback about the feature I have mentioned above. Thanks.
Perhaps I've misunderstood something, then. Is build.sh not the process we run to create the bash shell, at least on Windows?

If so, then that means this'll potentially do an update implicitly when you open the next bash shell, which might be while something else is running. Addressing that may mean separating the process behind Crowd from the process behind our environment.

Also, the original bug was posed as:

* Given that we will have a bash shell across all platforms we can add an initialization routine to the .profile file. It would check dependent on a simple text file on mozqa.com[...and download the new version]

...which is to say, update the version on login (or opening a login shell, which is when .profile is executed).

So afaik I have commented on the issue at hand as well as the code that's in front of me, with all knowledge you mention above in mind. 

I don't forget about nearly as much as you seem to think I do in the last couple of bugs regarding environment. Rather, I think you just need to seek clarification when you don't understand my thought process rather than assuming I got it wrong. :)

If one of my assumptions is incorrect, or if you've opted to post a fix that doesn't do what you said you would be doing, then let me know. Otherwise, comments stand.
It does occur to me that I might be confusing build.sh with a run.sh script or something similar. 

If that's the case, mea culpa, but in that case can you clarify when build.sh would run (i.e. when would the update actually take place?) The bug was pretty clear it was on shell entry, but perhaps you changed strategies without clarifying?
(Assignee)

Comment 6

7 years ago
(In reply to Geo Mealer [:geo] from comment #4)
> Perhaps I've misunderstood something, then. Is build.sh not the process we
> run to create the bash shell, at least on Windows?

The build and execution process are completely different scenarios. The final usage of the environment will never see the build scripts. Those exist only to automatically create the environment. The only script the user or any automation tool has to use is run.sh. This script spawns a shell, which also is in a virtual environment on OS X and Linux.

> * Given that we will have a bash shell across all platforms we can add an
> initialization routine to the .profile file. It would check dependent on a
> simple text file on mozqa.com[...and download the new version]
> 
> ...which is to say, update the version on login (or opening a login shell,
> which is when .profile is executed).
> 
> So afaik I have commented on the issue at hand as well as the code that's in
> front of me, with all knowledge you mention above in mind. 

Right. That's something we want to get to, and further discussion is necessary. But there are multiple steps involved.

> I don't forget about nearly as much as you seem to think I do in the last
> couple of bugs regarding environment. Rather, I think you just need to seek
> clarification when you don't understand my thought process rather than
> assuming I got it wrong. :)

You have reviewed the initial patch and I thought the overall process was clear. But I can see your confusion now. So let me explain in detail:

The build process will only be triggered if base tools have to be changed, e.g. security updates. Otherwise I would kinda like to keep the version of the environment the same over a couple of months, even across a couple of Mozmill releases. Decoupling Mozmill would be kinda helpful and it would help our Mozmill Crowd users who do not have to download those large packages again and again whenever a new Mozmill version gets released. To create the environment just run build.sh. The zip archives will then end-up on mozqa.com.

For the execution the user or any process has to call run.sh. It's a wrapper script and executes Mozmill tasks inside a bash shell. As mentioned above it will be a virtual environment on OS X and Linux, but not Windows. It can be run scripted or you can open an interactive shell by sourcing run.sh. Without Mozmill being installed, a script called during launching the shell can check for the currently supported Mozmill version. It's just a version number in a file up on mozqa.com. If the local version doesn't match, Mozmill will be auto-updated. That will ensure that users are always using the supported version of Mozmill. 

For our automation needs or any other reason there should be an environment variable the user can set, which overrides this version check. That way you can force to stay on specific version, or test a new beta version of Mozmill.

For our auto-provisioning tasks those environments are located on the shared network storage, and should be copied to the VM whenever a new testrun starts. I really don't want to upgrade Mozmill inside the environment itself, because pip/easy_install are way too broken - instead of an upgrade a downgrade happened. And we have seen this in the past too often.

The environments we can prepare with the Mozmill version we want to use and set the env variable to force that version. That means whenever we replace those files on the network storage, all connected machines will work on the new environment. 

The patches on this bug will be:

1. Enable install/upgrade/uninstall of Python packages inside the environment
2. Version check and auto-upgrade of Mozmill with override capabilities

I hope that clears any confusion? Or is something still unclear? If yes, please tell me.
Comment on attachment 573886 [details] [diff] [review]
Allow to install/upgrade/uninstall modules v1

That's very clear, thanks. 

I really do apologize for the misunderstanding. The truth is that we don't have good documentation as to what the process for using these scripts are, and I wasn't involved in writing or designing them. 

So when I only see them through the litte window of a code review it's easy to not quite get what's going on. In this case I knew just enough to -think- I knew what was going on, which is an easy way to be aggressively wrong.

That's something we should talk about offline at some point. Understanding all this would be key to giving you better code reviews.

Back to this one:

My only concern was updating Mozmill implicitly in a script not otherwise meant to update the environment. Now that you've clarified that build.sh is explicitly to update the environment, there's no concern at all and I've f+'d.
Attachment #573886 - Flags: feedback?(gmealer) → feedback+
Comment on attachment 573886 [details] [diff] [review]
Allow to install/upgrade/uninstall modules v1

Looks good to me
Attachment #573886 - Flags: feedback?(dburns) → feedback+
(Assignee)

Comment 9

7 years ago
Ian, I ran into a problem with setuptools in our environment. Similar to the Addons SDK package, I have moved the site-packages subfolder to './python-lib/' It contains all the files which are necessary to install/upgrade/uninstall Python packages. The Python path has also been updated to check that folder first for modules. There is a setuptools.pth which points to the current version of setuptools in this environment:

mozmill-env$ ls python-lib/ -l
-rw-r--r-- 1 501 dialout    237 2011-11-17 09:35 easy-install.pth
drwxr-xr-x 1 501 dialout    136 2011-11-17 09:35 pip-1.0.2-py2.7.egg
-rw-r--r-- 1 501 dialout 332005 2011-11-17 09:35 setuptools-0.6c11-py2.7.egg
-rw-rw-r-- 1 501 dialout 332974 2011-11-17 09:35 setuptools-0.6c12dev_r88846-py2.7.egg
-rw-r--r-- 1 501 dialout     40 2011-11-17 10:06 setuptools.pth

I now want to let the user allow to use pip to install/upgrade/uninstall Python packages:

mozmill-env$ cat python-lib/setuptools.pth
./setuptools-0.6c11-py2.7.egg

Sadly the setuptools module cannot be found after I sourced into this environment:

$ python -c "import setuptools"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named setuptools

Given that any attempt to install/upgrade/uninstall packages via pip fail. Do you have an idea what's missing, so Python finds that module?

If you want to have a look at the latest state of the environment, I have uploaded one to my people account:

http://people.mozilla.com/~hskupin/downloads/mozmill-env/linux-test.zip

I have no idea what's wrong because I can import modules for other modules, which I have installed right after creating the virtualenv, and before the folder modifications have been applied. So any help would be kinda appreciated.

Thanks
I'm not 100% sure what is going on, but is sounds like you have a set of vendor libraries but those won't be automatically added to the environment.  When you fix up the individual scripts it's okay, but not with you run a script that just uses #!/usr/bin/python (like pip) it won't work.

Coincidentally I just wrote a script to build a virtualenv around a vendor library.  This makes it easier to manage the library, but you don't need it to use the libraries, just manage (install/uninstall/upgrade).  The script is here, though you'll probably want to make some modifications for your own process: https://github.com/mozilla/appsync-vendor/blob/master/setup-vendor.sh

Basic usage is:
- run vendor/setup-vendor.sh
- don't check in much of what this script creates (it tries to create a helpful .gitignore, but just think of that as a suggestion)
- run build-env/bin/pip install/uninstall/upgrade
(Assignee)

Comment 11

7 years ago
Talked with Ian on IRC and it's somewhat more complicated. His script is for developers who want to setup a custom virtualenv. It's similar to our build.sh script. But it's not targeted for end users who don't have any of the necessary build tools installed. 

Probably we have to punt further work on it until later this quarter and have to live with a fixed installation of Mozmill for now.
(Assignee)

Comment 12

7 years ago
Ian pointed me out to the following ticket: https://github.com/pypa/virtualenv/issues/168

I will have to investigate it after I return from vacation. Thanks!
(Assignee)

Updated

7 years ago
Whiteboard: [mozmill-env]
(Assignee)

Comment 13

7 years ago
Now covered by https://github.com/whimboo/mozmill-environment/issues/12
Status: ASSIGNED → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → INCOMPLETE
Product: Mozilla QA → Mozilla QA Graveyard
You need to log in before you can comment on or make changes to this bug.