1078362 - Create script that downloads a script and executes it

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Armen [:armenzg]]

Hello!
Thanks for looking into helping the Mozilla project. Your help is very much appreciated and essential to the sustainability of our support to the Mozilla mission. [1]

In order to contribute with this issue you should follow the "setup" instructions in here:
https://wiki.mozilla.org/Auto-tools/Projects/Mozharness#Setup

= The issue =
In bug 1066636 we want to create a way for people to have more freedom to experiment running new type of jobs on tbpl.mozilla.org (e.g. they want to add a new type of test).

In order to accomplish this, we need to write a Mozharness script that would download a file and execute the script.

In order to do this, you can use configtest.py as a starting point [2].
Strip the functions after the __init__ constructor as we won't need them.
To begin with, I would create two actions called "download_script" and "run_script".

To download the file, I would suggest you inherit from transfer.py [3].
I suggest adding a function similar to load_json_from_url, however, it would have to be called download_file_from_url (We could have called it "download_file", however, it collides with the one in ScriptMixin).
In fact, I would like to see the functionality of download_file from ScriptMixin to move to TransferMixin.

For the run_script action, I would like you to use run_command [4].

The final product would be run something like this:
python scripts/run_script_from_url.py --script-url http://hg.mozilla.org/mozilla-central/file/default/path/to/script.py

I see other work needed to integrated well with 1066636, however, I want to reach this first milestone first.

= Get help =
If you need help or guidance feel free to write a comment in this bug.
You can also chat with us by visiting our IRC channel https://chat.mibbit.com/?url=irc%3A%2F%2Firc.mozilla.org%2F%23ateam

To know more about mozharness read:
https://wiki.mozilla.org/Auto-tools/Projects/Mozharness

To know more about IRC you can watch:
http://codefirefox.com/video/irc

[1] https://www.mozilla.org/en-US/about/manifesto/
[2] http://hg.mozilla.org/build/mozharness/file/573e3991380b/scripts/configtest.py
[3] http://hg.mozilla.org/build/mozharness/file/cc4fd618a7d9/mozharness/base/transfer.py#l106
[4] http://hg.mozilla.org/build/mozharness/file/default/mozharness/base/script.py#l646

Comment 1

[Akshay Parelkar]

Hi Armen,

I would like to work on this bug. 

I have actually already implemented this in a file called run_script_from_url.py as suggested above. However, I am not sure how I upload my file here. 

Thanks,
Akshay

Comment 2

[Akshay Parelkar]

Attachment: run_script_from_url.py

I have created a potential script that addresses the request in the bug report.

Comment 3

[Armen [:armenzg]]

Hi Akshay Parelkar,
Thank you for your submission. I will be looking into it and testing it.
I should be able to get back to you by tomorrow during my work day (9am-5pm Toronto time)

Comment 4

[Armen [:armenzg]]

I can see that the script in itself seems to do the trick [1].
The reason is because you have written a __main__ entry and are treating the script as a normal python script rather than a mozharness script. I don't blame you, mozharness is not easy to get started with.

Let's try something cleaner. Let's forget about mozharness for now. Let's write a more pythonesque script.

Can you please write a script that does the following:
1) download a python script
2) create a work dir
3) optionally download a requirements.txt
4) create a temp work dir
5) create a temp virtualenv
5.a) If we have a requirements file specified install it inside the virtualenv
6) run the script from #1 inside of the virtualenv
7) remove the temp work dir and temp virtualenv

The command could look like this:
python run_script_from_url.py --script-url htpp://abc.com/my_script.py --requirements-url htpp://abc.com/my_requirements.txt
 
To create the virtualenv and install the modules go and look at python.py [2] for some code on how to do it.
The mozharness code there might be a bit daunting but I think you should be able to grasp it.
If we typed it by hand we would do something similar to this [3]

I hope this makes a bit of more sense!

[1]
armenzg-thinkpad mozharness hg:[default!] $ python scripts/run_script_from_url.py --script-url https://dl.dropboxusercontent.com/u/129925/hello_world.py
17:54:27     INFO - MultiFileLogger online at 20141020 17:54:27 in /home/armenzg/repos/mozharness
17:54:27     INFO - Run as scripts/run_script_from_url.py --script-url https://dl.dropboxusercontent.com/u/129925/hello_world.py
https://dl.dropboxusercontent.com/u/129925/hello_world.py
hello_world.py
17:54:28     INFO - Running command: python hello_world.py
17:54:28     INFO -  Hello world!
17:54:28     INFO - Return code: 0

[2]
http://hg.mozilla.org/build/mozharness/file/7a3e1849c31d/mozharness/base/python.py#l277
http://hg.mozilla.org/build/mozharness/file/7a3e1849c31d/mozharness/base/python.py#l172

[3]
armenzg-thinkpad mozharness hg:[default!] $ cat my_requirements.txt 
redo
armenzg-thinkpad mozharness hg:[default!] $ virtualenv temp_venv
New python executable in temp_venv/bin/python
Installing Setuptools..............................................................................................................................................................................................................................done.
Installing Pip.....................................................................................................................................................................................................................................................................................................................................done.
armenzg-thinkpad mozharness hg:[default!] $ source temp_venv/bin/activate
(temp_venv)armenzg-thinkpad mozharness hg:[default!] $ pip freeze
argparse==1.2.1
wsgiref==0.1.2
(temp_venv)armenzg-thinkpad mozharness hg:[default!] $ pip install -r my_requirements.txt 
Downloading/unpacking redo (from -r my_requirements.txt (line 1))
  Downloading redo-1.4.tar.gz
  Running setup.py egg_info for package redo
    
Installing collected packages: redo
  Running setup.py install for redo
    
    Installing retry script to /home/armenzg/repos/mozharness/temp_venv/bin
Successfully installed redo
Cleaning up...
(temp_venv)armenzg-thinkpad mozharness hg:[default!] $ pip freeze
argparse==1.2.1
redo==1.4
wsgiref==0.1.2

Comment 5

[Akshay Parelkar]

Hi Armen,

Thanks for your reply. I misinterpreted the script format previously.

I will work on the script as you described above.

-Akshay

Comment 6

[Armen [:armenzg]]

Hi Akshay,
How has it been going?
Feel free to jump on the #ateam IRC channel and discuss any issues you might find. I'm happy to help!

Comment 7

[Akshay Parelkar]

Attachment: My progress so far

Hi Armen, 

Here is my progress so far. 

I have some difficulty understanding how to pass input as the virtualenv "activate" script requires interactive input.

I looked at the run_command method called from create_virtualenv[line 362]. However, I am not sure how input is passed into the virtualenv spawned at line 724.

I will get on the #ateam channel sometime this week. I should most likely be on sometime after 6PM EST.

Thanks,
Akshay

Comment 8

[Armen [:armenzg]]

Attachment: script_with_requirement.py

Comment 9

[Armen [:armenzg]]

Attachment: requirements.txt

Comment 10

[Armen [:armenzg]]

Hello Akshay,
Thanks for your script. It is heading on the right direction, however, we need to adjust it to make it sustainable and ready for production systems.

As I have been working on other parts of the releng code, I don't think we need to make this a mozharness script but just a normal python script.
I have reduced the scope and adjusted the bug's summary.

I have tried your script like this:
 python your_script.py \
 --script-url https://bugzilla.mozilla.org/attachment.cgi?id=8525528 \
 --requirements-url https://bugzilla.mozilla.org/attachment.cgi?id=8525529

Could you please look into making it work?

Comment 11

[Armen [:armenzg]]

Comment on attachment 8525052 [details]
My progress so far

Could you please give me diffs of the script? Once attached to bugzilla it makes it easier to do code review.
You can clone the tools repo and hg add the script there.
 hg clone http://hg.mozilla.org/build/tools
 cd tools/buildfarm/utils
 hg add run_script_from_url.py
 hg diff > /path/to/patch.diff

Here's the review of the code.

>from sys import argv
..
>    
>
>def getopts(argv):
>    opts = {}
>    while argv:
>        if argv[0][0] == '-':
>            opts[argv[0]] = argv[1]
>            argv = argv[2:]
>        else:
>            argv = argv[1:]
>    return opts        
>
>if __name__ == "__main__":
>    scriptArgs = getopts(argv)

Could you please use the following approach?
 import argparse
 
 if __name__ == "__main__":
     parser = argparse.ArgumentParser()
     parser.add_argument("--script-url", dest="script_url", required=True)

>    work_dir = "work_dir"

Can you please place all constants before the main function?
Could you please change this to "workdir"? (no underscore)

>    os.mkdir(work_dir)

This will fail with OSError if we run it more than once.

>    temp_work_dir = "temp_work_dir"
>    os.mkdir(temp_work_dir)
>
>    temp_virtualenv = "../temp_venv"

Let's create the venv under the workdir as "venv"

>    executable = ["virtualenv", temp_virtualenv]
>    subprocess.Popen(args=executable)

Can we please call it with --no-site-packages?

>    run_script = ["source "+temp_virtualenv+"bin/activate", scriptArgs['--script-url']]
>    p = subprocess.Popen(args=executable)
>

It would be awesome if we could do source activate with python like that, however, we can't as far as I know.

Could you please look into this code and see how I add into sys.path the site-packages that gets created under the venv.
http://hg.mozilla.org/build/mozharness/file/default/mozharness/mozilla/testing/mozpool.py#l42

To understand this better, run this before entering a virtualenv:
python -c "import sys; import pprint; pprint.pprint(sys.path)"

and then once you enter the virtualenv:
 armenzg@armenzg-thinkpad:~/moz/tmp/temp$ virtualenv --no-site-packages venv
 New python executable in venv/bin/python
 Installing setuptools, pip...done.
 armenzg@armenzg-thinkpad:~/moz/tmp/temp$ source venv/bin/activate
 (venv)armenzg@armenzg-thinkpad:~/moz/tmp/temp$ python -c "import sys; import pprint; pprint.pprint(sys.path)"
 ['',
  '/home/armenzg/moz/tmp/temp/venv/lib/python2.7',
  '/home/armenzg/moz/tmp/temp/venv/lib/python2.7/plat-x86_64-linux-gnu',
  '/home/armenzg/moz/tmp/temp/venv/lib/python2.7/lib-tk',
  '/home/armenzg/moz/tmp/temp/venv/lib/python2.7/lib-old',
  '/home/armenzg/moz/tmp/temp/venv/lib/python2.7/lib-dynload',
  '/usr/lib/python2.7',
  '/usr/lib/python2.7/plat-x86_64-linux-gnu',
  '/usr/lib/python2.7/lib-tk',
  '/home/armenzg/moz/tmp/temp/venv/local/lib/python2.7/site-packages',
  '/home/armenzg/moz/tmp/temp/venv/lib/python2.7/site-packages']

I think what we mainly care about is appending to sys.path the absolute path of "venv/lib/python2.7/site-packages".
>    
>        
>
>

Comment 12

[Akshay Parelkar]

Attachment: Updated script

Hi Armen,

Here is my latest iteration. 

I tested it with the script and requirements you had attached previously.

Thanks,
Akshay

Comment 13

[Armen [:armenzg]]

Comment on attachment 8528117 [details] [diff] [review]
Updated script

Review of attachment 8528117 [details] [diff] [review]:
-----------------------------------------------------------------

Hello Akshay,
Thank you for addressing all the previous comments.
I run it locally and it is doing what I would expect it to. Good job!
This is in the right track. It seems you got the hang of this!

You're getting so close and I would like to figure how do you prefer working together from here on?
This script will take a bit of time until I can integrate it into bug 1066636; this means that we have time in here to make it more robust and flexible.
Would you like me to keep on thinking of making the script more robust?
Or would you prefer me to give a very limited scope and let you find other bugs to work on?

On another note, I want to take a turn on how we do things for this script.
I would like us to setup two different virtual environments.
One for this script in itself and another one for the script that we download (as we have been doing so far).
We can call it run_script_from_url_requirements.txt and for now we can start adding mozprocess to it (http://mozbase.readthedocs.org/en/latest/mozprocess.html)
A reason for this is that we will be able to separate the environment of this script versus the one that we download.
In the future, I would also like to define which python binary to use but that is way into the future (not for this bug).

Can you also please remove the directories that have been generated?
Can you also please add some checks to prevent executing the script if the current work dir has any files or directories?
I want to catch scripts that pollute our working environment and I don't want future runs to have anything left behind.

For the next patch, could you please run pylint against your script? You can see the output below.

armenzg@armenzg-thinkpad:~/repos/tools$ pylint buildfarm/utils/run_script_from_url.py | head -n 33
No config file found, using default configuration
************* Module run_script_from_url
C: 23, 0: Trailing whitespace (trailing-whitespace)
C: 29, 0: Trailing whitespace (trailing-whitespace)
C: 34, 0: Trailing whitespace (trailing-whitespace)
C: 37, 0: Trailing whitespace (trailing-whitespace)
C: 51, 0: Line too long (82/80) (line-too-long)
C: 55, 0: Trailing whitespace (trailing-whitespace)
C: 56, 0: Line too long (90/80) (line-too-long)
C: 61, 0: Trailing whitespace (trailing-whitespace)
C: 64, 0: Trailing whitespace (trailing-whitespace)
C: 65, 0: Final newline missing (missing-final-newline)
C:  1, 0: Missing module docstring (missing-docstring)
C:  8, 0: Missing function docstring (missing-docstring)
W: 11,27: Redefining name 'workdir' from outer scope (line 30) (redefined-outer-name)
C: 11, 0: Missing function docstring (missing-docstring)
C: 12, 4: Invalid variable name "currDir" (invalid-name)
C: 16, 4: Invalid variable name "f" (invalid-name)
C: 25, 4: Invalid constant name "parser" (invalid-name)
C: 28, 4: Invalid constant name "args" (invalid-name)
C: 30, 4: Invalid constant name "workdir" (invalid-name)
C: 36, 4: Invalid constant name "script_file" (invalid-name)
C: 40, 8: Invalid constant name "requirements_file" (invalid-name)
C: 42, 4: Invalid constant name "temp_virtualenv" (invalid-name)
C: 43, 4: Invalid constant name "executable" (invalid-name)
C: 44, 4: Invalid constant name "p" (invalid-name)
C: 45, 4: Invalid constant name "exec_done" (invalid-name)
C: 48, 4: Invalid constant name "cwd" (invalid-name)
C: 56, 8: Invalid constant name "install_requirements" (invalid-name)
C: 57, 8: Invalid constant name "p" (invalid-name)
C: 61, 4: Invalid constant name "run_script" (invalid-name)
C: 62, 4: Invalid constant name "p" (invalid-name)
W:  1, 0: Unused import argv (unused-import)

::: buildfarm/utils/run_script_from_url.py
@@ +13,5 @@
> +    os.chdir(workdir)
> +
> +    #Open file locally for saving
> +    f = urllib2.urlopen(url, timeout=timeout)
> +    local_file = open(get_filename_from_url(url), "w+")

You can probably skip the switching of directories by using in here the following:
> local_file = open(os.path.join(workdir, get_filename_from_url(url)), "w+")

@@ +24,5 @@
> +if __name__ == "__main__":
> +    parser = argparse.ArgumentParser()
> +    parser.add_argument("--script-url", dest="script_url", required=True)
> +    parser.add_argument("--requirements-url", dest="requirements_url")
> +    args = parser.parse_args()

To my own personal sadness, today I had to switch a script I wrote from using argparse to optparse even though is deprecated.
See http://hg.mozilla.org/build/tools/rev/b53b63b5e135

It seems that the python interpreter that runs on the Mac 10.6 machines does not have argparse and it is too cumbersome to fix the machines.

Could you please it switch over? Thanks!

@@ +28,5 @@
> +    args = parser.parse_args()
> +   
> +    workdir = "workdir"
> +    if not os.path.exists(workdir):
> +        os.mkdir(workdir)

Please make workdir an option to the script and make it mandatory.

With optparse you will have to check that the options where passed (instead of using required=True) like this:
> if not options.workdir:
>    parse.error("blah blah")

Same thing as seen on my fix today: http://hg.mozilla.org/build/tools/rev/b53b63b5e135

@@ +32,5 @@
> +        os.mkdir(workdir)
> +
> +        
> +    download_file_from_url(workdir=workdir, url=args.script_url)
> +    script_file = get_filename_from_url(args.script_url)

Make download_file_from_url return the absolute path to the file download.
That way we don't have to call get_filename_from_url.

@@ +34,5 @@
> +        
> +    download_file_from_url(workdir=workdir, url=args.script_url)
> +    script_file = get_filename_from_url(args.script_url)
> +    
> +    if args.requirements_url is not None:

Instead of doing "is not None| do this:
> if not args.requirements_url:

It's easier to read and consistent with other automation code style.

@@ +36,5 @@
> +    script_file = get_filename_from_url(args.script_url)
> +    
> +    if args.requirements_url is not None:
> +        download_file_from_url(workdir=workdir, url=args.requirements_url)
> +        requirements_file = get_filename_from_url(args.requirements_url)

Same here wrt to not calling get_filename_from_url.

@@ +39,5 @@
> +        download_file_from_url(workdir=workdir, url=args.requirements_url)
> +        requirements_file = get_filename_from_url(args.requirements_url)
> +
> +    temp_virtualenv = workdir + "/venv"
> +    executable = ["virtualenv", temp_virtualenv, "--no-site-packages"]

I found a cool way of setting up a virtualenv in this file:
http://hg.mozilla.org/build/talos/file/f3179facd945/INSTALL.py

Have a look at it and let's bring of some of that goodness in here :)

@@ +43,5 @@
> +    executable = ["virtualenv", temp_virtualenv, "--no-site-packages"]
> +    p = subprocess.Popen(args=executable)
> +    exec_done = True
> +    while p.poll() is None:
> +        pass

Let's start using mozprocess in here instead of subprocess directly.
This will ensure that we have similar behaviors for Windows as on UNIX systems.

See:
http://mozbase.readthedocs.org/en/latest/mozprocess.html

@@ +49,5 @@
> +    os.chdir("./workdir/venv/bin")
> +
> +    sys.path.append(os.path.abspath("./workdir/venv/lib/python2.7/site-packages"))
> +    execfile("activate_this.py", dict(__file__="activate_this.py"))
> +    os.chdir(cwd)

After all, it seems that we won't need this.
We can install directly inside of a virtualenv if we call the pip install inside of it.
This means that we won't need this after all.

See the following output, it shows that mozprocess is not installed inside of the venv OR outside:
armenzg@armenzg-thinkpad:~/repos/tools$ source workdir/venv/bin/activate
(venv)armenzg@armenzg-thinkpad:~/repos/tools$ pip freeze | grep mozprocess | wc -l
0
(venv)armenzg@armenzg-thinkpad:~/repos/tools$ deactivate
armenzg@armenzg-thinkpad:~/repos/tools$ workdir/venv/bin/pip install mozprocess
Downloading/unpacking mozprocess
  Downloading mozprocess-0.21.tar.gz
  Running setup.py (path:/home/armenzg/repos/tools/workdir/venv/build/mozprocess/setup.py) egg_info for package mozprocess
    
Requirement already satisfied (use --upgrade to upgrade): mozinfo in ./workdir/venv/lib/python2.7/site-packages (from mozprocess)
Requirement already satisfied (use --upgrade to upgrade): mozfile>=0.12 in ./workdir/venv/lib/python2.7/site-packages (from mozinfo->mozprocess)
Installing collected packages: mozprocess
  Running setup.py install for mozprocess
    
Successfully installed mozprocess
Cleaning up...
armenzg@armenzg-thinkpad:~/repos/tools$ pip freeze | grep mozprocess | wc -l
0
armenzg@armenzg-thinkpad:~/repos/tools$ source workdir/venv/bin/activate
(venv)armenzg@armenzg-thinkpad:~/repos/tools$ pip freeze | grep mozprocess | wc -l
1

@@ +55,5 @@
> +    if args.requirements_url is not None: 
> +        install_requirements = ["pip", "install", "-r", workdir + "/" + requirements_file]
> +        p = subprocess.Popen(args=install_requirements)
> +        while p.poll() is None:
> +            pass

Please use mozprocess in here.

Could you also call it like this?
workdir/venv/bin/pip install -r requirements.txt --download-cache ~/venv/cache --timeout 120 --no-index --find-links http://pypi.pub.build.mozilla.org/pub

Once this job runs in production machines we don't want to be hitting pypi.python.org since it can be down while the internal pypi should be up and running.
This way we reduce intermittent issues.

@@ +61,5 @@
> +    run_script = ["python", workdir + "/" + script_file]    
> +    p = subprocess.Popen(args=run_script)
> +    while p.poll() is None:
> +        pass    
> +    print "Done"
\ No newline at end of file

Use mozprocess.

Please don't print "Done".

Comment 14

[Akshay Parelkar]

Attachment: Updated script to use optparse and mozprocess

Hi Armen,

I updated the script to use mozprocess and optparse. 

I am not sure how to create a virtualenv for this script as doing so would have to assume that the user has mozprocess installed on their system.

I am also looking into setting up the alternative way of creating the virtualenv. I need to look in to how to get mozprocess to accept input as I didn't see how to do that in the API.

I will continue to work on other bugs as well, but feel free to comment on additional features for this script anytime and I will work on them.

Thanks,
Akshay

Comment 15

[Akshay Parelkar]

Attachment: updated requirements.txt

The Mozilla pypi server has mozhttpd v0.5, but not v0.7, so I created an updated requirements.txt.

Comment 16

[Sreekar]

Hi Armen,
I've gone through the comments on this page and it seemed like there has been no activity for about six months. Has the bug been resolved? If not, is it open for me to give it a shot?

Thanks,
Sreekar

Comment 17

[Armen [:armenzg]]

Hi Shreekar,
I'm going to be closing this bug as two things happened in the last 9 months which make this work not necessary anymore.

What type of contributions are you interested in? Python? Automation? Releng?

Feel free to join on IRC on the #ateam channel to chat synchronously:
https://wiki.mozilla.org/IRC

I suggest IRCCloud as a way to connect.