610605 - unit test mozharness script

Status: RESOLVED DUPLICATE of bug 650880

(Release Engineering :: General, defect, P5)

Description

[Aki Sasaki (not active)]

mustafaj in #seneca has this one.

Filing to track.

Comment 1

[Aki Sasaki (not active)]

http://mustafaredha.wordpress.com/2010/11/08/moztest-day-6-0-1-1/
http://iraq.proximity.on.ca/hg/mozharness/rev/5bcf1ff06beb

> from base.config import BaseConfig

I don't think you use this... and probably won't need it (BaseScript uses it).

> from base.log import SimpleFileLogger

Or this.

>         config_options = [[
>          ["--system",],
>          {"action": "store",
>           "dest": "system",
>           "type": "string",
>           "help": """Sets which system the tests are run on.

We'll probably use something like this, but renamed... arch, maybe?
We should detect what OS we're running on, as a kind of a dummy check
to prevent attempting to install a dmg on windows, for example.

This is fine for now, though.  No need to massively scope creep yet.

>         #first make the baseDirectory for the test environment    
>         os.makedirs(baseDirectory)

You want to use self.mkdir_p().
One of the main reasons behind mozharness is to log *everything*.
Also, it allows you to use --noop (in theory).

>         for directory in directoryStructure:
>             os.makedirs(baseDirectory + "/" + directory)

Here too.

>         centralRepository = "http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-"

This should go in config at some point.
Also, mozharness is currently in camelcase/underscore schizoid mode atm.
I'm going to guess it'll go 100% underscores at some point.

>         downloadPrefix = centralRepository + system + "/" + buildStamp + "/"

This works, but do you know about the sprintf style of

downloadPrefix = "%s%s/%s/" % (centralRepository, system, buildStamp)

?

>         print ""
>         print "all sources and tests will be downloaded from build #: "
>         print downloadPrefix

self.info() or self.debug() :)

>         """
>         this gets tricky
>         i'd like to do *tests.zip
>         and *tar.bz2
>         or something, because each operating system will have a different
>         file url
>         """
>          
>         firefoxSuffix = "firefox-4.0b8pre.en-US.linux-x86_64.tar.bz2"
>         testSuffix = "firefox-4.0b8pre.en-US.linux-x86_64.tests.zip"

This is complex, but it can be config-file driven.
You can specify --browser-version or something at least, to replace the 4.0b8pre?

>         urllib.urlretrieve(firefox)

self.downloadFile() ?

You're making progress, both in python-land and mozharness land...

Comment 2

[mustafa]

http://mustafaredha.wordpress.com/2010/11/15/moztest-day-7-0-1-2/
http://iraq.proximity.on.ca/hg/mozharness/file/a4c3a156cae1/scripts/moztest.py

There are the links to the new updates, info as well as actual script.
This is just dealing with some of the previous comments, will soon switch all variables to use under_Scores ... but can we make sure that's what we're doing, since script.py likes to play both sides of the variable naming game.

todo: extraction, testrun, default config, additional options

Comment 3

[Aki Sasaki (not active)]

The variable name thing isn't a big deal, just pointing that out.  I'm going to avoid pointing out small nits at this point; it's more important to get something working first, and we can revisit those later.

Looks like you're starting to use the harness properly, and yes, a config file with your testing defaults will help a) remove hardcodes from the script itself and b) avoid having to type in a ton of command line arguments.

You can look at the various json files in mozharness/configs/.  Basically, you're creating a dictionary that will populate self.config.

(I'm pretty sure) configuration goes like this:

1) optparse defaults
2) initial_config_file settings
3) --config-file <filename> settings # (defaults to ./localconfig.json)
4) command line arguments

Each step overrides any settings in the previous step, so if you specify  --work-dir foo and your config file has "work_dir": "bar" in it, your work_dir will be foo (command line takes precedence over config file).

You might as well put everything you want to test in your config file, however, and make sure anything you want to be able to override is exposed in an optparse option.

Comment 4

[mustafa]

http://iraq.proximity.on.ca/hg/mozharness/
http://mustafaredha.wordpress.com/

Quick update to extraction.  I'm going to do little releases in spurts.  I know you're busy, so please don't feel obligated to comment unless you need me to change how I'm doing whatever I'm doing.

Updated extraction of unit tests file into the tests folder and some thoughts on where to go next.  I'll have bzip extraction working and the first test run by Friday and then onto adding the additional functionality for 0.25 before I document everything for 0.3

Comment 5

[Armen [:armenzg]]

Hi Mustafa,
Aki will be gone for the next two weeks and I will be helping you out during that time. I have caught up this week with reading mozharness code.

I am looking at rev 0def376d8401.

What I would want us to focus on for the next two weeks is the following (kind of on this order):
- complete and correct run
- correct usage of mozharness
- polishing (naming)
- add more test suites (maybe not this semester)

Few comments before the review:
* It seems that you have few changes that were not expected (you might want to do "hg diff" before "hg ci"):
http://iraq.proximity.on.ca/hg/mozharness/rev/0def376d8401#l1.19
http://iraq.proximity.on.ca/hg/mozharness/rev/0def376d8401#l1.34
* I still don't understand the different levels of logging. I will let aki review on that when he gets back.

Functionality:
##############
* The first thing I want to be able to do is the script to run to completion and right now I wasn't able to.
* I run the following:
> python scripts/moztest.py --repo=mozilla-central --arch=linux --buildStamp=12
082451/
* It tried to download the 64-bits package when that is supposed to be found in "mozilla-central-linux64" instead:
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1290082451/firefox-4.0b8pre.en-US.linux-x86_64.tests.zip
* How do you handle optimized builds VS debug builds? (you can see them in ftp at tinderbox-builds directory)
* Which test suite are you targeting from the ones I emailed you?
* I can see that you are not yet using the parameters beside the "buildStamp"
* Let's only work for getting one test suite working completely

Beginning of review:
####################
When I run the script I get the following:
> Traceback (most recent call last):
>  File "scripts/moztest.py", line 143, in <module>
>    moztest.downloadEverything()
>  File "scripts/moztest.py", line 97, in downloadEverything
>    repo = self.config['repo']
>KeyError: 'repo'
I should receive something more meaningful (I see that aki is not handling it yet).
In BaseConfig this should be handled dynamically depending on the values in config_options (see comment in BaseConfig's code "* checkRequiredSettings or something -- run at init, assert that these settings are set.").
You can get an idea how to do checks in here (there might be better examples out in there):
http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/runreftest.py#88
This work can be pushed to later on and there will be few things to note:
* there can be parameters that are optional and others that are required
* since it would live in BaseConfig it would have to be general

You can pass a usage parameter to BaseConfig through BaseScript. Give it a shot.
> print ""
> #usage example: ./moztest.py --repo=mozilla-central --arch=linux64 
> --buildStamp=1289732796/

>    - system checker to ensure correct arch is running
Later.
>    - variable names to underscores
As aki says don't get too worried right now. Let's get things working first.

>    - figure out how to download from latest build
It would be nice but for now it will just make your life harder as you would have to strip the unix timestamps in http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/$branch-$platform
I wish we have a file knowing this information.

>    - the user shouldnt specify firefox versiexiton, as if they're specifying a
>        timestamp then that is a given
I don't get it.
>    - Include download progress bar?
No need to worry for now.   
    
>    def setEnvironment(self):
>        """SetEnvrionment will create a directory structure for the tests"""

In general, each "action" (most function definitions are to be considered "actions") should take care of itself completely ignorant of any type of setup type action. Look at aki's script and see how after __init__ the function run() gets called and immediately calls all different actions. You do what run() does on __main__. Just move it inside of a run() function.
 
Please use the following style for setting up each "action" where needed:
>    def pull(self):
>        c = self.config
>        abs_work_dir = os.path.join(c['base_work_dir'],
>                                    c['work_dir'])

Don't use your suggested structure. We can't redefine the structure another time. For now, let's be as much as copycats as we can of what we run in production.    
>        directoryStructure = ("tars", "sources", "tests")
>        
>        for directory in directoryStructure:
>            self.mkdir_p("%s/%s" % (self.baseDirectory, directory))
Try to follow the following directory structure for now:
> bin      
> certs    
> symbols
> firefox-4.0b8pre.en-US.linux-i686.tar.bz2    
> firefox-4.0b8pre.en-US.linux-i686.tests.zip
> firefox  
> reftest
The reftest directory comes from choosing the test suite called "reftests". Depends on which test suite you are working this would be different.
Don't worry about "symbols" right now.
Remember you can use a log from tbpl.mozilla.org to see which files get unzipped for each test suite. Let's focus in only one of the test suites for this semester.
   
>    def downloadEverything(self):
>        centralRepository = "http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/"
Please keep this constant on a config file under "configs/"
Change the name of the action to "downloadFiles".

>        repo = self.config['repo']
>        arch = self.config['arch']
>        buildStamp = self.config['buildStamp']
>        downloadPrefix = "%s%s-%s/%s/" % (centralRepository, repo, arch, buildStamp)
You can do this:
> downloadPrefix = "%s%s-%s/%s/" % (centralRepository, \
>                                   self.config['repo'], \
>                                   self.config['arch'], \
>                                   self.config['buildStamp'])


>        #the suffixes will be incorporated into config file.  its a little more complicated than that.
We probably won't be doing what you say
>        firefoxSuffix = "firefox-4.0b8pre.en-US.linux-x86_64.tar.bz2"
>       testSuffix = "firefox-4.0b8pre.en-US.linux-x86_64.tests.zip"
Use wildcards "firefox-*.tests.zip" and "firefox-*.zip".
This would avoid you to have to know the version and or the naming scheme.
def downloadFila might need to be fixed to make it work. If you have trouble get back to me.
        
>        #i'm not sure how to reference the variable from the setEnv function
>        self.chdir("%s/%s" % (self.baseDirectory, "tars"))
Drop this structure as previously mentioned.
>        self.downloadFile(downloadPrefix + firefoxSuffix)
>        self.downloadFile(downloadPrefix + testSuffix)
Use "%s/%s" % (param1, param2) style.
Use "url" instead of "downloadPrefix" and "binary_filename" and "test_filename" for the others. You might want to might want them to be self.binary_filename and self.test_filename so you can use it on "extractStuff".
If downloadFile fails we should abort. Look what aki uses somewhere else:
>                if not self.downloadFile(deb_url, deb_name):
>                    self.addSummary("Can't download %s; skipping %s on %s" % \
>                                    (deb_url, locale, platform), level="error")
>                    self.failures.append('%s_%s' % (platform, locale))
>                    continue
     
>        #return to baseDirectory
>        #self.chdir(baseDirectory)
No need. Each action should not count on the previous to set things back.

>    def extractStuff(self):
Should be something more defined like "extractFiles".

We want to add a function in BaseScript to extract a single file. 
Look in http://hg.mozilla.org/build/buildbotcustom/file/tip/steps/misc.py#l376
to get an idea on how to handle this.

We still need an action called "runTest()" to actually run the test. Let's start with only one test suite.

This script is laying down the foundation of what will be the last product. You are doing well. Keep it up and yes, go ahead and get back to me in smaller iterations so my reviews won't take hours (now I have a base understanding so it should be faster) and I can get back to you faster and more times.

Let's hope we get something easy for aki to review ;)

Comment 6

[mustafa]

Hi Armen, Aki,

I mistakenly erased my main moztest.py, the most recent version.  I had a Nov 8th version on hand worked on that.

Armen, if you have the Nov 18th, can you e-mail it to me?  It has the extraction methods I wrote.

The repo itself is down, so I copy pasted the code into here directly at the bottom.  I will get a new hg repo up asap.

Here's what I've done in order from Armen's last comment:

> python scripts/moztest.py --repo=mozilla-central --arch=linux --buildStamp=12
082451/
* It tried to download the 64-bits package when that is supposed to be found in
"mozilla-central-linux64" instead:
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1290082451/firefox-4.0b8pre.en-US.linux-x86_64.tests.zip










#!/usr/bin/env python

import os
import sys
import urllib

#append the lib path
sys.path.insert(1, os.path.dirname(sys.path[0]))

from mozharness.base.script import BaseScript

print ""
#usage example: ./moztest.py --repo=mozilla-central --arch=linux64 --buildStamp=1289732796/

"""TODO:
    """

#moztest class
class moztest(BaseScript):
    def __init__(self, **kwargs):
        """Moztest will create a test environment (directories) to run
        firefox build unit tests, download the unit tests and firefox build
        to this environment, and begin running tests"""
        
        config_options = [[
         ["--arch",],
         {"action": "store",
          "dest": "arch",
          "type": "string",
          "help": """Sets which arch the tests are run on... ex: "linux" or "linux64" """
         }
        ],[
         ["--buildStamp",],
         {"action": "store",
          "dest": "buildStamp",
          "type": "string",
          "help": "Timestamp of the build against which you are running tests... ex: 13054689"
         }
        ],[
         ["--repo",],
         {"action": "store",
          "dest": "repo",
          "type": "string",
          "help": "Specifies the repository being used, eg: mozilla-central-linux, maple-macosx"
         }
        ]]
        
        BaseScript.__init__(self, config_options=config_options)
        
    def set_environment(self):
        """SetEnvrionment will create a directory structure for the tests"""
        
        #set full path for test env
        #Create base_directory for the test environment    
        #then create all child dirs
        #directory names stored in tuple "directory_structure"
        
        base_directory = os.path.join(self.config['base_work_dir'], \
        self.config['work_dir'])
        self.info("mozTest will run out of: %s" % base_directory)
        self.mkdir_p(base_directory)
       
        directory_structure = ("tars", "sources", "tests")
        
        for directory in directory_structure:
            self.mkdir_p("%s/%s" % (base_directory, directory))
    
    def download_files(self):
       
        central_repository = "http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/"
        
        url = "%s%s-%s/%s/" % (central_repository, \
		self.config['repo'], \
		self.config['arch'], \
		self.config['buildStamp'])

        binary_filename = "firefox-4.0b9pre.en-US.linux-x86_64.tar.bz2"
        test_filename = "firefox-4.0b9pre.en-US.linux-x86_64.tests.zip"  
        
        self.info("all sources and tests will be downloaded from build: %s" % url)  
       
	self.chdir("%s/%s/%s" % (self.config['base_work_dir'], self.config['work_dir'], "tars"))
	self.download_file("%s/%s" % (url, binary_filename))
	self.download_file("%s/%s" % (url, test_filename))

    def run(self):
        moztest.set_environment()
        moztest.download_files()       

 
if __name__ == '__main__':
	moztest=moztest()
	moztest.run();

Comment 7

[mustafa]

okay. I clicked save by mistake, so lets move on...

1.  Here is the usage: 
./moztest.py --repo=mozilla-central --arch=linux64 --buildStamp=1289732796/

You can't just specify linux, you have to specify 64 if you want 64 bit, or just linux for 32...

2. Move everything to a run() function.
Done

...Also went ahead and updated to use the new mozharness directory instead of lib

3. Setting up actions...c = self.config
- Do we really want to do that even if the method has a few different actions in itself?  for example 'download_files' has self.config, self.download, self.info, self.chdir ....should I do this for each, or only when the actions in a method are repeatedly used

4. I'll update the directory structure on the next update

5. Choosing the tests...
- sofar I've been doing a full download of all tests.  Once I get the extraction method nailed down, I'll change it to specify a test in the parameters, like test=runtest or test=all_tests...

6. json config file.  The majority of the variables used should go into a default json config file, including central_repository and a few others.  sofar I've just been relying on aki's localconfig.json

7. Changed the name of all variables to underscores _ syntax

8. Cleaned up download_files and its variable names

9. Using asterisks for Firefox and test url: I need to talk to you about this, no idea how to do it so that the asterisk is translated into the full url over http like it expands in shell.

-----done.

Next thing I want to do is get that extraction method and merge it back in, and get the extraction going. 

runtest will be the first test suite used, and that will be the easy part.

I'll need to clarify the few things above, and then fix the directory structure and have the test run.  

I won't focus on user friendly error responses and environment checker until I get the former stuff done.  

My semester is officially done, but like I told aki, I'd like to keep working on this over the next little while and see it through if you guys don't mind.

Comment 8

[Armen [:armenzg]]

Attachment: moztest.py - 357:5bcf1ff06beb - Nov. 8th

Not sure if this is the moztest.py you are looking for.

Comment 9

[Armen [:armenzg]]

as per IRC with mustafaj I am putting this back into the future pool as his hands are full right now.

Comment 10

[Aki Sasaki (not active)]

Duping forward for cleanliness.