707218 - write a script to create a talos.zip file with the dependencies from mozbase

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[Jeff Hammel]

Talos currently has two upstream dependencies from mozbase:

- mozdevice: devicemanagerADB.py, devicemanager.py, devicemanagerSUT.py
- mozhttpd.py, which in the mozbase directory now has a directory structure: https://github.com/mozilla/mozbase/tree/master/mozhttpd/mozhttpd

We also want to include mozinfo.py soon: https://bugzilla.mozilla.org/show_bug.cgi?id=699129

None of these three packages have any dependencies and may be used standalone

Currently, we version these separately for talos and take fixes as needed.  Instead we should use the upstream dependencies.  For the python package (e.g. `python setup.py develop`) case, we can add these to setup.py and have setuptools pull them down.  

For the talos.zip case, we can write a script that puts these files in the the talos directory and not have to change how talos imports things.  This is a good interim solution until we have a infrastructure pypi server that we can use to install these dependencies for buildbot.  The script can just point at these few files, put them in the appropriate place in the talos directory and zip it appropriately (and cleanup).

Note that this only works because none of these dependencies have dependencies.  For e.g. mozprocess and mozrunner, we will have to be more clever or be transitioned to a package index infrastructure.  But its at least something to build on.

Comment 1

[Jeff Hammel]

Note that we should update the instructions on how to deploy Talos once this script is written

Comment 2

[William Lachance (:wlach)]

Also should update (or consolidate):

https://wiki.mozilla.org/Mobile/Fennec/Android#talos
https://wiki.mozilla.org/Buildbot/Talos#Running_locally_-_Source_Code
https://wiki.mozilla.org/Auto-tools/Tools/Talos#How_To_Run_Automation_Style_.28aka_The_Way_Of_Pain.29

Comment 3

[Jeff Hammel]

pushed mozdevice to pypi as we'll need this to test/use the setup.py route: http://pypi.python.org/pypi/mozdevice

Comment 4

[Jeff Hammel]

Attachment: script to create talos zips and import changes and additions to .hgignore file

I've tested the resulting talos.zip with tsvg using the instructions at https://wiki.mozilla.org/Mobile/Fennec/Android#talos and things work fine.  Preliminary testing on staging (minus the down fedora slave) seem fine, though I'll run it again as I've made changes since my initial changes. I will also test the python setup.py develop method using ts and tsvg.  

Are there other tests that should be run before I put this up for review?  I will also work on updating deployment and setup instructions for this.

mozinfo.py isn't currently used.  I put it there such that Bug 699129 is more easily actionable.  However, I can remove if not desired at this time.

Comment 5

[Jeff Hammel]

I've tested this locally in a clean virtualenv (although with some things, like pageloader, already in talos).  python setup.py works fine and I was able to run ts.  Is there anything else I should test with this code path?

Comment 6

[Jeff Hammel]

Comment on attachment 579517 [details] [diff] [review]
script to create talos zips and import changes and additions to .hgignore file

putting up for review

Comment 7

[Mike Taylor [:bear]]

Comment on attachment 579517 [details] [diff] [review]
script to create talos zips and import changes and additions to .hgignore file

i'll have to make changes to the foopy environment (we symlink into the talos directory structure) and also to our helper tools - none of this is worthy of blocking this fine refactoring IMO

Comment 8

[Jeff Hammel]

I have created https://developer.mozilla.org/en/Python/Virtualenv , https://developer.mozilla.org/en/Python#section_4, and updated https://wiki.mozilla.org/Buildbot/Talos#Running_locally_-_Source_Code so that the documentation comes closer to reflecting reality.  More updates will need to be done wrt how talos.zip now works on landing.

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 579517 [details] [diff] [review]
script to create talos zips and import changes and additions to .hgignore file

Review of attachment 579517 [details] [diff] [review]:
-----------------------------------------------------------------

Just some general questions and nits, otherwise this looks great.

::: create_talos_zip.py
@@ +41,5 @@
> +        except:
> +            print url
> +            raise
> +        f = file(filename, 'w')
> +        f.write(contents)

what if the file exists and is locked?  can we put some try/catch around here?

@@ +73,5 @@
> +    # make the talos.zip file
> +    zip = zipfile.ZipFile('talos.zip', mode='w')
> +    talosdir = os.path.abspath(os.path.join(here, 'talos'))
> +    for dirpath, dirnames, filenames in os.walk(talosdir):
> +        filenames = [i for i in filenames if not i.endswith('.pyc')]

.pyc is in .hgignore, why filter it here?

Comment 10

[Jeff Hammel]

(In reply to Joel Maher (:jmaher) from comment #9)
> Comment on attachment 579517 [details] [diff] [review]
> script to create talos zips and import changes and additions to .hgignore
> file
> 
> Review of attachment 579517 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Just some general questions and nits, otherwise this looks great.
> 
> ::: create_talos_zip.py
> @@ +41,5 @@
> > +        except:
> > +            print url
> > +            raise
> > +        f = file(filename, 'w')
> > +        f.write(contents)

> what if the file exists and is locked?  can we put some try/catch around
> here?

Sure, I can add that.
 
> @@ +73,5 @@
> > +    # make the talos.zip file
> > +    zip = zipfile.ZipFile('talos.zip', mode='w')
> > +    talosdir = os.path.abspath(os.path.join(here, 'talos'))
> > +    for dirpath, dirnames, filenames in os.walk(talosdir):
> > +        filenames = [i for i in filenames if not i.endswith('.pyc')]
 
> .pyc is in .hgignore, why filter it here?

Oops, that was for an earlier incarnation where I didn't filter through .hgrc.  I'll remove.

Should I retest with these changes?

Comment 11

[Jeff Hammel]

(In reply to Jeff Hammel [:jhammel] from comment #10)
> (In reply to Joel Maher (:jmaher) from comment #9)
> > Comment on attachment 579517 [details] [diff] [review]
> > script to create talos zips and import changes and additions to .hgignore
> > file
> > 
> > Review of attachment 579517 [details] [diff] [review]:
> > -----------------------------------------------------------------
> > 
> > Just some general questions and nits, otherwise this looks great.
> > 
> > ::: create_talos_zip.py
> > @@ +41,5 @@
> > > +        except:
> > > +            print url
> > > +            raise
> > > +        f = file(filename, 'w')
> > > +        f.write(contents)
> 
> > what if the file exists and is locked?  can we put some try/catch around
> > here?
> 
> Sure, I can add that.

Actually, wait;  I'm not sure what is being asked here.  If the file is locked and you try to write to it, shouldn't it just err out (as it would as stands) vs trying to raise?

Comment 12

[Jeff Hammel]

Also, I don't currently create a pageloader.xpi inside of page_load_test.  This leads to the following:

(tmp)│python PerfConfigurator.py -e `which firefox` -a tsvg
(tmp)│ls
20111208_1416_config.yml  ffprocess_win32.py    places_generated_med
addon.config              ffprofile_unix.py     places_generated_min
base_profile              ffprofile_win32.py    post_file.py
bcontroller.py            ffsetup.py            remote.config
breakpad                  getInfo.html          remotePerfConfigurator.py
cmanager_linux.py         __init__.py           run_tests.py
cmanager_mac.py           mobile.config         sample.config
cmanager_remote.py        mobile_profile        scripts
cmanager_win32.py         mozdevice             startup_test
cycles.config             mozhttpd.py           ttest.py
ffprocess_linux.py        mozinfo.py            utils.py
ffprocess_mac.py          page_load_test        xperf.config
ffprocess.py              PerfConfigurator.py   xtalos
ffprocess_remote.py       places_generated_max
(tmp)│python run_tests.py 
(tmp)│python run_tests.py 20111208_1416_config.yml 
RETURN:s: qm-pxp01
RETURN:id:20111208031134
RETURN:<a href = "http://hg.mozilla.org/mozilla-central/rev/6785d3003414">rev:6785d3003414</a>
qm-pxp01: 
		Started Thu, 08 Dec 2011 14:16:35
Running test tsvg: 
		Started Thu, 08 Dec 2011 14:16:35
Failed tsvg: 
		Stopped Thu, 08 Dec 2011 14:16:35
FAIL: Busted: tsvg
FAIL: Found processes still running: firefox, plugin-container. Please close them before running talos.
Traceback (most recent call last):
  File "run_tests.py", line 591, in <module>
    main()
  File "run_tests.py", line 588, in main
    test_file(arg, options.screen, options.amo)
  File "run_tests.py", line 534, in test_file
    raise e
utils.talosError: 'Found processes still running: firefox, plugin-container. Please close them before running talos.'
(tmp)│python run_tests.py 20111208_1416_config.yml 
RETURN:s: qm-pxp01
RETURN:id:20111208031134
RETURN:<a href = "http://hg.mozilla.org/mozilla-central/rev/6785d3003414">rev:6785d3003414</a>
qm-pxp01: 
		Started Thu, 08 Dec 2011 14:16:49
Running test tsvg: 
		Started Thu, 08 Dec 2011 14:16:49
Traceback (most recent call last):
  File "run_tests.py", line 591, in <module>
    main()
  File "run_tests.py", line 588, in main
    test_file(arg, options.screen, options.amo)
  File "run_tests.py", line 520, in test_file
    browser_dump, counter_dump, print_format = mytest.runTest(browser_config, test)
  File "/home/jhammel/tmp/talos/ttest.py", line 238, in runTest
    browser_config['bundles'][bundlename])
  File "/home/jhammel/tmp/talos/ffsetup.py", line 276, in InstallBundleInBrowser
    zip_extractall(zipfile.ZipFile(bundle_path), destpath)
  File "/usr/lib/python2.7/zipfile.py", line 697, in __init__
    self.fp = open(file, modeDict[mode])
IOError: [Errno 2] No such file or directory: 'page_load_test/pageloader.xpi'

I'm going to assume we want this here? And add it???  Not sure what is desired here.

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

pageloader is not included in talos.zip proper, so please don't add it in.  

Regarding a file writing error, lets just catch the error and exit cleanly.

Comment 14

[Jeff Hammel]

(In reply to Joel Maher (:jmaher) from comment #13)
> pageloader is not included in talos.zip proper, so please don't add it in.  

Okay, thanks.

> Regarding a file writing error, lets just catch the error and exit cleanly.

From http://k0s.org/mozilla/hg/talos-patches/file/tip/create-zips-zip#l72 :

        try:
            f = file(filename, 'w')
            f.write(contents)
            f.close()
        except:
            print filename # the file may not be writable
            raise

Comment 15

[Jeff Hammel]

:bear, can you check to see if http://k0s.org/mozilla/talos/bug-707218/talos.zip works with the build infrastructure? It is generated with the latest version of the patch and checked locally.  If there are any problems please let me know.  Otherwise, with your go-ahead, I'd like to check in.

Comment 16

[Jeff Hammel]

wrt comment 15, now moved to 
http://people.mozilla.com/~jhammel/talos.zip

Comment 17

[Jeff Hammel]

:bear, any ETA for this?

Comment 18

[Jeff Hammel]

see also https://bugzilla.mozilla.org/show_bug.cgi?id=675117#c32

Comment 19

[Mike Taylor [:bear]]

I ran this thru a number of staging runs on the tegra side and found no problems

Comment 20

[Jeff Hammel]

So this will need to be unbitrotted and corrected to include mozinfo and kill the sys.path addition in run_tests.py.  In fact, mozinfo.py will be mirrored twice for talos.zip

Comment 21

[Jeff Hammel]

The following can be patched successfully:

│patch -p1 < ~/create-talos-zip.diff 
patching file .hgignore
patching file create_talos_zip.py
patching file talos/PerfConfigurator.py
patching file talos/devicemanager.py
patching file talos/devicemanagerADB.py
patching file talos/devicemanagerSUT.py
patching file talos/ffprocess_remote.py
patching file talos/mozhttpd.py
patching file talos/run_tests.py
Hunk #1 succeeded at 464 (offset 10 lines).

I will unbitrot the others by hand

Comment 22

[Jeff Hammel]

Attachment: unbitrot and update patch

This unbitrots the patch and updates it. Not sure what all testing we want to do.  I am staging it now.  Not sure which local tests I should run on my tablet?  Not sure if :bear wants another round of tegra testing?  Please comment.  test_install.py still works...that is something.  I am happy to add more tests if there is a need.  The staged zip is at http://k0s.org/mozilla/talos/tip/zips/talos.zip

Comment 23

[Mike Taylor [:bear]]

Comment on attachment 584664 [details] [diff] [review]
unbitrot and update patch

Review of attachment 584664 [details] [diff] [review]:
-----------------------------------------------------------------

it all looks good but I'm really wanting to see this run thru the try environment as that is a much better indicator than my tegra environment 

let me know if you need an assist on that - I can convince someone in releng to lend some brain cells to this

Comment 24

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 584664 [details] [diff] [review]
unbitrot and update patch

Review of attachment 584664 [details] [diff] [review]:
-----------------------------------------------------------------

overall call it a r+

::: .hgignore
@@ +10,5 @@
> +mozhttpd.py
> +talos.zip
> +mobile_tp4
> +remoteapp.ini
> +svg.manifest.develop
\ No newline at end of file

can this be *.develop?

Comment 25

[Jeff Hammel]

(In reply to Joel Maher (:jmaher) from comment #24)
> Comment on attachment 584664 [details] [diff] [review]
> unbitrot and update patch
> 
> Review of attachment 584664 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> overall call it a r+
> 
> ::: .hgignore
> @@ +10,5 @@
> > +mozhttpd.py
> > +talos.zip
> > +mobile_tp4
> > +remoteapp.ini
> > +svg.manifest.develop
> \ No newline at end of file
> 
> can this be *.develop?

WFM

Comment 26

[Jeff Hammel]

Staging with a corrected .hgignore file (not that that is important, but I cleaned it up for checkin. See http://k0s.org/mozilla/hg/talos-patches/rev/5287c5f15d5a

Comment 27

[Jeff Hammel]

All passing on staging and local tests.  Things seem to pass on try, minus a seemingly unrelated failure (https://tbpl.mozilla.org/php/getParsedLog.php?id=8218022&tree=Try&full=1) . Pushed: http://hg.mozilla.org/build/talos/rev/8031d427ef14

I will add instructions to https://wiki.mozilla.org/Buildbot/Talos wrt instructions on how to generate a talos.zip file now.

Comment 28

[Jeff Hammel]

Note that currently we pull the latest files from mozbase master: http://hg.mozilla.org/build/talos/file/8031d427ef14/create_talos_zip.py#l18 . This is fine as long as we stay in sync with them, but we should be double-careful that this is the case, and/or we should think about pegging to a version.

Comment 29

[Mozilla RelEng Bot]

Try run for b58b094081b0 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=b58b094081b0
Results (out of 79 total builds):
    success: 78
    failure: 1
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-b58b094081b0

Comment 30

[Jeff Hammel]

So I did not hitherto realize that the default for python's zipfile.ZipFile object is to not compress: http://docs.python.org/library/zipfile.html#zipfile-objects. This causes the talos.zip to be about 19M versus about 5.5M . I will post a follow-up patch

Comment 31

[Jeff Hammel]

Attachment: compress the zipfile

This restores the talos.zip size to 5.9M

Comment 32

[Jeff Hammel]

(In reply to Jeff Hammel [:jhammel] from comment #31)
> Created attachment 584873 [details] [diff] [review]
> compress the zipfile
> 
> This restores the talos.zip size to 5.9M

testing on try: https://tbpl.mozilla.org/?tree=Try&rev=1c11ce7c9f25

Comment 33

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 584873 [details] [diff] [review]
compress the zipfile

good find!

Comment 34

[Mozilla RelEng Bot]

Try run for 1c11ce7c9f25 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=1c11ce7c9f25
Results (out of 79 total builds):
    success: 77
    failure: 2
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-1c11ce7c9f25

Comment 35

[Jeff Hammel]

pan opt again burning on Android xul (timeout exceeded) and failure in tzoom:

getting files in '/mnt/sdcard/tests/profile/minidumps/'
Failed tzoom: 
		Stopped Thu, 29 Dec 2011 17:11:12
FAIL: Busted: tzoom
FAIL: unrecognized output format
Traceback (most recent call last):
  File "run_tests.py", line 603, in <module>
    main()
  File "run_tests.py", line 600, in main
    test_file(arg, options.screen, options.amo)
  File "run_tests.py", line 546, in test_file
    raise e
utils.talosError: 'unrecognized output format'
program finished with exit code 1

Not sure what that is about. From https://tbpl.mozilla.org/php/getParsedLog.php?id=8223480&tree=Try

I highly doubt its related to this fix.

Okay to push?

Comment 36

[Joel Maher ( :jmaher ) (UTC -8)]

yes, tpan is a common failure these days :(

Comment 37

[Jeff Hammel]

pushed follow up: http://hg.mozilla.org/build/talos/rev/d622555b654c