Closed Bug 650887 Opened 12 years ago Closed 11 years ago

desktop talos runner in mozharness

Categories

(Release Engineering :: Applications: MozharnessCore, defect, P1)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: aki, Assigned: aki)

References

Details

(Whiteboard: [mozharness][talos][mozharness+talos])

Attachments

(1 file, 3 obsolete files)

This should get complexity out of buildbotcustom and allow the a-team,
developers, and releng to all use the same scripts.

By "port" I mean wrap run_tests.py in mozharness, for now.
Blocks: 650890
Summary: port desktop talos to mozharness → desktop talos runner in mozharness
Blocks: 651974
Assignee: nobody → aki
No longer blocks: 651974
I'm not sure if this makes sense to do until bug 688604 is landed so that we can use the mozhttpd server.  If not, there is no good way of using a webserver that doesn't require root privileges since, at least on linux, you're not allowed to open port 80
a) i have a talos zipfile that lets me use --develop without being blocked
b) we can install mozhttpd directly
c) we can start a webserver and test

I don't think this blocks.  At most it blocks the ability to start a webserver automatically without a patched talos; there's plenty of other work that can happen here by using one of the workarounds above.
So it seems like the first step is to replace the existing buildbotcustom logic with a mozharness script.  Is this an accurate assessment?
Correct. They'll probably run in parallel until we can prove desktop talosrunner is stable, but the goal is to replace it. A secondary goal is to allow a user to run the same script locally.
Attached file WIP (obsolete) —
So I had incorrectly perceived the goal here to be getting something working for developers vs something to replace the existing buildbotcustom setup.  Here's what I have now, which should mostly work except for the webserver part, just for posterity.
Attached file WIP 2 (obsolete) —
So this is another WIP slated towards buildbot as a consumer.  It errs our at

10:27:23     INFO -  utils.talosError: 'Found processes still running: firefox. Please close them before running talos.'

I don't have any firefox processes running so something is awry
(In reply to Jeff Hammel [:jhammel] from comment #6)
> Created attachment 572882 [details]
> WIP 2
> 
> So this is another WIP slated towards buildbot as a consumer.  It errs our at
> 
> 10:27:23     INFO -  utils.talosError: 'Found processes still running:
> firefox. Please close them before running talos.'
> 
> I don't have any firefox processes running so something is awry

Ah, this is bug 700722.  This should probably be fixed before this script is usable.
Depends on: 700722
Thanks, Jeff!

Looks like installing talos installs pyyaml by default, but I'm a bit worried about production, which can't get to pypi.  Listing pyyaml explicitly might help us there.

For user-friendliness making this script and the other mozharness talos- and unittest- scripts "feel" the same in terms of actions and config naming would be good, but I'm fine leaving that for later.

It does sound like --develop and the process issue will make running this not so user-friendly, so getting those fixed properly would be good. If we want/need to roll something out before those are resolved, I think we can work around them.
Once https://github.com/escapewindow/mozharness/blob/talosrunner/scripts/device_talosrunner.py lands we should unify the options a bit, maybe have a common base class
Attached file WIP (obsolete) —
Attachment #572583 - Attachment is obsolete: true
Attachment #572882 - Attachment is obsolete: true
This is currently blocked by bug 701573.  The virtualenv-ed talos does not shell out to the correct python process, which is the fault of Talos.  I will fix that first.
Depends on: 701573
(In reply to Jeff Hammel [:jhammel] from comment #11)
> This is currently blocked by bug 701573.  The virtualenv-ed talos does not
> shell out to the correct python process, which is the fault of Talos.  I
> will fix that first.

I'm working around this by setting PATH in the run_tests.py call (where I need to change it to use os.pathsep).
I'd love a real fix.
https://github.com/escapewindow/mozharness/blob/talosrunner/scripts/device_talosrunner.py#L340
Attachment #573691 - Attachment is obsolete: true
Attachment #573963 - Flags: review?(aki)
Comment on attachment 573963 [details] [diff] [review]
basic mozharness for talos

>+    def _fetch(self, url, dest):
>+        """fetch a static resource from the web"""
>+        # TODO: could be upstreamed
>+        f = file(dest, 'w')
>+        resource = urllib2.urlopen(url)
>+        f.write(resource.read())
>+        resource.close()
>+        f.close()

self.download_file() ?
http://hg.mozilla.org/build/mozharness/file/cbcc9a5d5549/mozharness/base/script.py#l147

>+    def _set_talos_dir(self):
>+        """XXX this is required as talos must be run out of its directory"""
>+        python = self.query_python_path()
>+        self.talos_dir = self.get_output_from_command([python, '-c', 'import os, talos; print os.path.dirname(os.path.abspath(talos.__file__))'])
>+        if not self.talos_dir:
>+            self.fatal("Talos directory could not be found")
>+        self.info('Talos directory: %s' % self.talos_dir)

I'm more of a "check talos out from hg and know where the path is" type person rather than "install talos from pip and figure out where it's installed" type person.

However, if this works, I'm not opposed.

>+    def install_pageloader(self):
>+        """install pageloader"""
>+        if not hasattr(self, 'talos_dir'):
>+            self._set_talos_dir()
>+        dest = os.path.join(self.talos_dir, 'page_load_test', 'pageloader.xpi')
>+        self._fetch(self.config['pageloader_url'], dest)

I solved this by checking out hg.m.o/build/pageloader into the talos directory.  Tmtowtdi, but this may be part of what we want to standardize at some point.

https://github.com/escapewindow/mozharness/blob/talosrunner/configs/users/aki/tablet1.py#L42
https://github.com/escapewindow/mozharness/blob/talosrunner/scripts/device_talosrunner.py#L217

>+    def generate_config(self):
>+        """generate talos configuration"""

There may be more options we want here, but this is a good start.

There may be additional work that needs doing before this script is both fully featured for everything we do in production, and we have parity between the two talos scripts.  But other than possibly s,_fetch,download_file, I don't see anything glaringly wrong here.

Thanks for the script!
Attachment #573963 - Flags: review?(aki) → review+
(In reply to Aki Sasaki [:aki] from comment #14)
> Comment on attachment 573963 [details] [diff] [review] [diff] [details] [review]
> basic mozharness for talos
> 
> >+    def _fetch(self, url, dest):
> >+        """fetch a static resource from the web"""
> >+        # TODO: could be upstreamed
> >+        f = file(dest, 'w')
> >+        resource = urllib2.urlopen(url)
> >+        f.write(resource.read())
> >+        resource.close()
> >+        f.close()
> 
> self.download_file() ?
> http://hg.mozilla.org/build/mozharness/file/cbcc9a5d5549/mozharness/base/
> script.py#l147

Ah, I missed that somehow.  Thanks.  I'll change before I check in.

> >+    def _set_talos_dir(self):
> >+        """XXX this is required as talos must be run out of its directory"""
> >+        python = self.query_python_path()
> >+        self.talos_dir = self.get_output_from_command([python, '-c', 'import os, talos; print os.path.dirname(os.path.abspath(talos.__file__))'])
> >+        if not self.talos_dir:
> >+            self.fatal("Talos directory could not be found")
> >+        self.info('Talos directory: %s' % self.talos_dir)
> 
> I'm more of a "check talos out from hg and know where the path is" type
> person rather than "install talos from pip and figure out where it's
> installed" type person.

I would like to see a method that is "check out from hg/download from tarball and install via `setup.py`.  Would that be worth ticketing? I'm more inclined to use a direct .tar.gz/.zip download versus hg to avoid another dependency but agree that the way I've done things is a bit magic.  

Also note that there are also upstream talos issues here.  Talos should not require to be run from a specific directory, but it does.  Talos should also separate the software pieces from the data pieces but it does not.  We'll have to fix these too, probably this quarter.  I just haven't gotten around to figuring out what all needs to be done there.

> However, if this works, I'm not opposed.
> 
> >+    def install_pageloader(self):
> >+        """install pageloader"""
> >+        if not hasattr(self, 'talos_dir'):
> >+            self._set_talos_dir()
> >+        dest = os.path.join(self.talos_dir, 'page_load_test', 'pageloader.xpi')
> >+        self._fetch(self.config['pageloader_url'], dest)
> 
> I solved this by checking out hg.m.o/build/pageloader into the talos
> directory.  Tmtowtdi, but this may be part of what we want to standardize at
> some point.
> 
> https://github.com/escapewindow/mozharness/blob/talosrunner/configs/users/
> aki/tablet1.py#L42
> https://github.com/escapewindow/mozharness/blob/talosrunner/scripts/
> device_talosrunner.py#L217
> 
> >+    def generate_config(self):
> >+        """generate talos configuration"""
> 
> There may be more options we want here, but this is a good start.
> 
> There may be additional work that needs doing before this script is both
> fully featured for everything we do in production, and we have parity
> between the two talos scripts.  But other than possibly
> s,_fetch,download_file, I don't see anything glaringly wrong here.

Indeed, will fix the _fetch -> download_file issue and push

> Thanks for the script!
pushed: http://hg.mozilla.org/build/mozharness/rev/1d1144817b2a
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
(In reply to Jeff Hammel [:jhammel] from comment #15)
> I would like to see a method that is "check out from hg/download from
> tarball and install via `setup.py`.  Would that be worth ticketing?

bug 697462 ?

> I'm more
> inclined to use a direct .tar.gz/.zip download versus hg to avoid another
> dependency but agree that the way I've done things is a bit magic.  

The thing about the hg.m.o zip file is it's only tip, which means any checkin will get picked up immediately.  We may want that behavior, but we may also want to lock a particular revision to a branch.

We do have talos tarballs already, so this isn't something that needs solving immediately.

> Also note that there are also upstream talos issues here.  Talos should not
> require to be run from a specific directory, but it does.  Talos should also
> separate the software pieces from the data pieces but it does not.  We'll
> have to fix these too, probably this quarter.  I just haven't gotten around
> to figuring out what all needs to be done there.

I'd love to see those things fixed, great to hear!

We may want followup bugs for a) --download support so this is user-at-desktop friendly, and b) to make device_talosrunner and this script share more backend methods.
Reopened due to the reopening of https://bugzilla.mozilla.org/show_bug.cgi?id=700722 . The code is still valid, it will just no longer work because of the downstream talos snafu.  Unless there is a burning reason, I'd rather no backout the script
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
I'm fine with keeping the script checked in, and adding patches to it as needed; nothing in production is calling this script.
Blocks: 713003
Blocks: 713055
Priority: P4 → P1
Whiteboard: [mozharness][talos] → [mozharness][talos][mozharness+talos]
re comment 19, bug 700722 is now fixed; closing
Status: REOPENED → RESOLVED
Closed: 11 years ago11 years ago
Resolution: --- → FIXED
Depends on: 865311
No longer depends on: 865311
Product: mozilla.org → Release Engineering
Component: Other → Mozharness
You need to log in before you can comment on or make changes to this bug.