855554 - mozharness should properly deal with tar wildcards on Ubuntu OR use hg to obtain the talos bits

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Rail Aliiev [:rail]]

16:51:51     INFO - Running command: ['tar', 'zx', '--strip-components=1', '-f', '/builds/slave/test/build/talos-4a0e7514c66c.tar.gz', '**/talos/'] in /home/cltbld/talos-slave/talos-data
16:51:51     INFO - Copy/paste: tar zx --strip-components=1 -f /builds/slave/test/build/talos-4a0e7514c66c.tar.gz **/talos/
16:51:52     INFO -  tar: Pattern matching characters used in file names
16:51:52     INFO -  tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning
16:51:52     INFO -  tar: **/talos: Not found in archive
16:51:52     INFO -  tar: Exiting with failure status due to previous errors
16:51:52    ERROR - Return code: 2
16:51:52    FATAL - Halting on failure while running ['tar', 'zx', '--strip-components=1', '-f', '/builds/slave/test/build/talos-4a0e7514c66c.tar.gz', '**/talos/']


Adding --wildcards fixes the problem, but I'm not sure if this parameter is supported by other platforms (fedora 12 does).

Comment 1

[Armen [:armenzg]]

How does this bug block bug 820243? I think this was more for the Ubuntu switch over, no?

Comment 2

[Rail Aliiev [:rail]]

(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) from comment #1)
> How does this bug block bug 820243? I think this was more for the Ubuntu
> switch over, no?

It shouldn't. Blocking bug 713055 is enough.

Comment 3

[Jason Yeo[:jyeo]]

Attachment: use hgtool instead of depending on tar

Comment 4

[Jason Yeo[:jyeo]]

Attachment: add talos_revision attribute to talos.json

Comment 5

[Jason Yeo[:jyeo]]

09:30:06     INFO - rmtree: /home/cltbld/talos-slave/talos-data/talos
09:30:06     INFO - retry: Calling <bound method Talos._get_revision of <mozharness.mozilla.testing.talos.Talos object at 0xfe58d0>> with args: (<mozharness.base.vcs.hgtool.HgtoolVCS object at 0xe4af10>, '/builds/slave/talos-slave/test/build/talos_repo'), kwargs: {}, attempt #1
09:30:06     INFO - Running command: ['hgtool.py', '-r', '4a0e7514c66c', 'http://hg.mozilla.org/build/talos', '/builds/slave/talos-slave/test/build/talos_repo']
09:30:06     INFO - Copy/paste: hgtool.py -r 4a0e7514c66c http://hg.mozilla.org/build/talos /builds/slave/talos-slave/test/build/talos_repo
09:30:06    ERROR - caught OS error 2: No such file or directory while running ['hgtool.py', '-r', '4a0e7514c66c', 'http://hg.mozilla.org/build/talos', '/builds/slave/talos-slave/test/build/talos_repo']

The linux machines don't have hgtool? Maybe should use 'vcs': 'hg' instead?

Comment 6

[Aki Sasaki (not active)]

Using hg instead of hgtool is one fix.
Another is to get build/tools onto the box first via hg (or grabbing hgtool via wget), and then using hgtool for talos, but I'm not sure how much we win with that setup.

Comment 7

[Jason Yeo[:jyeo]]

Attachment: use hg instead of hgtool

using hg instead of hgtool

Comment 8

[Armen [:armenzg]]

(In reply to Jason Yeo [:jyeo] from comment #7)
> Created attachment 756644 [details] [diff] [review]
> use hg instead of hgtool
> 
> using hg instead of hgtool

I pushed it to ash-mozharness.

Comment 9

[Armen [:armenzg]]

It seems that it is checking out talos but then it has trouble unzipping the dirtyDB.zip (for Mac):
talos-r4-lion-039:talos-slave cltbld$ pwd
/Users/cltbld/talos-slave
talos-r4-lion-039:talos-slave cltbld$ ls -l talos-data/
total 8
-rw-rw-r--  1 cltbld  staff  2368 30 Mar  2011 count_and_reboot.py

https://tbpl.mozilla.org/php/getParsedLog.php?id=23641123&tree=Ash&full=1#error0
11:30:13     INFO - Downloading addons...
11:30:13     INFO - Downloading http://build.mozilla.org/talos/profiles/dirtyDBs.zip to /Users/cltbld/talos-slave/test/build/dirtyDBs.zip
11:30:13     INFO - retry: Calling <bound method Talos._download_file of <mozharness.mozilla.testing.talos.Talos object at 0x10bcd1c10>> with args: (u'http://build.mozilla.org/talos/profiles/dirtyDBs.zip', u'/Users/cltbld/talos-slave/test/build/dirtyDBs.zip'), kwargs: {}, attempt #1
11:30:13     INFO - Downloaded 7765237 bytes.
11:30:13    FATAL - Can't run command ['unzip', '-q', '-o', u'/Users/cltbld/talos-slave/test/build/dirtyDBs.zip'] in non-existent directory '/Users/cltbld/talos-slave/test/../talos-data/talos'!

Comment 10

[Aki Sasaki (not active)]

Comment on attachment 756577 [details] [diff] [review]
use hgtool instead of depending on tar

>-class Talos(TestingMixin, BaseScript):
>+class Talos(TestingMixin, BaseScript, VCSMixin):

You can use VCSScript (or MercurialScript) instead of BaseScript + VCSMixin.  It's just shorthand.
Either way works.

>@@ -335,29 +336,32 @@ class Talos(TestingMixin, BaseScript):
>         c = self.config
>         talos_url = self.query_talos_url()
>         if not c.get('webroot') or not talos_url:
>             self.fatal("Both webroot and talos_url need to be set to populate_webroot!")
>         self.info("Populating webroot %s..." % c['webroot'])
>         talos_webdir = os.path.join(c['webroot'], 'talos')
>         self.mkdir_p(c['webroot'], error_level=FATAL)
>         self.rmtree(talos_webdir, error_level=FATAL)
>-        tarball = self.download_file(talos_url, parent_dir=self.workdir,
>-                                     error_level=FATAL)
>-        if self._is_windows():
>-            tarball = self.query_msys_path(tarball)
>-        command = c.get('webroot_extract_cmd')
>-        if command:
>-            command = command % {'tarball': tarball}
>-        else:
>-            tar = self.query_exe('tar', return_type='list')
>-            command = tar + ['zx', '--strip-components=1', '-f', tarball,
>-                             '**/talos/']
>-        self.run_command(command, cwd=c['webroot'],
>-                         error_list=TarErrorList, halt_on_failure=True)
>+
>+        # clone talos' repo
>+        talos_repo_path = os.path.join(self.workdir, 'talos_repo')
>+        repo = {
>+            'repo': 'http://hg.mozilla.org/build/talos',
>+            'vcs': 'hgtool',
>+            'dest': os.path.join(self.workdir, talos_repo_path),
>+            'revision': self.talos_json_config['global']['talos_revision']
>+            }
>+        self.vcs_checkout(**repo)
>+
>+        src_talos_path = os.path.join(talos_repo_path, 'talos')
>+        dest_talos_path = os.path.join(self.workdir, 'talos')
>+        mv_command = ['mv', src_talos_path, dest_talos_path]
>+        self.run_command(mv_command, cwd=self.workdir)

a) maybe use self.move() instead of run_command(['mv' ...])?
b) I don't think you're actually populating c['webroot'], which will be problematic when you actually run the tests.

Comment 11

[Aki Sasaki (not active)]

Comment on attachment 756644 [details] [diff] [review]
use hg instead of hgtool

You'll get this for free if you use MercurialScript.
But this is a valid way of doing it.

Comment 12

[Aki Sasaki (not active)]

(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) from comment #9)
> It seems that it is checking out talos but then it has trouble unzipping the
> dirtyDB.zip (for Mac):
> talos-r4-lion-039:talos-slave cltbld$ pwd
> /Users/cltbld/talos-slave
> talos-r4-lion-039:talos-slave cltbld$ ls -l talos-data/
> total 8
> -rw-rw-r--  1 cltbld  staff  2368 30 Mar  2011 count_and_reboot.py
> 
> https://tbpl.mozilla.org/php/getParsedLog.
> php?id=23641123&tree=Ash&full=1#error0
> 11:30:13     INFO - Downloading addons...
> 11:30:13     INFO - Downloading
> http://build.mozilla.org/talos/profiles/dirtyDBs.zip to
> /Users/cltbld/talos-slave/test/build/dirtyDBs.zip
> 11:30:13     INFO - retry: Calling <bound method Talos._download_file of
> <mozharness.mozilla.testing.talos.Talos object at 0x10bcd1c10>> with args:
> (u'http://build.mozilla.org/talos/profiles/dirtyDBs.zip',
> u'/Users/cltbld/talos-slave/test/build/dirtyDBs.zip'), kwargs: {}, attempt #1
> 11:30:13     INFO - Downloaded 7765237 bytes.
> 11:30:13    FATAL - Can't run command ['unzip', '-q', '-o',
> u'/Users/cltbld/talos-slave/test/build/dirtyDBs.zip'] in non-existent
> directory '/Users/cltbld/talos-slave/test/../talos-data/talos'!

This is probably since we're not populating c['webroot'] anymore (see comment 10).

Comment 13

[Jason Yeo[:jyeo]]

Attachment: Actually populates webroot

Comment 14

[Aki Sasaki (not active)]

Comment on attachment 756853 [details] [diff] [review]
Actually populates webroot

>+        cp_command = ['cp', '-rf', dest_talos_path, talos_webdir]
>+        self.run_command(cp_command, cwd=self.workdir)

I think self.copytree() does what you want.
Hopefully this works for you.

Comment 15

[Armen [:armenzg]]

(In reply to Jason Yeo [:jyeo] from comment #13)
> Created attachment 756853 [details] [diff] [review]
> Actually populates webroot

I've pushed it to Ash and re-trigger talos other jobs.

Comment 16

[Jason Yeo[:jyeo]]

I'm seeing jobs turning green for mac builds. This should be fixed right?

https://tbpl.mozilla.org/?tree=Ash&jobname=talos

Comment 17

[Aki Sasaki (not active)]

If it's green on all platforms, including ubuntu :) , then yes.

Comment 18

[Armen [:armenzg]]

We have the crash on Ubuntu and we have hg weirdness on Windows.

Comment 19

[Jason Yeo[:jyeo]]

Green on all platforms. Fixed.