1246264 - failure to mach build with an artifact build

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

jmaher@jmaher-ThinkPad-X230:~/mozilla/mozilla-inbound$ ./mach build
 0:00.16 Running |mach artifact install|.
 0:00.16 /home/jmaher/mozilla/mozilla-inbound/mach artifact install
 0:03.49 Ignoring exception unpickling cache file /home/jmaher/.mozbuild/package-frontend/artifact_url-cache.pickle: IOError(2, 'No such file or directory')
 0:03.49 Ignoring exception unpickling cache file /home/jmaher/.mozbuild/package-frontend/artifact_tree-cache.pickle: IOError(2, 'No such file or directory')
 0:05.85 Installing from remote pushhead e36505acb6f3e7451d0b5a435fa6bf65ed026384 on mozilla-inbound
 0:05.85 Installing from https://queue.taskcluster.net/v1/task/XC9Aiu1HTueC8HV03PeIbQ/artifacts/public%2Fbuild%2Ffirefox-47.0a1.en-US.linux-x86_64.common.tests.zip
 0:05.85 Ignoring exception unpickling cache file /home/jmaher/.mozbuild/package-frontend/fetch-cache.pickle: IOError(2, 'No such file or directory')
 0:05.85 Downloading to temporary location /home/jmaher/.mozbuild/package-frontend/4608132da4558757-public%2Fbuild%2Ffirefox-47.0a1.en-US.linux-x86_64.common.tests.zip
Error running mach:

    ['artifact', 'install']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

OSError: [Errno 2] No such file or directory: '/home/jmaher/.mozbuild/package-frontend/tmpzCERCP.tmp'

  File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/mach_commands.py", line 1488, in artifact_install
    return artifacts.install_from(source, self.distdir)
  File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/artifacts.py", line 835, in install_from
    return self.install_from_hg(source, distdir)
  File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/artifacts.py", line 819, in install_from_hg
    if self.install_from_url(url, distdir):
  File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/artifacts.py", line 800, in install_from_url
    filename = artifact_cache.fetch(url)
  File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/artifacts.py", line 418, in wrapper
    result = method(self, *args, **kwargs)
  File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/artifacts.py", line 615, in fetch
    dl.wait()
  File "/home/jmaher/mozilla/mozilla-inbound/objdir-frontend/_virtualenv/lib/python2.7/site-packages/mozregression/download_manager.py", line 97, in wait
    self.raise_if_error()
  File "/home/jmaher/mozilla/mozilla-inbound/objdir-frontend/_virtualenv/lib/python2.7/site-packages/mozregression/download_manager.py", line 162, in _download
    dir=os.path.dirname(dest)) as temp:
  File "/usr/lib/python2.7/tempfile.py", line 462, in NamedTemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags)
  File "/usr/lib/python2.7/tempfile.py", line 239, in _mkstemp_inner
    fd = _os.open(file, flags, 0600)
Error running mach:

    ['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Process executed with non-0 exit code 1: [u'/home/jmaher/mozilla/mozilla-inbound/mach', u'artifact', u'install']

  File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/mach_commands.py", line 412, in build
    self._run_mach_artifact_install()
  File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/mach_commands.py", line 645, in _run_mach_artifact_install
    pass_thru=True, ensure_exit_code=True)
  File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/base.py", line 642, in _run_command_in_srcdir
    return self.run_process(cwd=self.topsrcdir, **args)
  File "/home/jmaher/mozilla/mozilla-inbound/python/mach/mach/mixin/process.py", line 147, in run_process
    raise Exception('Process executed with non-0 exit code %d: %s' % (status, args))

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

if this helps shed light on file system issues, it seems as though I have the correct paths:
 ls -la ~/.mozbuild
total 48
drwx------ 10 jmaher jmaher  4096 Feb  5 12:42 .
drwxr-xr-x 70 jmaher jmaher 12288 Feb  4 12:25 ..
drwxrwxr-x  3 jmaher jmaher  4096 Aug 31 10:27 android-device
drwxr-xr-x 10 jmaher jmaher  4096 Mar 19  2013 android-ndk-r8e
drwxrwxr-x 10 jmaher jmaher  4096 Aug 26 13:30 android-sdk-linux
drwxrwxr-x  5 jmaher jmaher  4096 Jan 18 06:17 hgwatchman
drwxrwxr-x  3 jmaher jmaher  4096 Jul 28  2014 mercurial
drwxrwxr-x  2 jmaher jmaher  4096 Aug 12 15:31 mozboot
drwxrwxr-x  2 jmaher jmaher  4096 Feb  5 12:42 package-frontend
drwxrwxr-x 14 jmaher jmaher  4096 Jan 28 04:13 version-control-tools
jmaher@jmaher-ThinkPad-X230:~/mozilla/buildbot-configs$ ls -la ~/.mozbuild/package-frontend/
total 20
drwxrwxr-x  2 jmaher jmaher 4096 Feb  5 12:42 .
drwx------ 10 jmaher jmaher 4096 Feb  5 12:42 ..
-rw-rw-r--  1 jmaher jmaher  228 Feb  5 12:42 artifact_tree-cache.pickle
-rw-rw-r--  1 jmaher jmaher  403 Feb  5 12:42 artifact_url-cache.pickle
-rw-rw-r--  1 jmaher jmaher    6 Feb  5 12:42 fetch-cache.pickle
jmaher@jmaher-ThinkPad-X230:~/mozilla/buildbot-configs$

Comment 2

[Nick Alexander :nalexander [he/him]]

So, we use mozregression to download artifacts, but the version is pinned.  Code is around https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/artifacts.py#632.  This almost looks like a transient download error not being surfaced.  jmaher, is this reproducible?

Comment 3

[Chris Manchester (limited bugmail, email directly)]

:parkouss, does the stack in comment 0 ring any bells from the mozregression side?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

I got this working on try #3,  iam fine closing this as resolved/worksforme.

:parkouss there is a needinfo info in comment 3 for you!

Comment 5

[Julien Pagès (:parkouss)]

Oh! Looks like there is some mozregression code used by mozbuild:

https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/artifacts.py#72-77

Wow. I wasn't aware of that. Hm, that seems a bit dangerous because mozregression is not in tree (at least as far as I know ?)! Maybe I should take the code of download_manager, and put it in a third party package on pypi, but anyway if it is used in tree it should live in tree no (e.g. a copy in tree) ?

To answer the question, I would say /home/jmaher/.mozbuild/package-frontend/ does not exists or is not a directory, reading the error

OSError: [Errno 2] No such file or directory: '/home/jmaher/.mozbuild/package-frontend/tmpzCERCP.tmp'

Comment 6

[Nick Alexander :nalexander [he/him]]

(In reply to Julien Pagès (:parkouss) from comment #5)
> Oh! Looks like there is some mozregression code used by mozbuild:
> 
> https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/
> artifacts.py#72-77
> 
> Wow. I wasn't aware of that. Hm, that seems a bit dangerous because
> mozregression is not in tree (at least as far as I know ?)! Maybe I should
> take the code of download_manager, and put it in a third party package on
> pypi, but anyway if it is used in tree it should live in tree no (e.g. a
> copy in tree) ?

Artifacts pull a published version of mozregression: https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/mach_commands.py#1461.

> To answer the question, I would say /home/jmaher/.mozbuild/package-frontend/
> does not exists or is not a directory, reading the error
> 
> OSError: [Errno 2] No such file or directory:
> '/home/jmaher/.mozbuild/package-frontend/tmpzCERCP.tmp'

Hmm, it's possible we're not careful about arranging the path to download into.  In fact, it looks like we're not!  That's a bug.  I'll try to look further shortly.

Comment 7

[Julien Pagès (:parkouss)]

(In reply to Nick Alexander :nalexander from comment #6)
> (In reply to Julien Pagès (:parkouss) from comment #5)
> > Oh! Looks like there is some mozregression code used by mozbuild:
> > 
> > https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/
> > artifacts.py#72-77
> > 
> > Wow. I wasn't aware of that. Hm, that seems a bit dangerous because
> > mozregression is not in tree (at least as far as I know ?)! Maybe I should
> > take the code of download_manager, and put it in a third party package on
> > pypi, but anyway if it is used in tree it should live in tree no (e.g. a
> > copy in tree) ?
> 
> Artifacts pull a published version of mozregression:
> https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/
> mach_commands.py#1461.

I just saw! Sorry for the noise :)

> > To answer the question, I would say /home/jmaher/.mozbuild/package-frontend/
> > does not exists or is not a directory, reading the error
> > 
> > OSError: [Errno 2] No such file or directory:
> > '/home/jmaher/.mozbuild/package-frontend/tmpzCERCP.tmp'
> 
> Hmm, it's possible we're not careful about arranging the path to download
> into.  In fact, it looks like we're not!  That's a bug.  I'll try to look
> further shortly.

Maybe the folder in which the downloads should persist have been removed at some point ?
The DownloadManager should create the dirsetory if required (it does on master - I *think* it does in 2.0.1 too):

https://github.com/mozilla/mozregression/blob/master/mozregression/download_manager.py#L233).

Comment 8

[Julien Pagès (:parkouss)]

Oh, I see that you are using 1.0.2 and not 2.0.1 - there is probably not that code, so maybe this is the cause of the issue (download dir not created).

Comment 9

[Chris Manchester (limited bugmail, email directly)]

I can reproduce this reliably by removing the destination folder, should be an easy fix.

Comment 10

[Nick Alexander :nalexander [he/him]]

(In reply to Chris Manchester [:chmanchester] from comment #9)
> I can reproduce this reliably by removing the destination folder, should be
> an easy fix.

Agreed.  Just need to `ensureParentDirs` in the right place.  Let's throw this one into the mentored bug pond.

Comment 11

[Sambuddha Basu [:sambuddhabasu1]]

Attachment: Ensure cache directory exists for artifacts installation

Comment 12

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8718504 [details] [diff] [review]
Ensure cache directory exists for artifacts installation

Review of attachment 8718504 [details] [diff] [review]:
-----------------------------------------------------------------

Thank you for the patch. This is a simple bug, but fixing it is very valuable. Let me know if you have any question about the comment below.

::: python/mozbuild/mozbuild/mach_commands.py
@@ +1467,5 @@
>  
>          state_dir = self._mach_context.state_dir
>          cache_dir = os.path.join(state_dir, 'package-frontend')
>  
> +        # Ensure the cache directory exists.

Not sure this comment is necessary.

@@ +1469,5 @@
>          cache_dir = os.path.join(state_dir, 'package-frontend')
>  
> +        # Ensure the cache directory exists.
> +        if not os.path.exists(cache_dir):
> +            os.makedirs(cache_dir)

The preferred way to do this is to attempt creation, catch any OSError, and re-raise for any error condition other than the directory already existing. This is most robust, because it avoids the "check then act" concurrency error. I believe there are some examples of this in the tree.

Comment 13

[Sambuddha Basu [:sambuddhabasu1]]

Attachment: Ensure cache directory exists for artifacts installation

Thank you for the explanation. I have made the necessary changes.

Comment 14

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8718544 [details] [diff] [review]
Ensure cache directory exists for artifacts installation

Review of attachment 8718544 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good. Usually things can be landed by setting the "checkin-needed" keyword on the bug, but I have something else to land this evening, and the tree appears open, so I'll go ahead and get this in.

Comment 15

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/f11b6e1d652e

Comment 16

[Sambuddha Basu [:sambuddhabasu1]]

(In reply to Chris Manchester [:chmanchester] from comment #14)
> Comment on attachment 8718544 [details] [diff] [review]
> Ensure cache directory exists for artifacts installation
> 
> Review of attachment 8718544 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Looks good. Usually things can be landed by setting the "checkin-needed"
> keyword on the bug, but I have something else to land this evening, and the
> tree appears open, so I'll go ahead and get this in.

Thanks for the review! I will remember to add the "checkin-needed" keyword to bugs from now onward.

Comment 17

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/f11b6e1d652e