Closed Bug 1246264 Opened 4 years ago Closed 4 years ago

failure to mach build with an artifact build

Categories

(Firefox Build System :: General, defect)

defect
Not set

Tracking

(firefox47 fixed)

RESOLVED FIXED
mozilla47
Tracking Status
firefox47 --- fixed

People

(Reporter: jmaher, Unassigned, Mentored)

References

(Blocks 1 open bug)

Details

(Whiteboard: [lang=python][good first bug])

Attachments

(1 file, 1 obsolete file)

jmaher@jmaher-ThinkPad-X230:~/mozilla/mozilla-inbound$ ./mach build
 0:00.16 Running |mach artifact install|.
 0:00.16 /home/jmaher/mozilla/mozilla-inbound/mach artifact install
 0:03.49 Ignoring exception unpickling cache file /home/jmaher/.mozbuild/package-frontend/artifact_url-cache.pickle: IOError(2, 'No such file or directory')
 0:03.49 Ignoring exception unpickling cache file /home/jmaher/.mozbuild/package-frontend/artifact_tree-cache.pickle: IOError(2, 'No such file or directory')
 0:05.85 Installing from remote pushhead e36505acb6f3e7451d0b5a435fa6bf65ed026384 on mozilla-inbound
 0:05.85 Installing from https://queue.taskcluster.net/v1/task/XC9Aiu1HTueC8HV03PeIbQ/artifacts/public%2Fbuild%2Ffirefox-47.0a1.en-US.linux-x86_64.common.tests.zip
 0:05.85 Ignoring exception unpickling cache file /home/jmaher/.mozbuild/package-frontend/fetch-cache.pickle: IOError(2, 'No such file or directory')
 0:05.85 Downloading to temporary location /home/jmaher/.mozbuild/package-frontend/4608132da4558757-public%2Fbuild%2Ffirefox-47.0a1.en-US.linux-x86_64.common.tests.zip
Error running mach:

    ['artifact', 'install']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

OSError: [Errno 2] No such file or directory: '/home/jmaher/.mozbuild/package-frontend/tmpzCERCP.tmp'

  File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/mach_commands.py", line 1488, in artifact_install
    return artifacts.install_from(source, self.distdir)
  File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/artifacts.py", line 835, in install_from
    return self.install_from_hg(source, distdir)
  File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/artifacts.py", line 819, in install_from_hg
    if self.install_from_url(url, distdir):
  File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/artifacts.py", line 800, in install_from_url
    filename = artifact_cache.fetch(url)
  File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/artifacts.py", line 418, in wrapper
    result = method(self, *args, **kwargs)
  File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/artifacts.py", line 615, in fetch
    dl.wait()
  File "/home/jmaher/mozilla/mozilla-inbound/objdir-frontend/_virtualenv/lib/python2.7/site-packages/mozregression/download_manager.py", line 97, in wait
    self.raise_if_error()
  File "/home/jmaher/mozilla/mozilla-inbound/objdir-frontend/_virtualenv/lib/python2.7/site-packages/mozregression/download_manager.py", line 162, in _download
    dir=os.path.dirname(dest)) as temp:
  File "/usr/lib/python2.7/tempfile.py", line 462, in NamedTemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags)
  File "/usr/lib/python2.7/tempfile.py", line 239, in _mkstemp_inner
    fd = _os.open(file, flags, 0600)
Error running mach:

    ['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Process executed with non-0 exit code 1: [u'/home/jmaher/mozilla/mozilla-inbound/mach', u'artifact', u'install']

  File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/mach_commands.py", line 412, in build
    self._run_mach_artifact_install()
  File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/mach_commands.py", line 645, in _run_mach_artifact_install
    pass_thru=True, ensure_exit_code=True)
  File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/base.py", line 642, in _run_command_in_srcdir
    return self.run_process(cwd=self.topsrcdir, **args)
  File "/home/jmaher/mozilla/mozilla-inbound/python/mach/mach/mixin/process.py", line 147, in run_process
    raise Exception('Process executed with non-0 exit code %d: %s' % (status, args))
if this helps shed light on file system issues, it seems as though I have the correct paths:
 ls -la ~/.mozbuild
total 48
drwx------ 10 jmaher jmaher  4096 Feb  5 12:42 .
drwxr-xr-x 70 jmaher jmaher 12288 Feb  4 12:25 ..
drwxrwxr-x  3 jmaher jmaher  4096 Aug 31 10:27 android-device
drwxr-xr-x 10 jmaher jmaher  4096 Mar 19  2013 android-ndk-r8e
drwxrwxr-x 10 jmaher jmaher  4096 Aug 26 13:30 android-sdk-linux
drwxrwxr-x  5 jmaher jmaher  4096 Jan 18 06:17 hgwatchman
drwxrwxr-x  3 jmaher jmaher  4096 Jul 28  2014 mercurial
drwxrwxr-x  2 jmaher jmaher  4096 Aug 12 15:31 mozboot
drwxrwxr-x  2 jmaher jmaher  4096 Feb  5 12:42 package-frontend
drwxrwxr-x 14 jmaher jmaher  4096 Jan 28 04:13 version-control-tools
jmaher@jmaher-ThinkPad-X230:~/mozilla/buildbot-configs$ ls -la ~/.mozbuild/package-frontend/
total 20
drwxrwxr-x  2 jmaher jmaher 4096 Feb  5 12:42 .
drwx------ 10 jmaher jmaher 4096 Feb  5 12:42 ..
-rw-rw-r--  1 jmaher jmaher  228 Feb  5 12:42 artifact_tree-cache.pickle
-rw-rw-r--  1 jmaher jmaher  403 Feb  5 12:42 artifact_url-cache.pickle
-rw-rw-r--  1 jmaher jmaher    6 Feb  5 12:42 fetch-cache.pickle
jmaher@jmaher-ThinkPad-X230:~/mozilla/buildbot-configs$
Blocks: artifact
So, we use mozregression to download artifacts, but the version is pinned.  Code is around https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/artifacts.py#632.  This almost looks like a transient download error not being surfaced.  jmaher, is this reproducible?
Flags: needinfo?(jmaher)
:parkouss, does the stack in comment 0 ring any bells from the mozregression side?
I got this working on try #3,  iam fine closing this as resolved/worksforme.

:parkouss there is a needinfo info in comment 3 for you!
Flags: needinfo?(jmaher) → needinfo?(j.parkouss)
Oh! Looks like there is some mozregression code used by mozbuild:

https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/artifacts.py#72-77

Wow. I wasn't aware of that. Hm, that seems a bit dangerous because mozregression is not in tree (at least as far as I know ?)! Maybe I should take the code of download_manager, and put it in a third party package on pypi, but anyway if it is used in tree it should live in tree no (e.g. a copy in tree) ?

To answer the question, I would say /home/jmaher/.mozbuild/package-frontend/ does not exists or is not a directory, reading the error

OSError: [Errno 2] No such file or directory: '/home/jmaher/.mozbuild/package-frontend/tmpzCERCP.tmp'
Flags: needinfo?(j.parkouss)
(In reply to Julien Pagès (:parkouss) from comment #5)
> Oh! Looks like there is some mozregression code used by mozbuild:
> 
> https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/
> artifacts.py#72-77
> 
> Wow. I wasn't aware of that. Hm, that seems a bit dangerous because
> mozregression is not in tree (at least as far as I know ?)! Maybe I should
> take the code of download_manager, and put it in a third party package on
> pypi, but anyway if it is used in tree it should live in tree no (e.g. a
> copy in tree) ?

Artifacts pull a published version of mozregression: https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/mach_commands.py#1461.

> To answer the question, I would say /home/jmaher/.mozbuild/package-frontend/
> does not exists or is not a directory, reading the error
> 
> OSError: [Errno 2] No such file or directory:
> '/home/jmaher/.mozbuild/package-frontend/tmpzCERCP.tmp'

Hmm, it's possible we're not careful about arranging the path to download into.  In fact, it looks like we're not!  That's a bug.  I'll try to look further shortly.
(In reply to Nick Alexander :nalexander from comment #6)
> (In reply to Julien Pagès (:parkouss) from comment #5)
> > Oh! Looks like there is some mozregression code used by mozbuild:
> > 
> > https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/
> > artifacts.py#72-77
> > 
> > Wow. I wasn't aware of that. Hm, that seems a bit dangerous because
> > mozregression is not in tree (at least as far as I know ?)! Maybe I should
> > take the code of download_manager, and put it in a third party package on
> > pypi, but anyway if it is used in tree it should live in tree no (e.g. a
> > copy in tree) ?
> 
> Artifacts pull a published version of mozregression:
> https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/
> mach_commands.py#1461.

I just saw! Sorry for the noise :)

> > To answer the question, I would say /home/jmaher/.mozbuild/package-frontend/
> > does not exists or is not a directory, reading the error
> > 
> > OSError: [Errno 2] No such file or directory:
> > '/home/jmaher/.mozbuild/package-frontend/tmpzCERCP.tmp'
> 
> Hmm, it's possible we're not careful about arranging the path to download
> into.  In fact, it looks like we're not!  That's a bug.  I'll try to look
> further shortly.

Maybe the folder in which the downloads should persist have been removed at some point ?
The DownloadManager should create the dirsetory if required (it does on master - I *think* it does in 2.0.1 too):

https://github.com/mozilla/mozregression/blob/master/mozregression/download_manager.py#L233).
Oh, I see that you are using 1.0.2 and not 2.0.1 - there is probably not that code, so maybe this is the cause of the issue (download dir not created).
I can reproduce this reliably by removing the destination folder, should be an easy fix.
Assignee: nobody → cmanchester
(In reply to Chris Manchester [:chmanchester] from comment #9)
> I can reproduce this reliably by removing the destination folder, should be
> an easy fix.

Agreed.  Just need to `ensureParentDirs` in the right place.  Let's throw this one into the mentored bug pond.
Assignee: cmanchester → nobody
Mentor: nalexander, cmanchester
Whiteboard: [lang=python][good first bug]
Comment on attachment 8718504 [details] [diff] [review]
Ensure cache directory exists for artifacts installation

Review of attachment 8718504 [details] [diff] [review]:
-----------------------------------------------------------------

Thank you for the patch. This is a simple bug, but fixing it is very valuable. Let me know if you have any question about the comment below.

::: python/mozbuild/mozbuild/mach_commands.py
@@ +1467,5 @@
>  
>          state_dir = self._mach_context.state_dir
>          cache_dir = os.path.join(state_dir, 'package-frontend')
>  
> +        # Ensure the cache directory exists.

Not sure this comment is necessary.

@@ +1469,5 @@
>          cache_dir = os.path.join(state_dir, 'package-frontend')
>  
> +        # Ensure the cache directory exists.
> +        if not os.path.exists(cache_dir):
> +            os.makedirs(cache_dir)

The preferred way to do this is to attempt creation, catch any OSError, and re-raise for any error condition other than the directory already existing. This is most robust, because it avoids the "check then act" concurrency error. I believe there are some examples of this in the tree.
Attachment #8718504 - Flags: review?(cmanchester)
Thank you for the explanation. I have made the necessary changes.
Attachment #8718504 - Attachment is obsolete: true
Attachment #8718544 - Flags: review?(cmanchester)
Comment on attachment 8718544 [details] [diff] [review]
Ensure cache directory exists for artifacts installation

Review of attachment 8718544 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good. Usually things can be landed by setting the "checkin-needed" keyword on the bug, but I have something else to land this evening, and the tree appears open, so I'll go ahead and get this in.
Attachment #8718544 - Flags: review?(cmanchester) → review+
(In reply to Chris Manchester [:chmanchester] from comment #14)
> Comment on attachment 8718544 [details] [diff] [review]
> Ensure cache directory exists for artifacts installation
> 
> Review of attachment 8718544 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Looks good. Usually things can be landed by setting the "checkin-needed"
> keyword on the bug, but I have something else to land this evening, and the
> tree appears open, so I'll go ahead and get this in.

Thanks for the review! I will remember to add the "checkin-needed" keyword to bugs from now onward.
https://hg.mozilla.org/mozilla-central/rev/f11b6e1d652e
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla47
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.