Last Comment Bug 808280 - [mach] mozbuild uses multiprocess.synchronize via pymake which is unavailable on OpenBSD
: [mach] mozbuild uses multiprocess.synchronize via pymake which is unavailable...
Status: RESOLVED FIXED
:
Product: Core
Classification: Components
Component: Build Config (show other bugs)
: Trunk
: Other OpenBSD
: -- normal (vote)
: mozilla21
Assigned To: Gregory Szorc [:gps]
:
: Gregory Szorc [:gps]
Mentors:
Depends on: 808357
Blocks:
  Show dependency treegraph
 
Reported: 2012-11-03 03:26 PDT by Landry Breuil (:gaston)
Modified: 2013-12-18 16:12 PST (History)
6 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
Remove unused pymake imports, v1 (983 bytes, patch)
2013-01-17 10:30 PST, Gregory Szorc [:gps]
ted: review+
landry: feedback+
Details | Diff | Splinter Review

Description Landry Breuil (:gaston) 2012-11-03 03:26:43 PDT
As of now mach fails on OpenBSD with :

Traceback (most recent call last):
  File "mach", line 53, in <module>
    import mach.main
  File "/src/mozilla-central/python/mach/mach/main.py", line 20, in <module>
    from mozbuild.base import BuildConfig
  File "/src/mozilla-central/python/mozbuild/mozbuild/base.py", line 10, in <module>
    import pymake.parser
  File "/src/mozilla-central/build/pymake/pymake/parser.py", line 37, in <module>
    import data, functions, util, parserdata
  File "/src/mozilla-central/build/pymake/pymake/data.py", line 6, in <module>
    import parserdata, parser, functions, process, util, implicit
  File "/src/mozilla-central/build/pymake/pymake/process.py", line 429, in <module>
    class ParallelContext(object):
  File "/src/mozilla-central/build/pymake/pymake/process.py", line 435, in ParallelContext
    _condition = multiprocessing.Condition()
  File "/usr/local/lib/python2.7/multiprocessing/__init__.py", line 189, in Condition
    from multiprocessing.synchronize import Condition
  File "/usr/local/lib/python2.7/multiprocessing/synchronize.py", line 59, in <module>
    " function, see issue 3770.")
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.

Consider this an OS deficiency or not, but if pymake becomes a had requirement through mozbuild & mach, can we use an alternative for those needed features ?
Comment 1 Landry Breuil (:gaston) 2012-11-03 03:29:03 PDT
Grm See Also doesnt support bugs.python.org... so see also http://bugs.python.org/issue3770 for more details
Comment 2 Landry Breuil (:gaston) 2012-11-03 03:37:44 PDT
Note that python2.7 build/pymake/make.py shows the same python traceback, so the problem is in pymake itself, not really mach..
Comment 3 Ted Mielczarek [:ted.mielczarek] 2012-11-03 10:06:18 PDT
We shouldn't be using Pymake anywhere but on Windows. It's not a perf win anywhere else.
Comment 4 Gregory Szorc [:gps] 2012-11-03 10:24:16 PDT
Pymake is entering the equation on non-Windows because we use the pymake API to parse the output of mozconfig2mk. This is the mechanism in mozbuild (which mach uses) to discover the object directory.

The obvious solution is not to use pymake here. I would love to improve the "parsing" of mozconfigs in mozbuild so that it doesn't involve so many steps. But, if we fix that, we still have an issue of multiprocessing not working on OpenBSD. And, that will surely bite us when things like test runners eventually utilize multiprocessing for parallel execution. I would like to figure out whether it is possible to make multiprocessing work on OpenBSD. If not, we're going to be in a world of hurt.

Landry: What OpenBSD are you running? Are you running Python 2.7.3? If not 2.7.3, I'm curious if that fixes it. If you are on 2.7.3, I'm tempted to ask you to compile Python from source to see if it does a better job than whatever is in Ports. Could you verify that |python -c 'import multiprocess'| is a sufficient test case?
Comment 5 Landry Breuil (:gaston) 2012-11-03 10:55:37 PDT
(In reply to Gregory Szorc [:gps] from comment #4)
> Landry: What OpenBSD are you running? Are you running Python 2.7.3? If not
> 2.7.3, I'm curious if that fixes it. If you are on 2.7.3, I'm tempted to ask
> you to compile Python from source to see if it does a better job than
> whatever is in Ports. Could you verify that |python -c 'import
> multiprocess'| is a sufficient test case?

I'm running OpenBSD -current as always, python 2.7.3. We have no special patches in ports (see http://www.openbsd.org/cgi-bin/cvsweb/ports/lang/python/2.7/patches/) and compiling stuff directly from source is usually a pain. The multiprocess thing is disabled upstream in the configure env where it detects sem_open is not available on OpenBSD, that was a consequence of python issue 3770

$python2.7 -c 'import multiprocess' 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named multiprocess
Comment 6 Gregory Szorc [:gps] 2012-11-03 11:08:08 PDT
I just installed OpenBSD 5.2. Installed Python from ports release. It was Python 2.7.3. However:

$ /usr/local/bin/python2.7 -c 'import multiprocessing.synchronize'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/multiprocessing/synchronize.py", line 59, in <module>
    " function, see issue 3770.")
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.


Note that 'import multiprocessing' works just fine. It's when you need the synchronization primitives that the world blows up. This is most unfortunate.
Comment 7 Gregory Szorc [:gps] 2012-11-03 11:13:12 PDT
Similar error on Python 3.2 from ports:

Python 3.2.3 (default, Nov  3 2012, 11:11:20)
[GCC 4.2.1 20070719 ] on openbsd5
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing
>>> import multiprocessing.synchronize
Traceback (most recent call last):
  File "/usr/local/lib/python3.2/multiprocessing/synchronize.py", line 54, in <module>
    from _multiprocessing import SemLock
ImportError: cannot import name SemLock

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.2/multiprocessing/synchronize.py", line 59, in <module>
    " function, see issue 3770.")
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.
Comment 8 Gregory Szorc [:gps] 2012-11-03 12:39:12 PDT
In the Python bug tracker, the following issues are all relevant:

* http://bugs.python.org/issue3770
* http://bugs.python.org/issue10348
* http://bugs.python.org/issue5725

Essentially the Python people realized that sem_open() didn't work on BSDs. It was close to release time. So, they just disabled the bits that required it. It appears the intent was to eventually support BSDs. Unfortunately, nobody ever followed up. Even Python 3.3 lacks this functionality.

In a nutshell, multiprocessing (at least the useful parts not provided by subprocess) won't work on OpenBSD (and possibly other BSDs). What does this mean for Mozilla? Well, if we want to support OpenBSD, we can't require multiprocessing. This is most unfortunate as multiprocessing is the only sane way in Python's standard library to perform simple cross-process communication, synchronization, etc. I would hate to not have multiprocessing in my tool belt. I already had ideas of rewriting some test runners to support parallel execution with it.

I assume we'll want the ability for in-tree Python tools to spin up multiple processes. If we can't use multiprocessing, we'll need to find something to replace it with. Maybe there is something on PyPI we can import. Maybe we can roll our own using something like 0MQ. I dunno.

I think the ideal solution is for Python to implement support for this feature. The relevant files in the Python tree are in Modules/_multiprocessing/. Someone would "just" need to throw in some ifdefs and implement the Python API using whatever makes sense on OpenBSD. I wouldn't be surprised if there were a patch floating around on the internets that did just this. If not, I highly encourage someone to write it. http://semanchuk.com/philip/sysv_ipc/ might be useful as a reference.

Until then, it might be worth rewriting the mozconfig integration code in mozbuild to not use pymake and thus not use multiprocessing. The easy solution is to write a crude Makefile parser (it actually won't be that hard since I think we just need to parse out a simple variable assignment). The proper and long-term solution likely involves refactoring mozconfig2mk and friends to be more Python friendly (kind of like what Mike Hommey did with config.status). I'm thinking that instead of evaluating mozconfig scripts through different run-time environments, we just have a single driver that outputs something consumed by Python which then extracts the bits it cares about. But, the long-term solution is likely out of scope for this bug.

What a mess.
Comment 9 Mike Hommey [:glandium] 2012-11-03 12:50:08 PDT
Why does the pymake parsing API need multiprocessing anyways?
Comment 10 Mike Hommey [:glandium] 2012-11-03 12:54:00 PDT
Said differently, things can probably be rearranged to avoid using something that is unsupported. As a matter of fact, the reason why the failure in comment 0 is happening is because there is a "static" class member _condition in ParallelContext that is initialized to multiprocessing.Condition(). If it was lazily initialized, maybe it would just work, if nothing in the parser API actually uses ParallelContext.
Comment 11 Gregory Szorc [:gps] 2012-11-03 13:13:48 PDT
We could probably hack pymake to work, sure. I've been meaning to refactor mozbuild to not import pymake though. I'd happily go with the quick and dirty parsing route to extract MOZ_OBJDIR.
Comment 12 Jeff Hammel 2012-11-06 11:25:05 PST
> The easy solution is to write a crude Makefile parser (it actually won't be that hard since I think we just need to parse out a simple variable assignment). 

I would hate to duplicate effort/code here :/
Comment 13 Gregory Szorc [:gps] 2012-11-06 11:37:44 PST
(In reply to Jeff Hammel [:jhammel] from comment #12)
> > The easy solution is to write a crude Makefile parser (it actually won't be that hard since I think we just need to parse out a simple variable assignment). 
> 
> I would hate to duplicate effort/code here :/

Bug 808357 is related and apparently has blessing from Mike Hommey to move forward. So, I don't think this is a big concern.

The bigger concern is lack of multiprocessing for *anything*. This would be a huge loss for any tool wishing to do true parallelism [without having to worry about the GIL].
Comment 14 Gregory Szorc [:gps] 2013-01-17 10:30:15 PST
Created attachment 703392 [details] [diff] [review]
Remove unused pymake imports, v1

This is fixing the regression reported in bug 784841 since this bug was already on file.

Anyway, when we implemented mozconfig "parsing" in Python in bug 808357, we stopped using the pymake modules. These modules were the source of the multiprocessing import that made BSDs unhappy. Unfortunately, we forget to remove the imports!

I'm /pretty/ sure that this will enable BSDs to work with mach and the new virtualenv code.

ted: Please rubber stamp trivial change.
gaston: Please verify this actually fixes the problem.
Comment 15 Gregory Szorc [:gps] 2013-01-17 11:04:03 PST
https://hg.mozilla.org/mozilla-central/rev/d29ffd335728

Landry: Please reopen if this doesn't fix things for you.
Comment 16 Landry Breuil (:gaston) 2013-01-17 12:15:16 PST
Comment on attachment 703392 [details] [diff] [review]
Remove unused pymake imports, v1

That allow m-c to pass configure & start building (so fixes the regression from 784841), and i can even use mach now \o/
Comment 17 Landry Breuil (:gaston) 2013-12-18 13:08:40 PST
I dont want to open a new bug about it (since there's nothing to fix) but i just wanted to let ppl in cc know that OpenBSD has a working sem_open() implementation since some weeks, so now python2.7 -c 'import multiprocessing.synchronize' works on OpenBSD, and as far as i'm concerned mach can use it as much as it wants :)
Comment 18 Gregory Szorc [:gps] 2013-12-18 13:15:00 PST
(In reply to Landry Breuil (:gaston) from comment #17)
> I dont want to open a new bug about it (since there's nothing to fix) but i
> just wanted to let ppl in cc know that OpenBSD has a working sem_open()
> implementation since some weeks, so now python2.7 -c 'import
> multiprocessing.synchronize' works on OpenBSD, and as far as i'm concerned
> mach can use it as much as it wants :)

\o/

Does that mean multiprocessing.Pool works as well?!
Comment 19 Landry Breuil (:gaston) 2013-12-18 14:00:46 PST
(In reply to Gregory Szorc [:gps] from comment #18)
> (In reply to Landry Breuil (:gaston) from comment #17)
> > I dont want to open a new bug about it (since there's nothing to fix) but i
> > just wanted to let ppl in cc know that OpenBSD has a working sem_open()
> > implementation since some weeks, so now python2.7 -c 'import
> > multiprocessing.synchronize' works on OpenBSD, and as far as i'm concerned
> > mach can use it as much as it wants :)
> 
> \o/
> 
> Does that mean multiprocessing.Pool works as well?!

Hmm... it seems rome wasnt built in a day.

simple test taken from http://stackoverflow.com/questions/4413821/multiprocessing-pool-example

$cat t.py
from multiprocessing import Pool

def  f(x):
    return x*x

if __name__ == '__main__':
    pool = Pool(processes=4)              # start 4 worker processes
    result = pool.apply_async(f, [10])    # evaluate "f(10)" asynchronously
    print result.get(timeout=1)           # prints "100" unless your computer is *very* slow
    print pool.map(f, range(10))          # prints "[0, 1, 4,..., 81]"

$python2.7 t.py 
Traceback (most recent call last):
  File "t.py", line 7, in <module>
    pool = Pool(processes=4)              # start 4 worker processes
  File "/usr/local/lib/python2.7/multiprocessing/__init__.py", line 232, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild)
  File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 138, in __init__
    self._setup_queues()
  File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 233, in _setup_queues
    self._inqueue = SimpleQueue()
  File "/usr/local/lib/python2.7/multiprocessing/queues.py", line 352, in __init__
    self._rlock = Lock()
  File "/usr/local/lib/python2.7/multiprocessing/synchronize.py", line 147, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1)
  File "/usr/local/lib/python2.7/multiprocessing/synchronize.py", line 75, in __init__
    sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 78] Function not implemented

I'll prod our local python/semaphore hackers to see what's possible. Iirc we still lack something (named semaphores or something ?) that kde4 wants, so this is in the works.
Comment 20 Landry Breuil (:gaston) 2013-12-18 14:20:48 PST
Argh. I'll take that back, i was testing on the wrong box without sem_open/fixed python. The same exact test works fine on a fully up to date box, so yes multiprocessing.Pool works :)
Comment 21 Gregory Szorc [:gps] 2013-12-18 16:12:05 PST
This is all terrific news!

Note You need to log in before you can comment on or make changes to this bug.