900015 - talos on mozharness results in a lot of errors while installing PyYAML to the virtualenv

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=25958737&full=1&branch=birch#error0

01:25:06     INFO -    Running setup.py install for PyYAML
01:25:06     INFO -      checking if libyaml is compilable
01:25:06     INFO -      gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/tools/python27/include/python2.7 -c build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c -o build/temp.macosx-10.4-x86_64-2.7/check_libyaml.o
01:25:06    ERROR -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:2:18: error: yaml.h: No such file or directory
01:25:06     INFO -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c: In function 'main':
01:25:06    ERROR -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:5: error: 'yaml_parser_t' undeclared (first use in this function)
01:25:06    ERROR -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:5: error: (Each undeclared identifier is reported only once
01:25:06    ERROR -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:5: error: for each function it appears in.)
01:25:06    ERROR -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:5: error: expected ';' before 'parser'
01:25:06    ERROR -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:6: error: 'yaml_emitter_t' undeclared (first use in this function)
01:25:06    ERROR -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:6: error: expected ';' before 'emitter'
01:25:06  WARNING -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:8: warning: implicit declaration of function 'yaml_parser_initialize'
01:25:06    ERROR -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:8: error: 'parser' undeclared (first use in this function)
01:25:06  WARNING -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:9: warning: implicit declaration of function 'yaml_parser_delete'
01:25:06  WARNING -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:11: warning: implicit declaration of function 'yaml_emitter_initialize'
01:25:06    ERROR -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:11: error: 'emitter' undeclared (first use in this function)
01:25:06  WARNING -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:12: warning: implicit declaration of function 'yaml_emitter_delete'
01:25:06  WARNING -      libyaml is not found or a compiler error: forcing --without-libyaml
01:25:06     INFO -      (if libyaml is installed correctly, you may need to
01:25:06     INFO -       specify the option --include-dirs or uncomment and
01:25:06     INFO -       modify the parameter include_dirs in setup.cfg)
01:25:06     INFO -    Running setup.py install for talos
01:25:06     INFO -      Installing PerfConfigurator script to /builds/slave/talos-slave/test/build/venv/bin
01:25:06     INFO -      Installing talos script to /builds/slave/talos-slave/test/build/venv/bin
01:25:06     INFO -      Installing talos-results script to /builds/slave/talos-slave/test/build/venv/bin
01:25:06     INFO -  Successfully installed datazilla httplib2 mozcrash mozdevice mozfile mozhttpd mozinfo mozlog mozprocess oauth2 PyYAML talos

It appears to be setup correctly and it functions for our purposes.  We need to find a way to add --without-libyaml to the installation in the virtualenv.

Comment 1

[Jason Yeo[:jyeo]]

Two ways to fix this:
- Install the libyaml c library.
- Do `pip install PyYAML --global-option='--without-libyaml'` before installing talos
- some other way to pass --without-libyaml to `pip install talos` step?

Comment 2

[Jeff Hammel]

we could, at least in theory, move away from YAML for talos configuration; IIRC, JSON is supported and working now

Comment 3

[Armen [:armenzg]]

(In reply to Jeff Hammel [:jhammel] from comment #2)
> we could, at least in theory, move away from YAML for talos configuration;
> IIRC, JSON is supported and working now

That would be great as any of the options that jyeo mentions sound hacky (with all due respect).

Comment 4

[Jason Yeo[:jyeo]]

Attachment: Pass global options to install_module, add --without-libyaml as global-option for PyYAML

Comment 5

[Aki Sasaki (not active)]

Comment on attachment 785141 [details] [diff] [review]
Pass global options to install_module, add --without-libyaml as global-option for PyYAML

I'm not a big fan of tuples for this type of thing, as the order is never intuitive.  Pretty sure the python style guide agrees with me.

I prefer {'module': ..., 'url': ..., 'options': [...]}.  However, this will require a lot of backporting since we already use the {modulename: url} construct in a decent number of places.

If this error is a massive problem that needs immediate fixing, I'm ok with this  (marking r+ with this caveat).  If we can live with it a bit longer to fix it in a better way, I would lean towards either getting rid of yaml or going with the above dict approach.

Comment 6

[Jason Yeo[:jyeo]]

Attachment: Pass global options to install_module, add --without-libyaml as global-option for PyYAML v2

Using dicts instead of tuples

Comment 7

[Jason Yeo[:jyeo]]

Attachment: Pass global options to install_module, add --without-libyaml as global-option for PyYAML v3

Change {'module1': 'path/to/module'} to {'name': module1, 'url': 'path/to/module'}

Comment 8

[Aki Sasaki (not active)]

Comment on attachment 786325 [details] [diff] [review]
Pass global options to install_module, add --without-libyaml as global-option for PyYAML v3

Awesome, thanks for doing this!

>             if isinstance(module, dict):
>-                (module, module_url) = module.items()[0]
>+                module_name = module.get('name', None)

I'm currently leaning towards making module_name required... install_module() seems to need it.
What do you think?

If so, we should self.fatal() here if module_name is None.

Comment 9

[Jason Yeo[:jyeo]]

Attachment: Pass global options to install_module, add --without-libyaml as global-option for PyYAML v4

> I'm currently leaning towards making module_name required... install_module() seems to need it.
> What do you think?

Yep, I was thinking about that too. Better to be defensive.

Comment 10

[Aki Sasaki (not active)]

Merged to production.

Comment 11

[Jason Yeo[:jyeo]]

I don't see any warnings related to libyaml. https://tbpl.mozilla.org/php/getParsedLog.php?id=26306899&tree=Mozilla-Central&full=1

FIXED. :)